A coarse-to-fine dynamic layer pruning framework for parameter-efficient fine-tuning
(2026) In Neurocomputing 698.- Abstract
The Parameter Efficient Fine-Tuning (PEFT) method freezes the original pre-trained model parameters and introduces a small number of trainable parameters from Adapter or LoRA to adapt models to downstream tasks. Current PEFT methods mainly enhance LLMs’ cross-task generalization through rigid pruning and global pruning strategies. However, these approaches suffer from coarse-grained adaptation and suboptimal efficiency in complex tasks and ultimately cause negative transfer in LLMs. To address this issue, this paper proposes a coarse-to-fine dynamic layer pruning framework for PEFT (CF-PEFT). Specifically, CF-PEFT first employs a gradient-free Broad Learning System (BLS) for initial layer classification based on task information gain... (More)
The Parameter Efficient Fine-Tuning (PEFT) method freezes the original pre-trained model parameters and introduces a small number of trainable parameters from Adapter or LoRA to adapt models to downstream tasks. Current PEFT methods mainly enhance LLMs’ cross-task generalization through rigid pruning and global pruning strategies. However, these approaches suffer from coarse-grained adaptation and suboptimal efficiency in complex tasks and ultimately cause negative transfer in LLMs. To address this issue, this paper proposes a coarse-to-fine dynamic layer pruning framework for PEFT (CF-PEFT). Specifically, CF-PEFT first employs a gradient-free Broad Learning System (BLS) for initial layer classification based on task information gain between adjacent layers. It quickly prunes task-agnostic layers to achieve coarse-grained efficient adaptation. It then devises parameter update magnitude-guided greedy strategy to progressively freeze task-specific layers to ensure fine-grained specialized adaptation. CF-PEFT strikes a better balance between task performance and parameter efficiency by gradient-free initialization and Coarse-to-Fine layer pruning strategy, which prevents performance degradation from rigid pruning and avoids the excessive computational cost from global pruning. Extensive experimental results demonstrate the effectiveness of CF-PEFT across tasks of varying-complexity and show its superior efficiency with only 0.03% training parameters in LLMs.
(Less)
- author
- Zhang, Xin ; Li, Shuzhen LU ; Liu, Zhulin and Chen, C. L.Philip
- organization
- publishing date
- 2026-10-14
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- Broad learning system, Gradient-free initialization, Layer pruning, Parameter efficient fine-tuning
- in
- Neurocomputing
- volume
- 698
- article number
- 134299
- publisher
- Elsevier
- external identifiers
-
- scopus:105042721038
- ISSN
- 0925-2312
- DOI
- 10.1016/j.neucom.2026.134299
- language
- English
- LU publication?
- yes
- id
- 34429181-3cc3-4fa6-8eb6-b95a94022f6d
- date added to LUP
- 2026-09-30 16:39:01
- date last changed
- 2026-09-30 16:39:42
@article{34429181-3cc3-4fa6-8eb6-b95a94022f6d,
abstract = {{<p>The Parameter Efficient Fine-Tuning (PEFT) method freezes the original pre-trained model parameters and introduces a small number of trainable parameters from Adapter or LoRA to adapt models to downstream tasks. Current PEFT methods mainly enhance LLMs’ cross-task generalization through rigid pruning and global pruning strategies. However, these approaches suffer from coarse-grained adaptation and suboptimal efficiency in complex tasks and ultimately cause negative transfer in LLMs. To address this issue, this paper proposes a coarse-to-fine dynamic layer pruning framework for PEFT (CF-PEFT). Specifically, CF-PEFT first employs a gradient-free Broad Learning System (BLS) for initial layer classification based on task information gain between adjacent layers. It quickly prunes task-agnostic layers to achieve coarse-grained efficient adaptation. It then devises parameter update magnitude-guided greedy strategy to progressively freeze task-specific layers to ensure fine-grained specialized adaptation. CF-PEFT strikes a better balance between task performance and parameter efficiency by gradient-free initialization and Coarse-to-Fine layer pruning strategy, which prevents performance degradation from rigid pruning and avoids the excessive computational cost from global pruning. Extensive experimental results demonstrate the effectiveness of CF-PEFT across tasks of varying-complexity and show its superior efficiency with only 0.03% training parameters in LLMs.</p>}},
author = {{Zhang, Xin and Li, Shuzhen and Liu, Zhulin and Chen, C. L.Philip}},
issn = {{0925-2312}},
keywords = {{Broad learning system; Gradient-free initialization; Layer pruning; Parameter efficient fine-tuning}},
language = {{eng}},
month = {{10}},
publisher = {{Elsevier}},
series = {{Neurocomputing}},
title = {{A coarse-to-fine dynamic layer pruning framework for parameter-efficient fine-tuning}},
url = {{http://dx.doi.org/10.1016/j.neucom.2026.134299}},
doi = {{10.1016/j.neucom.2026.134299}},
volume = {{698}},
year = {{2026}},
}