基于滤波衰减的自知识蒸馏压缩算法
DOI:
作者:
作者单位:

taptap点点官方注册信息工程学院

作者简介:

通讯作者:

中图分类号:

TP391.4

基金项目:

国家自然科学(62067002,61967006,62062033);江西省交通厅科技项目(2022X0040)


Self knowledge distillation compression algorithm based on filter decaying
Author:
Affiliation:

Fund Project:

National Natural Science Foundation project (620670026196700662062033); Jiangxi Provincial Department of Transportation Science and Technology Project (2022X0040)

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    【目的】为了解决模型剪枝后性能损失严重的问题。【方法】提出了一种基于滤波衰减和自知识蒸馏的压缩算法。大多数传统的剪枝方法通过直接消除网络中的冗余滤波器或将其权重置为零。然而,这些方法存在缺陷,因为被消除的冗余滤波器仍然保留着一定价值的信息。本方法通过滤波衰减机制来保留冗余滤波器的信息,进而缩小剪枝前后的模型差异,降低剪枝导致的性能损耗。同时,在剪枝过程中引入一个退火衰减函数,使得滤波器的衰减呈现动态变化,进而能够快速高效地搜索模型的最佳子结构,提高模型的收敛速度。此外,还利用自知识蒸馏技术在预训练模型和压缩模型之间进行知识转移。【结果】结果表明,该压缩算法在减少VGG-16模型37.3%FLOPs的条件下,将模型精度提升了0.12%。【结论】该方法能够为卷积神经网络提供一种更稳定、更高效的模型压缩方法。

    Abstract:

    [Purpose] To address the severe performance loss after model pruning. [Method] A compression algorithm based on filter attenuation and self-knowledge distillation is proposed. Most traditional pruning methods directly eliminate redundant filters in the network or set their weights to zero. However, these methods have shortcomings as the eliminated redundant filters still retain valuable information. This method utilizes a filter attenuation mechanism to preserve the information of redundant filters, thereby minimizing the model difference before and after pruning and reducing the performance loss caused by pruning. Meanwhile, an annealing attenuation function is introduced during the pruning process to dynamically change the attenuation of filters, enabling a fast and efficient search for the optimal substructure of the model and improving the convergence speed of the model. Additionally, self-knowledge distillation is employed for knowledge transfer between the pre-trained model and the compressed model. [Result] The results show that this compression algorithm improves the model accuracy by 0.12% while reducing the FLOPs of the VGG-16 model by 37.3%. [Conclusion] This method provides a more stable and efficient model compression approach for convolutional neural networks.

    参考文献
    相似文献
    引证文献
引用本文
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2024-03-15
  • 最后修改日期:2024-05-29
  • 录用日期:2024-05-30
  • 在线发布日期:
  • 出版日期:
Baidu
map
关闭