KaydedilenlerDerginizi ekleyin

Dergiler / Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım ve Teknoloji / 2022 / Cilt: 10 - Sayı: 3

Improved Knowledge Distillationwith Dynamic Network Pruning

Eren ŞENER
Emre AKBAŞ
Sayfa
650–665
Yayın Tarihi
2022-09-30
DOI
—
🏛️Bu makalenin bibliyografik verileri ve tam metin bağlantısı DergiPark üzerinden sağlanmıştır.
DergiPark Kaydı ↗
✨

Yapay Zeka Hızlı Bakış & Çıkarımlar

AI Insight
⚡ TL;DR(1 Cümlede Hızlı Çıkarım)

Compared to KD, our method produces better accuracy and more compact models.

📌 3 Maddede Bu Makale

🎯
Amaç & Hipotez:Deploying convolutional neural networks to mobile or embedded devices is often prohibited by limited memory and computational resources.
📊
Temel Bulgular & Veri:In KDDP, we train the student network with supervision from the teacher network, while applying L1 regularization on the neuron activations in a fully-connected layer.
💡
Sonuç & Katkı:Compared to KD, our method produces better accuracy and more compact models.
🔬Tür:Modelleme & Sayısal Simülasyon
📍Kapsam:Genişletilmiş Akademik Örneklem
⚙️Değişkenler:Knowledge Distillation · Neural Network Compression · Image Classification

Abstract

Deploying convolutional neural networks to mobile or embedded devices is often prohibited by limited memory and computational resources. This is particularly problematic for the most successful networks, which tend to be very large and require long inference times. Many alternative approaches have been developed for compressing neural networks based on pruning, regularization, quantization or distillation. In this paper, we propose the “Knowledge Distillation with Dynamic Pruning” (KDDP), which trains a dynamically pruned compact student network under the guidance of a large teacher network. In KDDP, we train the student network with supervision from the teacher network, while applying L1 regularization on the neuron activations in a fully-connected layer. Subsequently, we prune inactive neurons. Our method automatically determines the final size of the student model. We evaluate the compression rate and accuracy of the resulting networks on an image classification dataset, and compare them to results obtained by Knowledge Distillation (KD). Compared to KD, our method produces better accuracy and more compact models.

🔖

Bu Makaleye Atıf Yap

ŞENER, E., AKBAŞ, E. (2022). Improved Knowledge Distillationwith Dynamic Network Pruning. Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım ve Teknoloji, Cilt: 10 - Sayı: 3, 650–665. https://ajindex.com/articles/667298

📄 Tam Metin PDF Okuyucu

📄

PDF görüntüleyici hazırlanıyor...

Kaynakça (10)

  1. Y. LeCun, J. S. Denker, S. A. Solla, R. E. Howard, and L. D. Jackel, “Optimal brain damage.,” in Advances in Neural Processing Systems (NIPS Conference), vol. 2, pp. 598–605, 1989
  2. B. Hassibi and D. Stork, “Second order derivatives for network pruning: Optimal brain surgeon,” in Advances in Neural Information Processing Systems 5 (NIPS Conference), pp. 164–171, 1992
  3. S. Srinivas and R. V. Babu, “Data-free parameter pruning for deep neural networks,” in Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015, pp. 31.1–31.12, 2015 [4] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in Advances in Neural Information Processing Systems (NIPS Conference), pp. 1135–1143, 2015
  4. H. Zhou, J. M. Alvarez, and F. Porikli, “Less is more: Towards compact cnns,” in European Conference on Computer Vision (ECCV), pp. 662–677, Springer, 2016
  5. W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in Advances in Neural Information Processing Systems (NIPS Conference), pp. 2074–2082, 2016
  6. Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2736–2744, 2017
  7. J. Jin, A. Dundar, and E. Culurciello, “Flattened convolutional neural networks for feedforward acceleration,” in 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015
  8. F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and [10] T. Li, B. Wu, Y. Yang, Y. Fan, Y. Zhang, and W. Liu, “Compressing convolutional neural networks via factorized convolutional filters,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3977–3986, 2019
  9. E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” in Advances in Neural Information Processing Systems 27 (NIPS Conference), December 8-13, Montreal, Quebec, Canada, pp. 1269–1277, 2014
  10. H. Kim, M. U. K. Khan, and C.-M. Kyung, “Efficient neural network compression,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVP