Use of Stochastic Switching in the Training of Binarized Neural Networks
(2022) EITM01 20221Department of Electrical and Information Technology
- Abstract
- Most prior research into the field of Binarized Neural Networks (BNNs) has been motivated by a desire to optimize Deep Learning computations on traditional hardware. These
methods rely on bit-wise operations to drastically decrease computation time, however
to address the resulting loss in accuracy it is common practice to reintroduce continuous
parameters and train using batch normalization. Here a separate application for BNNs is
investigated to find if it is a feasible application to build and train BNNs on hardware based
on binary memristors with stochastic switching. To do this BNNs have been trained without batch normalization and multiple new algorithms have been proposed which attempt
to utilize the stochastic switching... (More) - Most prior research into the field of Binarized Neural Networks (BNNs) has been motivated by a desire to optimize Deep Learning computations on traditional hardware. These
methods rely on bit-wise operations to drastically decrease computation time, however
to address the resulting loss in accuracy it is common practice to reintroduce continuous
parameters and train using batch normalization. Here a separate application for BNNs is
investigated to find if it is a feasible application to build and train BNNs on hardware based
on binary memristors with stochastic switching. To do this BNNs have been trained without batch normalization and multiple new algorithms have been proposed which attempt
to utilize the stochastic switching ability in place of storing multiple bits of information
per weight. The results show that BNNs without batch normalization are limited to lower
accuracies than conventional CNNs, and it is essential to include an element of gradient
accumulation for stochastic switching to work. To adress instability issues with stochastic
switching based training an undo function is introduced which is shown to stabilize training well. Future research should look into accumulating gradients using stochastically
switching memristors. (Less) - Popular Abstract
- When only allowing the connections between neurons to have two possible strengths
it is no longer possible to improve the network by making small changes to the connections. Here, the proposed solution is to use random chance to instead make large
changes with a low probability.
Please use this url to cite or link to this publication:
http://lup.lub.lu.se/student-papers/record/9098286
- author
- Åkesson, Svante LU
- supervisor
-
- Mattias Borg LU
- organization
- course
- EITM01 20221
- year
- 2022
- type
- H2 - Master's Degree (Two Years)
- subject
- keywords
- Machine Learning BNN Binarized Neural Network Deep Learning Convolutional Batchnormalization Stochastic Switching Training Gradient Accumulation Undo Binary Weights Memristor FTJ Hardware
- report number
- LU/LTH-EIT 2022-887
- language
- English
- id
- 9098286
- date added to LUP
- 2022-09-08 14:17:10
- date last changed
- 2022-09-08 14:17:10
@misc{9098286, abstract = {{Most prior research into the field of Binarized Neural Networks (BNNs) has been motivated by a desire to optimize Deep Learning computations on traditional hardware. These methods rely on bit-wise operations to drastically decrease computation time, however to address the resulting loss in accuracy it is common practice to reintroduce continuous parameters and train using batch normalization. Here a separate application for BNNs is investigated to find if it is a feasible application to build and train BNNs on hardware based on binary memristors with stochastic switching. To do this BNNs have been trained without batch normalization and multiple new algorithms have been proposed which attempt to utilize the stochastic switching ability in place of storing multiple bits of information per weight. The results show that BNNs without batch normalization are limited to lower accuracies than conventional CNNs, and it is essential to include an element of gradient accumulation for stochastic switching to work. To adress instability issues with stochastic switching based training an undo function is introduced which is shown to stabilize training well. Future research should look into accumulating gradients using stochastically switching memristors.}}, author = {{Åkesson, Svante}}, language = {{eng}}, note = {{Student Paper}}, title = {{Use of Stochastic Switching in the Training of Binarized Neural Networks}}, year = {{2022}}, }