Skip to main content

LUP Student Papers

LUND UNIVERSITY LIBRARIES

Use of Stochastic Switching in the Training of Binarized Neural Networks

Åkesson, Svante LU (2022) EITM01 20221
Department of Electrical and Information Technology
Abstract
Most prior research into the field of Binarized Neural Networks (BNNs) has been motivated by a desire to optimize Deep Learning computations on traditional hardware. These
methods rely on bit-wise operations to drastically decrease computation time, however
to address the resulting loss in accuracy it is common practice to reintroduce continuous
parameters and train using batch normalization. Here a separate application for BNNs is
investigated to find if it is a feasible application to build and train BNNs on hardware based
on binary memristors with stochastic switching. To do this BNNs have been trained without batch normalization and multiple new algorithms have been proposed which attempt
to utilize the stochastic switching... (More)
Most prior research into the field of Binarized Neural Networks (BNNs) has been motivated by a desire to optimize Deep Learning computations on traditional hardware. These
methods rely on bit-wise operations to drastically decrease computation time, however
to address the resulting loss in accuracy it is common practice to reintroduce continuous
parameters and train using batch normalization. Here a separate application for BNNs is
investigated to find if it is a feasible application to build and train BNNs on hardware based
on binary memristors with stochastic switching. To do this BNNs have been trained without batch normalization and multiple new algorithms have been proposed which attempt
to utilize the stochastic switching ability in place of storing multiple bits of information
per weight. The results show that BNNs without batch normalization are limited to lower
accuracies than conventional CNNs, and it is essential to include an element of gradient
accumulation for stochastic switching to work. To adress instability issues with stochastic
switching based training an undo function is introduced which is shown to stabilize training well. Future research should look into accumulating gradients using stochastically
switching memristors. (Less)
Popular Abstract
When only allowing the connections between neurons to have two possible strengths
it is no longer possible to improve the network by making small changes to the connections. Here, the proposed solution is to use random chance to instead make large
changes with a low probability.
Please use this url to cite or link to this publication:
author
Åkesson, Svante LU
supervisor
organization
course
EITM01 20221
year
type
H2 - Master's Degree (Two Years)
subject
keywords
Machine Learning BNN Binarized Neural Network Deep Learning Convolutional Batchnormalization Stochastic Switching Training Gradient Accumulation Undo Binary Weights Memristor FTJ Hardware
report number
LU/LTH-EIT 2022-887
language
English
id
9098286
date added to LUP
2022-09-08 14:17:10
date last changed
2022-09-08 14:17:10
@misc{9098286,
  abstract     = {{Most prior research into the field of Binarized Neural Networks (BNNs) has been motivated by a desire to optimize Deep Learning computations on traditional hardware. These
methods rely on bit-wise operations to drastically decrease computation time, however
 to address the resulting loss in accuracy it is common practice to reintroduce continuous
parameters and train using batch normalization. Here a separate application for BNNs is
investigated to find if it is a feasible application to build and train BNNs on hardware based
 on binary memristors with stochastic switching. To do this BNNs have been trained without batch normalization and multiple new algorithms have been proposed which attempt
 to utilize the stochastic switching ability in place of storing multiple bits of information
per weight. The results show that BNNs without batch normalization are limited to lower
 accuracies than conventional CNNs, and it is essential to include an element of gradient
 accumulation for stochastic switching to work. To adress instability issues with stochastic
switching based training an undo function is introduced which is shown to stabilize training well. Future research should look into accumulating gradients using stochastically
 switching memristors.}},
  author       = {{Åkesson, Svante}},
  language     = {{eng}},
  note         = {{Student Paper}},
  title        = {{Use of Stochastic Switching in the Training of Binarized Neural Networks}},
  year         = {{2022}},
}