Developing a spiking neural model of Long Short-Term Memory architectures

Abstract

Current advances in Deep Learning have shown significant improvements in common Machine Learning applications such as image, speech and text recognition. Specifically, in order to process time series, deep Neural Networks (NNs) with Long Short-Term Memory (LSTM) units are widely used in sequence recognition problems to store recent information and to use it for future predictions. The efficiency in data analysis, especially when big-sized data sets are involved, can be greatly improved thanks to the advancement of the ongoing research on Neural Networks (NNs) and Machine Learning for many applications in Physics. However, whenever acquisition and processing of data at different time resolutions is required, a synchronization problem for which the same piece of information is processed multiple times arises, and the advantageous efficiency of NNs, which lack the natural notion of time, ceases to exist. Spiking Neural Networks (SNNs) are the next generation of NNs that allow efficient information coding and processing by means of spikes, i.e. binary pulses propagating between neurons. In this way, information can be encoded in time, and the communication of information is activated only when the input to the neurons change, thus giving higher efficiency.

In the present work, analog neurons are used for training and then they are substituted with spiking neurons in order to perform tasks. The aim for this project is to find a transfer function which allows a simple and accurate switching between analog and spiking neurons, and then to prove that the obtained network performs well in different tasks. At first, an analytical transfer function for more biologically plausible values for some neuronal parameters is derived and tested. Subsequently, the stochastic nature of the biological neurons is implemented in the neuronal model used. A new transfer function is then approximated by studying the stochastic behavior of artificial neurons, allowing to implement a simplified description for the gates and the input cell in the LSTM units. The stochastic LSTM networks are then tested on Sequence Prediction and T-Maze, i.e. typical memory-involving Machine Learning tasks, showing that almost all the resulting spiking networks correctly compute the original tasks.

The main conclusion drawn from this project is that by means of a neuronal model comprising of a stochastic description of the neuron it is possible to obtain an accurate mapping from analog to spiking memory networks, which gives good results on Machine Learning tasks.

Popular Abstract

Spiking neurons communicate with each other by means of a telegraph-like mechanism: a message is encoded by a neuron in binary events–called spikes–that are sent to another neuron, which decodes the incoming spikes of signal by means of the same coding originally used by the sending neuron. The problem addressed in this project then was: is it possible to make a group of such neurons remember things in the short-term but for long-enough time that they are able to solve tasks that require memory?

Imagine you are driving to work through a road you have never taken before, and your task is to turn right at the next traffic light. The memory-tasks we wanted the neural networks to learn and solve are of this sort, and no spiking networks exist that can do this. With regards to this goal, the approach we opted for was to train a network of standard artificial neurons and then, once the network learned how to perform the task, we would switch the standard neurons with our modeled spiking neurons. In order to do this, of course, there are some constraints, in particular the two types of neurons (standard and spiking) have to encode signals in the same way, meaning that they need to have the same coding policy.

In this project, I had to find an adequate coding policy for the spiking neurons, in order to give the same policy to a network of standard neurons and to test this superposition. Turned out, after the standard networks had learned the tasks and then switched with spiking units, the spiking neurons were indeed able to remember short-term information (such as looking for a traffic light before turning right) and to perform well in such memory tasks, allowing useful computation over time.

One of the scientific fields in need of improvement is, in fact, signal processing over time. Nowadays most of the detection instruments collect signal during a time window, meaning that the signal collected in a small time range is considered as a whole, instead of detecting in continuous time. In the first case, a buffer of the history of the time windows (the information gathered before meeting the traffic light) is stored, while when information is processed in continuous time only relevant information (the time at which the traffic light is encountered) is needed. Being able to classify signals as soon as they are detected is a characteristic of asynchronous detection, an example of which is our sight, or hearing. The brain, in fact, is one of the most efficient and powerful systems existent. So why not studying a computation method inspired by the brain?

Spiking neurons are exactly that: artificial units performing brain-like computation. Hence, these neurons potentially offer efficient computation and an advantageous method for continuous-time signal processing, which hopefully will be implemented in many research fields in the future.

• author: Pozzi, Isabella
• supervisor: Mattias Ohlsson
• organization: Department of Physics; Combustion Physics
• course: FYSM30 20172
• year: 2018
• type: H2 - Master's Degree (Two Years)
• subject: Physics and Astronomy
• language: English
• id: 8935203
• date added to LUP: 2018-02-09 11:52:14
• date last changed: 2018-02-09 11:52:14