Two-layer training
For the researchers, the main challenge was to integrate the key components needed for on-chip training onto a single neural chip. “The main task that had to be solved was to include electrochemical random access memory (EC-RAM) components, for example. These are components that simulate the storage and release of electrical charges attributed to neurons in the brain,” says van de Burg.
The researchers built a two-layer neural network based on organic random-access memory components, and tested the devices using an evolution of the widely used back-propagation with gradient descent training algorithm. “The traditional algorithm is often used to improve the accuracy of neural networks, but that’s not compatible with our devices, so we came up with our own version,” Stevens says.
Moreover, with AI in many fields rapidly becoming an unsustainable drain on energy resources, the opportunity to train neural networks on hardware components at a fraction of the energy cost is an enticing prospect for many applications — from ChatGPT to weather forecasting.
Future need
While researchers have proven that the new training approach works, the next logical step is to go bigger, bolder, and better.
“We have proven that this works with a small two-layer network,” says Van de Burgh. “In the next phase, we would like to involve industry and other major research labs so that we can build much larger networks of devices and test them with real-life data problems.”
The next step will allow the researchers to demonstrate that these systems are highly effective in training, as well as running, useful neural networks and AI systems. “We want to apply this technology in many practical cases,” says Van de Burgh. “My dream is for such techniques to become the norm in AI applications in the future.”
Full paper details
“Physical implementation of backpropagation using gradient descent for local training of multilayer neural networks“Evelyn R. W. van Doremayele, Tim Stevens, Stijn Ringling, Simone Spollauer, Marco Fattori, and Yuri van de Borght, Science Advances, (2024).
Evelyn R. W. van Doremelle and Tim Stevens contributed equally to the research and are both considered first authors of the paper.
Tim Stevens currently works as a mechanical engineer at Microalinea company co-founded by Marco Fattori.
