As scientists push the boundaries of machine learning, the amount of time, energy, and money required to train increasingly complex neural network models is getting more and more complex. A new field of artificial intelligence called analog deep learning promises faster computation with a fraction of the power usage.
Programmable resistors are the building blocks of analog deep learning, just like transistors are the basic elements of digital processors. By replicating arrays of programmable resistors in complex layers, researchers can create a network of analog artificial “neurons” and “synapses” that perform computations just like a digital neural network. This network can then be trained to achieve complex AI tasks such as image recognition and natural language processing.
An interdisciplinary team of MIT researchers set out to push the speed limits for a type of human-made analog synapse that they had Developed previously. They used an inorganic process material in the manufacturing process that enables their devices to run a million times faster than previous versions, and it’s also about a million times faster than the synapses in the human brain.
Moreover, this inorganic material makes the resistor significantly energy efficient. Unlike the materials used in the previous version of their device, the new material is compatible with silicon manufacturing technologies. This change enabled the fabrication of nanometer-scale devices and could pave the way for integration into commercial computing devices for deep learning applications.
“With this key insight, and the very powerful nanofabrication techniques we have in them MIT NanoWe were able to put these pieces together and demonstrate that these devices are intrinsically very fast and operate at reasonable voltages,” says senior author Jesús A. del Alamo, Professor Donner in the Department of Electrical Engineering and Computer Science (EECS) at MIT. This is really working these devices at a point where they now look really promising for future applications.”
“The mechanism of action of the device is the electrochemical introduction of the smallest ion, the proton, into an insulating oxide to modulate its electronic conductivity. Since we are working with very thin devices, we can speed up the movement of this ion using a strong electric field, and push these ionic devices into a nanosecond operating order,” explains the senior author Bilge Yildiz, Breene M. Kerr Professor in the Departments of Nuclear Science and Engineering and Materials Science and Engineering.
Senior author Joe Lee, Battelle Energy Alliance Professor of Nuclear Science and Engineering and Professor of Materials Science and Engineering, says, “Here we apply up to 10 volts across a special nano-thick solid glass film that conducts protons, without permanently damaging it. The stronger the field, The speed of the ionic devices increased.”
These programmable resistors greatly increase the speed at which a neural network is trained, while significantly reducing the cost and energy to perform such training. This could help scientists develop deep learning models more quickly, which could then be applied in uses such as self-driving cars, fraud detection, or medical image analysis.
“Once you have an analog processor, you will no longer be training the networks that everyone else is running on. You will be training networks of unprecedented complexity that no one else can afford, thus vastly outperforming them all. In other words, this is not a faster car, It’s a spacecraft,” adds lead author and MIT postdoctoral expert Murat Onen.
Co-authors include Frances M. Ross, Ellen Swallow Richards Professor in the Department of Materials Science and Engineering. postdocs Nicola Emond and Bauming Wang; and Difei Zhang, a graduate student from EECS. The research was published today in Sciences.
Deep learning acceleration
Analog deep learning is faster and more energy efficient than its digital counterpart for two main reasons. “First, the computation is done in memory, so huge amounts of data aren’t transferred back and forth from memory to the processor.” Analog processors also perform operations in parallel. If the size of the array is expanded, then the analog processor will not need more time to complete the new operations because all the arithmetic operations occur simultaneously.
A key component of MIT’s new analog processor technology is known as the PPR. These resistors, measured in nanometers (one nanometer equals one billionth of a metre), are arranged in a matrix, like a checkerboard.
Learning in the human brain occurs due to the strengthening and weakening of the connections between nerve cells, which are called synapses. Deep neural networks have always adopted this strategy, in which network weights are programmed through training algorithms. In the case of this new processor, increasing and decreasing the electrical conductivity of the proton resistors enables analog machine learning.
Conduction is controlled by the motion of the protons. To increase conduction, more protons are pushed into a channel in the resistor, while protons are pushed out to decrease conduction. This is accomplished by using an electrolyte (similar to a battery) that conducts protons but blocks electrons.
To develop an ultrafast, highly energy-efficient programmable proton resistor, the researchers looked at different electrolyte materials. While other devices use organic compounds, Onen focused on inorganic phosphosilicate glass (PSG).
PSG is basically silicon dioxide, a powdered desiccant found in small bags that come in the box with new furniture to remove moisture. It has been studied as a conductor of the proton in moisture conditions for fuel cells. It is also the most well-known oxide used in silicon processing. To make PSG, a tiny amount of phosphorous is added to silicon to give it special proton-conducting properties.
Onin hypothesized that the improved PSG could have a high proton conductivity at room temperature without the need for water, making it an ideal solid electrode for this application. He was right.
PSG enables ultra-fast proton movement because it contains many nanometer-sized pores whose surfaces provide pathways for proton diffusion. It can also withstand extremely strong, pulsed electric fields. This is critical, Onin explains, because applying more effort to the device enables the protons to move at blinding speeds.
“The speed was definitely surprising. Normally, we wouldn’t apply such extreme fields across devices, so as not to turn them into ash. But instead, the protons ended up moving at tremendous speed through the device stack, specifically a million times faster than what we had before And this motion doesn’t harm anything, thanks to the protons’ small size and low mass. It’s almost like teleportation,” he says.
“The nanosecond time scale means we are close to a ballistic tunneling system or even a proton quantum tunneling system, under this extreme field,” Lee adds.
Because the protons do not damage the material, the resistor can last for millions of cycles without breaking. This new electrolyte enabled a programmable proton impedance to be millions of times faster than their previous devices and could operate effectively at room temperature, which is important for its incorporation into computing devices.
Thanks to the dielectric properties of PSG, almost no electric current passes through the material as the protons move. Onen adds that this makes the device significantly more energy efficient.
Now that they have demonstrated the effectiveness of these programmable resistors, the researchers plan to re-engineer them for mass manufacturing, Del Alamo says. Then they can study the properties of the resistance matrices and scale them so that they can be integrated into the systems.
At the same time, they plan to study the materials to remove the bottlenecks that limit the effort required to efficiently transfer protons to, through and from the electrolyte.
“Another exciting direction that these ionic devices can enable is energy-efficient devices to simulate neural circuits and rules of synaptic plasticity that are inferred in neuroscience, beyond analog deep neural networks. We have already started such collaboration with neuroscience, supported by MIT quest for intelligenceYildiz adds.
“The collaboration that we have will be essential for innovation in the future. The way forward is still very challenging, but at the same time very exciting,” says Del Alamo.
“Interaction reactions such as those in lithium-ion batteries have been extensively explored for memory devices. This work demonstrates that proton-based memory devices provide impressive and surprising switching speed and endurance,” says William Chueh, associate professor of materials science and engineering at the University of Stanford, who was not involved in this research. “It lays the foundation for a new class of memory devices to run deep learning algorithms.”
This work demonstrates an important breakthrough in biologically inspired resistive memory devices. These all-solid-state proton devices rely on remarkable atomic-scale control of protons, similar to biological synapses but at faster rates,” says Elizabeth Dickey, the Teddy Wilton Hawkins Distinguished Professor and Chair of Materials Science and Engineering at Carnegie Mellon University, who was not involved in this. Work.” “I commend MIT’s multidisciplinary team for this exciting development, which will enable future-generation computing devices.”
This research is funded in part by the MIT-IBM Watson AI Lab.