Harnessing sound waves for photonic neuromorphic computing

Optical neural networks could bring about sustainable, energy-efficient AI, but fulfilling this vision requires further advances in scalability and reconfigurability. Using sound waves in optical fibres and PICs offers one path to flexible, configurable, photonic deep learning units.

By Birgit Stiller, professor at Leibniz University and group leader at the Max Planck Institute for the Science of Light

Last year, Microsoft made headlines when it signed a deal to buy energy from Three Mile Island, a nuclear power plant infamous for the accident that took place there in 1979. And Microsoft is not the only tech company seeking new ways to fulfil its soaring energy needs. Google and Meta have also been making deals to purchase power from nuclear facilities, while Google and Microsoft have even signed long-term agreements with nuclear fusion start-ups to secure future fusion-generated electricity.

These actions underscore just how much energy these companies are expecting to need for their power-hungry datacentres and AI applications. Large language models such as ChatGPT have already proven extremely useful across numerous industries and in our daily lives. A technology that might have seemed far-fetched just a few years ago has been adopted remarkably quickly, and the demand for AI services will only continue to grow. But providing access to them will require new technologies that speed up data processing and reduce energy consumption.

One potential solution arises from using optical devices, such as PICs or hybrid photonic-electronic chips, as the basis for new computing architectures. By using light instead of electrical signals to represent data, optical neural networks could handle large volumes of information at high speeds and with great energy efficiency, forming the backbone of future AI systems.

Figure 1. Part of Birgit Stiller’s research group, from left to right: Steven Becker, Niklas Braband, Jesus Humberto Marines Cabello, Andreas Geilen, Birgit Stiller. Photo: Susanne Viezens.

Electrical signals travel comparatively quickly over short distances, meaning that the speed of light itself does not necessarily offer a huge advantage for computing. A more significant gain comes from the fact that light has many more parameters that can encode information; frequency, spatial modes, optical phase, and polarisation can all carry data simultaneously – an advantage that is already being harnessed in optical communications.

To date many of the experimental approaches to implementing optical neural networks have relied on fixed components and devices configured to manipulate light in specific ways.

These components may include resonator structures on a chip, huge interferometric networks made of photonic waveguides, arrangements of diffractive phase plates, or even metamaterials. It is possible to tune and train these elements, but for many of these setups, once a structure or device is fabricated there is limited room for reconfiguration. This lack of flexibility means that some devices can usually perform the task they were designed for and cannot be adapted for broader functions – an essential feature of the most useful AI models.

Now, let’s take a break from photonics and think about sound: about acoustic waves. Acoustic waves have a completely different nature from optical signals, which are electromagnetic waves. Optical waves travel at high speed through materials and also through vacuums like outer space. Acoustic waves, on the other hand, travel a million times more slowly than light in air, and they do not travel through a vacuum at all; they need a material to propagate through. Optical and sound waves also have very different frequency ranges, with the former generally oscillating millions of times more per second than the latter.

Given all these dissimilarities, it may seem unlikely that pristine optical information could be captured in such a bulky, macroscopic thing as an acoustic wave. Yet the two forms of energy can in fact interact very efficiently via the refractive index of a material.

For example, in an acousto-optic modulator, sound waves periodically distort the material and change its refractive index, scattering incoming optical waves. Additionally, in certain materials, light waves can cause charged particles in the structure to move (a physical effect called electrostriction), and the resulting deformation can produce acoustic waves. These interactions offer an opportunity to modulate and control optical signals with sound.

Figure 2. In photonic systems, sound waves can serve several different purposes, such as storing data in a temporary memory, amplifying certain signals, or introducing functions like recurrency and nonlinearity.

Sound as memory
In our research on neuromorphic computing at the Leibniz University Hannover and Max Planck Institute for the Science of Light, our vision is to create neuromorphic hardware based on optical fibres and photonic chips that can be built and reconfigured by acoustic waves. To this end, we use stimulated Brillouin scattering – a phenomenon by which light and sound waves affect one another via the refractive index of a material.

By bringing the waves into interaction, we have found a way to build reconfigurable building blocks based on sound waves for photonic machine learning. With optical control pulses, we can generate coherent acoustic waves, which can then be used for various functions.

For example, they can be used to store information, serving as a temporary memory, to amplify certain signals, or to enable other functionalities such as recurrency and nonlinearity. These building blocks can be implemented in hair-thin optical fibres, already used globally for fast internet connections, or in photonic chips for potential hybrid integration with electronics.

The optoacoustic memory uses the fact that acoustic waves are slow and can therefore act like a parking slot for optical information. Additionally, these waves fulfil several other key requirements for an optical high-performance random-access memory: they are coherent, they are compatible with photonic chips, they can be tuned to different frequencies, and they have a high bandwidth.

The mechanism for transferring information from light waves to sound waves involves an optical data stream being sent into the optical fibre or photonic chip, while a second optical control pulse is sent from the other side. When these two counterpropagating waves collide, they create an acoustic wave that encodes the information from the data stream. This information is preserved in the acoustic domain until a second optical control pulse enters the medium and “reads” the sound wave, creating a corresponding optical wave again.

Figure 3. Nonlinear activation functions are essential for neural networks to learn complex patterns in data and to perform complex tasks. Acoustic waves can effectively mediate these types of functions in photonic architectures through their nonlinear interactions with light waves.

We have demonstrated that this memory is coherent in amplitude and phase, meaning that the data does not get lost or scrambled, and provides a high bandwidth up to the GHz range. The memory can also be accessed at high speed and be operated at different frequency channels at the same time with negligible crosstalk.

Because the acoustic waves only travel in one specific direction, this type of photonic memory is nonreciprocal, meaning that it is unaffected by counterpropagating signals and only interacts with the optical control pulse intended to read it. This property also allows for the nonreciprocal manipulation and potentially storage of specific types of light beams called orbital angular momentum modes, which can carry more information than standard light signals.

Recurrency and nonlinearity
Being so slow, acoustic waves remain in the medium much longer than optical waves, which makes them perfect candidates for recurrent neural networks – a type of architecture that is designed to process sequential data, in which the order of the data is important.

For example, in human language, the order of words in a sentence affects its meaning. The two sentences “The cat hunts the mouse” and “The mouse hunts the cat” consist of the same words but have different meanings, indicated by the word orders.

A traditional fully connected neural network on a computer faces difficulties capturing this context because it requires access to memory. But recurrent operators feed the output of one calculation back as an additional input for the next, linking a series of computational steps and thus providing context for each single calculation performed. By equipping neural networks with recurrent operations, we can give systems internal memory and enable them to capture contextual information, overcoming the challenges associated with processing sequential data.

We have now used sound waves to implement such a recurrent operator. Our Optoacoustic REcurrent Operator (OREO) harnesses the intrinsic properties of optical waveguides, meaning we can enable recurrent operations in the existing physical structures, without needing to add an artificial reservoir or any additional components. Unique to OREO is the advantage of being entirely controlled by light, making the optoacoustic computer programmable on a pulse-by-pulse basis, and eliminating the need for complicated structures or transducers to convert between signal types.

Figure 4. As well as enabling new architectures for AI, the optoacoustic phenomena being investigated by Stiller’s research group could one day be used for quantum communication and quantum information processing.

Another key ingredient of neural networks is an activation function – a nonlinear mathematical operation that takes the inputs to a particular neuron and determines its output. Examples of activation functions include ReLU, sigmoid, or tanh functions, which are commonly used to transform the weighted sum of inputs in an artificial neural network. Crucially, these functions introduce nonlinearity into the neuromorphic structure, which is essential for deep learning models to learn complex relationships between data and to be able to perform complex tasks.

In optical neural networks, nonlinear activation functions should ideally be implemented in the photonic domain, to avoid the typical conversion between optics and electronics, which would limit the benefits of using photonic hardware in the first place. When it comes to building larger optical neural networks, it might prove especially beneficial to have an all-optical activation function which can also be controlled all-optically, making the system more efficient and easier to scale.

For linear operations such as the weighted sum – a matrix operator – a plethora of photonic techniques already exist. But this is not the case for nonlinear activation functions, for which few approaches have been demonstrated experimentally. Here, acoustic waves could become an enabler because the effect is intrinsically nonlinear. Indeed, we have demonstrated that sound waves can be the mediator for an effective photonic activation function.

Once again, stimulated Brillouin scattering is the key; optical input information interacts with sound waves and undergoes a nonlinear change depending on the level of optical intensity. In this mechanism, the information does not have to leave the optical domain and is directly processed in optical fibres or photonic waveguides.

Besides avoiding electro-optic conversion and any associated loss or latency, including a photonic activation function in an optical neural network also preserves the bandwidth of the optical data and maintains the coherence of the signal. The versatile control of the nonlinear activation function with the help of sound waves allows the implementation of this process in existing optical fibre systems as well as photonic chips.

Future directions
With our recent European Research Council (ERC) Consolidator funding, we hope to scale up our approach and to achieve complete, reconfigurable computing architectures that might lead to fabrication-less AI. But, while AI may be our primary focus, it is not the only promising application of optoacoustics. Another research direction that the Stiller Lab is actively investigating is the quantum domain; here too, sound waves in optical waveguides could enable technological progress.

For instance, optically controlled acoustic waves could serve as a quantum memory for quantum information processing, as well as for quantum repeaters, which enable long-distance transmission of quantum information despite signal degradation. Interactions between light and sound waves could also generate the entanglement necessary for quantum communications.

In the first experimental demonstration of its kind, we have already proven that we can achieve strong coupling between optical and acoustic waves in waveguides – a prerequisite for quantum information processing.

Additionally, we have shown that our system exhibits high cooling rates, meaning it can cool down very quickly between operations, suppressing thermal noise that could interfere with quantum states. These experiments will ultimately enable optoacoustic memory for quantum states and entanglement between photons and phonons – “quanta” of vibrational energy that can be thought of as the sound-wave counterparts of photons.

Looking ahead, we believe that using acoustic waves for optical neural networks has great potential to unlock a new class of photonic neuromorphic computing which can be reconfigured spontaneously. On-chip implementations of optical neural networks can also benefit from this approach without additional electronic controls. Further down the line, combining both research avenues – quantum interactions and neuromorphic architectures – could create a powerful toolbox for quantum neuromorphic computing, helping to realise neural networks in photonic quantum hardware based on optical fibres or PICs.

As both the AI and quantum industries incorporate more photonics, adding in sound can help these transformative technologies to flourish in a sustainable way.