Hopfield memory: a magnet that remembers pictures

The picture is 64 × 64 pixels, and each pixel is a spin $s_i = \pm 1$ (gold or dark). Every pair of pixels is coupled with strength $J_{ij}$, and the energy is the Ising energy with those couplings:

A pixel feels a field $h_i = \sum_j J_{ij} s_j$ from all the others and, at zero temperature, simply points along it. Each flip can only lower the energy, so the picture keeps changing until it reaches a valley, a state where every pixel already agrees with its field.

To store pictures $\xi^1, \xi^2, \dots$, Hebb’s rule sets each coupling to how often the two pixels agree across them:

Pixels that are usually alike pull each other the same way; pixels that are usually opposite push apart. Each stored picture then sits at the bottom of its own valley. The rule is named after Donald Hebb, who suggested in 1949 that connections between neurons strengthen when the neurons fire together.

Every stored picture comes with its negative, every pixel flipped, which is a valley just as deep: the energy does not change when all the spins flip. Start from noise and you will sometimes land in one.

Hebb’s rule works well for unrelated random pictures: a network of N pixels holds about 0.14 N of them before recall breaks down (Amit, Gutfreund and Sompolinsky, 1985). Pictures that resemble each other are another matter. The letters here share most of their background, so their valleys overlap, and a damaged W can settle into a blend of letters that is none of them. Choose Hebb gets confused to see it.

The projection rule corrects for this. It measures how much each pair of stored pictures overlap and builds the couplings from the inverse of that table of overlaps, so that every stored picture is an exact fixed point however alike they are (Personnaz, Guyon and Dreyfus, 1985). The price is that the couplings no longer come from a simple local rule: each one depends on all the pictures together.

Above zero temperature a pixel sometimes points against its field. A little heat does no harm, and can even shake the picture out of a shallow spurious valley. Too much, and recall melts: the overlap with the memory never reaches 1 and eventually falls to zero. This is a phase transition of exactly the kind elsewhere in the lab, with the overlap as the order parameter.

• Raise Damage towards 50% and press Corrupt. Around 40% the W is barely visible, and still comes back.
• Rub out half of the O with the brush. The rest of it is enough to restore the whole.
• Draw your own picture, store it, then corrupt it. Simple bold shapes come back best.
• Add random memories three at a time under Hebb’s rule and watch recall of the letters get worse.

A Hopfield network is not how the brain stores memories in any detail, and it is not how modern image models work. It is the clearest example of an idea that runs through both: memory as an energy landscape, with recall as rolling downhill. Later work connected it closely to machine learning. Geoffrey Hinton, who shared the 2024 prize, built on it with the Boltzmann machine, and in 2020 Ramsauer and colleagues showed that a modern, continuous version of the Hopfield update is the attention step at the heart of transformer models.

• Ising model Up-or-down magnets that line up below one exact temperature, and lose it above.
• Clock and XY models Arrows that can point any way. No magnet forms, yet vortices pair up below T_BKT.
• Potts model q colours instead of two. Past four, the transition stops being gradual.
• Percolation and forest fires Plant trees at random and light one edge. Past 59.27% the fire gets across.
• Avalanches and crackling noise Disordered magnets flip in bursts of every size as the field rises.

New to all this? The lab’s introduction explains order parameters, critical points and how the simulations work.

• J. J. Hopfield (1982), Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. USA.
• D. J. Amit, H. Gutfreund and H. Sompolinsky (1985), Storing Infinite Numbers of Patterns in a Spin-Glass Model of Neural Networks, Phys. Rev. Lett..
• L. Personnaz, I. Guyon and G. Dreyfus (1985), Information storage and retrieval in spin-glass like neural networks, J. Physique Lett..
• H. Ramsauer and colleagues (2020), Hopfield Networks is All You Need, arXiv.