This article may be too technical for most readers to understand.(October 2021) |
Modern Hopfield networks[1][2] (also known as Dense Associative Memories[3]) are generalizations of the classical Hopfield networks that break the linear scaling relationship between the number of input features and the number of stored memories. This is achieved by introducing stronger non-linearities (either in the energy function or neurons’ activation functions) leading to super-linear[3] (even an exponential[4]) memory storage capacity as a function of the number of feature neurons. The network still requires a sufficient number of hidden neurons.[5]
The key theoretical idea behind the modern Hopfield networks is to use an energy function and an update rule that is more sharply peaked around the stored memories in the space of neuron’s configurations compared to the classical Hopfield network.[3]