Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks release_biectlv2ffec5mzkwklejc3g6y

by Giorgio Gosti, Viola Folli, Marco Leonetti, Giancarlo Ruocco

Published in Entropy by MDPI AG.

2019   Volume 21, Issue 8, p726

Abstract

In a neural network, an autapse is a particular kind of synapse that links a neuron onto itself. Autapses are almost always not allowed neither in artificial nor in biological neural networks. Moreover, redundant or similar stored states tend to interact destructively. This paper shows how autapses together with stable state redundancy can improve the storage capacity of a recurrent neural network. Recent research shows how, in an N-node Hopfield neural network with autapses, the number of stored patterns (P) is not limited to the well known bound 0.14 N , as it is for networks without autapses. More precisely, it describes how, as the number of stored patterns increases well over the 0.14 N threshold, for P much greater than N, the retrieval error asymptotically approaches a value below the unit. Consequently, the reduction of retrieval errors allows a number of stored memories, which largely exceeds what was previously considered possible. Unfortunately, soon after, new results showed that, in the thermodynamic limit, given a network with autapses in this high-storage regime, the basin of attraction of the stored memories shrinks to a single state. This means that, for each stable state associated with a stored memory, even a single bit error in the initial pattern would lead the system to a stationary state associated with a different memory state. This thus limits the potential use of this kind of Hopfield network as an associative memory. This paper presents a strategy to overcome this limitation by improving the error correcting characteristics of the Hopfield neural network. The proposed strategy allows us to form what we call an absorbing-neighborhood of state surrounding each stored memory. An absorbing-neighborhood is a set defined by a Hamming distance surrounding a network state, which is an absorbing because, in the long-time limit, states inside it are absorbed by stable states in the set. We show that this strategy allows the network to store an exponential number of memory patterns, each surrounded with an absorbing-neighborhood with an exponentially growing size.
In application/xml+jats format

Archived Files and Locations

application/pdf  588.0 kB
file_yw3lskkzuveozp7oska357c5hy
web.archive.org (webarchive)
res.mdpi.com (publisher)
Read Archived PDF
Preserved and Accessible
Type  article-journal
Stage   published
Date   2019-07-25
Language   en ?
DOI  10.3390/e21080726
PubMed  33267440
Container Metadata
Open Access Publication
In DOAJ
In ISSN ROAD
In Keepers Registry
ISSN-L:  1099-4300
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 263f22b6-3d7f-4371-ade1-04e19a9aa77b
API URL: JSON