Recurrent Entity Network
Recurrent Entity Network
The **Recurrent Entity Network** is equipped with a dynamic long-term memory which allows it to maintain and update a representation of the state of the world as it receives new data. For language understanding tasks, it can reason on-the-fly as it reads text, not just when it is required to answer a question or respond as is the case for a [Memory Network](https://paperswithcode.com/method/memory-network). Like a [Neural Turing Machine](https://paperswithcode.com/method/neural-turing-machine) or Differentiable Neural Computer, it maintains a fixed size memory and can learn to perform location and content-based read and write operations.  However, unlike those models it has a simple parallel  architecture in which several memory locations can be updated simultaneously. 

The model consists of a fixed number of dynamic memory cells, each containing a vector key $w_j$ and a vector value (or content) $h_j$. Each cell is associated with its own processor, a simple gated recurrent network that may update the cell value given an input. If each cell learns to represent a concept or entity in the world, one can imagine a gating mechanism that, based on the key and content of the memory cells, will only modify the cells that concern the entities mentioned in the input. There is no direct interaction between the memory cells, hence the system can be seen as multiple identical processors functioning in parallel, with distributed local memory. 

The sharing of these parameters reflects an invariance of these laws across object instances, similarly to how the [weight tying](https://paperswithcode.com/method/weight-tying) scheme in a CNN reflects an invariance of image statistics across locations. Their hidden state is updated only when new information relevant to their concept is received, and remains otherwise unchanged. The keys used in the addressing/gating mechanism also correspond to concepts or entities, but are modified only during learning, not during inference.
Tracking the World State with Recurrent Entity Networks
https://paperswithcode.com/paper/tracking-the-world-state-with-recurrent
Tracking the World State with Recurrent Entity Networks