Abstract
A recurrent Transformer with fixed-size memory and coupled prefiller-decoder training improves long-context modeling while enabling efficient parallel training and reduced inference cost.
We introduce , a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. consists of two coupled models: a prefiller Q, which leverages full attentionIn practice, we use interleaved full and sliding-window attention for Q, as this yields stronger performance. The essential requirement is that Q be more expressive than P, with access to the full history. to produce memory targets m'_t, and a decoder P, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories m_t for next-token prediction. We train with a memory consistency loss that aligns m_t with m'_t, allowing inference to use P alone. Empirically, improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between P and Q reduces parameter memory while preserving most of the gains.
Community
We introduce Maglev, a recurrent Transformer architecture with fixed-size memory
that generalizes sliding-window attention while remaining parallelizable during
training. Maglev consists of two coupled models: a prefiller Q, which leverages
full attention to produce memory targets m′_t, and a decoder P, which uses only
sliding-window attention and recurrent K/V injection to produce decoder memories
m_t for next-token prediction. We train Maglev with a memory consistency loss
that aligns m_t with m′_t, allowing inference to use P alone. Empirically, Maglev improves validation loss and downstream pretraining benchmarks over slidingwindow and latent recurrent transformer baselines. Moreover, sharing parameters between P and Q reduces parameter memory while preserving most of the gains.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling (2026)
- The State-Prediction Separation Hypothesis (2026)
- Training Hybrid Block Diffusion Language Models with Partial Bidirectionality (2026)
- Test-Time Training with Next-Token Prediction (2026)
- Learning What to Remember: Test-Time Training via Context Distillation (2026)
- Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models (2026)
- A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2608.02870 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper