# Catastrophic Forgetting by Language models

**URL:** <https://discuss.huggingface.co/t/catastrophic-forgetting-by-language-models/173863>\
**Category:** Research\
**Created:** [February 27, 2026, 6:08pm UTC](https://discuss.huggingface.co/t/catastrophic-forgetting-by-language-models/173863 "2026-02-27T18:08:43Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fourwheels2512](https://avatars.discourse-cdn.com/v4/letter/f/e9c0ed/32.png) [@Fourwheels2512](https://discuss.huggingface.co/u/Fourwheels2512)\
**Post date:** [February 27, 2026, 6:08pm UTC](https://discuss.huggingface.co/t/catastrophic-forgetting-by-language-models/173863/1 "2026-02-27T18:08:43Z")

</div>

To all the awesome experts in AI/ML out there. i need a favor.  
I realized there is a gap in Language Models (SLMs/LLMs) remembering the data continuously which is termed as ‘catastrophic forgetting’.

To solve that problem I came up with an adapter called Constrained Residual Mixing Adapter (CRMA) that enables continual learning. I tested it on TinyLlama 1.1B and Mistral 7B — the result: -0.1% drift across 4 sequential  
domains. Essentially zero forgetting.

CRMA: -0.1% drift. Naive: +351% forgetting. Same model, same data, same hardware.

Holds at both 1.1B and 7B. No replay, no EWC, no KD needed.  
● CRMA Modular vs Naive — Mistral 7B (4 sequential domains)

┌─────────┬────────────┬──────────────────┐  
│ Task │ CRMA Drift │ Naive Forgetting │  
├─────────┼────────────┼──────────────────┤  
│ Medical │ -0.2% │ +228% │  
├─────────┼────────────┼──────────────────┤  
│ Legal │ -0.1% │ +593% │  
├─────────┼────────────┼──────────────────┤  
│ Code │ -0.1% │ +233% │  
├─────────┼────────────┼──────────────────┤  
│ Finance │ +0.0% │ — │  
├─────────┼────────────┼──────────────────┤  
│ Average │ -0.1% │ +351% │  
└─────────┴────────────┴──────────────────┘

Now the favor - If you’re interested in independently verifying these results, I’d love to hear from you. DM me and I’ll share what you need to reproduce it. Thank you. and best wishes
