What’s the best strategy for fine-tuning a large language model (LLM) on domain-specific data without catastrophic forgetting?

Beginners
1

How can we fine-tune a large model for a specific domain while still keeping its general knowledge and skills?

2

Methods include teaching while incorporating conventional knowledge during fine-tuning and excluding (freezing) certain parts of the network from fine-tuning, and these are often used in combination.

3

Good question, the answer depends on which version of “keep general knowledge” you actually need, and it’s worth separating the two:

  1. You need the facts available but not reasoned over → don’t fine-tune, use RAG. Retrieval leaves the base model untouched, so there’s zero forgetting by definition. Best when your domain is mostly lookup.

  2. You need the model to actually internalize the domain (reason and generate in it) → you have to touch the weights, and that’s where forgetting bites. The standard tools (as John6666 said) are mixing general data back in during training + freezing layers. They work, but they’re fiddly — you keep a general-data replay set around and tune how much to mix, and freezing trades away how much the model can actually learn.

A lower-friction route in that second case is a constrained adapter: instead of replaying or penalizing to recover general knowledge after the fact, you constrain the update so it can’t overwrite base capabilities in the first place. No replay set, no penalty coefficient to tune. In our continual-learning tests this held ~0% drift on prior knowledge while still learning the new domain, where plain sequential LoRA degraded several hundred percent.

If you share your model + domain I’m happy to point you at the exact setup, or run a quick before/after so you can see the retention numbers on your own data.

-– Ashwin (working on this at ModelBrew — modelbrew.ai)