I’m sharing two working papers on the internal geometric dynamics of Large Language Models, focusing on trajectory instability, regime classification, and methodological robustness.
While most evaluations focus on output metrics (perplexity, benchmarks), this work investigates the hidden state trajectories during autoregressive generation to understand how models process information internally.
Paper 1: Four Dynamical Regimes in LLMs
Link: Four Dynamical Regimes in large Language Models : An Empirical Phase Map
We introduce ct_t, a token-level instability metric, and identify four consistent dynamical regimes across 10 models:
Underactive: Rigid, low variance (e.g., TinyLlama).
Adaptive: Balanced flux/stability (e.g., Qwen-2.5 family).
Transition: Boundary zone.
Chaotic: High instability spikes (e.g., Gemma-2B, GPT-2).
Key Finding: Qwen models consistently maintain an “Adaptive” regime, suggesting a structural balance between flexibility and stability that correlates with their robust performance in fine-tuning and reasoning tasks.
Paper 2: Methodological Audit of Trajectory Instability
Link: Conditional Dynamic Signatures in Large Language Models :
A large-scale audit (n=17 models) examining the conditional nature of these measurements. We document:
Normalisation Sensitivity: How CLIPPED_MAD vs RAW impacts regime discrimination.
Variance Decomposition: Architecture explains only 7% of variance, while prompt category explains 17%, and residual noise 76%.
Fragmented Topology: At scale (n=17), clean model “families” dissolve into a continuous spectrum with only two robust local pairs (OPT-Pythia, Phi-Qwen).
Documented Falsifications: Six small-panel hypotheses that did not survive scale-up, reported explicitly to support methodological discipline.
Key Insights for the Community
Stability is Conditional: A model’s dynamic regime depends heavily on the prompt category (e.g., scientific reasoning amplifies architectural differences).
Collapse is Cyclical: We observe a robust COLLAPSE-RIVALRY cycle where models exit low-entropy states 84% of the time, suggesting a self-correcting mechanism rather than permanent failure.
Dynamic ≠ Semantic: Our control audits (V19) show that while we can influence dynamic stability via hidden-state interventions, this does not guarantee semantic correction. Dynamic control is necessary but insufficient for output quality.
Code & Data
All data, metrics, and reference implementations are available in the associated Zenodo repositories. We welcome feedback, replication attempts, and discussions on extending these metrics to larger (7B+) or MoE architectures.
Best regards,
Jean-Denis Bosange Batuli
CEO – IDChain SRL (Unbind)
1 Like