Smallayallm: a Malayalam chat LoRA for Gemma 4 12B
Smallayallm is a LoRA adapter (rank 16, alpha 32, 65.6M trainable parameters, 0.545% of the model) fine-tuned for Malayalam chat on a single NVIDIA DGX Spark (GB10). It is the v0 release of a research project; the full method, measurements, and limitations are in the paper in the GitHub repository: https://github.com/vaishaks/smallayallm
One caveat is stated plainly because it matters for reproduction: training started from the unsloth/gemma-4-12b-it checkpoint, which is the instruct model rather than the raw base checkpoint. Google's base repository was gated at the time, and the Unsloth mirror was the accessible source.
Training
- Data: 61,594 SFT records (Aya Malayalam 1,594; IndicAlign 40,000; BPCC-Human 20,000 across both translation directions). Sources and licences are listed in the dataset card: https://huggingface.co/datasets/vaishaks/smallayallm-sft-v0
- Configuration: bf16, maximum length 1024, completion-only loss, learning rate 1e-4 with cosine schedule, length-grouped batching, effective batch 16, 2 epochs (7,700 steps).
- Measured on the Spark: 140,407 s (39.0 h) wall time, loss from 4.27 (first logged step) to 0.673 (final), peak memory 54.4 GB.
Evaluation
Frozen 48-prompt set (8 categories of 6), base and fine-tuned under identical greedy decoding. Only the automatic and checklist layers are reported. The subjective layer (fluency and register scoring by a native speaker) is pending and is not replaced by an LLM judge.
- Mean script purity, Malayalam-expected prompts (45 prompts): 81.9% -> 89.7%
- Math exact answers: 6/6 -> 6/6
- Constrained-format checks passed: 12 -> 12
- Automatic checklist passes overall: 43 -> 56 (failures 22 -> 9)
- Translation chrF++ against BPCC-Human references: en->ml 25.57 -> 38.73, ml->en 33.45 -> 61.08
- Manglish script discipline (purity): COLL-01 0.0% -> 82.5%, COLL-04 0.0% -> 0.0% (one of the two Manglish prompts fixed, the other unchanged)
- Known regressions in the same run: summarization mean script purity fell 96.3% -> 81.6% (one fine-tuned output, SUMM-02, was a single period character), and script purity regressed on 5 of the 45 Malayalam-expected prompts while improving on 37.
Full per-prompt results are in eval_v0_results.json in the GitHub repository.
Demo
A free Gradio demo Space is linked here once it is live: https://huggingface.co/spaces/vaishaks/smallayallm-demo. ZeroGPU usage is quota'd per visitor account (documented free quota is 5 GPU-minutes per day), so each visitor gets a handful of conversations a day.
Limitations
The human-written share of the training mix is small (the Aya portion plus a thin slice of IndicAlign); most of the mix is Wikipedia-grounded synthetic conversation and gold translation pairs. Six prompts per category is a smoke-test sample, so the evaluation ranks failure modes and cannot support tight statistical claims.