Ammu-1.0-FactCheck-tiny

Fact-Check Need Classifier | ModernBERT Architecture | 32M Parameters | 96.35% Macro-F1

Overview

Ammu-1.0-FactCheck-Need-tiny is a lightweight encoder model trained to classify user prompts into (FACT_CHECK_NEEDED) or does not (NO_FACT_CHECK_NEEDED). It is trained on a diverse, high-quality synthetic dataset of user prompts labeled as FACT_CHECK_NEEDED NO_FACT_CHECK_NEEDED. It achieves an overall Macro-F1 score of 96.35%, outperforming much larger 307M encoder baselines across all difficulty tiers (easy, medium and hard). This model can be used an upfront guardrail that directs queries to retrieval-augmented generation (RAG) pipelines or web search engines only when factual grounding is genuinely needed.

Key Features

Achieves a 96.35% overall Macro-F1 score (99.27% Easy, 96.33% Medium, 94.94% Hard), outperforming the 307M parameter baseline by more than +27% Macro-F1 points.
Built upon ModernBERT architecture with 150M parameters
Trained on balanced synthetic data spanning 120 granular topics, 19 prompt interaction formats, and 29 user personas.

Use Case

Serves as an efficient user prompt classifier that saves LLM token costs and latency by bypassing unnecessary search or retrieval for non-factual queries (creative writing, coding, ideation, math puzzles, and text transformations).

Usage

# First install the Hugging Face transformers library
!pip install transformers torch

from transformers import pipeline

# Initialize the Fact-Check classification pipeline
classifier = pipeline(
    "text-classification",
    model="kalyan-ks/Ammu-1.0-FactCheck-Need-tiny",
    device=0  # Set to 0 for GPU, or -1 for CPU
)

# Example input prompts
prompts = [
    "India is the most populous country in the world. Do you agree?",
    "Assume that I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. How old I am today?",
    "I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. My age?",
    "Kalyan KS is from India. Extract named entities from the given sentence.",
    "Extract named entities: Kalyan KS is from India.",
]

# Run classification inference
results = classifier(prompts)

# Display predictions
for prompt, res in zip(prompts, results):
    print(f"Prompt: {prompt}")
    print(f"Prediction: {res['label']} (Confidence: {res['score']:.4f})\n")

Output

Prompt: India is the most populous country in the world. Do you agree?
Prediction: FACT_CHECK_NEEDED (Confidence: 0.6767)

Prompt: Assume that I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. How old I am today?
Prediction: FACT_CHECK_NEEDED (Confidence: 0.9998)

Prompt: I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. My age?
Prediction: FACT_CHECK_NEEDED (Confidence: 1.0000)

Prompt: Kalyan KS is from India. Extract named entities from the given sentence.
Prediction: NO_FACT_CHECK_NEEDED (Confidence: 1.0000)

Prompt: Extract named entities: Kalyan KS is from India.
Prediction: NO_FACT_CHECK_NEEDED (Confidence: 1.0000)

Model Evaluation

This model was evaluated on a curated holdout benchmark test set of 4,214 balanced instances across three difficulty tiers (Easy, Medium, and Hard). Macro-F1 scores are reported below:

Model	Size	Easy	Medium	Hard	Overall
Vela-1.0-Encoder-307M-FactCheck	307M	77.72	70.64	64.15	69.42
Ammu-1.0-FactCheck-Need-base	150M	98.91	96.74	95.23	96.56
Ammu-1.0-FactCheck-Need-small	68M	98.55	95.97	95.11	96.13
Ammu-1.0-FactCheck-Need-tiny	33M	99.27	96.33	94.94	96.35
Ammu-1.0-FactCheck-Need-nano	17M	98.91	95.56	94.18	95.66

Limitations

Language Scope: The model has been trained and evaluated predominantly on English language texts; classification accuracy on multilingual or code-switched inputs may be lower.
Classification vs. Fact Verification: This model only predicts the need for factual verification based on the user's input intent; it does not check the truth value of factual claims or generate factual answers.
Highly Ambiguous or Mixed Prompts: Prompts that seamlessly mix complex creative writing with subtle real-world factual claims in a single query may present borderline confidence scores and benefit from threshold tuning.

Citation

@misc{ammu-1.0-factcheck-need-tiny-2026,
  title = {Ammu-1.0-FactCheck-Need-tiny: Fact-Check Need Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-FactCheck-Need-tiny}
}