Grammar Corrector with a Small Language Model

What you'll build

Fine-tune a small open-source language model (SmolLM2-360M or Qwen2.5-0.5B) to correct grammatical errors in short English sentences written by learners. Train on a learner-English corpus (JFLEG or BEA-2019), evaluate properly with M2 score (the official Grammatical Error Correction metric) and GLEU, and compare against an open rule-based tool (LanguageTool) and a hosted Large Language Model (LLM) used zero-shot. The clear before/after demo plus the existence of well-defined metrics makes this an ideal first NLP project.

What goes in, what comes out

source:  I am very happy because I do not has anything to do.
refs:    [
  "I am very happy because I do not have anything to do.",
  "I am very happy because I have nothing to do.",
  "I am very happy because I do not have anything to do.",
  "I'm very happy because I have nothing to do."
]

source:  She go to school every day with her friend, but yesterday she don't go.
refs:    [
  "She goes to school every day with her friend, but yesterday she did not go.",
  "She goes to school every day with her friend, but yesterday she didn't go.",
  "She goes to school with her friend every day, but yesterday she did not go.",
  "She goes to school every day with her friend, but yesterday she didn't."
]

Approach                        GLEU    M2 F0.5    Latency (ms/sent)
----------------------------    -----   --------   -----------------
No-change (copy input)          40.5     0.00            ~0
LanguageTool (rule-based)       49.2     34.1            ~30
SmolLM2-360M SFT                55.7     49.4            ~80
Qwen2.5-0.5B SFT                57.1     51.0            ~95
LLM zero-shot                   58.4     53.2           ~600

Per error type (SmolLM2-360M SFT, F0.5):
  Missing article ("a/the")         0.62  ← strong
  Wrong verb tense                  0.51
  Subject-verb agreement            0.55
  Wrong preposition                 0.32  ← still weak
  Spelling errors                   0.41
  Word order                        0.28  ← hardest

Datasets

Tools you'll need

These are suggestions, not requirements. If your team is more comfortable with a different library, model, or framework that achieves the same goal, use it — and briefly explain the choice in your README.

Python: Python 3.10 or newer. Compute: Colab T4 / Kaggle P100 (16 GB GPU) is comfortable for SmolLM2-360M / Qwen2.5-0.5B SFT. Inference runs on a laptop CPU in real time.

• transformers — Loads HuggingFaceTB/SmolLM2-360M or Qwen/Qwen2.5-0.5B with its tokeniser.
• trl — Provides SFTTrainer for the supervised fine-tuning loop.
• accelerate — Mixed-precision and device placement.

• language-tool-python — Wraps the LanguageTool rule-based grammar checker. The traditional baseline.
• openai / anthropic / groq — For the zero-shot hosted-LLM baseline.

• errant — The official ERRANT scorer for GEC. Produces M2 scores and a per-error-type breakdown.
• sacrebleu — GLEU is implemented inside sacrebleu (or via the original gleu.py script).
• evaluate — Wrapper for ROUGE-style metrics.

How to approach it

One reasonable path through the project. Specific tools (UMAP, HDBSCAN, BERTopic, etc.) are examples — feel free to swap them for alternatives you know better.

1. Pick the base model. SmolLM2-360M as the default. Qwen2.5-0.5B is a strong alternative.
2. Load training data. BEA-2019 + W&I + LOCNESS for training (around 40k pairs); JFLEG for dev / test. Optionally pretrain on Lang-8 first if compute allows.
3. Format with the chat template. Each row becomes user "Correct the grammar in this sentence:\n[source]" → assistant "[target]". Use tokenizer.apply_chat_template.
4. Run SFT. trl.SFTTrainer, 2–3 epochs, learning rate 2e-5, bf16 if supported.
5. Baseline 1 — Copy. Score the no-change baseline (output = input). This is the absolute floor.
6. Baseline 2 — LanguageTool. Run LanguageTool on the test set, take its top suggestion, score.
7. Baseline 3 — Zero-shot LLM. Prompt a hosted LLM with the same instruction. Be deterministic.
8. Score with ERRANT. Run errant_parallel to align corrections, then compute M2 F0.5 against gold references. Also compute GLEU.
9. Break down per error type. ERRANT tags edits as M (missing), R (replace), U (unnecessary) per type. Compute per-type F0.5 to see what the tuned model is good and bad at.

What to deliver

• The fine-tuned model checkpoint (HuggingFace repo or local artifacts).
• A reproducible training + inference notebook.
• A results table with GLEU, M2 F0.5, and latency for all four approaches.
• A per-error-type breakdown for the tuned model, with at least 5 example corrections (good and bad) per type.
• A short README documenting base model, training data, hyperparameters, and the team's observations.

References

• ERRANT scorer (Bryant et al., 2017)
• JFLEG dataset card
• BEA-2019 shared task overview
• GLEU paper (Napoles et al., 2015)
• SmolLM2 model family
• TRL SFTTrainer guide
• LanguageTool