← Work

I trained a LoRA adapter over SmolLM2-360M to extract typed factors from football match narratives, and then it failed the gate I had written before training it. Four thresholds were declared up front: minimum F1 gain, no attribution regression, a bounded increase in fallback rate, and schema validity. The adapter cleared one of them. It got marginally better at classifying factor type (+0.001 F1 against a required +0.15), regressed on team attribution, and nearly doubled the fallback rate because it emitted near-zero severity and certainty, which the downstream zero-weight check correctly flagged. The base model plus a deterministic fallback stayed in production. Both the adapter and the dataset are published anyway — the threshold table and the frozen test set outlive the weights, and a negative result nobody can check is not a result.

Most fine-tuning write-ups are success stories, which is how you can tell most of them are missing a gate. If the only outcome a project can produce is "it worked," the evaluation was built after the training run and shaped to fit it.

This one has the opposite shape. I wrote the thresholds first, trained the adapter, ran it against a frozen test set, and it lost to the model it was supposed to beat.

The task

The Underdog Lab forecaster needs match narratives turned into something numeric. A sentence like "Norway are without their first-choice keeper and travel on two days' rest" has to become typed factors — each with a severity, a certainty, and the span of text it came from — drawn from a fixed 14-factor taxonomy.

The part that makes it hard is the part that isn't extraction. The schema has two slots beyond factors:

• unsupported_claims — assertions in the text that must not become factors
• ambiguities — things the text genuinely leaves open, which must be marked rather than resolved

A model that scores well by confidently inventing plausible factors is exactly the model this task is designed to reject.

The gate, written first

Four checks, with thresholds fixed before the training run:

Training data was 861 compositional synthetic scenarios covering negation, ambiguity, irrelevant commentary, unsupported claims, and prompt-injection attempts. Evaluation ran against a frozen 98-example test set that was never used for prompt iteration or training.

The adapter itself is unremarkable on purpose: LoRA at r=16, alpha=32, dropout 0.05, on q_proj and v_proj over SmolLM2-360M-Instruct. Both models were quantised to Q8_0 and served through the same runtime, so the comparison isolates the weights.

The result

The gate report is a committed artifact, not a note I wrote afterwards:

{
  "computed_decision": "NO-SHIP",
  "declared_decision": "NO-SHIP",
  "failed_checks": [
    "minimum_f1_gain",
    "no_attribution_regression",
    "fallback_rate_controlled"
  ],
  "selected_runtime": "base_plus_fallback",
  "claim": "The adapter is not shipped. The base model plus deterministic fallback remains the production extraction path."
}

Why it failed

The adapter got slightly better at the easy half of the task — naming which of the 14 factor types a sentence contains — and worse at everything that required judgement.

The specific failure is instructive: it learned to emit near-zero severity and certainty on almost every factor. Under the scoring contract, a factor with zero weight carries no information, and the downstream zero-weight check treats a run of them as a hallucination signal and falls back to the deterministic parser. So the fallback rate nearly doubled, which is why a model that looks fractionally better on F1 is materially worse in production.

Fine-tuning taught it the vocabulary and cost it the calibration.

A negative result nobody can check is not a result.

What shipped instead

The base model plus a deterministic fallback. That was already the production path, and it stayed the production path.

Both artifacts are public anyway — the adapter and the dataset — with the threshold table and the comparison on the cards. The weights are the least interesting output of this project. The gate, the frozen set, and the SHA-256-pinned evaluation contract are reusable; the adapter is a data point about what a 360M model does when you ask it for calibrated confidence.

What I'd do differently

Three things, in order of how much I think they'd matter:

1. Put severity and certainty in the loss explicitly. They were treated as ordinary tokens in a JSON blob. A task where calibration is the whole point should not learn calibration incidentally.
2. Review the training data. All 861 records are still review_status: pending. The dataset's own README says metrics are not claim-ready until they're reviewed, and the gate inherited that weakness.
3. Check the ceiling before training. Absolute F1 is low for both models. That is a signal the task is not reachable at 360M parameters, and I should have established the ceiling with a larger model before spending a run on the small one.

None of that makes the outcome a failure. The gate did exactly what it was built to do, which was stop me from shipping this.

Related notes

• Evals are compressed production failures
• Fine-tuning is a last resort, not a first move