Fine-tuning is the easy part
What a 2-hour hackathon taught us about small models, and about labels
Most teams reach for the common frontier models for every language task (gpt, claude, gemini), through someone else's API. For a fixed task that runs thousands of times, that default costs money per call, ties the work to a provider, sends the data out of the organisation that holds it, and uses far more computation than the task needs. We wanted to see how close a small model you own could get, on a real production task.
So we ran a 2-hour internal hackathon at Baobab Tech (yes hours, …its 2026), followed by about 3 hours of clean-up and documentation. The model part turned out to be easy. What we learned is mostly about the data.
The task
EvalExplorer collects evaluation reports from international development organisations. When a report arrives, its first pages go to a large LLM (gpt-oss-120b, 117B parameters), which returns five labels: the evaluation's approach , type , timing , themes and countries .
An evaluation report's first pages go into a fine-tuned 2B model, which returns five labels as JSON
We took 1,420 reports the pipeline had already labelled and fine-tuned 9 small open models (350M to 26B parameters) on 1,148 of them with LoRA on Hugging Face Jobs. Counting zero-shot baselines, reinforcement-learning variants and quantized exports, we scored 55 variants on 134 held-out reports: how often does each give the same labels as the pipeline?
It works, on average
| Model | Agreement with the pipeline | Before fine-tuning | Training cost |
|---|---|---|---|
| Qwen3.5 2B | 84.7 | 45.8 | $3.20 |
| Qwen3.5 4B | 84.7 | 67.1 | $1.60 |
| Gemma 4 26B-A4B | 84.4 | 70.1 | $2.90 |
| LFM2.5 350M | 79.2 | 20.9 | $0.29 |
Agreement is the mean over the five fields (1 or 0 for approach, type and timing, F1 for themes and countries), 0 to 100. The 2B figure includes a GRPO step on top of fine-tuning; fine-tuning alone scores 84.2 for $0.89.
Agreement with the pipeline before and after fine-tuning, per model: every LLM jumps to 79-85, and size stops mattering past about 2B parameters
A 2B model, about 60 times smaller than the pipeline's LLM, matches it as well as models 2 and 13 times its size. As a 4-bit GGUF file, Qwen3.5-4B is 2.8 GB and still scores 84.1: small enough to run on hardware an organisation already has, with its documents never leaving it. That is the sovereign and frugal case for small models (on AI's footprint, see Baobab Tech's sourced corpus research-ai-energy).
The full results, run by run, are on the results page.
If we had stopped here, the story would be "a 2B model replaces a 117B one for under a dollar". It would be true on average, and misleading.
The average hides the failures
Only about 1 report in 4 has all five fields right, and the errors are not spread evenly:
"Labellers agree": how far the pipeline and a 2-of-3 majority of three newer LLMs agree on that code across all 1,420 reports. "Model finds it": recall of the fine-tuned 2B model on the test reports (rough for rare codes, which have 2 to 10 test examples). Every code is on the results page.
One dot per code: training examples against labeller agreement. The codes the model misses are rare, loosely defined, or both
Common, clearly defined codes are found 80-95% of the time. Codes that are rare, or that the labellers themselves cannot agree on, are found far less often. For a search tool, those are exactly the codes that matter: a missed "rapid evidence assessment" hides a report from everyone filtering for one.
Why: the labels, not the model
Some codes are loosely defined. Each code reached the labelling LLM as a one-line definition, and some overlap almost word for word: economic development is "development finance, infrastructure"; international finance is "development finance, private sector". With 287 training examples, labellers still agree on economic development only 31% of the time. A model cannot learn a distinction its labels do not make consistently.
The full definitions never made it in. Our original taxonomy has full definitions and long keyword lists for every theme. The pipeline used one-line summaries, and the models were trained on code names alone. The full taxonomy also overlaps in places (growth and economic development share their trade and economy keywords), so it needs work too.
Some codes are rare. Developmental has 16 training examples, rapid evidence assessment 34. Rare codes with a clear definition are learned (nature and environment: 32 examples, all 3 test cases found); rare and loosely defined ones are not.
The labels are LLM output, and we treated them as gold. That was the right shortcut for a hackathon. Three newer LLMs (GLM-5.3-Flash, DeepSeek-V4.1-Flash, Qwen3.8-2.4T-A95B) relabelled all 1,420 reports: they agree with each other at 86-88 and with the pipeline at 74-76. But agreement is not correctness. On the 36 reports a person had checked, all three leave the evaluation approach blank where people gave one. Without human-verified labels, we can measure how consistent the labels are, not whether they are right.
Pairwise agreement of the four label sets: the three newer LLMs agree at 86-88, the pipeline at 74-76 with each
What we learned
- Fine-tuning a small model for a defined task is now a few hours and a few dollars. All 55 variants together cost about $45 of compute. Supervised fine-tuning did almost all the work; reinforcement learning helped only with a targeted reward, and 8-bit quantization cost nothing.
- An average score is not enough to decide. Report per field and per code, and decide which errors matter for the use.
- The model can only be as good as its labels. On well-defined codes, a 2B model is as good as a 117B one. On loosely defined ones, no model is good, because the labels are not.
- Data preparation is the work. In order: a precise codebook (full definitions, an example and a counter-example for each pair of neighbouring codes, a rule for blanks); human verification of a few hundred reports, weighted towards the weak codes; at least 50 verified examples of every code, rare ones included; a test set that can measure each code. Then retrain, which is the cheap part.
What's next
We are not putting this model into production. It is not good enough on the codes that matter most for search, and the fix is more and better data. As more reports come in, we will build a better training and test set (the full taxonomy in the prompts, its overlaps resolved, human-verified labels for every code) and keep experimenting until the model scores high enough on every code, not just on average. We are also exploring the same approach for excerpt tagging: labelling findings, recommendations and methods inside each report rather than the whole document.
If you have a defined task and about a thousand labelled examples, try it. Then spend the time you saved on your labels.
AI use disclaimer: AI (Claude Opus 5.5) was used extensively to build the training harness, documentation and writing.
On Hugging Face
Everything is public:
- Results page : the story, the best run per model, the per-code reliability table, and a browser for all 55 runs.
- Collection : every model, dataset and Space in one place.
- Experiments : one report per run with predictions and scores, the handover, the label-quality follow-on and all the code.
- Data : the 1,420 reports, the pipeline's labels and the three LLM relabellings.
- GGUF files : the fine-tuned models ready for llama.cpp, including Qwen3.5-4B at 2.8 GB.
- Fine-tuned adapters : for example Qwen3.5-4B and Qwen3.5-2B; the GRPO variants are in