Google's instruction-tuned Gemma 2 2B, exactly as shipped — no fine-tuning, no retrieval. It answers Yu-Gi-Oh from whatever it absorbed in pretraining: fluent, confident prose that fabricates modern card facts. This is the control the whole experiment measures against.
Ask the untouched base anything and read its answer. It replies from pretraining memory only — no fine-tuning, no lookup — so it is fluent but fabricates modern card facts and stays faithful to real card text only 0.18/2 of the time.
The system reads your question and writes an answer straight from what it memorised in pretraining — nothing is looked up.
Where this system lands against the other two on the 60 held-out biased questions, scored blind by a reference-grounded judge out of 10. Green is this system; tan are the other two.
Measured mean judge /10 on the 60 held-out (biased set), scored blind by a reference-grounded judge (gemini-3.1-flash-lite). This system in green; the other two in tan.
| System | Biased 60 | Unbiased 60 |
|---|---|---|
| A · base | 3.98 | 1.93 |
| B · fine-tune | 5.25 | 2.62 |
| C · +retrieval | 8.05 | 8.25 |
Mean /10 on the biased and the equal-split unbiased 60. Paired deltas — fine-tuning A→B: +1.27 (p=0.007 biased; +0.68 marginal, p=0.041 unbiased). Retrieval B→C: +2.80 (p<0.001 biased; +5.63 unbiased).
Each rung keeps everything below it and adds one thing — but the big jump in real facts only comes at the top, with retrieval.
All three systems are the same Gemma 2 2B — 26 layers, grouped-query attention (8 heads sharing 4 KV), a sliding 4,096-token window, and a GeGLU feed-forward. Run 4-bit at inference.
| Class | Gemma2ForCausalLM |
| Layers | 26 |
| Hidden size | 2,304 |
| Attention | 8 heads / 4 KV · dim 256 · GQA |
| Feed-forward | GeGLU · inner 9,216 |
| Attention window | sliding 4,096 · logit soft-cap |
| Norm | RMSNorm |
| Context | 8,192 tokens |
| Vocabulary | 256,128 |
| Embeddings | tied input/output |
| Inference | 4-bit NF4 |
The Gemma 2 2B configuration shared by all three systems.
The prompt is turned into numbers, passed through the same small stack of maths 26 times, then turned back into the next word.
The corpus and pipeline that built Systems B and C — shown here for context, since the base itself does no training of ours and was imported free.
System A is the untouched base — no training of ours. Loaded 4-bit (NF4) at inference; the corpus and pipeline shown here are what Systems B and C are built from.
System A is the imported base — no training of ours; the QLoRA rows apply to Systems B and C.
Measured corpus composition by source · 27.5 MB total, 78% free-licensed Yugipedia prose + 22% YGOPRODeck card facts.
Free web text is cleaned, a teacher writes practice questions, a strict judge throws out the weak ones, and what survives trains the model.