Gali0.6 preview · draft

When the model copies the answer from its context

A study, Gali 0.6 preview. Elamur, 20 November 2026. A note on a negative result, not a full paper.

Setup. One Gali — a 20-block trunk of width 512 with two branches, 89 million parameters — trained in social scenes: alone, listening to another copy of Gali, talking, being interrupted, overhearing two others; a background stream of related Wikipedia paragraphs at times. In most scenes the input holds, beside the source sentence, a noisy copy of an answer written by another copy of Gali. Translation pairs are human ones from Tatoeba across many languages.

Control. The same model given another source sentence of the same direction. A model that reads its source scores higher with the right one.

afterchains of numbersword F1, right sourceword F1, another source
12 000 steps72 %0.0890.092
24 000 steps78 %0.0750.072
+ 20 000 with half the items plain73 %0.1360.133

Reading. The source reaches Gali — chains of numbers need it and are mostly right — but translation ignores it: copying the other streams was the cheapest way to lower the loss. Making half of the translation items plain, with the source alone, raised the score on both sides equally, so the model learnt the target languages' typical words, not translation. The next suspect is the number of directions at once (128 in the evaluation); on English and French alone a small Gali followed its source clearly (0.63 against 0.13).

Lesson. Any context that contains a version of the answer needs its own control: the right source against another source, with everything else in the input kept. Related. Copying the streams is a shortcut in the sense of Geirhos et al. (2020): a cue that lowers the training loss without the skill the task was meant to teach. Translation is scored with word F1 and chrF (Popović 2015) on Tatoeba pairs (Tiedemann 2020).

References