Gali0.6 preview · draft

Continuing a multilingual autoencoder as it grows

A study, Gali 0.6 preview. Elamur, 19–20 November 2026. A note on measurements, not a full paper.

Question. The autoencoder keeps gaining tasks and languages. Should it start over each time, or continue from its checkpoint?

Practice. We continue while the vocabulary layout and the architecture stay the same, and start over only when they change. Each change is followed by a dip, then by recovery:

changeskill watchedbeforeafter 5 000 stepslater
seeds and roles planted in every streamchains of numbers95 %47 %78 % at 10 000
from 30 to 76 languageschains of numbers78 %39 %58 % at 15 000
from 30 to 76 languagesmarkup, mixed held-out set70 %76 %73 % at 15 000
from 76 to about 140 languageschains of numbers58 %64 %67 % at 10 000

Replay is already in. The old tasks are never dropped: every continuation keeps numbers, shared prompts and text of all the earlier languages in the same mix (1 : 1 : 3), and only adds. So the dip is not forgetting for lack of old data. It comes from the format: seeds and roles put new tokens into every stream, old and new, so every skill has to relearn where its tokens go. Weight protection such as elastic weight consolidation (Kirkpatrick et al. 2017) is the next thing to try if the dips grow.

Reading. A new token role disturbs everything the decoder writes, so the skills that depend on exact sequences, such as numbers, fall hardest and come back within some thousands of steps. New languages cost the old skills less than new roles do. Markup is measured on a held-out set drawn from the current data, which changes with the languages; it is a trend, not a series of the same test.

Why continue. A start from scratch spends the same steps relearning the bytes, n-grams and markup that the checkpoint already has; the dip costs less than that. The same holds for Gali: a new autoencoder checkpoint does not restart Gali, which continues from its own.

Limits. No run from scratch was made for comparison; the cost of starting over is an estimate from the first runs' learning curves. Related. The dip is the catastrophic interference of McCloskey and Cohen (1989), softened here by keeping the old tasks in the mix rather than by protecting weights as elastic weight consolidation does (Kirkpatrick et al. 2017). Adding languages to a fixed model also spreads its capacity thinner (Conneau et al. 2020).

References