Gali0.6 preview · draft

The multilingual autoencoder that reads and writes for Gali

Draft, Gali 0.6 preview (tied to Mova 0.6). Elamur, November 2026. Open to edits until Mova 0.6 is released.

Abstract

Gali never reads or writes characters itself. One encoder–decoder transformer, trained before Gali and then frozen, turns tokens of every language into states and states back into tokens. It learns to rebuild a text from its bare words together with the markup Mova would write for it — dependency relations, pragmatics, morphological features, case and WordNet concepts — and two synthetic tasks that align the languages. Because the markup weighs only a tenth of its loss, its confident disagreements with Mova point at parser errors, which made it a working tool for Mova before Gali was ready (in practice). This page describes the model, its tasks and what we measured; the main paper describes Gali between its halves, and the vocabulary paper the 8192 token ids it uses.

One set of weights for many languages follows multilingual sequence-to-sequence pretraining: mBART (Liu et al. 2020), T5 and mT5 (Raffel et al. 2020; Xue et al. 2021), and byte-level ByT5 (Xue et al. 2022), whose lesson that bytes make any text writable is kept in the 256 byte ids of every bank. Cross-lingual encoders such as XLM-R (Conneau et al. 2020) and LASER (Artetxe and Schwenk 2019) show a shared space appearing without parallel text, and the «curse of multilinguality» (Conneau et al. 2020) — capacity spread over too many languages — is what we watch as the language count grows. Rebuilding markup from words is a form of structured denoising; reading the encoder's layers with linear probes follows Alain and Bengio (2017) and Belinkov (2022). Using a model's confident disagreements to find annotation errors follows confident learning (Northcutt et al. 2021) and annotation error detection (Dickinson and Meurers 2003; Klie et al. 2023).

Architecture

The autoencoder is an encoder–decoder transformer with one set of weights for every language. Its encoder reads a sequence of tokens bidirectionally; its decoder writes tokens causally and cross-attends to states it is given. Both work in the same vocabulary of 8192 ids (see the vocabulary).

Width384 numbers per position, 6 attention heads
Depth12 encoder blocks and 12 decoder blocks
Blockspre-norm (RMSNorm), self-attention with rotary positions, a GELU feed-forward of 4 × the width; in the decoder, cross-attention without positions
Embeddingone table, shared by the input and the output (tied)
Parameters52.7 million

It is trained to rebuild a text of any of its languages from its bare words, restoring the markup Mova would write — tree relations, pragmatics, features, case and concepts — and on two synthetic tasks described in training. After that it is frozen. Gali uses its encoder to read and its decoder to write; nothing in it changes while Gali learns.

One experiment in this preview adds a long skip inside the encoder: the output of block 4 goes, through a linear map that starts at zero, into the input of block 8. A probe of the encoder's layers showed the token identity fading from 99.6 % at the embedding to 92.7 % after block 12, most of it after block 4 (see measurements). The skip is compared with the same model trained the same way without it.

Training

The autoencoder's tasks: bare words to text with markup, a number chain from one language's tokens to another's, a shared sequence with one language's filler words to the same sequence with another's; and the check of Mova's parsesBare wordsn-grams only, no caseAEencode, decodeText with markuptree, pragmatics, features, conceptsDigits in A's tokensa chain of up to 8 numbersAEencode, decodeThe same digits in B's tokensB's tag seeded inShared ids + A's wordsmarkup, signs, conceptsAEencode, decodeSame shared ids + B's wordsany word of B is rightthe check of Mova's parsesMova's markupon held-out sentencesDisagreementsthe AE at p ≥ 0.9Read by a personreal errors → rulesMova's rules change; the next checkpoint is checked again
Above, the three training tasks; markup weighs a tenth of the text loss, which leaves the autoencoder free to disagree with Mova. Below, how the disagreements are used.
Texta sentence or paragraph of one of its languages, given as its bare words, rebuilt with its markup
Numbersa chain of numbers in one language's tokens, rewritten in another language's tokens
Shared prompta random sequence of shared tokens diluted with one language's words, rewritten one to one diluted with another's

Text. English and French sentences come from Tatoeba with Mova's parses written in Mova Logic 0.6: the input keeps only the words' n-grams, the target has the whole markup — the tree, pragmatics, features, case and WordNet concepts (the most frequent sense of each noun and verb). For 28 more languages the text is Wikipedia prose, rendered with every template expanded, coded by the language's bank; its target adds case and the concepts of linked articles. English Wikipedia in Mova Logic is the source of the @work role, the more technical register Gali writes its work in.

Numbers. A chain of up to 8 numbers, each up to 10¹², is written in Arabic digits in language A's tokens; the target is the same digits in language B's tokens, with B's tag seeded in. Since every language cuts digits into n-grams in its own way, this aligns the lexical halves of the languages.

Shared prompt. A random sequence drawn from the shared half — markup, signs, concepts and their codes — is diluted with random words of language A; the target is the same shared sequence in the same order diluted with words of language B. The filler words are random, so at a filler position any word of language B is right: the loss there is minus the log of the probability of all of B's lexical ids together.

Seeds. Every stream carries its language tag and a role token — input, output, work, or the streams of a numbered copy of Gali — planted at the start and then at growing gaps (Gali's seeds). The autoencoder learns to read them and to write them back, so that Gali can be told the language and the role of what it writes.

Languages. The first 30 languages were the Wikipedia editions of the Elamur sites; then 46 more with a bank of their own; then 66 small languages from the human-written prose of their Wikipedias, a Ruthenian corpus and Lojban texts (the banks). Each language has a tag that is a path in a language graph: a dialect is marked by its standard's tag followed by its own code, and a variety with several parents carries each of them. The grammars that describe these languages are Mova's, and Lojban's grammar is there too.

Continuing. The autoencoder is not trained from scratch when tasks or languages are added; it continues from its last checkpoint, and quality dips first and then recovers (the study).

Measurements

The 12 + 12-block autoencoder of width 384, after 50 000 steps (19 November 2026), on held-out data: chains of numbers rewritten in another language, 92 % exactly; shared prompts rewritten one to one, 100 %; text rebuilt with the decoder given the target so far, 99.7 % of the words and 94.3 % of the markup right, 19 % of whole lines without an error.

The same tasks on a 5 + 5-block autoencoder of width 256 trained on English and French text only reached 97.4 % of the markup and 42 % of whole lines: the larger model now spreads over 30 languages.

Does the encoder forget?

A linear probe per encoder layer learns to read, at every position, the token there and its neighbours, from the states of lines it was not trained on (59 394 training and 20 474 test positions, all 30 sources).

layerthe token itselfthe previous tokenthe next token
embedding99.6 %5.3 %5.8 %
398.6 %30.2 %9.4 %
597.4 %29.2 %18.1 %
895.1 %27.4 %17.5 %
1292.7 %26.7 %16.6 %

The identity fades slowly, by 7 points over 12 layers, most of it after block 4; the context of the neighbours is built by blocks 3 to 5 and then kept. This is what the skip from block 4 into block 8 is tested against.

The skip was then tested against the same model without it (the study): it changes where the identity is held, not how much of it survives at the top, and whole lines come out better.

Limitations

52.7 million parameters are few for about 140 languages, and the small languages have a few thousand articles each. Concept labels come from the most frequent WordNet sense and from Wikipedia links, so they are noisy; the markup of languages other than English and French is case, layout and concepts, since Mova parses only those two fully. Evaluation sets follow the data, so numbers across continuations are trends.

References