Marking languages, continua and dialects in Gali's tokens
Draft, Gali 0.6 preview (tied to Mova 0.6). Elamur, November 2026. Open to edits until Mova 0.6 is released.
Abstract
Every stream Gali reads or writes carries a tag that says its language (the vocabulary). For a few dozen standard languages one token would do; for continua, dialects, varieties with several parents and historical written languages it does not. In Gali 0.6 a tag is a path in a language graph: the root language's tag, then a two-token code for each node down to the variety, and, for a variety with more than one parent, a marker and each further parent's tag. The graph also decides which bank of tokens a text is written with. Drawing the graph is not a purely linguistic task: where one language ends and another begins is often settled by states, schools, scripts and markets, and what we can support is settled by the corpora that exist. We describe the scheme, the decisions it needed, and what each costs.
1. A language is not a fixed unit
That a language is «a dialect with an army and a navy» (Weinreich 1945) is a joke linguists repeat because it is true often enough. Kloss (1967) separated two reasons for calling a speech form a language: distance from its neighbours (Abstand) and its own development as a standard (Ausbau). Serbian and Croatian are close by distance and separate by elaboration; the spoken varieties of Arabic are far apart by distance and share one written standard. Codes reflect these decisions rather than make them: ISO 639-3 has individual languages and macrolanguages, ISO 639-5 collective codes for families, IETF BCP 47 private-use subtags for everything else, and Glottolog (Hammarström et al. 2025) a genealogical tree whose leaves are languages and dialects.
A model has to choose too. A tag that is too coarse mixes norms a reader can tell apart; a tag that is too fine has no data behind it. Gali's choices follow four rules:
- Real use decides, not law: a language that is alive or has a corpus gets content gathered and translation planned; official status alone says little (Ukrainian is formally official in places where it is hardly used)
- A dialect is marked under its standard, never with a single token of its own, and gets its own chapters in the standard's grammar, not mixed into it
- Script can split a standard: where one continuum has two written norms in two scripts, each script is a standard
- A corpus is a condition: a node with no text is kept as frame or removed, and is never trained on as if it had data
2. The scheme
Tags are paths
The shared half of the vocabulary has 200 single-token language tags and 200 ids that build codes of two or more tokens. A standard language with much text has a single token. Every other node of the graph has a two-token code, and its tag is the path from its root:
| Modern Standard Arabic | ar — 1 token |
|---|---|
| Egyptian Arabic | ar → Egyptian–Sudanese continuum → Egyptian Arabic — 4 tokens |
| A local variety | ar → Levantine → North Levantine → Aleppo — 7 tokens |
| Rusyn | uk → Rusyn — 3 tokens |
The path is resolved recursively from three tables: dialects attached to a standard, varieties inside a continuum, and deeper paths. The same mechanism serves Chinese (zh → Wu → Shanghainese) and Catalan (ca → Valencian → …).
Several parents
A language graph is not a tree. Surzhyk mixes Ukrainian and Russian; Algherese is Catalan with Sardinian and Italian contact; Ruthenian, the written language of the Grand Duchy of Lithuania, descends from the same East Slavonic speech as Ukrainian and Belarusian and writes much of Church Slavonic. A variety may therefore list several parents. With a main line, its tag is the path through the first parent, then the marker @also and the full tag of each further parent: Surzhyk is uk-path, its own code, @also, ru — 5 tokens. With parents of equal standing, as Ruthenian's are, there is no main line: its own code comes first, then @also and each parent's tag in alphabetical order, 9 tokens in all.
Every stream carries the whole tag
A tag is planted twice at the start of a stream with its role token and then repeated at growing gaps (seeds). The whole path is repeated each time, never shortened to root and leaf: the tag says what the text is, and the model is never left to guess which part was dropped.
Banks follow the graph
A text is written with one bank of 4096 lexical tokens, never a mix: banks reuse the same ids, so a mixed stream would give one id two meanings. A node uses its own bank, else its nearest ancestor's — along the main line first. A dialect reads with its standard's bank, so the standard's words keep their tokens; a continuum has one bank for all its varieties; a variety in another script than its parent gets its own; a variety with equal parents inherits none (the banks in practice).
3. Decisions the scheme needed
Each of these was decided by the people building Elamur, or derived from their principles and open to revision, with the evidence in the language inventory: Wikidata's statements on official status and scripts, Glottolog's classification, speaker numbers and endangerment levels, and the corpora we have.
- Serbo-Croatian: one continuum split by script — the Cyrillic standard (sr) and the Latin standard (hr); Bosnian and Montenegrin in Latin script are norms in the Latin branch, their Cyrillic forms in the Cyrillic one; Serbian in Latin script has no node of its own and is read either as Serbian, transliterated into Cyrillic, or as Croatian
- Chinese: two standards by script, simplified and traditional; Cantonese, Wu, Min Nan, Hakka, Gan, Min Dong and Classical Chinese are varieties inside the continuum
- Hindi and Urdu: two languages by script and written tradition, though one by everyday speech
- Norwegian: everything under Bokmål; Nynorsk and the dialects are norms under its tag
- Catalan: one pluricentric language with one supported standard; Valencian, Balearic and the others are varieties inside it
- Arabic: Modern Standard Arabic is the root; the spoken varieties are continua inside it, marked to any depth
- Galician, Afrikaans, Frisian, Luxembourgish, Macedonian: separate languages; Moldovan a variety of Romanian; Dari of Persian, and Tajik its Cyrillic norm with a bank of its own; Scots of English, not written by Gali; Low German and Silesian under German and Polish, Dutch Low Saxon under Dutch; Western Punjabi in Shahmukhi under Punjabi, with its own bank
- Azerbaijani and Uzbek: the southern varieties (South Azerbaijani in Perso-Arabic script, Southern Uzbek) are dialects under their standards and are not written by Gali for now; South Azerbaijani keeps a bank of its own for its script
- Kurdish: a meta node with three branches, as Arabic has — Kurmanji in Latin script, Sorani in Arabic script, and the Southern Kurdish continuum; the meta node has no bank, each branch reads with its script's
- Zazaki and Gorani: a separate Zaza–Gorani continuum, not under Kurdish
- Berber: a meta node too — the Moroccan continuum with its standard in Tifinagh (a bank of its own), the Algerian one with Kabyle, the most alive of its varieties, Tuareg when there is text, and the small eastern varieties as frame
- Occitan: one continuum; Aranese, official in Catalonia, a dialect within it
- Gallo-Italic: one Italianised continuum (Piedmontese, Lombard, Ligurian, Emilian, Romagnol), used for training; Venetian, Corsican and Sardinian separate
- Sámi: one continuum, the Sámi languages as varieties inside it, however different from one another
- Southern Kurdish: a continuum we read
- Rusyn, Moravian, East Slovak, Kashubian and Võro: dialects under Ukrainian, Czech, Slovak, Polish and Estonian; Rusyn and Kashubian also get grammars of their own
These are not linguistic truths. Some follow political realities — an official standard in its own script is a standard even when speakers understand each other. Some are economic: Elamur's sites are translated into 79 languages, and a language becomes a target of translation when there are readers to serve and text to learn from. Small languages are read, banked, autoencoded and described in grammars, but not yet written by Gali.
4. Corpora decide what a node is worth
A node with a corpus can be trained; a node without one is frame. Egyptian Arabic has its own Wikipedia, so its 4-token tag carries text; Aleppo has none, and its node waits. A test node for the mixed speech of Southern Bessarabia — Russian with Romanian, Gagauz, Bulgarian, Ukrainian and Albanian, six parents — showed that the scheme carries such a tag (14 tokens) and also what it costs: with its role token the seed is 15 tokens, and on 200 tokens of text seeds became 56 % of the stream against 15 % for Russian alone. With no text of it anywhere, the node was removed. Ruthenian, by contrast, has a corpus of about 1.3 million tokens — transcribed statutes, charters and polemics, and uncorrected scans of documents — so it got a bank of its own, fitted on the transcriptions only, without the scans.
Where corpora are thin, quality is uneven: the bot-made stubs that fill some small Wikipedias would teach a bank template phrases, and are left out. Where they are rich, the scheme can go deeper than any table of codes.
5. Limitations
The graph is drawn by people and will be redrawn. A long tag is expensive in a short text; we keep the whole tag because a partial one invites guessing, and will measure the cost when real long tags appear in a corpus. Varieties with no written norm are rarely written at all, so the levels below a standard are mostly frame. Two banks cannot be mixed in one stream, so code-switching inside a sentence is written in one language's bank with the other language's words spelled through its n-grams.
References
- Hammarström, H., Forkel, R., Haspelmath, M., Bank, S. (2025). Glottolog. Leipzig: Max Planck Institute for Evolutionary Anthropology.
- IETF (2009). BCP 47: Tags for identifying languages (RFC 5646).
- ISO 639-3 (2007) and ISO 639-5 (2008): codes for the representation of names of languages and language families.
- Johnson, M. et al. (2017). Google's multilingual neural machine translation system: enabling zero-shot translation.
- Joshi, P. et al. (2020). The state and fate of linguistic diversity and inclusion in the NLP world.
- Kloss, H. (1967). «Abstand languages» and «Ausbau languages». Anthropological Linguistics 9(7).
- Lewis, M. P., Simons, G. F. (2010). Assessing endangerment: expanding Fishman's GIDS.
- Moseley, C. (ed.) (2010). Atlas of the World's Languages in Danger. UNESCO.
- Weinreich, M. (1945). Der YIVO un di problemen fun undzer tsayt. Yivo Bleter 25(1).