Gali0.6 preview · draft

Marking languages, continua and dialects in Gali's tokens

Draft, Gali 0.6 preview (tied to Mova 0.6). Elamur, November 2026. Open to edits until Mova 0.6 is released.

Abstract

Every stream Gali reads or writes carries a tag that says its language (the vocabulary). For a few dozen standard languages one token would do; for continua, dialects, varieties with several parents and historical written languages it does not. In Gali 0.6 a tag is a path in a language graph: the root language's tag, then a two-token code for each node down to the variety, and, for a variety with more than one parent, a marker and each further parent's tag. The graph also decides which bank of tokens a text is written with. Drawing the graph is not a purely linguistic task: where one language ends and another begins is often settled by states, schools, scripts and markets, and what we can support is settled by the corpora that exist. We describe the scheme, the decisions it needed, and what each costs.

1. A language is not a fixed unit

That a language is «a dialect with an army and a navy» (Weinreich 1945) is a joke linguists repeat because it is true often enough. Kloss (1967) separated two reasons for calling a speech form a language: distance from its neighbours (Abstand) and its own development as a standard (Ausbau). Serbian and Croatian are close by distance and separate by elaboration; the spoken varieties of Arabic are far apart by distance and share one written standard. Codes reflect these decisions rather than make them: ISO 639-3 has individual languages and macrolanguages, ISO 639-5 collective codes for families, IETF BCP 47 private-use subtags for everything else, and Glottolog (Hammarström et al. 2025) a genealogical tree whose leaves are languages and dialects.

A model has to choose too. A tag that is too coarse mixes norms a reader can tell apart; a tag that is too fine has no data behind it. Gali's choices follow four rules:

2. The scheme

Tags are paths

The shared half of the vocabulary has 200 single-token language tags and 200 ids that build codes of two or more tokens. A standard language with much text has a single token. Every other node of the graph has a two-token code, and its tag is the path from its root:

Modern Standard Arabicar — 1 token
Egyptian Arabicar → Egyptian–Sudanese continuum → Egyptian Arabic — 4 tokens
A local varietyar → Levantine → North Levantine → Aleppo — 7 tokens
Rusynuk → Rusyn — 3 tokens

The path is resolved recursively from three tables: dialects attached to a standard, varieties inside a continuum, and deeper paths. The same mechanism serves Chinese (zh → Wu → Shanghainese) and Catalan (ca → Valencian → …).

Several parents

A language graph is not a tree. Surzhyk mixes Ukrainian and Russian; Algherese is Catalan with Sardinian and Italian contact; Ruthenian, the written language of the Grand Duchy of Lithuania, descends from the same East Slavonic speech as Ukrainian and Belarusian and writes much of Church Slavonic. A variety may therefore list several parents. With a main line, its tag is the path through the first parent, then the marker @also and the full tag of each further parent: Surzhyk is uk-path, its own code, @also, ru — 5 tokens. With parents of equal standing, as Ruthenian's are, there is no main line: its own code comes first, then @also and each parent's tag in alphabetical order, 9 tokens in all.

Every stream carries the whole tag

A tag is planted twice at the start of a stream with its role token and then repeated at growing gaps (seeds). The whole path is repeated each time, never shortened to root and leaf: the tag says what the text is, and the model is never left to guess which part was dropped.

Banks follow the graph

A text is written with one bank of 4096 lexical tokens, never a mix: banks reuse the same ids, so a mixed stream would give one id two meanings. A node uses its own bank, else its nearest ancestor's — along the main line first. A dialect reads with its standard's bank, so the standard's words keep their tokens; a continuum has one bank for all its varieties; a variety in another script than its parent gets its own; a variety with equal parents inherits none (the banks in practice).

3. Decisions the scheme needed

Each of these was decided by the people building Elamur, or derived from their principles and open to revision, with the evidence in the language inventory: Wikidata's statements on official status and scripts, Glottolog's classification, speaker numbers and endangerment levels, and the corpora we have.

These are not linguistic truths. Some follow political realities — an official standard in its own script is a standard even when speakers understand each other. Some are economic: Elamur's sites are translated into 79 languages, and a language becomes a target of translation when there are readers to serve and text to learn from. Small languages are read, banked, autoencoded and described in grammars, but not yet written by Gali.

4. Corpora decide what a node is worth

A node with a corpus can be trained; a node without one is frame. Egyptian Arabic has its own Wikipedia, so its 4-token tag carries text; Aleppo has none, and its node waits. A test node for the mixed speech of Southern Bessarabia — Russian with Romanian, Gagauz, Bulgarian, Ukrainian and Albanian, six parents — showed that the scheme carries such a tag (14 tokens) and also what it costs: with its role token the seed is 15 tokens, and on 200 tokens of text seeds became 56 % of the stream against 15 % for Russian alone. With no text of it anywhere, the node was removed. Ruthenian, by contrast, has a corpus of about 1.3 million tokens — transcribed statutes, charters and polemics, and uncorrected scans of documents — so it got a bank of its own, fitted on the transcriptions only, without the scans.

Where corpora are thin, quality is uneven: the bot-made stubs that fill some small Wikipedias would teach a bank template phrases, and are left out. Where they are rich, the scheme can go deeper than any table of codes.

5. Limitations

The graph is drawn by people and will be redrawn. A long tag is expensive in a short text; we keep the whole tag because a partial one invites guessing, and will measure the cost when real long tags appear in a corpus. Varieties with no written norm are rarely written at all, so the levels below a standard are mostly frame. Two banks cannot be mixed in one stream, so code-switching inside a sentence is written in one language's bank with the other language's words spelled through its n-grams.

References