Gali0.6 preview · draft

An autoencoder that finds errors in a deterministic parser

A production example, Gali 0.6 preview. Elamur, 20 November 2026. Draft, open to edits until Mova 0.6 is released.

The setting

Mova parses text deterministically: dependency trees, morphology, pragmatics. Its rules are written and tested by hand, so its errors are systematic: a rule that misfires does so on every sentence of a kind. Gali's autoencoder learns to rebuild Mova's markup from bare words, but its loss gives the markup only a tenth of the weight. It is therefore free to disagree with Mova where the rest of the data points the other way, and a confident disagreement means that one of the two is wrong.

The procedure

  1. Take held-out English sentences the autoencoder never trained on, with Mova's parse.
  2. Give the autoencoder the bare words, and give its decoder Mova's markup up to each position.
  3. At every markup position, compare the autoencoder's most likely label with Mova's. Seed tokens are left out of the comparison, and so is case, since the input has none.
  4. Keep the disagreements where the autoencoder's label has probability 0.9 or more; read a sample by hand; send the likely Mova errors to Mova's maintainers with the sentence.

What it found

On 430 sentences, 39 775 markup positions gave 4 408 disagreements, and 379 of them were confident. A random 25 of the 379, read by hand, split into 7 where the autoencoder was right, 10 where Mova was right and 8 unclear. Six were sent as likely Mova errors; four were real:

The two others were the autoencoder's mistakes: a possessive with its apostrophe missing in the source text itself, and a preposition attached by a different treebank convention.

What it changed

Mova's maintainers fixed the four with general rules, not with exceptions for these sentences. Measured on the GUM and PUD treebanks after the change, 67 more tokens got the right labelled head and 40 more the right head; the English Web Treebank's development set moved by 0 to 3 tokens on each column, and none of the 379 flagged rows moved the wrong way.

How much it catches

A control plants errors: 5 % of Mova's labels are replaced by another label of the same family (part of speech, relation, head distance, a feature of the same name) before the autoencoder reads them. Of 1526 planted errors it flagged 64 % at probability 0.9 or more, and its own most likely label was Mova's original in 83 %. The head is the weakest family, flagged in 46 % of cases; pronoun type and verb form are the strongest, at 95 % and 83 %.

Lessons

Using a model's confident disagreements to find label errors is confident learning (Northcutt et al. 2021); finding errors in part-of-speech and treebank annotation goes back to Dickinson and Meurers (2003) and is surveyed by Klie et al. (2023). The labels checked are Universal Dependencies (de Marneffe et al. 2021).

References