Where a long skip keeps the token
A study, Gali 0.6 preview. Elamur, 19 November 2026. A note on a measurement, not a full paper.
Question. Does a skip that adds block 4's states into block 8 of the autoencoder's 12-block encoder make it forget less of its input?
Setup. Two continuations of the same parent, 20 000 steps each, same data in the same order, one with the skip (initialised at zero, so it starts as the parent) and one without; 64 held-out examples per task; then a linear probe per layer that reads the token at each position from the layer's states, on lines the probe did not train on.
| measure | with the skip | without |
|---|---|---|
| whole lines rebuilt without an error | 35.9 % | 26.6 % |
| markup | 80.4 % | 83.5 % |
| chains of numbers | 95.3 % | 96.9 % |
| the token itself, block 7 | 94.9 % | 98.1 % |
| the token itself, block 8 | 97.2 % | 97.4 % |
| the token itself, block 12 | 91.6 % | 91.9 % |
Reading. The skip does not keep more of the token at the top. Blocks 5 to 7 let it fade faster, as if they can count on the copy coming back at block 8, and block 8 restores it. Whole lines were ahead with the skip at every one of four evaluations; the markup was behind at the end. We kept the skip: a few blocks that work without the input's hints in between, with a long connection that brings the input back.
Limits. One pair of runs and 64 examples per task: a single pair cannot tell a small difference from the noise of the seed. The markup and numbers differences are within that noise; even the whole-line difference, consistent over four evaluations, needs repeats. We report it as one observation, not as a result. Related. The probe follows Alain and Bengio (2017); Belinkov (2022) discusses what such probes can and cannot show. A skip of this kind is a sparse case of DenseFormer's depth-weighted averaging (Pagliardini et al. 2024).
References
- Alain, G., Bengio, Y. (2017). Understanding intermediate layers using linear classifier probes.
- Belinkov, Y. (2022). Probing classifiers: promises, shortcomings, and advances.
- Pagliardini, M. et al. (2024). DenseFormer: enhancing information flow in transformers via depth weighted averaging.