Description du poste
AbstractAIs monitoring, reporting or restricting other AIs is a really interesting direction in control research. This can be done on several levels. For instance, behavior (what actions the AI is taking), reasoning traces (what's the AI thinking out loud), and white box (what's the AI's thinking internally).
Natural language autoencoders (NLAs) are an interesting thing in the third category. They explain in natural language an LLM's internal thoughts (more precisely activation at a particular layer at a particular token). An issue is the explanation can contain hallucinations.
NLAs are trained unsupervised with an encoder-decoder setup. The encoder converts the activation to natural language, the decoder converts back to an activation, and the reconstruction error drives the training.
The paper found that a larger mean drop in fraction of variance explained (FVE), so worse reconstruction, occurs from deleting true claims than false claims. This could be a useful monitor if usable to filter individual claims.
I ran experiments on two open NLAs (Qwen2.5-7B Layer 20 and Qwen3.6-27B Layer 42) with the following findings
1. despite mean deletion gap, individual claim discrimination is weak for 7B, and a coin flip for 27B (AUROC near 0.5)
2. claims associated with the final sentence have better discrimination (0.67 and 0.70 AUROC)
3. paraphrasing (syntax changed, semantics held constant) does not add additional discriminative value
Code, data, and results in Github.
Main result• Data: 100 Re-DocRED Wikipedia introductions, each cut at a word boundary; the activation is read at its final token• Claims: explanations are split into atomic true, false, and irrelevant claims against the prefix and Re-DocRED annotations; see the annotation guide.• Edits: deletions remove exact claim text spans, and paraphrases rewrite a sentence preserving semantics• Analysis: calculate FVE-drop per claim and means, confidence intervals, per-claim AUROC, claims labeled with category (the