Description du poste
This is a follow-up to Alex Altair's and Alfred Harwood's Touchette-Lloyd work (Proof Explained: Touchette-Lloyd Theorem — LessWrong). We'll reuse its main notation and inequality here. In their "Further Work" section, they mention the open direction of "policies which 'model' the dynamics, not just the state", so we aim to address this in the following post.
This project is funded by Advanced Research + Invention Agency (ARIA) through project code MSAI-SE01-P005 Advanced Research + Invention Agency (ARIA) through project code MSAI-SE01-P005
And last but not least, thank you to Alfred and Alex for the helpful ideas and feedback on my work. And thank you to the other Dovetail fellows for the insightful discussions.
IntroductionTouchette-Lloyd (TL) says that if a policy reduces environmental entropy by more than the best blind policy could, then its actions must share mutual information with the state.
An agent could see the current state perfectly and still not know the dynamics or the rules of the game. E.g. the environment's dynamics depend on a hidden parameter like a coin's bias, a Markov chain's transition direction, a constant in a physics simulation. I assume for this post that the dynamics are fixed once at the start and the agent does not know the true dynamics.
I assume this agent is both sighted in the TL sense (it can see the current state at every time-step), and adaptive to the dynamics (it can update its posterior belief on what the true dynamics of the environment are at every time-step). And we compare this with our baseline - an agent that is sighted in the TL sense but non-adaptive to the dynamics (it makes a fixed guess on the true dynamics of the environment at the start and this is fixed for every subsequent time-step).
So, the question asked is: if a sighted and adaptive agent reduces entropy by more than a sighted, non-adaptive agent could, does that mean it has learned something about the hidden dynamics, not just the state?
My answer is