Mortal Universal Agents
- Other resources on wireheading in AI

Mortal Universal Agents

In a recent work [1][2] with Mark Ring (these papers have received the Solomonoff AGI Theory Prize 2011 for the strongest contribution to Artificial General Intelligence theory), we considered several kinds of Universal Mortal Agents, like AIXI but with different utility functions:

a reinforcement-learning (RL) agent (say, AIXI),
a goal-seeking (GS) agent, that tries to achieve a given goal (achievement is tested by any computable pattern-matching criterion),
a prediction-seeking (PS) agent, the direct translation of Solomonoff Induction from the passive prediction setting to the active one. It tries to predict its future as best as possible,
a knowledge-seeking (KS) agent, that tries to maximize its knowledge about the whole world (its Kolmogorov complexity about its knowledge of the environment).

The agent not only outputs a action for the environment, but also its own source code for the next step; i.e., this output source code will be the definition of the agent on the next step. This allows the agent to modify itself in any desired way. However, for agents that are initially universally optimal, this is of little interest.

But let's consider additionally that the environment has read-access to this code [1], allowing it to base its returns depending on the definition of the agent.

We can define an additional survival agent, which utility function is defined so as to maximize the number of future steps the agent is identical to its initial description (apart from its “memory” of the past).

Now the environment proposes a (dangerous) game to the agent, called the Simpleton Gambit: Would the agent accept to modify itself into a unintelligent agent if the environment could (almost) guarantee that this would maximize its utility function?

We found the following results:

All agents accept the Simpleton Gambit, except the survival agent, since this is in direct contradiction with its utility function.
The RL agent accepts it quite enthusiastically, if it can be sure the deal is genuine, and so does the goal-seeking agent under some circumstances.
The prediction-seeking agent doesn't care much (though it doesn't care much about pretty much anything…), but should accept it most often. If death yields highly predictable outcomes, it could even choose death over life…
The knowledge-seeking agent accepts the gambit only if not accepting it leads to a predictable, uninteresting world.

Let's move on to the next stage [2]. We offer the agents the access to a delusion box, a kind of remote control that the agents can program to entirely modify their input signals (but not to modify their brain!). This delusion box is an abstraction for a generalization of the wirehead problem: intelligent agents will always find shortcuts to maximize their utility, shortcuts that are generally not intended by the designers, e.g. by directly stimulating (but not modifying) the “reward area” inside its brain. Another possibility is for the agent to acquire (by all means!) the “reward remote control” that humans may use to control the agents' behavior.

Would the different agents find such delusion box interesting? Would they use or abuse it?

Let us first consider the case where the agent are immortal. We found the following somewhat surprising results:

The RL, GS, and PS agents will use and abuse the delusion box, up to the point that its utility function becomes useless. They don't necessarily become unintelligent, but they then don't care about what goals we might want to give them. The put all their intelligence into keeping control over the delusion box.
The KS agent is different: once it has understood how this box works (it is supposedly not very complex), it becomes disinterested in it, and turn onto something else, where there is more knowledge to acquire.

Note that from the point of view of the agents, there is absolutely nothing wrong in using this delusion box, this is simply how they are defined.

Now if the agents are mortal again. Mortality can change everything, since the agents might not want to become “junkies”, since this may threaten their own lives. We found the following results:

The survival agent doesn't care about the delusion box.
The RL agent still abuses the box, but becomes the identical of the survival agent. Think of a junkie that cares about its health, but nothing else.
The GS case is a bit different: on the contrary to the RL agent, the GS agent cannot both modify its inputs and still get information about the world that would ensure their survival. It may try to “bring” the delusion box somewhere safe enough to use it for sufficiently long as to make it think it has achieved its goal.
The mortal PS agent is again strange, because it may find death appealing. Or it could shut its sensors down.
The KS agent still doesn't care much about the delusion box, but this time, being mortal, it will also ensure its own survival, in order to be able to continue choosing intelligent actions.

All in all, the knowledge-seeking agent seems to be the most interesting one, and behaves according to expectation, i.e. it tries to understand the world as deeply as possible.

[1] Orseau L., Ring M., “Self-Modification and Mortality in Artificial Agents”, Artificial General Intelligence (AGI) 2011, Springer, 2011. (pdf)

[2] Ring M., Orseau L.,“Delusion, Survival, and Intelligent Agents”, Artificial General Intelligence (AGI) 2011, Springer, 2011. (pdf)

Slides and video.