What is mild optimization?

A soft optimizer is a kind of AI that, informally speaking, doesn’t “optimize too hard”. For example, if you tell a soft optimizer to paint one car, it just paints one car once - it doesn’t paint every single car in the world, or slather a particular car with paint in the hopes of driving the probability of the car being “painted” to 1.

One particular version of a soft optimizer that has been discussed in the AI safety literature is the concept of a quantilizer. Mathematically, a quantilizer takes a random action from the qth percentile of what it judges to be the best actions ranked by future reward (where q is a tunable parameter). There are a few theoretical properties that quantilizers can be proven to have, but they may not transfer well into the real world.

As of July 2024, it’s unclear whether quantilizers can actually be implemented safely. It has been argued that imitators may potentially be a better way to perform tasks safely than quantilizers could.

Further reading:

• Mild Optimization tag on Alignment Forum
• Arbital article (by Eliezer Yudkowsky)