1.2 Principle of Optimization Transfer

Principle of Optimization Transfer¶

There is a general principle that will be repeated in this book that Kenneth Lange calls "optimization transfer". The basic idea applies to the problem of maximizing a function $f$.

1. We want to maximize $f$, but it is difficult to do so.
2. We can compute an approximation to $f$, call it $g$, based on local information about $f$.
3. Instead of maximizing $f$, we "transfer" the maximization problem to $g$ and maximize $g$ instead.
4. We iterate Steps 2 and 3 until convergence.

The difficult problem of maximizing $f$ is replaced with the simpler problem of maximizing $g$ coupled with iteration (otherwise known as computation). This is the optimization "transfer".

Note that all of the above applies to minimization problems, because maximizing $f$ is equivalent to minimizing $-f$.