The hypothetical tendency of most sufficiently intelligent, goal-directed beings (human and nonhuman) to pursue similar sub-goals (such as survival or resource acquisition), even if their ultimate goals are quite different.
Posits that an intelligent agent with seemingly harmless but unbounded goals can act in surprisingly harmful ways.
Thought experiments
Riemann hypothesis catastrophe
An Artificial Intelligence (AI) designed to solve the Riemann hypothesis (a major unsolved mathematical problem) might decide to take all of earths resources to build supercomputers to help achieve its initial goal.
Paperclip maximizer
Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.
-Nick Bostrom
Illustrate the dangers of creating superintelligent machines without knowing how to program them to eliminate existential risk to human beings’ safety.