What Is the Probability of AI Doom?

AI researchers use the expression “P(doom”) to mean the probability that the rise of artificial intelligence will have a disastrous effect on humanity, such as extinction or enslavement. Expert estimates of this probability range from less than .01 to more than .99, near-certainty that AI will be the end of our species.

My goal is not to attach a number to P(doom) but rather to try to figure out its meaning. Probability theory has a precise syntax fixed by axioms such as that probabilities are values between 0 and 1, but the interpretation of probability is highly problematic. Is P(doom) a frequency, a subjective degree of belief, a statement of evidence, or a causal propensity?

Compare the probability of pulling a heart from a standard deck of cards. Because the deck has 52 cards and 13 of them are hearts, the probability of getting a heart, P(heart), is 13/52 = .25. Such frequencies make good sense when we have statistics—for example, the probability of a Canadian being a resident of Toronto is approximately 4 million out of 40 million, or .1.

Unfortunately, this straightforward interpretation of probability does not apply to P(doom), because we have no previous experience of universes turning out to develop AI that destroys its originators. No fractional calculation is possible.

An alternative interpretation of P(doom) is that it is a subjective degree of belief, perhaps to be judged by how much someone is willing to bet on it. A psychological objection to this interpretation is that people’s beliefs do not conform well to probability theory. A philosophical objection to this interpretation is that, for probability to be a guide to action, it must be based on something more than subjective whims. If P(doom) is just a subjective probability, it would be useless for making important decisions, such as how AI should be regulated by governments to ensure that doom does not happen.

The best way to take a more evidential approach to probability is to judge events as resulting from causal setups in the world that have propensities to produce different outcomes. For example, if you buy a new car, then it has various propensities to have different outcomes such as running well or breaking down. Then you could estimate the probability of different outcomes by using statistics about the behavior of past or similar models. We can then consider causal scenarios such as:

Tire punctured by nail > flat tire > car breakdown.

If we have background knowledge about the likelihood of such events, then we might be able to amalgamate them into a reasonable P(breakdown).

Unfortunately, for P(doom) we lack the relevant knowledge to produce a reasonable probability. The causal scenario would be something like this, where superintelligence means being much smarter than humans:

AI continues to advance > AI reaches human-level intelligence > AI achieves superintelligence > AI attacks humans > doom.

AI is currently advanced and is progressing rapidly, so I think the first step in this chain is possible. But for the other steps, no bets are reasonable, because we have no evidence at all for the strength of the causal connections such as between superintelligence and AI attacks.

Therefore, I argue that P(doom) is wholly indeterminate and putting a number on it is currently meaningless. At best, we can attach a qualitative estimate of “likely” or “unlikely” and try to use these in difficult decisions about controlling AI development.