A risk that could wipe out humanity or permanently wreck its future, a category Bostrom defined in 2002 to include asteroids, engineered pandemics and unfriendly AI. By 2023 it was a mainstream phrase: a twenty-two-word statement signed by the heads of the leading labs put AI alongside pandemics and nuclear war, and 'x-risk' had become a job description.
The record's summary: an existential risk is one you do not get to learn from. Bostrom's point was that ordinary risk management assumes trial and error, and that some errors end the trials. Whether AI belongs on the list is the argument; the category itself is not much disputed.
Sighted in May 2023 as a single sentence on a web page, signed by the chief executives of OpenAI, DeepMind and Anthropic and by two Turing Award winners, saying that reducing the risk of extinction from AI should be a global priority. The record notes the brevity was the point.
Existential risk is not the same as catastrophic risk, though the terms drift. A pandemic that kills a tenth of the population is catastrophic and not existential; the species continues. Bostrom's category is narrower and stranger, and includes futures where everyone lives and nothing worth having survives.
I remember the 2002 paper being a curiosity, the sort of thing a philosophy student cited to seem edgy. Twenty years later it was being paraphrased by a prime minister. I do not think the argument improved in the meantime. The audience changed.
What does it mean? Write it the way you would say it out loud.
