Most of you have heard of Pascal’s Mugging. The mugger claims, without evidence, that you must give him five dollars or something incredibly bad will happen. Sure, he’s probably lying, but the tiny chance of truth times the massive consequences if so make more than the marginal value of five dollars, for most reasonable utility functions.
There are various ways to quibble with the scenario, but they’re mostly addressable, and it’s generally regarded as a fundamental issue with the whole concept of expected utilities. At least unless you’re willing to assert a finite bound on utility, which has its own issues. But I digress.
To disgress some more, probabilities can get quite low before Pascalian issues emerge. The probability of a serious crash on a short car journey is around one in a hundred million, but wearing a seatbelt is still positive EV under most value systems and still a good idea under most circumstances.
There’s a common, albeit disingenuous criticism of the AI safety movement, saying that it relies on Pascalian reasoning. The strawman goes that it doesn’t matter how unlikely AGI is, because the impact is so high that we need to prepare. And as most of you know, that’s not a thing any major AI safety advocate has said.
But as developments progress. As capabilities rise and alignment work doesn’t follow. As organizations abandon what safety plans they had. As yesterday’s “No one would be so stupid as to” becomes today’s “Hey look everybody, I just”…
Then Pascal cuts both ways. If instead of preventing a catastrophe the begger offers to bring about a utopia, the logic is the same. And if our odds are indeed “far underwater in the basement of the logistic success curve”, and if we indeed need logarithms to track them without floating point underflow or its emotional equivalent…
Then appealing to how bright a bright future could be or even to how much we might value continued existence is just another pascalian fallacy.
And we could decide that we might as well operate in forced optimism because what do we have to lose? We might have and lose our last opportunity to eat, drink and be merry before the darkness closes in. Our last chance for La Vie Boheme. That and our commitment to truth, incompatible with any forced viewpoint.
Is that where truth-seeking really leads us, though? If we could think clearly about everything we know, would our odds actually be that low? There’s still a lot of blank map, and a lot of room for unknown unknowns beyond that. And if our purely inside-the-best model probability is considerably darker than our overall probability, that has its own emotional consequences but we shouldn’t get them mixed up. And if we’re facing emotional underflow, well, that varies from person to person but none of us are exactly running IEEE 754.
So if we’re looking at emotional underflow, rather than decision-theoretic, well, remember that line I quoted earlier about the logistic success curve? It’s from something Eliezer posted on April first, 2022. April first is a significant date to post things, so take with a suitable quantity of salt, but he rights:
I would suggest reframing the problem - especially on an emotional level - to helping humanity die with dignity, or rather, since even this goal is realistically unattainable at this point, die with slightly more dignity than would otherwise be counterfactually obtained.
It is more dignified for humanity - a better look on our tombstone - if we die after the management of the AGI project was heroically warned of the dangers but came up with totally reasonable reasons to go ahead anyways.
Or, failing that, if people made a heroic effort to do something that could maybe possibly have worked to generate a warning like that but couldn’t actually in real life because the latest tensors were in a slightly different format and there was no time to readapt the methodology. Compared to the much less dignified-looking situation if there’s no warning and nobody even tried to figure out how to generate one.
The measuring units of dignity are over humanity’s log odds of survival - the graph on which the logistic success curve is a straight line. A project that doubles humanity’s chance of survival from approximately 0% to still approximately 0% is helping humanity die with one additional information-theoretic bit of dignity.
Like I said, this was posted on April First. I don’t know how seriously to take it. I don’t know to what extent Eliezer endorses this mental framing or this understanding of “dignity” and, despite spending a while staring at it, I don’t know to what extent I do.
And of course I don’t know to what extent any of you do.
It’s just… an option.
edit