In her recent essay “Wishing It So,” on AI and social existence, writer Megyn O’Glieblyn describes an episode at a recent conference presentation:
Some rogue audience member raised his hand and asked, in a calm and reasonable voice, ‘Who the fuck wants this technology?’ The speaker, an expert on existential risk, had been enumerating all the things that could go wrong in the attempt to build artificial general intelligence. In response to the question, he explained that we were in the midst of what was known as a ‘perverse incentive structure.’ Humanity as a whole might not benefit from superintelligence, but no single nation or corporation benefited from not building it – a problem that created a positive feedback loop. It was a rather elaborate way of expressing what I took to be a simpler answer: the market wants it.
[Simone] Weil believed the complexity of modern life had birthed … a global system of economic and political incentives that moved us like puppets and whose motives were not always our own. ‘It came to be realized,’ she wrote, ‘that society itself is a force of nature, as blind as the others, as dangerous for man if does not succeed in mastering it,’ and which prevents us ‘from stamping the world with the seal of our will.’ In one of her most striking metaphors, she writes, ‘On the whole, our present situation more or less resembles that of a party of absolutely ignorant travelers who find themselves in a motor-car launched at full speed and driverless across a broken country.’
I remember, still with some alarm, listening to an engineering professor from the University of British Columbia highlight the main themes from a climate-change conference that had just finished. After listing these themes, he noted that he wasn’t really worried that climate change would destroy the world and human civilization.
“The technological imperative will get us first.” It’s irresistible.
