The AI anxiety: our preoccupation with superintelligence
Source: Joel Achenbach

Nick Bostrom, philosopher at the Future of Humanity institute in Oxford, believes in killer robots. Photo: Tom Pilston

The world's spookiest philosopher is Nick Bostrom, a thin, soft-spoken Swede. Of all the people worried about runaway artificial intelligence, and Killer Robots, and the possibility of a technological doomsday, Bostrom conjures the most extreme scenarios. In his mind, human extinction could be just the beginning.


"We are in a race that we need to win. It's a race between the growing power of the technology and the growing wisdom we need to manage it. Right now, almost all the resources tend to go into growing the power of the tech." -- Max Tegmark

Bostrom's favourite apocalyptic hypothetical involves a machine that has been programmed to make paper clips (although any mundane product will do). This machine keeps getting smarter and more powerful, but never develops human values. It achieves "superintelligence." It begins to convert all kinds of ordinary materials into paper clips. Eventually it decides to turn everything on Earth ― including the human race (!!!) ― into paper clips.
Max Tegmark, a Physics professor at MIT, stands behind the Schršdinger equation for quantum mechanics (top) and Einstein's general theory of relativity (bottom).

Max Tegmark, a Physics professor at MIT, stands behind the Schršdinger equation for quantum mechanics (top) and Einstein's general theory of relativity (bottom). Photo: Josh Reynolds

"You could have a superintelligence whose only goal is to make as many paper clips as possible, and you get this bubble of paper clips spreading through the universe," Bostrom calmly told an audience in Santa Fe, New Mexico, earlier this year.

He added, maintaining his tone of understatement, "I think that would be a low-value future."

Bostrom's underlying concerns about machine intelligence, unintended consequences and potentially malevolent computers have gone mainstream. You can't attend a technology conference these days without someone bringing up the AI anxiety. It hovers over the tech conversation with the high-pitched whine of a 1950s-era Hollywood flying saucer.

People will tell you that even Stephen Hawking is worried about it. And Bill Gates. And that Elon Musk is investing heavily in research on how to keep machine intelligence under control. All that is true.

How this came about is as much a story about media relations as it is about technological change. The machines are not on the verge of taking over. This is a topic rife with speculation and perhaps a whiff of hysteria.

But the discussion reflects a broader truth: We live in an age in which machine intelligence has become a part of daily life. Computers fly planes and soon will drive cars. Computer algorithms anticipate our needs and decide which advertisements to show us. Machines create news stories without human intervention. Machines can recognise your face in a crowd.

New technologies ― including genetic engineering and nanotechnology ― are cascading upon one another and converging. We don't know how this will play out. But some of the most serious thinkers on Earth worry about potential hazards ― and wonder whether we remain fully in control of our inventions.

Science fiction pioneer Isaac Asimov anticipated these concerns when he began writing about robots in the 1940s. He developed rules for robots, the first of which was: "A robot may not injure a human being or, through inaction, allow a human being to come to harm."

People still talk about Asimov's rules. But they talk even more about what they call "the Singularity."

The idea dates to at least 1965, when British mathematician and code-breaker I.J. Good wrote, "An ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind."

In 1993, science fiction author Vernor Vinge used the term "the Singularity" to describe such a moment. Inventor and writer Ray Kurzweil ran with the idea, cranking out a series of books predicting the age of intelligent, spiritual machines.

Kurzweil, now a director of engineering at Google, embraces such a future; he is perhaps the most famous of the techno-utopians, for he believes that technological progress will culminate in a merger of human and machine intelligence. We will all become "transhuman."

Whether any of this actually will happen is the subject of robust debate. Bostrom supports the research but worries that sufficient safeguards are not in place.

Imagine, Bostrom says, that human engineers programmed the machines to never harm humans ― an echo of the first of Asimov's robot laws. But the machines might decide that the best way to obey the harm-no-humans command would be to prevent any humans from ever being born.

Or imagine, Bostrom says, that superintelligent machines are programmed to ensure that whatever they do will make humans smile. They may then decide that they should implant electrodes into the facial muscles of all people to keep us smiling.

Bostrom isn't saying this will happen. These are thought experiments. His big-picture idea is that, just in the past couple of hundred years, we've seen astonishing changes in the human population and economic prosperity. In Bostrom's view, our modern existence is an anomaly ― one created largely by technology. Our tools have suddenly overwhelmed the restrictions of nature. We're in charge now, or seem to be for the moment.

But what if the technology bites back?
The future of life

There is a second Swede in this story, and even more than Bostrom, he's the person driving the conversation. His name is Max Tegmark. He's a charismatic 48-year-old professor in the physics department at the Massachusetts Institute of Technology. He's also a founder of something called the Future of Life Institute, which has been doling out Elon Musk's $US10 million donation for research on making AI safer.

Tegmark is something of a physics radical, the kind of scientist who thinks there may be other universes in which not only the speed of light and gravity are different but the mathematical underpinnings of reality are different. Tegmark and Bostrom are intellectual allies. In Tegmark's recent book, Our Mathematical Universe: My Quest for the Ultimate Nature of Reality, he writes about meeting Bostrom at a conference in 2005 in California:

"After some good wine, our conversation turned to doomsday scenarios. Could the Large Hadron Collider create a miniature black hole that would end up gobbling up Earth? Could it create a 'strangelet' that could catalyse the conversion of Earth into strange quark matter?"

In addition to taking the what-could-go-wrong questions seriously, Tegmark and Bostrom entertain optimistic scenarios. Perhaps, they say, Earth is the only planet in the universe that harbours intelligent life. We have a chance to take this startling phenomenon of intelligence and spread it to the stars ― if we don't destroy ourselves first with runaway technology.

"The future is ours to shape. I feel we are in a race that we need to win. It's a race between the growing power of the technology and the growing wisdom we need to manage it. Right now, almost all the resources tend to go into growing the power of the tech," Tegmark said.
Overpopulation on Mars

Reality check. More than half a century of research on artificial intelligence has yet to produce anything resembling a conscious, willful machine. We still control this technology. We can unplug it.

Just down Vassar Street from Tegmark's office is MIT's Computer Science and Artificial Intelligence Laboratory, where robots are aplenty. Daniela Rus, the director, is an inventor who just nabbed $US25 million in funding from Toyota to develop a car that will never be involved in a collision.

Is she worried about the Singularity?

"It rarely comes up," Rus said. "It's just not something I think about."

With a few exceptions, most full-time AI researchers think the Bostrom-Tegmark fears are premature. A widely repeated observation is that this is like worrying about overpopulation on Mars.

Rus points out that robots are better than humans at crunching numbers and lifting heavy loads, but humans are still better at fine, agile motions, not to mention creative, abstract thinking.

"The progress has not been as steady as people say, and the machine skills are really far from being ready to match our skills," she said. "There are tasks that are very easy for humans ― clearing your dinner table, loading the dishwasher, cleaning up your house ― that are surprisingly difficult for machines."

The future is implacably murky when it comes to technology; the smartest people on the planet fail to see what's coming. For example, many of the great sages of the modern era didn't anticipate that computers would get smaller rather than bigger.

Anyone looking for something to worry about in the near future might want to consider the opposite of superintelligence: superstupidity.

In our increasingly technological society, we rely on complex systems that are vulnerable to failure in complex and unpredictable ways. Deepwater oil wells can blow out and take months to be resealed. Nuclear power reactors can melt down. Rockets can explode. How might intelligent machines fail ― and how catastrophic might those failures be?

Often there is no one person who understands exactly how these systems work or are operating at any given moment. Throw in elements of autonomy, and things can go wrong quickly and disastrously.

Such was the case with the "flash crash" in the stock market in 2010, when, in part because of automated, ultra-fast trading programs, the Dow Jones industrial average dropped almost 1000 points within minutes before rebounding.

"What we're doing every day today is producing super stupid entities that make mistakes," argues Boris Katz, another artificial intelligence researcher at MIT.

"Machines are dangerous because we are giving them too much power, and we give them power to act in response to sensory input. But these rules are not fully thought through, and then sometimes the machine will act in the wrong way," he said.

"But not because it wants to kill you."
It could have already happened

A living legend of the AI field and the MIT faculty is Marvin Minsky, 88, who helped found the field in the 1950s. It was his generation that put us on this road to the age of smart machines.

Minsky, granting an interview in the living room of his home a few miles from campus, flashed an impish smile when asked about the dangers of intelligent machines.

"I suppose you could write a book about how they'll save us," he said. "It just depends upon what dangers appear."

After Nick Bostrom's lecture in Santa Fe, held at the New Mexico School for the Deaf, he went to a book-signing event across town at the School for Advanced Research. His book is a meticulously reasoned, rather dense tract titled, Superintelligence: Paths, Dangers, Strategies.

Bostrom, standing at the edge of a courtyard, held forth amid a small cluster of party guests. Then he sat down for an hour-long interview. Reserved, intensely focused on his ideas, the 42-year-old Bostrom seemed apprehensive about whether his ideas could be fully grasped by someone who is not an academic philosopher. He was distracted by the possibility that a gnat, or fly, or some such insect had invaded his water glass.

Asked if there was something he now wishes he had done differently with his book, he said he should have been clear that he supports the creation of superintelligence. Unsurprisingly, most readers missed that key point.

"I actually think it would be a huge tragedy if machine superintelligence were never developed," he said. "That would be a failure mode for our Earth-originating intelligent civilisation."

In his view, we have a chance to go galactic ― or even intergalactic ― with our intelligence. Bostrom, like Tegmark, is keenly aware that human intelligence occupies a minuscule space in the grand scheme of things. The Earth is a small rock orbiting an ordinary star on one of the spiral arms of a galaxy with hundreds of billions of stars. And at least tens of billions of galaxies twirl across the known universe.

Artificial intelligence, Bostrom said, "is the technology that unlocks this much larger space of possibilities, of capabilities, that enables unlimited space colonisation, that enables uploading of human minds into computers, that enables intergalactic civilisations with planetary-size minds living for billions of years."

There's a bizarre wrinkle in Bostrom's thinking. He thinks a superior civilisation would possess essentially infinite computing power. These superintelligent machines could do almost anything, including create simulated universes that include programs that precisely mimic human consciousness, replete with memories of a person's history ― even though all this would be entirely manufactured by software, with no real-world, physical manifestation.

Bostrom goes so far as to say that unless we rule out the possibility that a machine could create a simulation of a human existence, we should assume that it is overwhelmingly likely that we are living in such a simulation.

"I'm not sure that I'm not already in a machine," he said calmly.