benchmark for intelligence

It was a crisp December morning—the kind that could refresh you and perhaps fill you up with a renewed energy for the day ahead. But this was not that kind of a day. In just a few hours you are going to face the world champion in a sport that you have never played before. That should bring chills down your spine or give you a panic attack. The sport is chess, the world champion is a software named Stockfish, and you are a new program called AlphaZero, developed by a little known startup named Deep Mind.

Why would you do this? Or rather, how would you do this? Can you even do this? Having never played chess, not even know how to play chess, and you get faced with none other than the world’s best chess player? Sure, that chess player is not a human, but perhaps that’s even scarier—it will not get tired or intimidated, and it has insurmountable computing ability. That’s true—Stockfish could compute over 70 million positions per second. Humans can’t come close to matching it. In fact, humans have been long left out of these world champion debates when it comes to chess.

Back in 1997, Garry Kasparov, a Russian chess grandmaster and then the reigning world champion, was famously defeated by IBM’s supercomputer Deep Blue. Since then, chess programs continued evolving—getting faster, smarter, and efficient. You didn’t need a supercomputer anymore to run these programs. Sure, you still need a very capable computer, but the computers were getting faster, cheaper, and much more capable. By the time the world overcame the fears of Y2K, the best chess players in the world were not humans; they were computer programs and Stockfish was the best one.

Move forward 20 years and you have yet another challenger on the block, except that this new software had no clue how to play chess at all. And yet, here we are two decades after that historical milestone with Kasparov vs. Deep Blue, witnessing another significant moment in the history of chess—Stockfish vs. AlphaZero. Out of 100 games, AlphaZero won 28 and drew 72. And remember, it didn’t even know how to play chess just a few hours before the first game. A year later, AlphaZero defeated Stockfish in a 1,000 games match.

The whole chess world was in disbelief. Kasparov said, “It’s a remarkable achievement…. It approaches the ‘Type B,’ human-like approach to machine chess dreamt of by Claude Shannon and Alan Turing instead of brute force.” Compared to Stockfish’s 70 million positions/second ability, AlphaZero could only do 80,000/second. Sure, this is still way more than any human can do, but the point is: the way to develop abilities and perhaps intelligence is not necessarily through more computational power. But more importantly, what exactly is the benchmark for intelligence?

Kasparov referred to Turing, who gave us the famous Turing Test. According to that test, if while conversing with a system behind a curtain we can’t tell if it’s a human or a machine, then that system passes the test of intelligence. There are several problems with this kind of testing, some of which I have already outlined before, so I won’t go into them right now. But what matters is for us to think about what really is intelligence. That’s the first question of many questions we need to ask as the world around us has started to get flooded with increasingly more AI-driven systems.

Read More