Challenges of the 20th century

This isn't about anything particular on the web, but I mean to refer back to it, so bear with me ...

The 20th century saw major technological advances and massive efforts to solve problems that were barely even conceivable in the 19th. I want to look at two such problems: Sending a rocket to the moon, and curing cancer.

In both cases, there was broad agreement that humanity (or in the case of the moonshoot, at least the U.S.) both could and should take up the task. By the mid 20th century rocket technology had been advancing steadily, to the point that when the Soviet Union announced the launch of Sputnik, the U.S. felt bound to respond with Apollo, whether out of a feeling of destiny or just to keep from losing face. In the field of medicine, antibiotics, vaccines and other medical developments had seen diseases that had been inevitable tragedies of life largely controlled or eradicated. Why not suppose, then, that cancer could be similarly dispatched?

A few decades on, sending a rocket to the moon is a solved problem, so much so that we now set our sights on Mars and the outer solar system and it doesn't seem outlandish to sponsor a prize for the first privately-sponsored lunar landing. Success rates in space ventures aren't perfect, by any means, and neither is our knowledge of space travel, but at least we have a general idea of why things fail and what can be done about it.

On the other hand, not only is cancer still a fact of life, there are still basic questions unanswered or poorly understood despite all the massive effort expended. This is not to say that the effort has been wasted. There have been dramatic improvements in treatment and prevention and in many cases there is now much hope where there had been little. Nonetheless, we can definitively say "We can send a rocket to the moon," but we are not even close to saying "We have cured cancer."

What's the difference, then? The clearest one is this: Before the space race even started, everyone knew what a rocket was and where the moon was. We didn't know what cancer was. In fact, one of the major results of the decades of cancer research is that there isn't really any such thing as "cancer" per se. There are a number of diseases with a number of different causes, symptoms, treatments and prognoses, all of which can potentially vary from person to person.

The two quests are qualitatively different. One was a feat of engineering, for the most part applying well-understood principles to a well-defined end. The other has been an ongoing field of research in which new principles and goals are still being uncovered.

Now, coming back to the field of computing, consider two very similar-looking problems that are getting significant attention: speech recognition and natural-language processing. Speech recognition is like sending a rocket to the moon. It has required major engineering effort and the results aren't perfect. However, there are commercial applications both for recognizing a small vocabulary from a broad range of speakers and for recognizing a broad vocabulary from a small range of speakers. Success rates aren't perfect, but at least we have a general idea of why things fail and what can be done about it.

Natural language processing, on the other hand, is an area of research comprising any number of different problems (including speech recognition, as it happens), requiring any number of different techniques and principles, some of which we haven't discovered yet. As with cancer, one of the major results has been understanding that there is much more to the problem than was originally thought. By the same token, there has been significant progress in some areas.

As things currently stand, one could plausibly argue "Computers can recognize spoken words," but we're not even close to saying "Computers can process language as well as humans," or even clearly defining what "process language" or "as well as humans" might mean here.

All of which sets the stage for bit of musing on search engines ...