Space Reliability Engineering

In a previous post on the Apollo 11 mission, I emphasized the role of software architecture, and the architect Margaret Hamilton in particular, in ensuring the success of the Apollo 11 lunar landing.  I stand by that, including the assessment of the whole thing as "awesome" in the literal sense, but as usual there's more to the story.

Since that non-particularly-webby post was on Field Notes, so is this one.  What follows is mostly taken from the BBC's excellent if majestically paced podcast 13 Minutes to the Moon [I hope to go back and recheck the details directly at some point, but searching through a dozen or so hours of podcast is time-consuming and I don't know if there's a transcript available -- D.H.], which in turn draws heavily on NASA's Johnson Space Center Oral History Project.

I've also had a look at Ars Technica's No, a "checklist error" did not almost derail the Apollo 11 mission, which takes issue with Hamilton's characterization of the incident and also credits Hal Laning as a co-author of the Executive portion of the guidance software which ultimately saved the day (to me, the main point Hamilton was making was that the executive saved the day, regardless of the exact cause of the 1202 code).

Before getting too far into this, it's worth reiterating just how new computing was at the time.  The term "software engineer" didn't exist (Hamilton coined it during the project -- Paul Niquette claims to have coined the term "software" itself and I see no reason to doubt him).  There wasn't any established job title for what we now call software engineers.  The purchase order for the navigation computer, which was the very first order in the whole Apollo project, didn't mention software, programming or anything of the sort.  The computer was another piece of equipment to be made to work just like an engine, window, gyroscope or whatever.  Like them it would have to be installed and have whatever other things done to it to make it functional.  Like "programming" (whatever that was).

In a way, this was a feature rather than a bug.  The Apollo spacecraft have been referred to, with some justification, as the first fly-by-wire vehicles.  The navigational computer was an unknown quantity.  At least one astronaut promised to turn the thing off at the first opportunity.  Flying was for pilots, not computers.

This didn't happen, of course.  Instead, as the podcast describes so well, control shifted back and forth between human and computer depending on the needs of the mission at the time, but it was far from obvious at the beginning that this would be the case.

Because the computer wasn't trusted implicitly, but treated as just another unknown to be dealt with, -- in other words, another risk to be mitigated -- ensuring its successful operation was seen as a matter of engineering, just like making sure that the engines were efficient and reliable, and not a matter of computer science.  This goes a long way toward explaining the self-monitoring design of the software.

Mitigating the risk of using the computer included figuring out how to make it as foolproof as possible for the astronauts to operate.  The astronauts would be wearing spacesuits with bulky gloves, so they wouldn't exactly be swiping left or right, even if the hardware of the time could have supported it.  Basically you had a numeric display and a bunch of buttons.  The solution was to break the commands down to a verb and a noun (or perhaps more accurately a predicate and argument), each expressed numerically.  It would be a ridiculous interface today.  At the time it was a highly effective use of limited resources [I don't recall the name of the designer who came up with this. It's in the podcast --D.H.].

But the only way to really know if an interface will work is to try it out with real users.  Both the astronauts and the mission control staff needed to practice the whole operation as realistically as possible, including the operation of the computer.  This was for a number of reasons, particularly to learn how the controls and indicators worked, to be prepared for as many contingencies as possible and to try to flush out unforeseen potential problems.  The crew and mission control conducted many of these simulations and they were generally regarded as just as demanding and draining as the real thing, perhaps moreso.

It was during one of the simulations that the computer displayed a status code that no one had ever seen before and therefore didn't know how to react to.  After the session was over, flight director Gene Kranz instructed guidance software expert Jack Garman to look up and memorize every possible code and determine what course of action to take when it came up.  This would take a lot of time searching through the source code, with the launch date imminent, but it had to be done and it was.  Garmin produced a handwritten list of every code and what to do about it.

As a result, when the code 1202 came up with the final opportunity to turn back fast approaching, capsule communicator (CAPCOM) Charlie Duke was able to turn to guidance controller Steve Bales, who could turn to Garman and determine that the code was OK if it didn't happen continuously.  There's a bit of wiggle room in what constitutes "continuously", but knowing that the code wasn't critical was enough to keep the mission on track.  Eventually, Buzz Aldrin noticed that the code only seemed to happen when a particular radar unit was being monitored.  Mission Control took over the monitoring and the code stopped happening.


I now work for a company that has to keep large fleets of computers running to support services that billions of people use daily.  If a major Google service is down for five minutes, it's headline news, often on multiple continents.  It's not the same as making sure a plane or a spaceship lands safely or a hospital doesn't lose power during a hurricane, but it's still high-stakes engineering.

There is a whole profession, Site Reliability Engineer, or SRE for short, dedicated to keeping the wheels turning.  These are highly-skilled people who would have little problem doing my job instead of theirs if they preferred to.  Many of their tools -- monitoring, redundancy, contingency planning, risk analysis, and so on -- can trace their lineage through the Apollo program.  I say "through" because the concepts themselves are considerably older than space travel, but it's remarkable how many of them were not just employed, but significantly advanced, as a consequence of the effort to send people to the moon and bring them back.

One tool in particular, Garman's list of codes, played a key role at a that critical juncture.  Today we would call it a playbook.  Anyone who's been on call for a service has used one (I know I have).



In the end, due to a bit of extra velocity imparted during the maneuver to extract the lunar module and dock it to the command module, the lunar module ended up overshooting its intended landing place.  In order to avoid large boulders and steep slopes in the area they were now approaching, Neil Armstrong ended up flying the module by hand in order to find a good landing spot, aided by a switch to increase or decrease the rate of descent.

The controls were similar to those of a helicopter, except the helicopter was flying sideways through (essentially) a vacuum over the surface of the moon, steered by precisely aimed rocket thrusts while continuing to descend, and was made of material approximately the thickness of a soda can which could have been punctured by a good jab with a ball-point pen.  So not really like a helicopter at all.

The Eagle landed with eighteen seconds of fuel to spare.  It helps to have a really, really good pilot.

Hats off to New Horizons

A few years ago, around the time of the New Horizons encounter with Pluto (or if you're really serious about the demotion thing, minor planet 134340 Pluto), I gave the team a bit of grief over the probe having to go into "safe mode" with only days left before the flyby, though I also tried to make clear that this was still engineering of a very high order.

Early on New Year's Day (US Eastern time), New Horizons flew by a Kuiper Belt object nicknamed Ultima Thule (two syllables in Thule: THOO-lay).  I'm posting to recognize the accomplishment, and this post will be grief-free.

The Ultima Thule encounter was much like the Pluto encounter with a few minor differences:

• Ultima Thule is much smaller.  Its long axis is about 1-2% of Pluto's diameter
• Ultima Thule is darker, reflecting about 10% of light that reaches, compared to around 50% for Pluto. Ultima Thule is about as dark as potting soil.  Pluto is more like old snow.
• Ultima Thule is considerably further away (about 43 AU from the sun as opposed to about 33 AU for Pluto at the time of encounter -- an AU is the average distance from the Sun to the Earth)
• New Horizons passed much closer to Ultima Thule than it did to Pluto (3,500 km vs. 12,500 km).  This requires more accurate navigation and to some extent increased the chances of a disastrous collision with either Ultima Thule or, more likely, something near it that there was no way to know about.  At 50,000 km/h, even a gravel-sized chunk would cause major if not fatal damage.
• Because Ultima Thule is further away, radio signals take proportionally longer to travel between Earth and the probe, about six hours vs. about four hours.
• Because Ultima Thule is much smaller, much darker and significantly further away, it's much harder to spot from Earth.  Before New Horizons, Pluto itself was basically a tiny dot, with a little bit of surface light/dark variation inferred by taking measurements as it rotated.  Ultima Thule was nothing more than a tinier dot, and a hard-to-spot dot at that.
• We've had decades to work out exactly where Pluto's orbit goes and where its moons are.  Ultima Thule wasn't even discovered until after New Horizons was launched.  Until a couple of days ago we didn't even know whether it had moons, rings or an atmosphere (it appears to have none).  [Neither Pluto nor Ultima Thule is a stationary object, just to add that little additional degree of difficulty.  The Pluto flyby might be considered a bit more difficult in that respect, though.  Pluto's orbital speed at the time of the flyby was around 20,000 km/h, while Ultima Thule's is closer to 16,500 km/h.  I'd think this would mainly affect the calculations for rotating to keep the cameras pointed, so it probably doesn't make much practical difference.]

In both cases, New Horizons had to shift from pointing its radio antenna at Earth to pointing its cameras at the target.  As it passes by the target at around 50,000 km/h, it has to rotate to keep the cameras pointed correctly, while still out of contact with Earth (which is light-hours away in any case).  It then needs to rotate its antenna back toward Earth, "phone home" and start downloading data at around 1,000 bits per second.  Using a 15-watt transmitter slightly more powerful than a CB radio.  Since this is in space, rotating means firing small rockets attached to the probe in a precise sequence (there are also gyroscopes on New Horizons, but they're not useful for attitude changes).

So, a piece of cake, really.

Seriously, though, this is amazing engineering and it just gets more amazing the more you look at it.  The Pluto encounter was a major achievement, and this was significantly more difficult in nearly every possible way.

So far there don't seem to be any close-range images of Ultima Thule on the mission's web site (see, this post is actually about the web after all), but the team seems satisfied that the flyby went as planned and more detailed images will be forthcoming over the next 20 months or so.  As I write this, New Horizons is out of communication, behind the Sun from Earth's point of view for a few days, but downloads are set to resume after that.  [The images started coming in not long after this was posted, of course --D.H. Jul 2019]

Margaret Hamilton: 1 New Horizons: 0

A bit more on Pluto, from a compugeek perspective if not a full-on web perspective ...

The New Horizons flyby was not completely without incident.  Shortly before the flyby itself, the craft went into "safe mode", contact was lost for a little over an hour and a small amount of scientific data was lost.  The underlying problem was "a hard-to-detect timing flaw in the spacecraft command sequence".  This quite likely means what's known in the biz as a "race condition", where two operations are going on at the same time, the software behaves incorrectly if the wrong one finishes first and the developers didn't realize it mattered.

Later investigation concluded that the problem happened when "The computer was tasked with receiving a large command load at the same time it was engaged in compressing previous science data."  This means that the CPU would have been both heavily loaded and multitasking, making it more likely that various "multithreading issues" such as race conditions would be exposed.

Now, before I go on, let me emphasize that bugs like this are notoriously easy to introduce by accident and notoriously hard to find if they do creep in, even though there are a number of well-known tools and techniques for finding them and keeping them out in the first place.

The incident does not in any way indicate that the developers involved can't code.  Far from it.  New Horizons made it through a ten-year, five billion kilometer journey, arriving within 72 seconds of the expected time, and was able to beam back spectacularly detailed images.  That speaks for itself.  It's particularly significant that the onboard computers were able to recover from the error condition instead of presenting the ground crew with an interplanetary Blue Screen of Death.  More on that in a bit.

Still ...

It's July 20, 1969.  The Apollo 11 lunar lander is three minutes from landing on the Moon when several alarms go off.  According to a later recounting by the leader of the team involved

Due to an error in the checklist manual, the rendezvous radar switch was placed in the wrong position. This caused it to send erroneous signals to the computer. The result was that the computer was being asked to perform all of its normal functions for landing while receiving an extra load of spurious data which used up 15% of its time.

This is a serious issue.  If the computer can't function, the landing has to be aborted.  However,

The computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing. It then sent out an alarm, which meant to the astronaut, I'm overloaded with more tasks than I should be doing at this time and I'm going to keep only the more important tasks; i.e., the ones needed for landing ... Actually, the computer was programmed to do more than recognize error conditions. A complete set of recovery programs was incorporated into the software. The software's action, in this case, was to eliminate lower priority tasks and re-establish the more important ones.

This is awesome.  Since "awesome" is generally taken to mean "kinda cool" these days, I'll reiterate: The proper response to engineering on this level is awe.  Let me try to explain why.

Depending on where you start counting, modern computing was a decade or two old at the time.  The onboard computer had "approximately 64Kbyte of memory and operated at 0.043MHz".  Today, you can buy a system literally a million times faster and with a million times more memory for a few hundred dollars.

While 64K is tiny by today's standards, it still leaves plenty of room for sophisticated code, which is exactly what was in there.  It does, however, mean that every byte and every machine cycle counts, and for that reason among others the code itself was written in assembler (hand-translated from a language called MAC and put on punch cards for loading).  Assembler is as low-level as it gets, short of putting in raw numbers, flipping switches or fiddling with the wiring by hand.

Here's a printout of that code if you're curious.  The dark bands are from printing out the listing on green-and-white-striped fanfold paper with a line printer such as used to be common at computer centers around the world.  The stripes were there to help the eye follow the 132-character lines.  Good times.  But I digress.

Just in case writing in assembler with an eye towards extremely tight code isn't enough, the software is asynchronous.   What does that mean?  There are two basic ways to structure a program such as this one that has to deal with input from a variety of sources simultaneously: the synchronous approach and the asynchronous approach.

Synchronous code essentially does one thing at a time.  If it's reading temperature and acceleration (or whatever), it will first read one input, say temperature from the temperature sensor, then read acceleration from the accelerometer (or whatever).  If it's asking some part of the engine to rotate 5 degrees, it sends the command to the engine part, then waits for confirmation that the part really did turn.  For example, it might read the position sensor for that part over and over until it reads five degrees different, or raise an alarm if doesn't get the right reading after a certain number of tries.

Code like this is easy to reason about and easy to read.  You can tell immediately that, say, it's an error if you try to move something and its position doesn't reach the desired value after a given number of tries.  However, it's no way to run a spaceship.  For example, suppose you need to be monitoring temperature continuously and raise a critical alarm if it gets outside its acceptable range.  You can't do that if you're busy reading the position sensor.

This is why high-performance, robust systems tend to be asynchronous.  In an asynchronous system, commands can be sent and data can arrive at any time.  There will generally be a number of event handlers, each for a given type of event.  The temperature event handler might record the temperature somewhere and then check to make sure it's in range.

If it's not, it will want to raise an alarm.  Suppose the alarm is a beep every five seconds.  In the asynchronous world, that means creating a timer to trigger events every five seconds, and creating an event handler that sends a beep command to the beeper when the timer fires (or, you can set a "one-shot" timer and have the handler create a new one-shot timer after it sends the beep command).

While all this is going on, other sensors will be triggering events.  In between "the temperature sensor just reported X" and "the timer for your beeper just went off", the system might get events like "the accelerometer just reported Y" and "the position sensor for such-and-such-part just read Z".

To move an engine part in this setup, you need to send it a command to move, and also create a handler for the position sensor's event.  That handler has to include a counter to remember how many position readings have come in since the command to move, along with the position the part is supposed to get to (or better, a time limit and the expected position).

A system like this is very flexible and doesn't spend time "blocked" waiting for things to happen, but it's also harder to read and reason about, since things can happen in any order and the logic is spread across a number of handlers, which can come and go depending on what the system is doing.

And then, on top of all this, the system has code to detect and recover from error conditions, not just in the ship it's controlling but in its own operation.  Do-it-yourself brain surgery, in other words.


I report my occupation as "software engineer" for tax purposes and such, but that's on a good day.  Most of us spend most of our time coding, that is, writing detailed instructions for machines to carry out.  True software engineering means designing a robust and efficient system to solve a practical problem.  The term was coined by Margaret Hamilton, the architect of the Apollo 11 control systems quoted above and a pioneer in the design of asynchronous systems.  As the story of the lunar landing demonstrates, she and her team set a high bar for later work.

New Horizons ran into essentially the same sort of problem that Apollo 11 did, but handled it less robustly (going to "safe mode" and then recovering, as opposed to automatically re-prioritizing), all building on techniques that Hamilton and her team helped develop, and using vastly more powerful equipment and development tools based on decades of collective experience.  So, with all due respect to the New Horizons team, I'd have to say Apollo 11 wins that one.

Latency, bandwidth and Pluto

As you may well know, the New Horizons spacecraft flew by Pluto last month, dramatically increasing our knowledge of Pluto and its moons (let's not even get into whether Pluto and Charon jointly constitute a "binary dwarf planet" or whatever).  There are even a few pictures on the web.

But wait ... that's not very many pictures for a ten-year mission.  Even worse, if you were watching at the time you'll know that New Horizons went completely dark for most of a day, right when it was flying by Pluto.  Isn't this the modern web, where everything is available everywhere instantly?  What gives, NASA?

Part of the problem is the way New Horizons is designed.  It's expensive to accelerate mass to the speed New Horizons is going, and since you can't exactly send a repair crew out to Pluto, it's good to have as few moving parts as possible.  As a result, the ship has a small battery and both the antenna and the cameras are mounted firmly in place.  If you want to turn the antenna toward Earth, you have to move the whole ship, using some of the small store of remaining fuel dedicated to course corrections and attitude control.  If you want to point the cameras toward Pluto, you have to turn the ship that way.  You can't do both.

That explains why the ship went dark for the duration of the flyby, but actually it effectively went dark for considerably longer than that.  It takes about 4.5 hours for a signal to travel the distance between Earth and Pluto.  That means the sequence of events is, more or less

• t + 0: Flight control sends commands to New Horizons to point the cameras at Pluto, take pictures, orient the antenna toward Earth, and report back.
• t + 4.5 hours: New Horizons gets the command and starts re-orienting and taking pictures
• t + 9 hours: Last time at which any signal from before the pointing operation will reach Earth
• t + 25.5 hours (more or less): New horizons, now with the antenna pointed back toward earth, sends "Phone home" message reporting status.
• t + 30 hours (more or less): "Phone home" message arrives

The important thing to note here is that, while the ship is actually out of contact for 21 hours, it's 25.5 hours from the time the command is sent to the time the ship is reachable again, and 30 hours before the ground crew knows it's reachable again.   If the phone home signal hadn't arrived, it would be 9 more hours, at a minimum, before they knew if any corrective action they'd taken had worked.  By internet standards this is ridiculously high latency, but anyone who's played a laggy video game or been on a conference call with people on the opposite side of the world has experienced the same problem on a smaller scale.

So part of the reason pictures have been slow in coming is latency.  It's going to take a minimum of 4.5 hours to beam anything back, and longer if the ground crew has to send instructions.

The other problem is bandwidth.  Pluto is about 5 billion kilometers (about 3 billion miles) away.  Signal strength drops off as the square of the distance, so, for example, a signal from Pluto is about 160 times weaker than a signal from Mars and, since the power source has to be small (around 15 watts), the signal is not going to be extremely powerful to start with.  Lower power means a lower signal to noise ratio and less bandwidth (or at least that's my dim software-engineer understanding of it -- in real life "bandwidth" doesn't exactly mean "how many bits you can transmit", and I'm sure there's lots more I'm glossing over).

Put all that together and on a good day we have about 2Kbps coming back from Pluto.  That's about what you could get out of a modem in the mid 1980s.   Internet technology has progressed just a bit since then, but internet technology doesn't have to cope with vast distances and stringent mass limitations.  At 2Kbps, one raw image from LORRI (the hi-res black-and-white camera) takes close to two hours to transmit.  This is why, if all goes well, we'll be getting Pluto pictures (and other data) well into 2016.

I'd still say that New Horizons is "on the web" in some meaningful sense, but the high latency and low bandwidth make it a great example of Deutsch's fallacies in action.

[Update: Not only is New Horizons sending back data slowly, it's not sending back particularly pretty data at the moment.  From their main page:

Why hasn’t this website included any new images from New Horizons since July? As planned, New Horizons itself is on a bit of a post-flyby break, currently sending back lower data-rate information collected by the energetic particle, solar wind and space dust instruments. It will resume sending flyby images and other data in early September.

-- D.H. 24 Aug 2015]

Curiosity online

The Curiosity rover, formally the Mars Science Laboratory, landed on Mars late on August 5th (or early on the 6th, depending on your time zone).

This was a major accomplishment.  Mars has a habit of eating space probes.  The majority have failed, sometimes quietly and sometimes spectacularly, at least once due to the kind of basic coding error that would send a regular geek cursing back to the keyboard.  Except when it comes to interplanetary travel there's generally no "next release".

The Curiosity landing was more complex than any Mars landing before it, and parts of it had never been realistically tested, much less the overall sequence.  This was not out of negligence or cut corners but a simple necessity.  Mars's atmosphere is much, much thinner than Earth's.  At its thickest it's equivalent to Earth's atmosphere at about 35 kilometers (about 20 miles, or about four times as high as the summit of Mount Everest).  Mars's gravity is about 40% of Earth's.  There's simply no practical way to re-create those conditions at the necessary scale anywhere near Earth.

This thin atmosphere is a real problem.  There's just enough of it you can't ignore it, but not enough for a parachute to take the lander all the way to the surface safely.  To land Curiosity, which is considerably larger and more massive than the Spirit and Opportunity rovers before it, NASA put together a landing sequence that Rube Goldberg might have appreciated:

• The probe slams into the atmosphere, protected by a heat shield and pulling upwards of 10g of deceleration, possibly as high as 15g (This is from NASA's press kit from before the landing.  I don't know what the actual numbers were, but clearly it's not going to work with humans aboard.)
• Once the the probe has slowed enough not to need the heat shield, the heat shield is jettisoned.
• The probe then deploys a parachute, which slows it to about 300 km/h (about 180 mph).
• The parachute is then jettisoned and a rocket-powered descent stage takes over, carefully avoiding the parachute and the back shell it's attached to.  If the rockets don't work, the probe will hit the surface, hard, in about 20 more seconds.
• The descent stage lowers the rover on nylon cords while it slowly descends, since landing completely under rocket power risks kicking up enough dust to damage the rover.
• While the descent stage is doing this, the rover gets its landing gear into place.
• When the rover is safely on the ground, the cords are cut loose, using explosives, and the descent stage flies off to crash land some distance away.

What could possibly go wrong?  Mind, this is a simplified description (and any inaccuracies are mine).  The full details include several more maneuvers, through six different configurations in all, with 76 pyrotechnic devices, ballast jettisoned at various points and dozens of people sweating bullets in the control room and elsewhere.

"Seven minutes of terror", they called it.  Damn impressive engineering, I call it.

OK, so now that we've paused to admire NASA/JPL's chops, what does this have to do with the web? 

I'm old enough to sorta kinda remember Apollo and grainy black-and-white TV coverage.  And Tang.  Cool, and, it must be said, more culturally significant than today's missions, but technically not even close to what we have today.  This is true not only of the vehicle itself, but of the communications technology.  In the 70s we had grainy black-and-white video.  For better imagery you had to wait for the astronauts to bring back the film.

Now, using essentially the same technology as a cell phone camera, NASA is able to capture digital images and put them up on its web site for the world to see immediately (as many have pointed out, much more quickly than NBC saw fit to broadcast Olympic events).  The web gallery includes not only the pretty press-release pictures but also the raw frames they were made from, including pictures of Curiosity staring at its feet and other such that didn't make the cut.  There are plenty of other goodies as well.  I particularly like the "white-balanced" images, which have been post-processed to show what the same terrain would look like under sunlight on Earth.

And there are, of course, a Twitter hashtag (#MSL) and (obligatory plug) updates on NASA's Google Plus page.

As inspiring as it may have been to watch thrilling news coverage with Walter Cronkite narrating, there's something much more intimate about being able to visit a web site from time to time and watch the story unfold directly from the source.

Now here's what you'd call a niche market

What happens when you combine social media with a highly successful crowdsourcing project? Well, from a ridiculously unrepresentative sample of one tweet, you get a pointer to a "very useful web site", http://www.howmanypeopleareinspacerightnow.com. Useful, I suppose, if you're the type of person that really needs to know how many people are in space right now, but somehow doesn't already know the answer (it doesn't change that often, and when it does, it's on the news).

Even then I have doubts. When I chased the link it was still showing 12 (6 on the ISS and 6 on STS-132), even though Atlantis had landed for the last time hours before. So our hypothetical seeker of space population data would still have had to look elsewhere for the truth.

[The site is still there ... and they now have an app.  At this writing, the number of people in space is: 6 --D.H. Dec 2015]

Will a solid rocket booster fit in my shopping cart?

Nearly forty years after its inception in 1972, NASA's space shuttle program is finally ending this year. That leaves the question of what to do with the million or so items of inventory that have accumulated over those decades. In today's web-enabled world, what else to do but put it all on line, eBay style?

Well, actually they're not putting everything online, or even close to it, at least not yet. And it's not open to the public, and you don't exactly bid on items. But, if you have a login, you can put requests for items in your shopping cart and if NASA selects you and you can defray the transportation costs, the item is yours.

If the item in question is an actual space shuttle, transportation costs will come to about $30 million. So far there are 20 or so applicants. Get while the gettin's good.

Challenges of the 20th century

This isn't about anything particular on the web, but I mean to refer back to it, so bear with me ...

The 20th century saw major technological advances and massive efforts to solve problems that were barely even conceivable in the 19th. I want to look at two such problems: Sending a rocket to the moon, and curing cancer.

In both cases, there was broad agreement that humanity (or in the case of the moonshoot, at least the U.S.) both could and should take up the task. By the mid 20th century rocket technology had been advancing steadily, to the point that when the Soviet Union announced the launch of Sputnik, the U.S. felt bound to respond with Apollo, whether out of a feeling of destiny or just to keep from losing face. In the field of medicine, antibiotics, vaccines and other medical developments had seen diseases that had been inevitable tragedies of life largely controlled or eradicated. Why not suppose, then, that cancer could be similarly dispatched?

A few decades on, sending a rocket to the moon is a solved problem, so much so that we now set our sights on Mars and the outer solar system and it doesn't seem outlandish to sponsor a prize for the first privately-sponsored lunar landing. Success rates in space ventures aren't perfect, by any means, and neither is our knowledge of space travel, but at least we have a general idea of why things fail and what can be done about it.

On the other hand, not only is cancer still a fact of life, there are still basic questions unanswered or poorly understood despite all the massive effort expended. This is not to say that the effort has been wasted. There have been dramatic improvements in treatment and prevention and in many cases there is now much hope where there had been little. Nonetheless, we can definitively say "We can send a rocket to the moon," but we are not even close to saying "We have cured cancer."

What's the difference, then? The clearest one is this: Before the space race even started, everyone knew what a rocket was and where the moon was. We didn't know what cancer was. In fact, one of the major results of the decades of cancer research is that there isn't really any such thing as "cancer" per se. There are a number of diseases with a number of different causes, symptoms, treatments and prognoses, all of which can potentially vary from person to person.

The two quests are qualitatively different. One was a feat of engineering, for the most part applying well-understood principles to a well-defined end. The other has been an ongoing field of research in which new principles and goals are still being uncovered.

Now, coming back to the field of computing, consider two very similar-looking problems that are getting significant attention: speech recognition and natural-language processing. Speech recognition is like sending a rocket to the moon. It has required major engineering effort and the results aren't perfect. However, there are commercial applications both for recognizing a small vocabulary from a broad range of speakers and for recognizing a broad vocabulary from a small range of speakers. Success rates aren't perfect, but at least we have a general idea of why things fail and what can be done about it.

Natural language processing, on the other hand, is an area of research comprising any number of different problems (including speech recognition, as it happens), requiring any number of different techniques and principles, some of which we haven't discovered yet. As with cancer, one of the major results has been understanding that there is much more to the problem than was originally thought. By the same token, there has been significant progress in some areas.

As things currently stand, one could plausibly argue "Computers can recognize spoken words," but we're not even close to saying "Computers can process language as well as humans," or even clearly defining what "process language" or "as well as humans" might mean here.

All of which sets the stage for bit of musing on search engines ...