Hyperlinks vs. the web

In the previous post on the 1968 Mother of all Demos by Doug Engelbart and company, I mentioned stumbling on the fact that Andries van Dam and Ted Nelson had put together HES, which is generally regarded as the first hypertext system, the year before.

All the parts of that were familiar. I've written extensively, though not always favorably, about Nelson's later project, Xanadu. Foley, van Dam et. al.'s Computer Graphics: Principles and Practice was on the shelf at the first place I coded for money, and I'm pretty sure I've run across the idea of hypertext at some point over the years. What I hadn't realized was that Nelson and van Dam had worked together, and that hypertext went back quite that far.

To be fair, I wasn't exactly shocked that hypertext dated to 1967, particularly in the context of Engelbart & co.'s demo, which included what were recognizably hyperlinks. What did catch my attention was that all the building blocks of Web 1.0 were in place in 1968, twenty-three years before the first actual web servers appeared in 1991:

• the concept of hyperlinks and hypertext
• the realization of that concept in running code
• connectivity between computers in different physical locations (ARPANET itself would come along a bit later, but computers were already talking to each other)
• interactive graphic displays
• the mouse

The bullet point I didn't quite add is file servers, but FTP dates to 1971. I haven't dug up solid evidence that there was such a thing as a file server in 1968, but I certainly wouldn't bet against it. If you prefer to put the "Web 1.0 could just as well have happened" point at 1971 and say it was only twenty years later that it actually happened, fine.

What took so long? According to van Dam's account, the people who saw the Mother of all Demos were generally impressed, but the overall reaction was to say "Wow, that was some demo" and then get back to work. Was this a failure of the imagination, that the computer researchers of the time couldn't wrap their heads around something that wasn't 80-column punched cards, even if they'd seen it with their own eyes?

Possibly, but there were also more mundane concerns. That list of pieces in place comes with a few disclaimers:

• Connectivity was generally at 1200 Baud, or approximately one millionth of a gigabit per second. This will deliver text faster than you can read it, but it amounts to a megabyte every two minutes. You can actually fit quite a bit of information into a megabyte, and 1.5Mb/s T1 lines were available, (for a hefty charge, generally to institutions or large corporations), but you're not going to run YouTube or Netflix on the bandwidth available at the time.
• Interactive graphic displays were a thing, but they were normally vector-based (think Asteroids, if you've heard of that) and in any case they were expensive specialized equipment. Graphic displays (bitmapped) didn't become commonplace until the 80s. Even then they weren't cheap and they looked absolutely primitive by today's standards

I think it's fair to say that in 1968 you could have put something together that looked quite a bit like Web 1.0, but it would have been a curiosity: slow, expensive and without anywhere near enough content to make for a compelling experience.

The first actual web servers were intended as an easier way to get to the content that had accumulated on various FTP/Gopher/UUCP/... servers over the years, which was getting to the point where it was hard to just know where something you were interested in was located. Not too long after that (AltaVista came along in 1995), there were enough index pages that it was hard to keep track of where a good index for what you were looking for was, much less the data itself, and web search was born.

So it wasn't simply a matter of the world turning its back on the wonderful potential of Engelbart's NES and Nelson and van Dam's HES and then suddenly waking up in 1991 to realize what they should have known all along. Things were happening in the meantime:

• Computing power, storage and bandwidth were increasing exponentially (as in, actually exponentially, at a more-or-less constant proportion per unit time, and not just "by a lot")
• As both a driving cause and an effect, the number of people with access to computing power, and the amount of data they wanted stored, also increased exponentially
• People continued to experiment with ways of organizing information and navigating complex webs of connections (1987's HyperCard comes to mind, but it's not the only example)

I think this is a common thread in technology in general: Things take a while to develop. It's not enough just to have a good concept, or even a working realization of it. You generally need a certain amount of infrastructure and a growing need that your new concept will meet, and some partial successes along the way.

Some other examples that come to mind:

• SketchPad, which anticipated several major developments, particularly the Graphical User Interface, was written in 1963, GUIs weren't really widespread until the 1980s
• The object-oriented language Simula came out in 1962 and SmallTalk in 1972. Objective-C was introduced in 1984, but OO languages didn't really get major traction until the mid 1990s
• The concept of a neural network dates back to 1943, at least. Perceptrons were introduced in 1969. Hopfield networks were a topic of research in the 1980s. Transformers were proposed in 2017, now eight years ago. ChatGPT came out five years later.

It shouldn't be surprising that concepts run ahead of implementation. Concepts are a lot easier to come up with. More interesting is how we get from (some) concepts to widespread implementation, and which concepts get there.

The future still isn't what it used to be: Doug Engelbart

If you're exploring the origins of the web and pondering how early ideas ended up incorporated -- or not -- into what we see now, sooner or later you're going to run into "The Mother of all Demos", a 1968 presentation by Doug Engelbart and company demonstrating the oN-Line System, or NLS for short, produced by the Augmentation Research Center at the Stanford Research Institute.

I watched it a few months ago. Having somehow missed it for all these years, I went in cold and -- deliberately -- without researching what I was supposed to be looking at. I took copious notes, in some cases replaying to try to make sure I got things right ... and then got distracted by other things. What follows is not fresh and detailed, but I want to at least include something about the demo, so here goes ...

NLS was directly inspired by Vannevar Bush's Memex concept, so if nothing else it's an interesting case study in what happens when you try to turn someone's architectural vision into a real system.

Everything's in glorious black and white, the text on the screens is UPPERCASE and clunky, with actual capital letters indicated by an OVERBAR, and some of the participants look more than a bit like the 1980s stereotype of nerds, because that's where the 1980s stereotype comes from. Actual 1980s nerds were hopelessly uncool in completely different ways (so I hear).

On the other hand, Engelbart is wearing a headset mic, moving a cursor (the team called it a "bug") around on the screen using a mouse, pointing, clicking, narrating the whole way through, linking up with colleagues in a different county in real time, cross-fading and compositing live video and generally giving the appearance of an ordinary livestream, just in black-and-white and what now qualifies as period costume, and without a discernible boss level.

About 20 minutes in,  Engelbart says "I'm going to do something called 'jump on a link'" and explains that a link points to a particular location in a particular file with a particular "view type" saying what kind of content you're looking at -- yeah, that's hypertext.

Physically, NLS is nothing like Bush's original description of a desk-shaped piece of furniture holding a library of microfilm and some sort of magnetic recording medium. More than twenty years had passed since Bush's original article and whole new technologies had developed in the interim, particularly in computing hardware.

NLS was running on a 24-bit SDS-940, clocked at somewhere on the order of 0.1MHz, with approximately 200KB of main memory, 5MB of swap space and 100MB of disk(-ish) storage. It had a text and graphic-capable display and a 1200-baud modem for connectivity.  It also ran QED (Quick EDitor), the very first text editor I learned to use (a bit later, though), and a direct ancestor of vi, which I still occasionally use.

Tiny as this might seem today, I'd argue that there's a bigger gulf between it and 1945's ENIAC than between it and equivalent commercially-available equipment from 1991 (as many years from NLS as NLS was from ENIAC): say a farm of a couple dozen 80MHz SPARCstation 2s, each with 128MB of ram and a few GB of disk. Yes, the SPARCstations could do a lot more, but the SDS has essentially the same pieces: Enough RAM to be meaningfully programmable (as opposed to requiring switches to be set manually), offline storage, connectivity^*, a graphical display, the ability to connect a variety of peripherals and so forth, which the ENIAC had essentially none of. The hardware available when NLS was developed was essentially a smaller, slower version of the hardware that Web 1.0 was built on. Nothing of the sort was available when Bush was thinking up Memex.

NLS is clearly recognizable, almost 60 years later, for what it is: a computer system. Memex, just as clearly, is a thought experiment, though one with enough detail to make it the next best thing to a demo.

To be sure, even if the difference between NLS hardware and today's is mainly a matter of quantity, the quantities have changed by a lot. Today's hardware is several orders of magnitude beyond the SDS, and there are several orders of magnitude more computers in the world now, and they are connected by networks several orders of magnitude faster than 1200 baud. That all has to make a difference, and it has. I hope to dig into this a little more deeply at some point.

There is another key difference between NLS and Memex. Memex is intended for a single person working alone. NLS is inherently multiuser and connected. While the actual demo takes place in San Francisco, the system itself is running in Menlo Park, about 50km/30mi away and probably about as much of a pain to drive to and from then as now. After a fair bit of introductory material, Engelbart is joined by Bill (English, or possibly Paxton?) in Menlo Park, and the two proceed to edit shared documents together.

In the post on Memex, I argued that this difference is one of the main reasons that as far as I'm concerned Bush did not invent the Web. I wouldn't say that Engelbart's team invented the Web, either, but just as the SDS-940 has much more in common with modern computers than with ENIAC, NLS has much more in common with the modern Web than with Memex.

Along with demonstrating the NLS hardware and software, Engelbart made a point of discussing the project that produced it and the approach behind it. That approach was

• Empirical: Try things out and see what works
• Evolutionary ("steepest ascent"): You can't see everything in advance, so take a heuristic approach and look for highest-impact changes at every point
• Whole system: While different people had different skills and responsibilities, the project itself encompassed hardware, software and anything else needed to produce a working system
• Bootstrapping (later called "eating you're own dogfood"): The people developing the system used the system. Past a certain early point, further development of the system was done in the system itself.

"It's a struggle doing it that way," Engelbart said, "but it's beginning to pay off."

I get the feeling both parts of that were knowing understatements.

What we now call dogfooding is a powerful approach that provides a constant reality check, but with the pitfall that it can skew toward the researcher's use cases. A system that's useful for developing software might or might not be useful for other things. Engelbart appears to have been aware of this hazard. One of his prominent examples is his own shopping/errand list. With its repetitions and loose categorization it looks natural. Maybe it wasn't his real list, but it feels like it could have been.

Besides making the system more relatable by giving an example that the audience would be familiar with, constructing a loosely-organized shopping/errand list on the fly demonstrates the system's ability to deal flexibly with free-form content.

This wasn't a given. I could easily imagine someone trying to demo a more rigidly structured system: "OK, now first I create a shopping list template ... Oh, I forgot about vegetables. Let me go back and add a vegetables section to the template. Now where was I?" as the audience's eyes glaze over. Engelbart just puts a list together. It doesn't hurt that he's a fairly engaging speaker and is intimately familiar with the system, but even taking that into account, it doesn't seem like the system is putting arbitrary barriers in his way.

That's not to say that using NLS would be simple for someone encountering it for the first time. After a while, you come to understand that Engelbart and company have memorized quite a few one-letter commands and established quite a few conventions to help find their way around. This is a classic ease-of-use vs. ease-of-learning tradeoff (it's not always a tradeoff, but I'll leave it at that before I start mumbling things about Pareto optimality).

There are some high-level goals that read a lot like what we would now call mission statements:

• Improve the effectiveness with which individuals work at intellectual tasks
• as a subgoal: better use of human capabilities
• Develop a system-oriented discipline for designing the means by which greater effectiveness is achieved

Again, Engelbart makes a point of calling that last point out. Just as important as producing a usable product is producing a way of developing such products.

NLS is meant to be "an instrument/vehicle helping humans to operate within the domain of complex information structures". While it's not explicit in this statement, the demo makes clear that, unlike Memex, NLS is intended for collaboration. It's also subtly larger in that it aims at "humans" rather than "men of science".

Engelbart puts forth several principles that sound surprisingly modern:

• "content represents concepts" Note the use of "content". It didn't start life as a marketing term.
• "structure represents relationships, or human-thought product"
• said structures are "too complex for direct human study", that is, the computer is taking on some of the weight of dealing with a complex structure
• there is a "control metalanguage" note use of "meta". Again, not a new term (well, it goes back to Aristotle's Metaphysics at least).

So it's 1968, Doug Engelbart has just demonstrated hypertext, computers are starting to get networked together, a bunch of standards ending in "TP" for "Transport Protocol" are about to be developed (FTP is announced in 1971) ... so we can expect the Web as We Know It to appear sometime around ... 1991? Wait, what? That's like, 20 years later.

Audience member Andries (Andy) van Dam's assessment of the impact of the demo was that computing mostly went on unchanged afterwards. Besides writing one of the most important early textbooks in computer graphics, van Dam, along with Xanadu founder Ted Nelson -- and I only just now learned this while researching this post -- developed HES, the Hypertext Editing System, at Brown University in 1967. As far as I can tell this was independent of the NLS work, though clearly the two teams would have known about each other.

Van Dam's assessment notwithstanding, several NLS alumni ended up at Xerox PARC (Palo Alto Research Center), which produced, among other things, the influential object-oriented language Smalltalk and a bitmapped mouse-driven GUI ... and these went nowhere until, so the story goes, Steve Jobs stopped by and basically grabbed the whole concept for the Apple Lisa.

The Lisa also went nowhere for a while, but was later reworked to the Macintosh, which eventually took off. I had the fortune of stumbling across a demo of a Lisa at school. This being a tech school, the hardware-oriented folks talked the Apple rep into taking the cover off so they could poke around. The Apple rep, though visibly nervous, was generally a good sport. But I digress.

The famous Macintosh ad was in 1984, 16 years after Engelbart's demo, itself 18 years after the project was initiated. We'd like to think we move much faster now, but do we, really, once you measure comparable points of development: someone's initial concept, start of project, first major demo, first real product, with error bars representing variability in how long things take?. I hope to dig a bit deeper into that as well.

*Though NLS used its modems to allow people to connect to it, computers were already connecting to each other. The ARPANET project was already underway and would be live in a year or two.

Tell us about the olden days

The previous post talked in generalities about how the web and internet may or may not have changed how we communicate and live. To go along with that, I thought it might be interesting to consider some specific examples. Since these are drawn from personal experience, this post will show my age more than most do, but so be it. If you find it amusing to append old man to the questions here, well, I don't suppose I can stop you.

I'm going to answer these on the assumption that you have no memory of anything before, say, the 1990s, so please bear with me if some of this is already obvious. I'm honestly not sure how much of this will be "wow, I didn't know that" to a typical reader and how much will be "well, yeah, no kidding." Also, though I'll generally write in past tense, many of the things I'll mention are still true. I'll call that out here and there, but not necessarily everywhere, so if you find yourself thinking "but ... they still have those", you're probably right.

If nothing else, this post will probably serve as a reminder that as much as I grumble about kidstechnology these days, a lot of this stuff is nice to have.

What did you do before GPS and mapping apps?

I grew up in a midwestern town that was built on a grid. The north/south blocks were long (eight to a mile) and the east-west blocks were short (twelve to a mile), so most addresses were on the north/south streets. First street was at the south end, running east-west, then second street and so on. The north-south streets had names in alphabetical order from east to west.

The first digit or digits of an address on a north-south street were the number of the nearest numbered street to the south. The last two digits of the address were 00 for the northeast corner lot and 01 for the northwest corner and generally increased by 4 per house from there. If I lived at 1234 Elm street and I knew your address was 2133 Maple, I knew to go nine blocks north (just over a mile) and, um, several blocks west, and your house would be on the west side of the street, toward the north end.

The main thoroughfares were a mile apart, since they'd started out as section roads, so you knew you could take 3rd, 11th, 19th or (later, as things got built out) 27th to get across town from east to west, and Cedar, Oak or (again later) Agate to get from north to south. A lot of towns were laid out using some version of this kind of scheme, and for that matter so were a lot of cities. San Francisco is a notable example -- a lot of folks would have built streets to follow the contour of the hills (to be fair, some do).

I say "were" and "could", but of course they didn't rename or renumber anything just because GPS came along, though it does certainly seem to matter less now. I currently live in an area with a large-scale grid of section roads, and many of the towns are on small-scale grids, but I've never bothered to learn the exact numbering schemes, even in my own neighborhood, because GPS is just easier. I do know the section roads reasonably well.

My first answer, in other words, was "you just got to know your way around town" and "the addresses were set up to make that easier".

That worked fine until I moved to an area on the East Coast where nothing was on a grid. At that point MapQuest was around, but I didn't have a smartphone. I ended up doing a fair bit of printing out directions off the web, trying to mostly memorize the way before starting out, peeking at the directions while stopped at stoplights and keeping a weather eye out for street signs and house numbers.

And getting lost fairly often.

Gradually, I learned the main roads and how they connected together, and how the smaller connectors connected to those, and where the main places I wanted to get to related to all that, and things got easier. People would also give general directions like "It's near Chestnut and Amethyst where the main library is. Turn left on Locust Street after the light and Smith Court will be a few streets down". If you already knew where the main library was, or even where Chestnut and Amethyst were, you had a pretty good shot.

There were also some clues like the common pattern of naming a main road after a city it was headed toward. For example, Richmond Road in Twickenham goes toward Richmond and Mortlake Road in Richmond goes toward Mortlake, and it's probably not a coincidence that in both cases you're heading toward London proper (there is no Twickenham Road in Richmond or Richmond Road in Mortlake, but on the other hand, back Stateside, Chapel Hill Road in Durham goes toward Chapel Hill, where it becomes Durham Road in Chapel Hill ...).

Later, I realized that learning the main road/smaller road pattern was something I'd dealt with before before, traveling in Europe, except that instead of main streets it was usually the public transit system -- get on the subway at your stop, follow the subway maps to your destination stop, find your actual destination from there.

My main problem then was that I don't have a great sense of overall direction. If a road takes a bend here and a curve there, I might think I'm headed pretty much the same direction I was before, when in fact I've turned almost 90 degrees.

So I really like having GPS available as a backup, even if I wish there were an easy way to say "yeah, I know this part, start giving me directions when we get to this part and just let me listen to my music until then."

The other part, of course, particularly before MapQuest, was knowing how to read a street or road map, which seems to be something of a lost art, a clear sign that GPS is just plain easier (particularly if your brain doesn't deal well with maps).

For long trips you had the Rand McNally Road Atlas, which showed all the interstates, federal highways and main state roads, along with cities and towns, with mileage shown on each segment. The distance between one town or exit and the next was shown as a number halfway between in one color. Some of those waypoints had a special dot in a different color, and the distance between those with special dots was shown in that color, so you didn't have to add up all the segments in between. There was also a schematic depiction of the interstate system with mileage numbers.

In other words, the road atlas encoded exactly the same kind of edge-weighted graph that mapping software uses, and you could use that to figure out the shortest route from point A to point B along main roads. If you had time, you could look for cutoffs on secondary roads. If you were adventurous, you could try to find local shortcuts and hope that at least you could find your way back to something that was on the map.

You might also carry smaller-scale state or regional maps, which you could get at any gas station (maybe still can). If you were staying in a city for a while, you'd pick up a city map, too. The road atlas also included maps of the main roads in most cities, and you could usually get by with that if you were just passing through [re-reading, I realize I forgot to mention that you could also ... stop and ask directions].

Any of these maps would be overlaid with a square grid with numbers in one direction and letters in the other, and there would be an index, so you could find out that Springfield was in square 5A and quickly find exactly where it was and figure out how to get there.

When I lived in the LA area, the Thomas Brothers map worked basically the same way (as does London's A to Z, along with, I'm sure, many, many others), so you could figure out that to get to the Sherman Oaks Galleria you take Wilshire to the 405, get off at the Ventura exit, hang a left on Sepulveda and a right onto Ventura and there you are.

But what about traffic? To this day, many local radio stations will provide frequent updates on road conditions and traffic, and make money off this information by selling ads. Just sayin'

Summing this all up

• Many places were designed to be easy to get around
• Pretty much any city has a system of main roads, secondary roads and side streets that you can just learn if you need to
• There are maps available at several scales. Larger scale maps include distance information and pretty much all have grids and indexes.
• Getting around is easier now, but it wasn't really that hard before smartphones and GPS, because there was already quite a lot of infrastructure to make it easier, particularly if maps are friendly to your brain.

What did you do before cell phones and texting?

Cell phones may have had the most noticeable effect on day-to-day life of all the web/internet/telecommunications advances of the past few decades.

Besides the clothes and hairstyles, one sure-fire sign that a movie is old (or the screenwriter is a bit behind the times) is a plot device that depends on a phone call. Our hero needs to get in touch with someone urgently. Can they make it to a payphone? Will the person they're calling be at home or at their desk? Will the line be busy? Will the wrong person answer the phone? Or maybe the right person is at home but they're afraid to pick up the phone because it might be the villain calling? If the hero had to leave a message, will the other person get home to check their answering machine in time? Will the wrong person overhear them leaving the message on the machine?

None of these really works today because

• Phones are now associated with people, while they used to be associated with places
• Today's phones can do more. For most of the landline era there was no caller ID and most phones could only handle one call at a time
• Messages can now be stored in the cloud rather than locally on analog tape

Since a landline is associated with a place, the vast majority of households had a single phone line, though there might be multiple phones in the house connected to it. If you called the number for that phone, you were calling the house. Someone would answer ("Hello?"), you'd say who you were ("Hi, it's Dave") and, if you wanted to talk to someone else at the house, who you wanted to talk to ("Could I speak to Earl?"). If that person was somewhere else, you could ask the person you were talking to to leave a message.

You could also just hang out and chat with them -- if you know Earl, you probably know Chris, the housemate, or Chris's good friend Sam, who doesn't live there but hangs out enough that everyone's comfortable with them answering the phone. The chance of talking to someone other than the person you were calling for wasn't necessarily a bad thing, though of course it could be.

The other half of a phone being associated with a place was that if you wanted to make a call, you had to get to a phone. That's why there were payphones (still are, here and there, I'm pretty sure). Or you could stop by a friend's house and ask to borrow their phone. In a pinch, you might be able to drop into a nearby business and ask to use their phone, but it had better be an emergency.

You could also call to a payphone, since they each had their own number, but that was pretty rare, to the point that a lot of people weren't aware that you could even do that. You'd mostly see it done in a movie, where the villain tells the hero to wait at the payphone at 12th and Main, and some innocent bystander steps in to make a call at just the wrong moment.

But that also meant that if you were away from a phone, no one could call you and no one expected to be able to. Earl's not home? Cool, I'll try later, or maybe I'll run into him. Likewise, no one expected you to be able to call them. The most likely answer to "Why haven't they called me back??" was "They're not home yet."

Honestly, this was kinda nice. I still miss it from time to time. Sure, you can unplug today, but it's not the default.

Voicemail today is mostly the same as it was fifty years ago. You could record whatever outgoing message you liked. When someone called your phone and the answering machine was turned on, it would play your outgoing message, beep and start recording whatever was on the line until the connection ended or (I'm pretty sure) until you picked up the phone on your end.

Depending on how a switch was set, it would also play what it was recording on its speaker, so you could hear the message that was being left. People who were at home could and did screen calls that way, so leaving a message might look like:

... Please leave your message at the beep

Hey, it's me ...

[picking up phone] Oh hey! I was hoping you'd call

but it might also look like:

... Please leave your message at the beep

Hey, it's me ...

[muttering to self and not picking up phone] Yeah well you can just take that phone and ...

... and I just wanted to say ... again ... I'm sorry I'm sorry I'm sorry

Sure, you can still screen calls and now even block people (which you couldn't do), but there's something special about listening in in real time.

Again, the main difference is that the answering machine is tied to a landline, which is tied to a place, so typically you'd check your answering machine for messages when you got home and either turn it off, or leave it on and screen calls. If you turned it off, you had to remember to turn it back on the next time you left ("Oh no, I'm sorry you couldn't leave a message. I forgot to turn my machine on."). It was also possible to access your voicemail by calling in and using a Touch-Tone™ keypad to put in a PIN, but to do that ... you'd have to get to a phone (and even late in the game, a lot of phones still had rotary dials, so not just any phone).

But of course, and this is the part that surprised me enough I remember discussing it in at least one post here, nobody leaves voicemail any more. I mean, you can still do it, but I'm not sure when I last left a voicemail for a person, as opposed to a business or doctor's office.

It took me a while to understand why. If you call someone and it goes to voicemail, surely it's easier to just say a message than to hang up and type out a text. Fair enough, but it's even easier to just type out the text without calling and waiting for an answer or voicemail. The setup for a voice connection is heavier weight than one might expect.

It's also a lot easier for the receiver to glance at a text than to access voicemail and then listen through. After hearing "Why didn't you just text me?" over and over, voicemail starts to look less and less attractive. With smart keyboards and speech-to-text, texting isn't that hard anyway, at least in my experience. And so came the return of telegram style and enough abbreviations, slang and conventions to (arguably) constitute a new dialect.

So the main differences here are that it's harder to unplug and ... text.

I was going to emphasize how utterly disruptive it is to always be connected but ... maybe not. Yes, I'm reachable by phone most of my waking hours, but I don't actually get that many phone calls. In particular, I don't get a lot of cold calls. Most of the time if someone calls me it's an actual person that's either a friend/family member or someone I'd asked to call me. I don't get a lot of spam calls to begin with, and if I do, I can either decline and let it go to voicemail, to check (and delete) at my leisure, or use a screening feature to ask them to leave a voicemail (which they never do).

I think some of this is regulatory, but spammers/scammers don't generally let regulation get in their way too much, so this must mostly be a matter of there being cheaper and more effective ways to spam and scam.

Most of the interruptions I get, by far, are notifications from apps which I chose to get. Or at least, I didn't diligently ask not to get. Many of these are email notifications. I don't get a ton of email, but I do get a steady stream through the day, just almost enough to want to Do Something About It.  I also get notifications for texts, which are generally from people I know, so I tend to look at them right away, and from a couple of news sources, which are pretty selective about only sending out alerts for major news. The main thing that's bugging me right now is the stream of "hey this movie just came out" notifications that I don't recall asking for, but that seems to have tapered off (or I turned them off?).

In other words, being "always on" doesn't seem to require being very "on", and there are a few things I could do to make it less disruptive yet. On the flip side, I can call or text pretty much any time I want, as long as I'm not driving, and even then it's usually not that hard to pull over. If I'm at the grocery store and I want to double-check what a household member wanted, that's easy. If I'm in an accident, I can call 911 (or if I can't, I have bigger problems). And so forth.

It's also easier to meet up, which is nice. I remember arranging to meet friends in Berlin not long after the Wall came down. After a few phone calls (and maybe even letters and postcards?), we arrived at a plan: meet on such-and-such date at the Kaiser-Wilhelm-Gedächtniskirche at high noon. If not everyone was there, come back at 1:00 and so forth, so that whoever was already there wouldn't have to sit around waiting. I don't think we set a time to give up waiting, but it was understood that if it got to be too late, whoever was there would just go on and see the city and whoever couldn't make it couldn't make it.

As it turned out, most of us were there at noon and the others showed up at 1:00 and off we went. During the visit, we'd occasionally arrange a rendezvous point to meet up at if we got separated, which may or may not have actually happened. This was pretty normal and it tended to work pretty well, but again, I'm not sure when was the last time I've made a plan like that, because why bother when you can just text or call? It's still a good trick to keep in mind, though, I think.

So staying connected is disruptive, but not really all that disruptive. Being more connected also has some conveniences, but is it really all that much more convenient? 

What did you do before search?

For the purposes of this post, let's assume that search Just Works: you can easily find any particular bit of information you need, assuming it's on the web somewhere (and "the web" basically means "whatever your search engine can find").

Sometimes, you'd end up just not finding something out. But there were options.

If you wanted a phone number or address, you could look someone up in the phone book. Since phone books were physical things, and fairly hefty ones in many cases, you could really only access them locally, though many libraries had phone books for major cities. In other words, there was an element of privacy protection built in, which tended to be enough for most purposes, though people did get unlisted numbers for various reasons.

Adding on to my claim that changes in how phones work have been among the most disruptive, that whole paragraph is from another age. If I want to call a business, their number is on their web page. If I want to call a person, we'll have exchanged phone numbers (likely by text, of course). My phone will remember my contacts, but that's actually not such a big deal. It doesn't take a lot of space to write down names and addresses of people, and you could get a miniature notebook for just such a purpose (I still have one somewhere).

The main convenience is being able to tap on an icon and have the phone place the call without even having to know the number -- there are only a few numbers I have memorized now, but mostly because I use them for supermarket loyalty programs and such, not because I dial them.

But what would one actually search for?

There are two main categories, I think. One is day-to-day information: Where is there a good restaurant that serves X kind of food? When is the DMV open? Does the local hardware store carry left-handed socket wrenches? (No, that's not a thing).

This is largely a matter of advertising (which, for the purposes of this post, at least, is distinct from search).  Businesses have an incentive to let as many people as possible know that they're around, so there's probably a local restaurant guide that will tell you who serves what, and there are probably multiple copies of it in various drawers in the house, or under the couch, because they just seem to keep turning up and, yikes, maybe they can reproduce?

You probably got something in the mail telling you when local government offices like the DMV are open, and they're probably listed in the yellow pages as well (back to phones ... there were actually two kinds of phone books: the white pages had residential listings for anyone with a phone who didn't opt out, and the yellow pages had paid listings for various businesses and similar entities. It's been so long since I've used one that I almost forgot that DMV would be in there).

Unless you lived in a major city, there probably weren't that many restaurants in town anyway, and it didn't take long to get to know them. As to hours, it was a good bet that anything that was open for business would at least be open between 10:00 and 4:00 on a weekday, and anything retail was at least open on Saturday (though maybe not Sunday, depending on where you were).  Again I say "was" and "would", but as far as I can tell, that's still mostly true.

In other words, if you search for "X restaurant near me", you're not asking something that could only be answered, or only be conveniently answered, once search engines came along. You're asking something that used to be reasonably easy to answer and is now somewhat easier, in principle.

As to what's available for sale where, some outlets would put out catalogs (the Sears Catalog is a famous example -- I hope that when you chase that link it says more about the cultural significance of that catalog) and many stores would put out flyers in the local newspaper saying what they had on sale that week.

Or (once more back to phones) you could call the hardware store and ask whether they had left-handed socket wrenches, and most likely someone would actually pick up the phone on the other end and tell you (and try not to giggle too loudly).

Long story short, most of the "Where can I find this in the physical world right now?" questions could be answered pretty easily, because people had an interest in making them easy to answer, just as they do now. The main difference is that there were more people involved. For example, there were more people working at a typical retail store to help customers and also to answer the phone if someone called. And that was kinda nice. I still miss it from time to time.

The other main thing I use search for is research, for example looking up material to put in a blog post. Having so much material online and searchable changes things considerably, but despite what the cartoon might suggest, it wasn't impossible to find things out.

To be sure, this wasn't something most people could do at home, if it wasn't in the dictionary or encyclopedia that were both much more commonplace then, or in some book or magazine that you happened to have on hand, and it helped, a lot, to have a university library or similar institution in your area. If you could get to one of those, though, there were definitely resources:

• There were probably microfilm copies of major newspapers and magazines plus local and regional publications
• Reference books like The Reader's Guide to Periodical Literature would tell you what was in those publications
• There would be copies of major scientific journals
• There would be a large reference section full of reference books on a wide variety of subjects
• And, of course, there would be a large collection of fiction, and non-fiction history, science, art, music and many other subjects
• Along with a card catalog to tell you where to find all of the above
• Your local school library would be a miniature of this, so you could practice finding books in the catalog and maybe even reading a microfilm copy of a news article you found in the Reader's Guide.

The main problem, besides not having access if you didn't live near such a library, is that it's harder to keep a collection of physical texts up to date. Even so, the library would have subscriptions to many major publications, and the Reader's Guide published updates biweekly, so you still could get a pretty good idea of the latest developments.

All this depended on your library carrying the type of information you were interested in and the major reference publishers indexing it. These being human endeavors and resources being limited, there was plenty of room for conscious bias, unconscious bias and plain old budgetary constraints to skew the picture.

Today, of course, you can search anything the major search engines index, major news sources update their pages continuously, preprints are up on ArXiv as soon as the authors want them to be, and information is generally available more widely and more quickly than it used to be. I'm going to steer well clear of how the current web-centric view of the world of information may be biased, and why, but I'll certainly acknowledge that it's a worthy topic of discussion.

But then, how many people are in the business of serious research? For us amateurs, how much does it really matter whether I find out about a new development right away, or in a month or a year when it finds its way into the library system, or a friend mails me a photocopy of someone's lecture notes, or whatever else. For the pros, even a good search engine will only get you so far.

Beyond that, as far as I can tell, you still need a good network of sources, whether primary sources or people who can point you to them or pull together, assess and summarize the information from primary sources. It remains to be seen what role LLMs will end up playing in that, particularly in professional-level research.

Search engines make day-to-day questions more convenient to answer, and they make the amateur researcher's job quite a bit easier, but were those all that hard to begin with?

What did you do before streaming?

Bought CDs, bought/rented DVDs or videotapes, watched TV, went out to movies, not to mention quite a few live shows.

Sometimes even talked to people.

What did you do before LLMs?

Dunno ... what did you do? It hasn't been that long!

On web standards and social networking

I started this blog, um, a few minutes ago, back when I was involved with a couple of the committees involved in developing standards for the web, and it seemed like one thing a person in my position was supposed to do was to blog about it, whether to try to make one's expertise generally available, or to promote one's brand or consulting services, or to become visible as part of a group of similar individuals (the term "blogroll" comes to mind), or to help set the future direction of the web by presenting brilliant analyses or ... something of that nature.

Fortunately, I soon separated the blog from my professional life and settled into just putting out whatever thoughts I had as I wandered the web.world, without a lot of concern for what, if anything, it might all lead to.  In the event, it hasn't led to all that much ... a few dozen followers, the occasional comment, and readership well below even thinking about trying to monetize anything ... but it has been a satisfying experience nonetheless.  I've learned all sorts of things, and writing about it has given me the opportunity to think things through, organize my thoughts and maybe even get better at writing.  The whole standards-committee thing seems long ago and far away.

Imagine my surprise, then, when an Ars Technica article on Mastodon popped up in my newsfeed and brought it all back.

When we last left our story, I was musing about how recent turbulence concerning cryptocurrencies and social media, and Twitter in particular, had gotten me thinking about what "web" even meant anymore and concluding that at least the core of the web, namely its interconnections, was alive and well, and also that it didn't have all that much to do with the turbulence.  Before that, I had said that there didn't really seem to be all that much to blog about.  Things were happening, but not a lot of things that I had much to say about.

But Mastodon is interesting, not just because of its role as a potential "giant killer" for Twitter -- I'm still not inclined to speculate on how that might shake out -- but because of what it is and how it was put together, both in its structure and in the process that led to it.  Allow me to take a walk down memory lane:

When I was involved in the standards process, the typical web standard went through a life cycle something like this:

• Someone, typically one of the Major Players, would create some new protocol or class of application to try to take advantage of the opportunities to be had on the rapidly expanding web.  This was after the major protocols like TCP, FTP, HTTP and the various email protocols were widespread.  Those tended to be less blatantly commercial, even though major corporations were generally involved.
• If the idea looked useful, other players, large and small, would try to work with it, using whatever documentation was available.
• This would often lead to a mess.  It's hard to write clear, complete documentation and it's almost guaranteed that once other people start using a system they'll try things that the original authors hadn't thought of.  When it's not clear what should happen in a particular situation, people will take their best guess, and different people will guess differently.
• People will also want to add new features to cover cases that the original authors hadn't thought of, or did think of but didn't have time to implement, or implemented in what seemed like a less-than-optimal way, or whatever.

In some ways, this is a good problem to have, since it means people are actually using the system, probably because it meets some need that hadn't been addressed before.  The flip side, though, is that in a networked world a system only works, or at least only works well, if people agree on how to use it.  If my version of AwesomeNet thinks that the command for "make everything awesome" is MKAWSM and yours thinks it's make-awesome, then chances are things won't be so awesome.

There are plenty of other minor disagreements that can happen, either because different people made an arbitrary choice differently, or because there's a genuine disagreement as to the best way to do something, or for any number of other reasons.  If this happens enough to be a real problem, there's usually a call for everyone to get together and create an agreement on which way to do things.  A standard, in other words.

Writing standards is a tricky thing.  You want to capture what people are currently doing, but in an abstract enough way to avoid just having a laundry list of things people currently do.  You want to establish boundaries that are tight enough to rein in the chaos, but not so tight to prevent people from finding new and better ways of doing things.  There are often several ways to do this.  For example, for the difference in command names you might

• Just pick one, say make-awesome, and "deprecate" the other, meaning that implementations of AwesomeNet aren't required to support it and if your code does use it, you should update your code.
• Pick one as preferred, require implementations to understand the other one, but forbid them from using it.  That means that a standard implementation of AwesomeNet will never use MKAWSM, but if someone talking to it does, it will understand (this is an example of Postel's principle).
• Allow both, usually by defining one as an "alias" for the other, so that most of the standard text can just talk about make-awesome, except for the part that says "you can use MKAWSM anywhere you can use make-awesome"
• Define an "abstract syntax" and talk about <make awesome> as a placeholder for "whatever name the implementation chooses to use for the make everything awesome command".  This means that there will need to be some sort of protocol for two implementations to tell each other which names they actually use.  That's probably not worth the trouble in most cases, but it's an option nonetheless.
• If "make everything awesome" is just a convenient shorthand for a bunch of other commands, define a general way of defining shorthands like that (often called "macros") and leave it out of the "core standard" entirely.  There are lots of ways to implement macros, since a macro language is essentially a programming language (even if it's not functionally complete), so this is probably only a good idea if at least some of the existing implementations already support it.
• There are probably several other approaches I'm not thinking of.

There are similar issues with features that only some of the existing implementations support.  Do you require everyone to support everyone's features?  Do you define a "core set" and make the rest optional?  Do you say that some existing features aren't allowed in the standard, so implementations that have them have to drop them?  Or maybe something else?

Another important question is "what happens in this particular special case?".  Sometimes the answer is "it's this particular behavior that you might not have thought of, and here's why".  Sometimes the answer is "that's an error, and the implementation should report it like this".  Sometimes, though, the answer is "the behavior is unspecified", either to leave room for new features or for the practical reason that different existing implementations do different things, or for whatever other reason.  This is more useful than it might seem.  It tells implementers that it's OK to do whatever's easiest, and not to expect any particular behavior in certain situations (and maybe try to avoid those situations altogether).

There are lots of other principles and techniques that experienced standards writers employ (learning some of these as a not-so-experienced standards committee member was one of the best parts of the experience).  One that sticks in mind is the principle that implementations should be able to safely ignore things they don't understand, so that it's safe to interact with something that implements a newer version of the standard.

In a typical standards process, there are dozens of questions to hammer out, and the answers are often interconnected.  It's a tricky technical problem and, of course, there are political considerations as well.  In my experience the politics are typically pretty civil, but most participants come in with certain "red lines" they won't cross, usually because it would cost more to implement them than the standard is worth to them since there are already customers out there using the not-yet-standardized products.  The usual approach is to stand firm on those and be prepared to be flexible on anything that doesn't cross a red line.  If two or more participant's red lines overlap, things can get tricky.

After all the meetings, if all goes well, the end result of this is a carefully crafted standards document full of MUST [NOT] and SHOULD [NOT], which are so important that there's actually a standard defining what they mean.

The new standard captures everything the committee could capture about what it means to implement and use whatever's being standardized.  With luck, the standard is at least self-consistent, but there will be parts that are deliberately left unspecified for reasons like those given above.  There will also be plain old mistakes, ideally small ones that can be corrected by errata (corrections, ideally short and specific, published in a separate section or document) but sometimes larger ones that will require more meetings and a new version of the standard.

Since standards need to be flexible, a standard doesn't always answer every question you'll need answered when coding to it.  Because of this, it's common for a separate "interoperation committee" to form once a standard is out.  These committees essentially fill in the blanks to create "profiles" that say how to use a particular standard in particular situations.  There might be a lightweight profile that's meant to provide only the core features so it's easier to implement and run.  There might be a security-oriented profile that specifies features to enable and parameters to use ("digital signatures are required for each message and private keys must be at least 2048 bits").  There might be a profile aimed particularly at business use, or whatever.

I've gone through all this in detail because, judging by the Ars article, all of these things either have happened with Mastodon, or may happen at some point because the same basic issues have come up.  More than that, Mastodon and similar services are, in fact, based on the ActivityPub standard, which specifies two protocols, one for clients to talk to instances  to interact with an inbox and outbox, and one for instances to send traffic to each other.

(An instance is a server in the sense of the client-server model, but not in the sense of a particular process or a server machine -- at least in theory, an instance could run on multiple physical servers and a single physical server could host multiple instances)

ActivityPub is meant to be very general-purpose, but in the context of social networking only some of the features are really important, so implementations in the Fediverse (the world of Mastodon and Mastodon-like services that talk to each other) will naturally tend to pay more attention to those features, which is probably fine until someone tries to take advantage of other features.  The standard for data encryption and signatures is not officially part of the system, so implementations that support it still have to deal with unencrypted data.  ActivityPub relies on a standard called WebFinger to discover what instances are out there, so implementing a real ActivityPub instance also means dealing with WebFinger.

And so on.

All of this is perfectly normal in the world of web standards.  A standard is meant to reduce uncertainty and make it clear where the remaining uncertainty is, and establish common vocabulary and practices.  It isn't meant to solve all problems with interoperation -- it's great if it can, but that's not always possible.  It isn't meant to answer all questions an implementer might have, and in fact it deliberately doesn't answer some questions so that implementers will be free to come up with better answers.  In general a standard is more about what than how (though a what in one context can be part of the how in another and vice versa).

In any case, ActivityPub only talks about how instances talk to each other and how clients interact with instances.  It doesn't, and shouldn't, talk about how people using it for social networking should interact.  That's a matter of community guidelines, that is, what the people running the actual instances will or won't allow on them.  Since the instances are deliberately federated, rather than owned by one entity as with Twitter, these decisions will be, too.

One particularly interesting point is whether posts to one instance can be kept private to that instance.  This proved contentions, so Hometown was forked off of Mastodon.  Hometown allows posts to be private to a particular instance.  Mastodon doesn't.

The good news, I think, is that federated services built on web standards, while messy in real life, can and do work.  Mastodon's success or failure depends more on how community guidelines shake out and whether people prefer to spend their time there than it does on the strength of the standards.  I only spent a lot of time on standards in this post because it was striking to me how much of what I learned back in those days is still relevant.

One other thing struck me while pondering the article.  This all seems a whole lot like Usenet, just with different underlying technology.

Usenet used an email-based model where a particular user could subscribe to various newsgroups, receiving emails as updates came in, and publish messages on those groups, either new messages or replies to a thread that was already going.  The main technical difference is that early usenet relied on a old protocol called UUCP (Unix-to-Unix Copy) to transfer files of messages point-to-point between sites (often but not always universities or corporations).  UUCP isn't used much any more, but, like many old standards, it is still in use here and there.

The main externally visible difference, besides the use of email, was that the newsgroups were owned by administrators (these might not be the server administrators, though server administrators could effectively veto a group by filtering it out).  Rather than tag a post with #this or #that, you'd post to comp.sci.this or alt.that.discuss or whatever.

From a social point of view, though, the similarities seem significant.  Just as site administrators could establish rules, such as by blocking people from posting or filtering out newsgroups, Mastodon instance owners can establish their own standards.  Just as different people would have different ideas as to what sort of discussion was acceptable and different sites and different newsgroups could have different standards, different instances have different rules of conduct.

Newsgroups allowed people with similar interests to meet up and discuss things they were interested in.  This included not only technical topics like scientific specialties or programming languages, but social topics like music and art and a fair bit of quite gamy content.  They also allowed otherwise marginalized people to gather together, and for the curious to find out a bit about an unfamiliar group of people, but also for trolls and worse to barge in if they could get past the admins.

In other words, from a social point of view, they had pretty much all the features, good and bad, of modern social networking communities.  Whether you want to see that as a glass half full -- people keep finding ways to get together -- or half empty -- even after decades we're still up against the same problems -- or a bit of both -- my general inclination -- is up to you.  If nothing else it might provide a counterweight to claims that any of what's going on now is unprecedented.

There's one other technical point that jumped out, though I doubt it will interest many people.  Two patterns of communication come up over and over again in distributed services.  One is the request/response pattern: I ask you a question or to do something, you respond with an answer or "I did it (and this happened)"/"I couldn't do that (for this reason)"/....  The other is the publish/subscribe pattern (pub/sub for short), where any number of parties can publish a message on a topic and any of those (including the publisher) can subscribe to messages on that topic.  Those aren't the only possibilities, but they account for a lot of traffic.  ActivityPub, as the name might suggest, is a pub/sub protocol.

The whole point of pub/sub is that publishers and subscribers don't know about each other, or whether any of them even exist.  I can publish to a topic that no one's listening to, and, depending on the type of service, either that message will be dropped on the floor, or it will be saved for delivery to anyone who does eventually subscribe.  In either case, each subscriber will see its own copy of the message.  The only difference is whether "each subscriber" means "each subscriber that was there when the message was published (more or less)" or "each subscriber that ever subscribes to this topic".

Since publishers and subscribers don't know about each other, there has to be some sort of intermediary.  For a large system, there will actually be many intermediaries communicating with each other in order to make sure published messages are delivered to subscribers.   Publishers and subscribers talk point-to-point with those intermediaries.  I tend to think of this as "I publish to the cloud, whatever's in the cloud does whatever it does, and the subscriber gets the message from the cloud".

Mastodon and company fit this perfectly: You send a message to the instance you're using, it talks to other instances, and people see your message.

The old Usenet followed the exact same pattern, just that the mechanism for the servers to talk to each other was quite a bit different.

Margaret Hamilton: 1 New Horizons: 0

A bit more on Pluto, from a compugeek perspective if not a full-on web perspective ...

The New Horizons flyby was not completely without incident.  Shortly before the flyby itself, the craft went into "safe mode", contact was lost for a little over an hour and a small amount of scientific data was lost.  The underlying problem was "a hard-to-detect timing flaw in the spacecraft command sequence".  This quite likely means what's known in the biz as a "race condition", where two operations are going on at the same time, the software behaves incorrectly if the wrong one finishes first and the developers didn't realize it mattered.

Later investigation concluded that the problem happened when "The computer was tasked with receiving a large command load at the same time it was engaged in compressing previous science data."  This means that the CPU would have been both heavily loaded and multitasking, making it more likely that various "multithreading issues" such as race conditions would be exposed.

Now, before I go on, let me emphasize that bugs like this are notoriously easy to introduce by accident and notoriously hard to find if they do creep in, even though there are a number of well-known tools and techniques for finding them and keeping them out in the first place.

The incident does not in any way indicate that the developers involved can't code.  Far from it.  New Horizons made it through a ten-year, five billion kilometer journey, arriving within 72 seconds of the expected time, and was able to beam back spectacularly detailed images.  That speaks for itself.  It's particularly significant that the onboard computers were able to recover from the error condition instead of presenting the ground crew with an interplanetary Blue Screen of Death.  More on that in a bit.

Still ...

It's July 20, 1969.  The Apollo 11 lunar lander is three minutes from landing on the Moon when several alarms go off.  According to a later recounting by the leader of the team involved

Due to an error in the checklist manual, the rendezvous radar switch was placed in the wrong position. This caused it to send erroneous signals to the computer. The result was that the computer was being asked to perform all of its normal functions for landing while receiving an extra load of spurious data which used up 15% of its time.

This is a serious issue.  If the computer can't function, the landing has to be aborted.  However,

The computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing. It then sent out an alarm, which meant to the astronaut, I'm overloaded with more tasks than I should be doing at this time and I'm going to keep only the more important tasks; i.e., the ones needed for landing ... Actually, the computer was programmed to do more than recognize error conditions. A complete set of recovery programs was incorporated into the software. The software's action, in this case, was to eliminate lower priority tasks and re-establish the more important ones.

This is awesome.  Since "awesome" is generally taken to mean "kinda cool" these days, I'll reiterate: The proper response to engineering on this level is awe.  Let me try to explain why.

Depending on where you start counting, modern computing was a decade or two old at the time.  The onboard computer had "approximately 64Kbyte of memory and operated at 0.043MHz".  Today, you can buy a system literally a million times faster and with a million times more memory for a few hundred dollars.

While 64K is tiny by today's standards, it still leaves plenty of room for sophisticated code, which is exactly what was in there.  It does, however, mean that every byte and every machine cycle counts, and for that reason among others the code itself was written in assembler (hand-translated from a language called MAC and put on punch cards for loading).  Assembler is as low-level as it gets, short of putting in raw numbers, flipping switches or fiddling with the wiring by hand.

Here's a printout of that code if you're curious.  The dark bands are from printing out the listing on green-and-white-striped fanfold paper with a line printer such as used to be common at computer centers around the world.  The stripes were there to help the eye follow the 132-character lines.  Good times.  But I digress.

Just in case writing in assembler with an eye towards extremely tight code isn't enough, the software is asynchronous.   What does that mean?  There are two basic ways to structure a program such as this one that has to deal with input from a variety of sources simultaneously: the synchronous approach and the asynchronous approach.

Synchronous code essentially does one thing at a time.  If it's reading temperature and acceleration (or whatever), it will first read one input, say temperature from the temperature sensor, then read acceleration from the accelerometer (or whatever).  If it's asking some part of the engine to rotate 5 degrees, it sends the command to the engine part, then waits for confirmation that the part really did turn.  For example, it might read the position sensor for that part over and over until it reads five degrees different, or raise an alarm if doesn't get the right reading after a certain number of tries.

Code like this is easy to reason about and easy to read.  You can tell immediately that, say, it's an error if you try to move something and its position doesn't reach the desired value after a given number of tries.  However, it's no way to run a spaceship.  For example, suppose you need to be monitoring temperature continuously and raise a critical alarm if it gets outside its acceptable range.  You can't do that if you're busy reading the position sensor.

This is why high-performance, robust systems tend to be asynchronous.  In an asynchronous system, commands can be sent and data can arrive at any time.  There will generally be a number of event handlers, each for a given type of event.  The temperature event handler might record the temperature somewhere and then check to make sure it's in range.

If it's not, it will want to raise an alarm.  Suppose the alarm is a beep every five seconds.  In the asynchronous world, that means creating a timer to trigger events every five seconds, and creating an event handler that sends a beep command to the beeper when the timer fires (or, you can set a "one-shot" timer and have the handler create a new one-shot timer after it sends the beep command).

While all this is going on, other sensors will be triggering events.  In between "the temperature sensor just reported X" and "the timer for your beeper just went off", the system might get events like "the accelerometer just reported Y" and "the position sensor for such-and-such-part just read Z".

To move an engine part in this setup, you need to send it a command to move, and also create a handler for the position sensor's event.  That handler has to include a counter to remember how many position readings have come in since the command to move, along with the position the part is supposed to get to (or better, a time limit and the expected position).

A system like this is very flexible and doesn't spend time "blocked" waiting for things to happen, but it's also harder to read and reason about, since things can happen in any order and the logic is spread across a number of handlers, which can come and go depending on what the system is doing.

And then, on top of all this, the system has code to detect and recover from error conditions, not just in the ship it's controlling but in its own operation.  Do-it-yourself brain surgery, in other words.


I report my occupation as "software engineer" for tax purposes and such, but that's on a good day.  Most of us spend most of our time coding, that is, writing detailed instructions for machines to carry out.  True software engineering means designing a robust and efficient system to solve a practical problem.  The term was coined by Margaret Hamilton, the architect of the Apollo 11 control systems quoted above and a pioneer in the design of asynchronous systems.  As the story of the lunar landing demonstrates, she and her team set a high bar for later work.

New Horizons ran into essentially the same sort of problem that Apollo 11 did, but handled it less robustly (going to "safe mode" and then recovering, as opposed to automatically re-prioritizing), all building on techniques that Hamilton and her team helped develop, and using vastly more powerful equipment and development tools based on decades of collective experience.  So, with all due respect to the New Horizons team, I'd have to say Apollo 11 wins that one.

All your IRQ are belong to us

I did some of my first real professional programming on an early IBM PC running MS-DOS.  Back then "DOS" stood for "Disk Operating System", as opposed to "Denial Of Service" and in the literal sense of "something that will operate a disk drive", that was accurate.  In other respects that "Operating System" implied, even at the time -- things like multitasking, so more than one program could run at once, or memory protection, so that a running program couldn't read or (worse) scribble on memory that didn't belong to it -- well, "Denial Of Service" might have been just as good a description.

Under MS-DOS's BIOS (Basic Input-Output System), applications talked to the system, and the hardware talked to the system, through "Interrupt Requests" or IRQs.  These were basically entries in a table of the form "When this happens, run the code at this address".  The entries in the table were called "vectors", and any particular IRQ had the address of a particular chunk of code, called an interrupt handler or interrupt routine.  For example, the IRQ for key clicks would be vectored to the code for dealing with a key click event.

Dealing with a key click event is not quite as simple as it sounds.  You had to do several things:

• "Debounce" the key click -- I forget whether the PC did this in hardware or software, but a when a human presses a key on a keyboard, the corresponding circuit doesn't just close, at least not on those early keyboards.  It would go through a period of milliseconds in which the circuit would bounce back and forth between open and closed.  Even to an early PC a millisecond is a fair bit of time, and you wouldn't want to interpret that bouncing as someone typing really fast.
• Keep track of which shift keys were pressed at the time.  You would do this by keeping a few bits around like "left shift key is up/down", "control key is up/down", etc.  The caps lock key acts differently from the other keys, of course.  Miss an event and you could get CAPITAL LETTERS when you wanted lowercase, or worse, control characters which could cause all kinds of fun.
• Buffer up key presses in a block of memory so that if the user typed several keys while the main program was thinking, they would still be there to read when it got done thinking.  Actual applications would read characters from a buffer, as 'H', 'e', 'l', 'l', 'o', rather than catching a series of events like left-shift-key-down, h-key-down, h-key-up, shift-key-up, l-key-down ... directly from the keyboard IRQ.
• Check for magic key sequences like "print screen" or the famous "ctrl-alt-del"

and this is not to mention things like actually displaying the typed character somewhere, or changing the state of a document being edited.  That was all done by the application code.

Keep in mind that the keyboard IRQ was just one IRQ.  There were IRQs for the system's internal timer, for communicating with the disk drive and the modem and printer ports, for applications to talk to the BIOS, and so forth, so imagine the discussion here multiplied by a dozen or so important IRQs.

I mentioned that most applications would be fine with just reading characters from the system's buffer, but some, for example many games, really were interested in the raw events.   There were also utilities you could buy that would allow you to do things like scroll back to text that had scrolled off the screen, or display a clock or check the spelling of what you'd just typed, if you hit a magic sequence of keys.  Because the DOS code sitting on top of the BIOS didn't directly support such things, such programs would "hook" the BIOS's IRQs by changing the IRQ to vector to their code.  Since DOS didn't do memory protection, anyone could Just Do That, and many did.

There are a couple of hazards to this approach.  For one, you didn't necessarily want to completely take over handling of the keyboard.  Many utilities just wanted to hook one magic key sequence to trigger what they did and pass the rest through untouched.  The usual approach to this was to "chain" -- the last thing that a newly-installed interrupt handler would do was to call the handler that had been there before it.  That means you don't care what happens down the line and you don't have to try to replicate what everything else was doing, but it leads to the second hazard.

Suppose I've written a nifty utility that pops up a calendar whenever the user presses ctrl-alt-C, and you've written a nifty utility that pops up a calculator whenever the user presses ctrl-alt-C.  Several things can happen if both of our utilities are installed:

• Maybe mine was installed last, so that the IRQ is vectored to my handler.  You'll see a calendar when you hit ctrl-alt-C, and you may or may not see anything else
• Most likely my handler will "eat" the magic keypress by only popping up the calendar,
• but it might choose to go ahead and chain to whatever handler was there before.  In that case, your handler could also get called, depending on whether any other handlers were installed between ours, and what they do.
• And likewise, of course, the other way around.

In other words, we have what is technically called "a mess" (or several other things you might imagine).  If your handler is installed last, it owns the world -- or at least the IRQ it handles.  If not, well, all kinds of things could happen, but a likely one is customers calling up saying "I installed your lousy utility and it doesn't work!"

The inevitable consequence: Every utility you bought would implore you to please, pretty please make sure that it's the last one in the AUTOEXEC.BAT script called at startup.  Or, more conveniently but also worse if you're trying to rein in this chaos, its handy installation script would edit AUTOEXEC.BAT to make sure it was the last one to run -- until the next utility with such an install script came along, or until you hand-edited AUTOEXEC.BAT to try to fix some conflict by moving some other utility to the bottom.

Ah, those wacky, wild and carefree days of the PC revolution.  Good thing this sort of thing doesn't happen any more in our modern, wonderful web.world.

Now where was I before this trip down memory lane?  Ah, right.  Cleaning up someone's system after  a couple of shiny-looking downloaded "utilities" reset the default browser, hijacked the search bar to point at a different search engine and left droppings in the startup folder offering to re-install something almost but not quite deleted.  Oh ... and fixing a driver that wasn't up to date.

Ah, progress.

A piece of history

Saw this on my g+ feed: CERN has re-posted the first web page ever (or at least the first one they put up for public consumption), 20 years after the fact.  I'm not sure if it's under the original URL, but http://info.cern.ch/hypertext/WWW/TheProject.html sure looks like it could be.  Anyone remember "hypertext"?

Like a lot of efforts in the early stages, much of the content is about the project itself.  For example, there's a "Line Mode" browser:

The LineMode Browser is suitable for use on dumb terminals, requiring no control sequences except for carriage return and line feed.

"Dumb terminals"?  "Control sequences"?  "Carriage return"?  What language are they speaking? (Of course, Lynx is still in business.)

In any case, it says something that CERN can bring back the original web page from the archives twenty years later, in all its textual glory, and it still works, and people really can see it, world-wide.

What do we mean "mobile device"?

It's pretty clear that mobile devices ... hang on a sec.  What's a mobile device?  According to Wikipedia, it's, um, a small electronic device you can carry around.  But not a laptop.  So a smart phone, a not-so-smart phone, a tablet computer, a camera, an MP3 player, a handheld video game, a pager ...

A few of those have been around a long time, at least by electronic standards.  Somehow, I don't think that most people have devices like this in mind when they speak of mobile devices.  For practical purposes, "mobile devices" means "smart phones, tablets and stuff like that".  More precisely, it's not just mobility that people care about.  It's mobile connectivity, the idea that your mobile device can connect to the world at large and interact with it in arbitrary ways.  The mobile web, that is.

So where was I?

It's pretty clear that mobile devices are playing a bigger and bigger role in people's lives these days.  Lots and lots and lots of people have cell phones, quite a few people have tablets, and more and more do every day(*).  It's also clear that people have adapted to having ready access to the web.  One sure way to know you're out in the boonies, whether for the good of getting away from it all or the ill of being cut off from it all, is not having any bars.

When I was a kid, not that long ago, I like to think, if you wanted to meet someone at a large public place, you would have to pre-arrange -- "Meet me on the west side of the station near the stairs for the subway line."  Now you can just call up your party and ask "Um, hey, where are you? ... oh, there I see you."  If you broke down at the side of the interstate, you'd have to wait for someone come by (unless you had a CB, and a lot of people did, though not necessarily for that particular reason).  Now you just call someone.  And, of course, all the behavioral changes brought on by the web, like pulling down news you're interested in instead of waiting for the evening paper or news broadcast, are possible whether or not you happen to be near home.

So if a mobile device is something mobile that can hook you up to the web, then what we have is a series of less-tethered-to-a-particular-place ways of connecting:

• Ancient times:  If you could connect to a remote system at all, it was through work, or a university or other such institution.  Maybe you could dial in to that system from home, using a honking big dumb terminal.  One way or another you were essentially going over phone lines (even the backbone of the time was a bunch of T1 lines, if I understand correctly).
• BBSs (Bulletin Board Systems) and online services such as Compu$erve begin to appear and personal computers with modems become commercially available.  Now you can connect from home, generally to a world completely different from what you'd encounter at work, assuming your line of work even involved the internet.
• Laptops become widespread.  Now you can connect from anywhere you can lug your laptop, assuming you can tie up a phone line.  By this time you can also plug your laptop in to people's local networks.  Cell phones exist, but using them to connect to the internet is cumbersome at best, and almost certainly very expensive.  Internet cafes pop up.
• WiFi becomes widespread.  With municipalities airports, hotels and commercial chains putting up hotspots here and there, the concept of an "internet cafe" becomes somewhat moot.  Many people can connect from wherever they are much of the time.  Phones are becoming webbier, but in a limited way.
• Present day: Smart phones become widespread.  Apps are developed so that you can interact with your favorite sites without squinting at a web site through a browser.  Phones have enough horsepower to provide a nice, snappy experience, at least where you have coverage.

If mobile connectivity is more important than whether a device will fit in your shirt pocket, and I think in this context it is, then mobility starts somewhere around the spread of laptops.  Certainly by the time WiFi is widespread and home "broadband" access is commercially available, the difference from the present day is more degree than kind (understanding that a big enough difference in degree is essentially a difference in kind).

That's not to say we're not entering a new phase.  We are.  A location-aware phone that is always on and always with you is significantly different from a laptop you have to plug in, power on, log in, etc.  From a technical point of view, designing for a small touch screen is significantly different from a laptop screen, much less a 30" monitor.  Nonetheless, the current phase is just the latest in a series of steps making it easier and easier to connect from anywhere.

(*) It's not always clear what "lots of people" or "widespread" should mean.  Widespread among affluent technophiles?  Lots of people in the developed world?  Widespread in a large portion of the world -- which may be ahead of parts of the developed world when it comes to mobile communication?  I'm bravely sidestepping such questions here, but I wanted to at least call them out.

Dennis Ritchie, 1941 - 2011

I have no intention of turning this blog into an obituaries column, and no desire to see "celebrity deaths come in threes" spill over into the tech world, but having noted the passing of Steve Jobs I feel obliged to note the passing of Dennis Ritchie as well.

You may or may not have heard of him before.  It took the major news outlets a while to pick up the story, and even then it wasn't front page.  For hours the main public source was colleague Rob Pike's Google+ page.  That's not too surprising.  CEO of major corporations and eminent computer scientist are two completely different gigs.  Nonetheless, Ritchie had as profound an effect on the Web As We Know It as anyone else, even though his groundbreaking work predates the web by a good measure.

It's fair to say that the web as we know it would not exist if not for Unix.  The first web server ran on NeXTSTEP,  which traces its roots to Unix [and, in fact, NeXT was run by the late Steve Jobs -- tech is a small world at times -- D.H. Nov 2018].  A huge number of present-day web servers, large and small, run on Linux/GNU which, even though the Linux kernel was developed from scratch and GNU stands for "GNU's Not Unix", provide an environment that's firmly in the Unix lineage.  The HTTP protocol the web runs on has its roots in the older internet protocols and belongs to a school of development in which Unix played a major role.

Ritchie was one of the original developers of Unix.

The Unix operating system, the Linux kernel, many of the GNU tools and countless other useful things (and at least one lame hack) are written in the C language, which is also one of the bases for C++, C#, Objective C and Java, among others.  All in all, C and its descendants account for a large chunk of the software that makes the web run, and for years, before the ANSI C standard, the de facto standard for the language was a book universally called "K&R" after its authors, Brian Kernighan and Dennis Ritchie.  That flavor of the language is still called "K&R C".

Ritchie continued to do significant work throughout his life and won various high honors, including the Association for Computing Machinery's top honor, the Turing award, and the US National Medal of Technology.  He was head of the Lucent Technologies System Software Research Department when he retired in 2007.  He may not have been a cultural icon, but in the world of software geekery he cast a long shadow.

RIP

Xanadu vs. the web: Part I - Prologue

I started out to write a post called "Hypertext vs. the web" which would try to compare early predictions of what hypertext was supposed to be against the way we use the web.  The idea was that, while we do indeed have documents with links embedded in them leading every which way, that's not the dominant way we use the web.  Wikipedia being an obvious and successful counterexample.

But you can't talk about early predictions of hypertext without talking about Xanadu.

So what was (or is) Xanadu?  In cold honesty and to the best of my understanding, Xanadu is a failed vision, but there's little to be learned from simply calling something a failure.  There can, however, be much to be gained from trying to understand what the vision in question was, and why it failed.  The full story is a long one, spanning decades and branching off in myriad directions, but I would like to take a few posts to explore the subject in broad outlines.  My aim is to compare a vision of what the web (or rather, its analog) might have been against what it actually turned out to be, and also to try to understand why things turned out the way they did.

As a bit of a disclaimer, I haven't used Xanadu (but that's part of the point: arguably no one has), nor have I seen a demo of it in its full form, nor am I closely or even not-so-closely acquainted with anyone involved, nor have I even corresponded with anyone involved.  In short, all I know about Xanadu is what I read on the web.  Given that the topic here is "Xanadu vs. the web", and there can be a certain antagonism between proponents of Xanadu and proponents of the web, this view is not without its distortions, but it's what I've got so I'm going with it.