The shape of the backbone

I was all set to write a somewhat philosophical post comparing the internet backbone to the phone system that came before it, when I realized I know very little about the internet backbone. Since I claim not just to be writing about the web, but actually figuring it out (however slowly) as I go along, that seemed like a pretty embarrassing lapse. So I've started doing a little basic digging and, so as to have something to post while I'm doing it, pass along some of my findings.


There was a British general once who, whenever he was sent to a new theater of battle, would start off by making a crude, not-painstakingly-to-scale map of the major roads and rail lines in the territory, noting the distances between the most important cities, junctions and other points of interest. This exercise would pay dividends later, not just in the day-to-day running of the campaign, but in preventing blunders on the order of depending on troops to cover a week's worth of distance in two days. It would also give some idea of the political structure of the place -- who was likely to be trading with whom, who might not care too much about what was going on where, and so forth -- and help greatly in deciding which supply lines to guard where, which enemy supply lines to attack where, etc., etc.

Along those lines (minus the military perspective), it seems useful to start with a very large-scale look at the international internet backbone, with an eye toward who's connected to whom and with what capacity. Physical distance is not as important here, though latency can matter in some cases.

Here's just such a map, dating from 2005, prepared by TeleGeography research for the International Telecommunications Union. Please don't look at the part that says "Proprietary and Confidential". There are probably more recent maps available* but this should give a reasonable impression. A few things that jump out:

• North America <-> Europe dominates, followed by North America <-> Asia with roughly 40% of that capacity, followed by everyone else, far behind. In particular, there is very little capacity, relatively speaking, directly linking Europe and Asia. If you're in Bangalore talking to Bucharest, you may well be going by way of San Francisco.
• Capacity to and from Africa is almost negligible on this map, but see below.
  
• Capacity on these scales is currently measured in Gbps (the document gives 5- and 6-figure numbers in Mbps, but most of that precision is probably spurious). That's pretty big compared to my home connection, but not really stunningly large considering how many people are involved. By comparison, the human visual system appears to have a capacity on the order of 1Gb/s, so the 500Gb/s pipe between North America and Europe is, in some notional sense, roughly equivalent to 500 pairs of eyes. Put more realistically, though, 500Gb/s is about 125,000 DVDs playing simultaneously.
• What these numbers actually mean is a different question entirely. Suppose for example that everyone in the US wants to follow the European football leagues online -- not that that's going to happen anytime soon. Would those millions of viewers saturate the pipe? Hardly, since the main pipe only has to carry a few video feeds at any given time.

As I said, the big missing piece in the 2005 picture is bandwidth between Africa and the rest of the world. As of this summer, that bandwidth has gone from practically nothing to about 1.3 Tb/s ,or 1300Gb/s to put it in the same units used above. That's over twice the size of the Europe <-> North America pipe, a classic example of leapfrogging technology assuming we're comparing apples to apples. Even if we're comparing apples to pears it still looks like a pretty big pipe.

(*) Back in the days that bang paths roamed the earth, one could subscribe to a newsgroup which published frequently updated maps of the backbone as it then existed, meaning mainly a bunch of T1 and similar lines. A T1 line could carry about 1.5 Mb/s, or .0015 Gb/s, or about 0.0003% of the Europe <-> North America link.

Digital vs. interactive vs. internet vs. broadcast

There's a lot I don't know about the brave new world of electronic media, but a few things are starting to stick in my brain:

• Video is the 500lb gorilla. Everything else we currently do takes much less space and bandwidth.
• Unless you have fiber or something similar, video is pushing the limits of your bandwidth (*)
• Even if you do have fiber, you can use up large chunks of it with bigger and shinier video, or more channels, or better-than-real-time downloads.
• TV in the US is set to go digital in the next couple of years. This frees up large chunks of bandwidth for other uses (**).

That last chunk of bandwidth is primo stuff. It was originally allocated to TV exactly because it's suited to transmitting video images into buildings and across wide swathes of rural land.

Now let's look at video in the internet world, as opposed to old-fashioned TV with rabbit ears (Dad, what are "rabbit ears"? Well, in the olden days TVs had antennas sticking up on top of them ...):

• It's digital. That means, subject to various legal and technical restrictions, you can make exact copies of it, save it, view it and edit it with a plain old home computer.
• It's available on demand (or at least, upon polite request with money attached). You don't have to wait for your show to be on.
• It can be interactive. You can do video conferencing, for example.

The interesting thing is that none of these requires the internet. It's been possible for years to record video digitally with a camcorder, make your own DVDs, and so forth. You can (again subject to legal and technical restrictions) digitally record TV programs received via cable, satellite or over the air. Video on demand is a standard feature of cable service. Video conferencing can be done (and in some cases is done) via closed circuit.

This is not to say that the internet can't add value, for example by making video conferencing available to everyone instead of just to parties who happen to have the infrastructure for it. The point is that much of the goodness of new media is not strongly tied to the net, nor is it, as is sometimes implied, a direct consequence of the net being involved.

The internet is inherently point-to-point. Your basic internet packet has a single destination address. You can also specify broadcast and multicast addresses, but those require extra thought and machinery to handle efficiently. The internet is also symmetrical in structure. Anyone can send to anyone. The other party might not listen, but that's a different story.

Broadcast TV, on the other hand, has a very different structure. One party broadcasts to many. It does so very efficiently and scalably. Someone puts up a tower. As many people as want to buy receivers. This can be and is being done for digital video as it was for analog. Traditional cable TV -- everything but the on-demand part -- is somewhat less easily scalable, but it still has the advantage of having only one fixed route: from the broadcast center out to the viewers.

Why should we persist in using old distribution machinery when we've got the infinite flexibility of the net? For one thing, people remain interested in live TV coverage. If I want to watch the NCAAs and a million other people do as well, then why bother the internet backbone about it when the network can just beam the games up to a satellite or broadcast them over the cable network.

The one-way broadcast model still works just fine in cases where you might well want digital content, but don't care about interactivity. In fact, it should generally work better, and using it offloads traffic that would otherwise have to go through the backbone.

Which leaves me to ponder what really ought to end up happening to all that prime broadcast bandwidth.

(* A standard DVD plays back about 4 hours of video from about 8GB of data, or about half a megabyte per second. This is curiously close to the bandwidth of my cable internet connection, and in fact video full-screen comes across more or less OK, but not dazzlingly well. It's interesting to note that the same cable can also keep at least 3 tuners happy showing and/or recording TV shows with good fidelity. It's almost as though cable companies are in the TV business, but I digress ...)

(** Most of it was bought up by Verizon, with Google only putting in a minimum bid. More could be said ...)

We'll be with you after a brief delay ...

One of Deutsch's distributed computing fallacies is that latency is zero -- messages arrive the moment you send them. In practice, it takes a bit of time for your message to get through your firewall to your ISP's router, onto the backbone, to your recipient's ISP, through their firewall and to their host.

Much of the time, this won't matter. If your mail client is polling every five minutes, who cares if the packets containing your email took some fraction of a second to get to your mail server? On the other hand, if you're trying to do something on the net that requires precise synchronization, things can get hairy.

The Network Time Protocol keeps good time (to within about 1/100 of a second over the internet), but NTP relies on propagating accurate timing out from a small set of central servers. It doesn't try to keep everyone in sync with each other directly.

Depending on the circumstances, people can notice delays as low as 20-40 milliseconds. For example, a lag of 40 milliseconds between hitting a key and hearing a sound is enough for a musician to notice. Echoes become perceptible around 100ms and extremely distracting not long after.

Latency on a local network can be quite low, often just a few milliseconds. The round-trip time for pinging a major service can be in the teens and twenties. This is partly because the major providers replicate their servers so that you're generally talking to a server close to you.

For example, I pinged www.google.com.au (Google Australia) and got a round-trip time of about 15ms. That's pretty impressive, given that I'm about 15,000km from the nearest point in Australia and light travels at 300,000km/s. That would give an absolute minimum time of 100ms for the 30,000km round-trip. However, the name www.google.com.au resolves (where I'm sitting) to a server in Mountain View, CA. That's fair game, as long as the Mountain View server looks enough like the "real" one Down Under.

However, if I try to ping the Australian Broadcasting Company (which probably has little reason to put a duplicate server in my neighborhood), I get a more believable time of 200ms or so. Depending on circumstances, that much delay can cause problems. For example, a badly placed speaker phone in a conference call between Oz and the US can render conversation nearly impossible.

As it turns out, most of the populated world has water directly opposite on the globe, but there are a few extreme cases, such as Hamilton, New Zealand and Córdoba, Spain. There are also plenty of less extreme cases, whether Europe/Australia, California/India or what-have-you, where even the best case may introduce noticeable delays.

The high-order bit here is that some level of humanly perceptible latency is likely to be with us, in particular situations, no matter how fast the hardware gets. Moore's law can make the pipes bigger and the processors faster, but it will do nothing to increase the speed of light.

Pumping data through the Hetch Hetchy Aqueduct

Paul Vixie made a very interesting speech to the Commonwealth Club in San Francisco last year. As it was aimed at a general audience, he started out with some familiar points. I'll repeat them here [with a couple of comments] because they lead up to the more interesting conclusions:

• Cash flow is more important than cash. The word for the day is monetize: to extract cash flow from.
• One's ability to monetize things in the physical world is governed by regulations, including anti-trust regulations, balancing the good of society against the good of the individual.
  
• Owning a physical CD or book or such is ownership in the dictionary sense.
  
• This is regulated by copyright law, a fair system which has made a lot of money for content creators while supplying a lot of content to people who might not have had it otherwise.
• This doesn't work in the virtual world, where perfect copying is cheap.
• In the case of music, this is mostly a concern to the major labels and their artists. Smaller artists give away music digitally to promote concerts and sell T-shirts [see also Radiohead's latest release]
• Big music, after trying to stop digital copying [actually, those lawsuits are still shaking out] has decided to try to make money off of digital music.
• They do this by letting you listen to what you want but controlling the means of playing it. A typical software player reports your downloading, maybe shows pop-ups, and ties you to one of the commercial OS's -- and its upgrades. Again, the word is "monetize".
  
• (For his part, Vixie just buys CDs and plays them on his Linux laptop. And respects copyrights.)

So far so good. But then he goes on:

• This model of monetizing content applies to all content on computers, even content you produce yourself for your own use, that is, your email, text documents, etc. How does that work? It works because ...
• Viruses, worms etc. are a major problem for computer users these days, a problem that many companies are trying to solve, that is, monetize.
• Consumers just want their computers to work, but there's not actually any money to be had in solving that problem. A computer that Just Works is like a razor blade that never gets dull.
• One part of the solution is to rent virus protection. In a fuller solution, OSs would ideally only run trusted applications registered with the OS vendor -- for a fee, of course. This is already how game boxes work.
• Carry this over to the application world, and you end up renting the ability to access your own content, which is now stored in encrypted and/or proprietary form.
• Fortunately, this is very difficult to pull off on current platforms. It would really require a whole new kind of hardware/software platform. Best to start small. Say, with music ...

And finally, Vixie draws a contrast between San Francisco's decision in the early twentieth century to build, operate and therefore control its own water supply, and its decision in the early twenty-first century not to build and operate its own wireless internet infrastructure, instead putting it out to bid.

In light of the points above, this may not be such a good idea. That doesn't mean that software vendors, record labels and so forth are evil, just that they're for-profit entities and must be expected to act as such.