Movie review, game review

While I'm on the VR theme ...

The other day I went to see Nightmare Before Christmas 3D. With its stop-motion animation, the original Tim Burton production of Nightmare might seem the perfect choice for a 3D adaptation. In fact, the 3D "remastering" added surprisingly little. Some in my party found it downright disappointing, having hoped for more dramatic 3D effects (which, IMHO, would have spoiled it). The high point for me wasn't the 3D, or even getting another look at director Henry Selick's visual inventiveness, but realizing what a fine score Danny Elfman had produced.

Don't get me wrong. Nightmare is a classic whichever way you look at it. But when the most notable feature of a 3D movie is the score, something is at least a little out of whack.

My experience with other 3D shows is similar (except for the Tim Burton/Elfman angle). Polarizing glasses are a great improvement over the classic red-green (both have been around for a while), but overall 3D just doesn't add that much.

There's a pretty likely explanation for this: We don't actually see in 3D. We do make use of the extra information from having a slightly offset pair of images to work with, but in a pinch we can make do with one. The image we construct either way isn't a full 3D rendering, but more of a "2.5D" schematic in which we are mainly concerned with things like what's in front of what, which way and how fast things are moving, and roughly how big and how far away things are.

The eye, that is, the massively complex machinery behind the eyes, can be fooled in any number of ways because of the assumptions it makes to construct that internal image. For example, it can be convinced that a flat canvas with paint on it is a three-dimensional space, or that a series of images quickly projected on a screen is a collection of objects moving in space.

That first illusion -- that a painting is the scene it represents -- is the big step. The rest is just refinement. That's why they called them "moving pictures". In all cases, the realism comes not from the fidelity of the image we present, but what the brain fills in. Which leads me to the game review.

With the very big caveat that my serious compugaming days are many years behind me, my nod for most immersive VR experience goes to .... NetHack. Yep, that one you might have seen somewhere with the ASCII-art rectangles connected with lines of #'s and an @ and a (usually) faithful d battling their way along.

Later versions got a bit baroque for my taste. Your mileage may vary, but somewhere along the line was a nearly perfectly-balanced concoction. What made the experience rich was the way the consequences were all worked out. If you tried something, you got a result that made sense, but not always the one you expected or wanted.

For example, there was the wand of polymorph. In the predecessor game Rogue, you could use one in desperation to try to zap the monster attacking you into something less deadly (a decent try against a xorn or umber hulk, not so good against a bat). In NetHack, you could still do that, but you could also zap a pile of rocks and sometimes get gems. You could zap anything and unless there was a good reason, it would change. Sometimes for the better, often not.

Much of the fun came from identifying all the magic items you came across. Every once in a while you might find a scroll of identify, but mostly you just had to try stuff and hope for the best. What does this wand do? I'll just zap it at that wall ... The lightning bolt bounces ... the lightning bolt hits you ... you die ... OK, next time I'll try zapping it at an angle ... Now what does this potion do?

That's not to mention the various ways of looting shops, or surviving the nearly-unsurvivable, or discovering what seemingly useless items did (why would I want to make the monster I'm fighting invisible?) or trying things like reading a scroll while confused.

It appears NetHack is still being actively developed. You hear maniacal laughter in the distance ...

Latency in a virtual stadium

Just to put a little perspective on my previous post on latency: Sound travels about 350 m/s. If the network round-trip time between the US and Australia is 200ms, then that's equivalent to a physical separation of about 35m, at least as far as sound is concerned. Being in a global virtual crowd is like being in a largish theater or small arena.

That's really not going to be a problem most of the time. The interesting thing to me is that there's a hard limit that is (just) humanly perceptible. For example, if you're assembling a virtual crowd, you'll hear the reactions from across the globe perceptibly later than those from your neighbors. The only way to even this out is to slow everybody down, which would have its own effects on the feedback loop.

Or consider a virtual marching band without any visual cues, which is effectively what you have if it takes just as long to see the baton as to hear the drum. Not impossible, but a bit more challenging.

We'll be with you after a brief delay ...

One of Deutsch's distributed computing fallacies is that latency is zero -- messages arrive the moment you send them. In practice, it takes a bit of time for your message to get through your firewall to your ISP's router, onto the backbone, to your recipient's ISP, through their firewall and to their host.

Much of the time, this won't matter. If your mail client is polling every five minutes, who cares if the packets containing your email took some fraction of a second to get to your mail server? On the other hand, if you're trying to do something on the net that requires precise synchronization, things can get hairy.

The Network Time Protocol keeps good time (to within about 1/100 of a second over the internet), but NTP relies on propagating accurate timing out from a small set of central servers. It doesn't try to keep everyone in sync with each other directly.

Depending on the circumstances, people can notice delays as low as 20-40 milliseconds. For example, a lag of 40 milliseconds between hitting a key and hearing a sound is enough for a musician to notice. Echoes become perceptible around 100ms and extremely distracting not long after.

Latency on a local network can be quite low, often just a few milliseconds. The round-trip time for pinging a major service can be in the teens and twenties. This is partly because the major providers replicate their servers so that you're generally talking to a server close to you.

For example, I pinged www.google.com.au (Google Australia) and got a round-trip time of about 15ms. That's pretty impressive, given that I'm about 15,000km from the nearest point in Australia and light travels at 300,000km/s. That would give an absolute minimum time of 100ms for the 30,000km round-trip. However, the name www.google.com.au resolves (where I'm sitting) to a server in Mountain View, CA. That's fair game, as long as the Mountain View server looks enough like the "real" one Down Under.

However, if I try to ping the Australian Broadcasting Company (which probably has little reason to put a duplicate server in my neighborhood), I get a more believable time of 200ms or so. Depending on circumstances, that much delay can cause problems. For example, a badly placed speaker phone in a conference call between Oz and the US can render conversation nearly impossible.

As it turns out, most of the populated world has water directly opposite on the globe, but there are a few extreme cases, such as Hamilton, New Zealand and Córdoba, Spain. There are also plenty of less extreme cases, whether Europe/Australia, California/India or what-have-you, where even the best case may introduce noticeable delays.

The high-order bit here is that some level of humanly perceptible latency is likely to be with us, in particular situations, no matter how fast the hardware gets. Moore's law can make the pipes bigger and the processors faster, but it will do nothing to increase the speed of light.

Take me out with the crowd

Previously I described the currently popular setup of crowd, performer and large display and said there must be some basic human reasons why the experience remains popular despite the obvious inconvenience. Here are some guesses. I suspect the real answer is a combination of them and other obvious stuff I've missed:

• Not all senses virtualize well or easily. If you're at the venue, you don't just hear and see the show. You jostle with people. You smell all manner of interesting (and not-so-interesting) aromas. You feel sound in your entire body, not just your ears, even without a high-powered sound system.
  
• Meeting people in person. People still like to meet face-to-face, for a number of reasons, including the previous point, even if it's only in small groups. Sure you can invite your friends over to watch the game on TV, but you can't invite the vendors or the random people you'll be seated around or run into in the parking lot.
• Ambient noise. To my knowledge, there aren't a lot of (or any?) virtual experiences that provide the murmur of 50,000 people chatting before the house lights go down, or the roar when the stage lights go up, or the sudden hush when something dramatic happens.
• Spontaneous mass expressions. There's probably a better name for this, but I mean things like chants, songs, the wave, synchronized clapping and so forth. Ambient noise could be simulated, but that would break the feedback loop behind these.
  
• Sense of community. When thousands brave winter weather to watch a crucial away game on the big screen in their home stadium, it's clear that they're there (and not at home watching TV) largely because they want to show that they care and be with people who care the same way. This has a lot in common with a mass political demonstration.

There is definitely overlap among these, but each has its own effect. Some of the items have analogs in the virtual world. Current home theater systems can bring the noise reasonably well. The slashdot effect is not unlike a mass reaction to something on the screen and there are word-of-mouth phenomena similar to a chant or wave getting started. Not all reactions in real crowds are spontaneous -- think applause signs or the organist/PA at a sports event -- and neither are they on the web -- think viral marketing.

Nonetheless, the combined effect (together with whatever else I missed) remains unique, which is why the virtual element of the big screen is more an adjunct than the main show. As far as that goes, the big screen is just an extension of the amplified sound of the PA, which has been around considerably longer than the net. Neither is necessary for a large crowd experience. Those have been around forever.

Personal datastores and Vixie's dystopia

Paul Vixie argues in his 2006 Commonwealth Club speech that OS vendors, in their quest to monetize their customer bases, will develop a system in which we effectively pay for access to our own data. At the root of this, I think, is the fact that managing our increasing piles of data and tracking ownership of it is not something most people want to take on. Think financial and health records, contacts, appointments, correspondence, home movies, commercial music, home-made mixes of commercial music, home movies with a mix of commercial music in the sound track ...

Software vendors are more than happy to supply the tools for managing this, and this is a very legitimate and useful thing for them to do. Service providers like banks are also happy to provide similar tools. Service providers like hospitals and clinics might want to, but at least in the US HIPPA will tend to prevent that.

The problem here is that the data tends to get tied to the tools and providers, leading to two kinds of lock-in:

• Weak lock-in due to proprietary formats. The maker of a word smasher has no particular incentive to make its format widely available. At best it has an incentive to read everyone else's format, write a few least-common-denominator formats (e.g., PDF) -- and make sure that its format has a few extras that no one else has.
• Strong lock-in due to restricted access. Unless I take steps to keep a copy, I can only get at my bank records by signing in at my bank. It may choose to limit my access to, say, only the previous year. My bank is also free to derive whatever secondary data it likes from my raw transaction record (e.g., whether or not I qualify for a given loan) without disclosing exactly how it arrived at that conclusion.

Neither of these inherently needs to keep people from accessing their own data. Proprietary formats are security through obscurity. Strong lock-in may work for content like music created by someone else (though I have my doubts), but shouldn't be an issue for content you create yourself. If the bank doesn't want to disclose its secret loan-qualification formula, that's its information, not mine, but I have the option to download my own transaction records every few months and store them where I choose..


The dystopia Vixie describes is pretty much the antithesis of the personal datastore. In that model, I control access to my information, probably paying someone to keep it safely backed up and to limit access to it. Software vendors sell tools for manipulating and searching that data. Service providers like banks and medical providers populate that data and may access parts of it with my permission.

Except, how does that really change the picture? Right now, if I want to save a document, I have it written to a file on my disk. In the personal datastore model, I have it written to a chunk of my datastore. In either case, a tool could refuse to write a version anything but it could read, and in either case, the antidote is to make sure buyers understand that this is happening and that there are alternatives.

It is important here to distinguish two key parts of the personal datastore model. The first is that the datastore is a network resource; it is safely maintained and visible from anywhere, but with fine-grained access control. The second is that it serves as a point of integration by keeping its data in open formats.

The two are largely orthogonal. I could, for example, carry a smartcard from health care provider to health care provider, and as long as the data on it is in a standard format I can keep my medical records all in one place as opposed to in separate silos. The smartcard is a point of integration but not a network resource.

On the other hand, I could keep my data on the net but in a format not even I can read directly. I wouldn't want to do this, but a vendor might want to do this, say by encrypting the data with a key only it knows. In that case I'd have a network resource that was considerably limited as a point of integration (unless I wanted to limit myself to other tools in that vendor's suite, of course ...)


In the end, I don't think that resistance to vendor lock-in will by itself be enough to make personal datastores happen, but it should at least make the environment more hospitable to them.

The multimedia crowd experience

Sometime, quite a while ago, big sports events changed. Time was, you would sit in a stadium and watch a game. If you were up in the nosebleeds you'd bring a pair of binoculars. If you missed a play, well, you missed it.

Then they started putting big screens in parks and you could watch what you were watching. If they panned the camera into the crowd, you could even watch people watch what you were watching. If you missed a play, you could watch a replay.

The same concept works for concerts. Even people up in the cheap seats can see the sweat drip off the performer's nose. In fact, you might have a better experience there than someone on the floor of the arena too far away to see the stage well and without a clear view of the screen.

The same basic setup of crowd, performer and big screen is also used in megachurches and even in larger college classes.

OK, so if you're basically watching the show on TV, why bother to go to the arena, particularly if it's a sports event that's on TV anyway? Why not just sit comfortably at home, steps away from your kitchen and bathroom, skip the freeway or train trip and take in the commentary. If you have a DVR you can even pause and replay that close call yourself, as many times as you want.

Clearly, being part of the crowd is important. Being able to react and experience everyone else's reactions is worth a lot. So is being able to say you were there. There's something basically human going on here, along the lines of Mike Chwe's theme of common knowledge.

I have no direct experience of the latest generation of virtual worlds, but my impression is that they're not immersive enough to deliver quite the same experience. The virtual stadium (minus the long lines at the restroom) is probably quite a ways a way, not to mention the virtual moshpit.

Pumping data through the Hetch Hetchy Aqueduct

Paul Vixie made a very interesting speech to the Commonwealth Club in San Francisco last year. As it was aimed at a general audience, he started out with some familiar points. I'll repeat them here [with a couple of comments] because they lead up to the more interesting conclusions:

• Cash flow is more important than cash. The word for the day is monetize: to extract cash flow from.
• One's ability to monetize things in the physical world is governed by regulations, including anti-trust regulations, balancing the good of society against the good of the individual.
  
• Owning a physical CD or book or such is ownership in the dictionary sense.
  
• This is regulated by copyright law, a fair system which has made a lot of money for content creators while supplying a lot of content to people who might not have had it otherwise.
• This doesn't work in the virtual world, where perfect copying is cheap.
• In the case of music, this is mostly a concern to the major labels and their artists. Smaller artists give away music digitally to promote concerts and sell T-shirts [see also Radiohead's latest release]
• Big music, after trying to stop digital copying [actually, those lawsuits are still shaking out] has decided to try to make money off of digital music.
• They do this by letting you listen to what you want but controlling the means of playing it. A typical software player reports your downloading, maybe shows pop-ups, and ties you to one of the commercial OS's -- and its upgrades. Again, the word is "monetize".
  
• (For his part, Vixie just buys CDs and plays them on his Linux laptop. And respects copyrights.)

So far so good. But then he goes on:

• This model of monetizing content applies to all content on computers, even content you produce yourself for your own use, that is, your email, text documents, etc. How does that work? It works because ...
• Viruses, worms etc. are a major problem for computer users these days, a problem that many companies are trying to solve, that is, monetize.
• Consumers just want their computers to work, but there's not actually any money to be had in solving that problem. A computer that Just Works is like a razor blade that never gets dull.
• One part of the solution is to rent virus protection. In a fuller solution, OSs would ideally only run trusted applications registered with the OS vendor -- for a fee, of course. This is already how game boxes work.
• Carry this over to the application world, and you end up renting the ability to access your own content, which is now stored in encrypted and/or proprietary form.
• Fortunately, this is very difficult to pull off on current platforms. It would really require a whole new kind of hardware/software platform. Best to start small. Say, with music ...

And finally, Vixie draws a contrast between San Francisco's decision in the early twentieth century to build, operate and therefore control its own water supply, and its decision in the early twenty-first century not to build and operate its own wireless internet infrastructure, instead putting it out to bid.

In light of the points above, this may not be such a good idea. That doesn't mean that software vendors, record labels and so forth are evil, just that they're for-profit entities and must be expected to act as such.

The end of text?

With help from friends, I finally tracked down the piece I said I was looking for. It was written in 1994 and, ironically enough, there only seems to be one copy of it on the web, or at least the Google-searchable part.

The thesis is that in 2020 a nine-year-old would have had no reason to learn how to read. The written word, after all, is just a technology for conveying ideas, and by 2020 video and other rich media will do the same job better.

The written word is a means to an end and not an end in itself. We use it to communicate with large groups and to preserve ideas, but we prefer the spoken word. In 2020world, with the ability to create, store and send audio and video as easily as written words, why would we need to read and write?

Look inside your own head. Do you store information as written words? Do you dream in written words? No, you don't. Visual images and spoken languages are our natural form of information. Writing is nothing more than a technology. It can be replaced by something better.

Clearly this is missing something. At the very least, the timing is off. If they're going to stop teaching reading in the schools in the next ten years, I'd expect to see serious signs right now that writing was on the way out. Perhaps a blog post written about an newspaper article is the wrong place to look, but I see no such signs. It's a separate question whether schools would change their curricula that quickly even in the face of irrefutable evidence.

So let's suppose that the date should have been 2050, or 2100. Technology predictions are notorious for assuming things will change faster than they really do. Is text really doomed to be obsolete?

Text is a means of recording words. True, we don't think in text. But neither do we think in words, at least not to the extent we sometimes say we do. Words, whether written or spoken, represent ideas. They do so digitally. Text is a sequence of discrete, arbitrary symbols. To a first approximation, so are spoken words. Otherwise text wouldn't work. Hand-copying of written text is among the oldest forms of digital data processing.

Text is compact. This post takes a small fraction of the space that an audible version would. Even in an age of abundant bandwidth, a difference of orders of magnitude will matter. Since text represents words digitally (as opposed to representing the waveforms of one particular utterance of those words), it is easily searched. At the very least, a usably searchable database of video and audio would have to use something much like text behind the scenes.

Text is faster for people because vision has more bandwidth than hearing and well-formatted text is tuned to take advantage of that. I can skim this post much faster than I could read it aloud. I can skim backwards easily. I can skip sentences and paragraphs easily and precisely. If I want to make a minor change to the second sentence of the third paragraph, I can easily locate that and I can easily change just the words I want to change. Text is thus more easily editable.

Since text can easily (from a human viewpoint) be accessed both randomly and sequentially, it is easier to organize. This is probably one reason speakers so often work from notes. Another is that the mere act of committing something to text encourages the writer to pay attention to its structure.

There are probably a few other relevant features of text that I've left out. There are also some that may not be particularly useful but whose implications are probably still worth understanding. For example, writing text is generally much slower than reading it, while speaking and hearing happen at the same speed.

In any case, my bet is that text will be around for quite some time, particularly on the net. If this still seems unlikely, consider how often text appears on TV. Video and text are by no means mutually exclusive.

Web usage in reality

Following onto the idea of a web-based hyperreality supplanting reality as we know it, some basic questions occur:

• How much of our information do we get from the web?
• By what measure?
• What are the trends over time for the population at large?
• What is the trend for a typical individual?
• What does the web tend to replace, or at least, what are the trends in other media?

I would think the answers to questions like these have a large bearing on how the web affects our perceptions now, and on what kinds of predictions we can make.

Is hyperreality the new reality?

While searching for a different article (on which I'll probably comment if I run across it) I found a piece by Daniel Rourke on "hyperreality". It expresses a notion that I've run across from time to time, one which seems compelling at first blush. Jumping right into the middle:

Could Wikipedia at its broadest boundaries be a metaphor for the future of human society? Take away our cultural memes and humanity would quickly revert to the simple cultures seen in our monkey and ape relatives. It was the evolution of language which bound humanity into a shared consciousness - a cultural brain which did more thinking than any individual identity could do alone.

This is practically self-evident, which makes it automatically suspect. Are other primate cultures really so simple? A primatologist might well disagree. Is human culture really that different from primate culture? There's more in common than we like to admit

Is language so big a factor in propagating culture? A good deal of cultural behavior consists in things we "just know to do" (or not do), and which we often have trouble putting into words. Writing an etiquette guide is a difficult endeavor; getting people to follow it even more so.

What is this shared consciousness that we all have? How much of it is inborn? How much of it is absorbed by immersion or learned by example? When we talk about how we do things, to what extent are we just verbalizing what the non-verbal parts of our brains are doing without our say-so?

What aspect of our shared consciousness does the web stand to change? Yes, the web can bring the same experiences and ideas to large numbers of people very quickly, but so can radio and television. To some extent, so can mass assemblies.

In all the above, I'm not saying the non-web world is the same as the web, but I want a bit more detail on how the web is different.


Rourke continues the theme of the net as an agent of profound change a bit further on

Over the next few years as the internet becomes ever more a totality of culture rather than simply a referent the lines bordering reality, hyper-reality and pure imagination will dissolve around us. I would go so far as to suggest that many generations from now cyber-entities once labeled 'human' will find it impossible to distinguish what was past-real, what is present-hyper-real and what will never be real in the seething masses of datum [sic] the internet will have become.

In other words, in a few years we will be so immersed in the internet that we won't know or care what's real and what's not. I think this misses out two important points.

One is that we are physical creatures. No matter how far off into cyberspace we float, at least for the near to medium future, we will need to eat. We will not be able to physically be two places at once. The laws of physics will still apply.

Even generations from now, if we have somehow slipped the bonds of our physical chains, our cyber-heirs will still be embedded in time. It seems unlikely that the distinction between past and present will cease to be useful.

The other point, closely related, is that our brains are very much shaped around physical reality. We automatically and subconsciously make any number of assumptions about what perceive based on what tends to work in the real, physical world.

These things we know that ain't necessarily so go by the general name of cognitive biases, and the striking thing about them is how many (along with their cousins like optical illusions) make sense in the context of an embodied being scrabbling to find food and mate in an environment of relative scarcity and danger.

In short, no matter how powerful the communicative machinery of the web becomes, or how great the bandwidth, we experience it (and will continue to experience it for some time) through our human wiring, with all its quirks and limitations. It seems to me an unproven assumption at best that, as scenarios like the one Rourke describes tend to assume, we have some hidden potential within us just waiting for something like the web to unlock.

Crowds and wisdom

Suppose you're measuring, say, the four walls of a room. Have twenty people each do the measurement with off-the-shelf tools. Then do the measurement very carefully with a laser interferometer or whatever. While we're in gedanken mode, assume that the room itself is very precisely joined, so that measuring to a few decimal places actually means something.

The central limit theorem tells us that the measurements will tend to fit a normal (i.e., "bell curve") distribution peaking very near the precise length. If you take the average of the imprecise measurements for each wall, the result will generally be quite close to the precise measurement. If you consider all four walls, the combined result -- the four averages -- will generally be closer to the precise measurement than the best individual set of four measurements, assuming the errors in the imprecise measurements are random.

Now take a typical "wisdom of crowds" example: the Oscar ™ party where each guest guesses who will win. A ballot consisting of the most popular choices almost always does better than the best individual ballot. Clearly this is not quite the same as the measurement problem above. At the very least you'd need a different metric. On the other hand, is this "wisdom", or just statistics at work? Two possible answers, not necessarily incompatible:

Wisdom of crowds is about more than Oscar ™ parties. Surowiecki's original book talks about situations like pedestrians optimizing traffic flows or markets setting prices. The key ingredients for crowd wisdom, he argues, are diversity of opinion, independence, decentralization and aggregation.

The party example is fairly low-powered. It might be explicable in terms of statistics, but something like cars not hitting each other or customers distributing themselves among popular restaurants may not.

Wisdom is more about statistics than we might like to think. The words "wise" and "wizened" come from the same root, having to do with age [In a comment to a post seemingly chosen at random, Earl points out that this etymology is hogwash. Dictionary.com confirms this. Nonetheless, the point still seems valid]. Wisdom is the judgment we (ideally) gain with life experience. It's a matter of gut feel or intuition, not of sequential reasoning. In other words, it seems more likely based on statistics than logic. Do we grow wiser mainly by accumulating more data points?

A stock example is chess mastership. Chess masters typically look at many fewer moves than beginners or even experts, but the moves they look at are better. A master will also typically be better at remembering positions from actual games, as opposed to random placements of pieces, while the rest of us will do about equally well at each. A master is drawing on a large store of game experience and using this to structure the analysis of the position at hand. Clearly there is more involved than a simple "this position looks like that one", but just as clearly that's part of the picture.

Whatever statistical analysis a master is subconsciously doing isn't simple enough to have been captured algorithmically. Computers can beat masters at chess, but they do it by bashing through vast numbers of hypothetical moves. Programs that try to "understand" positions beyond simple rules like "I have more material and my pieces are more centrally located" tend to fail.

If that doesn't muddy the waters enough, you might consider this viewpoint.

People v. UK cell phone providers

Following the recommendations of the IEGMP (a.k.a. the Stewart group), the UK government maintains a website called Sitefinder detailing the locations of cell phone base stations. The catch is that the database behind the site is populated voluntarily by the service providers.

Unfortunately, the providers stopped providing information to the database when the site's operators were told to make the stations' grid locations searchable online. The argument was that this would give sensitive information about the providers' networks.

Well, maybe, but I doubt this is a battle the providers will win. A nation known, among many other things, for trainspotting (the hobby, not the book/movie, or if you don't like that association, the Oxford English Dictionary) should have no trouble harnessing the efforts of interested individuals to compile an unofficial database comparable with, if not better than, the official version. "Crowdsourcing," they call it.

Taking a bite out of spam

One of the fascinating things about life on the web is, time and time again, that about the time you really start thinking "why don't they do this?", it turns out someone's been hard at work on it.

In this case, it's digitally signed email. As the BBC lays out in this piece, Yahoo! and eBay/PayPal are about to deploy a system that filters out unsigned mail to Yahoo! mail users (they don't use the term "private key" quite the way I understand it, but the gist is there). Underlying this is DKIM (RFC 4871), which came out in May of this year.

Once major players start to roll out signed mail, it shouldn't be long until all the major mail clients can handle the messages, and maybe not too much longer until ordinary folks start deciding to get keys. And then you'll be able to tell I wrote this (or at least, someone with my key did).

History tells us that this won't be the last chapter in the spam/phishing wars, but it should at least help.

OK, Computer

"Radiohead have made a record. So far, it is only available from this web site. You can pre-order it in these formats: Discbox and download."

With a plummy Oxbridge accent, a curiously garish homebrew website and a Thomas Pynchon reference included at no extra charge, Radiohead is doing its part to rock the digital content world. For a price of "It's up to you ... No really, it's up to you." you can have their latest, In Rainbows, straight from the band.

Or you can wait for your friends to get it and copy theirs. No really, it's up to you.

This isn't the first time a prominent act has bypassed the major labels (the artist formerly known as the artist formerly known as Prince comes to mind), but this one seems to be getting a fair bit of buzz. For example, Auntie says here that their server crashed from overwork.

It also says that most people are choosing to pay a reasonably normal price. I'm not surprised.

The general take I've seen on all this is that bands make most of their money from concert tickets and T-shirt sales anyway, so what's the big deal. Somewhere in the mix I hear record labels whistling in the dark, though interestingly Radiohead are currently said to be negotiating with their former label, Parlophone, and others for a new contract.

And speaking of trusting DNS

In the previous post, I argued that in practice people put a lot of trust in DNS. For another example, the standard anti-phishing advice is "Type the web address directly into your browser." In a short but potent article entitled DNS Complexity, Paul Vixie (whose name is on several DNS-related RFCs) gives a picture, as of April 2007, of just what a leap of faith this is. A couple of quotes:

[I]t is computationally trivial to pollute a caching name server with data that was never seen or published or approved by the editor of the administrative authority zone that it purports to have come from [...] DNSSEC (DNS Security Extensions) has been in production [sic] for 12 long years, without any production use to date, and we in the Internet standards community look forward to solving and re-solving this problem for many years to come. (But, I'm not bitter about it.) Meanwhile, the only reason that DNS isn't attacked more often is that nobody trusts its authenticity. (A catch-22, perhaps?)

Well, maybe no one who knows DNS intimately trusts its authenticity. Most people trust it without even knowing they're trusting it.

[T]he combination of things that were left unspecified in the protocol, things that were loosely specified in the protocol, and things that were unenforceably specified in the protocol - and implementations in the field that interpret the protocol specifications in all of their myriad ways - describes a rich and multidimensioned space where it's almost deliberately impossible to know exactly what's happening or exactly what would happen under describable circumstances. This holds true for the Internet's routing system as well, but in DNS there are more variables in every axis than in any other distributed system I've studied. We who work in the field think of DNS a little bit as though it were alive, and I hope that after reading this article, you can see why.

Indeed. Everyone together now: "It's aliiiive!"

How do you know I wrote this?

[This one came out a bit disjointed.  Re-reading, I'm not really sure why I mentioned TLS for SMTP servers as having much to do with digital signatures as a SPAM-fighting device.  I also seem to imply that DNS spoofing could point you at, say, a fake banking site without your browser knowing.  The certificate checking in TLS would catch that.  DNS spoofing could perhaps be used in more subtle ways to get your browser to accept a bogus update to its trusted certificates list, but that's a much smaller attack surface.  Anyway, the general drift, such as it is, is still good, as is the conclusion --DH 7 Sep 2010]

As I go along, one of the things I'm trying to figure out is why doesn't everyone have a cryptographically strong keyring? I mean, here's this really well-established, well-studied and as far as our best minds can figure out highly secure way of doing several useful things. In particular, it can establish to a practical certainty that whoever signed a particular piece of data knew a particular secret key.

But no one uses it.

OK, people do use it. The major example, not surprisingly, is that e-commerce sites generally use a certificate in their HTTPS handshake. Your browser will even tell you this if you know where to look (in Firefox you use the little lock icon next to the address at the top which, if you're like me, you forgot all about).

So as long as you remember to check for that, and no one's fooling around with DNS behind your back (see below) and your browser hasn't been compromised (see below), you're good. Note that this is meant to establish your trust in the server. The server will trust you on the basis of a small password (and your incentive not to give it away).

Outside that crucial and curiously asymmetric case, examples get a bit sparse.

At least one major online brokerage (and so probably all of them) will set you up with a digitally secure ID. Whatever overhead or unfamiliarity digital keys may have is worth it if there is enough money at stake.

I've mentioned that widespread use of signatures would probably take a pretty big bite out of spam (um, maybe that's not the best metaphor). I don't see any signs of traction on this, outside of some SMTP servers using certificate-based TLS (quick -- which ones and how do you tell?).

For a while I used to sign all my posts to a mailing list I was on. Then I switched to a different setup and never got around to turning signing back on. No one cared.

Even in an environment like a standards committee, where the members are speaking very publicly and representing companies with money at stake, the posts are generally not signed (at least not in the committees I've had direct experience with). You're trusting the committee members to keep their small passwords safe and the world at large not to care much about standards committees. Which, admittedly, is a pretty safe bet.

OS vendors use strong crypto to make sure that updates are from where they say they are, though I do recall once or twice seeing instructions to the effect that "You may see a message complaining about a bad or missing certificate. Ignore it." OK .... I'm getting a bit of a mixed message here, but OK ....

I couldn't remember whether my Firefox plug-ins were signed, and I couldn't find any indication of it for the ones I had installed, so I went to the Firefox extensions site and checked a couple more-or-less at random. These were in the web development section, so they did things like muck with the HTTP headers you send or munge the content coming back. Stuff you might want to be particularly sure was kosher before installing (or not -- the whole point of Trojan-horse-like attacks is that they can be disguised as anything).

Unsigned, of course. Even if they are signed, do I understand the signing protocol in question well enough to trust it? Not really. I take it on faith that the folks at Mozilla have thought it all through. I particularly hope they've thought the HTTPS stuff through. I'm sure I'd hear about it if they hadn't.

Not that most people will care, but Eclipse plug-in security is even more of a formality. I'm not sure I've ever seen a properly signed third-party plug-in. I probably have, but I don't remember where or when. Security of development tools is not of entirely academic interest. Ken Thompson had a bit to say about that.

What the heck is going on here? If you've got a reasonably mature technology and a problem it seems to fit well, yet no one seems to have adopted it, then either

• The problem isn't the problem it looked to have been
• There's more adoption than one might think
• Something else is solving the problem well enough to prevent major outrage
  

In this case I think it's a combination of the last two. In e-commerce, there actually is strong crypto involved, just woven in seamlessly enough you only see it if you're looking for it.

That's good, but SSL/TLS authenticate a server, not a person. To take a classic example, if I'm dealing with someone on eBay, their certificate means I can be pretty confident I'm talking to eBay. I can also be pretty confident that eBay at least made this person type in a password in order to sign on. I mean, I had to when I logged in. Beyond that I need eBay's feedback mechanism to help decide whether I want to do business with them.

Returning to the title, how do you know I wrote this? It probably goes like this:

• You believe that fieldnotesontheweb.blogspot.com is controlled by Blogger. We put remarkable trust in DNS, and despite attempts to poison it (or spoof by playing nasty games with Unicode), that trust seems to be repaid. So far.
• You believe that no one has stolen my blogger ID (or if they did, I would be able to get the account blocked until everything got straightened out).
• If you've read more than one post, this one (I hope) seems like it was written by the same person as the others.

Other situations behave similarly. I trust DNS to make sure my bank is my bank and that the servers involved in authenticating my HTTPS connection with them are who they say they are. We trust our OSs and browsers to do the right thing with updates. We exercise caution in dealing with phishy-looking emails.

We trust that email that says it's from someone we know really is, and we're generally right (the exception being mail from oneself -- either I sleepwalk or the occasional spammer is spoofing the From: line to be the same as the To: line). That makes a simple whitelist a reasonably good spam filter.

In short, we have a typical case of engineering in the real world. The status quo is a patchwork of partial solutions yielding results that are significantly non-optimal, but not quite bad enough to leave room for sweeping reforms. That's probably not going to change until the bar for going secure is very, very low, and/or people decide that having strong crypto is worthwhile even if no one else uses it.