A gigabit behind your eyes

OK, I thought this was going to be an easy one. At a lecture the other day I heard Edward Tufte (author, among other things, of The Visual Display of Quantitative Information and quite a bit of scathing invective aimed in the general direction of Power Point) claim that the human optic nerve has a capacity of around 20 megapixels per second. "And we have two of them!" he continued, pushing one of his major themes: people can easily and naturally process much more information visually than most graphics contain.

Leaving aside the questionable implication that two optic nerves allow us to process much more information than one -- unlikely both because the two eyes generally see almost the same image and because if they don't the result is generally less informative than the normal case -- I was happy to hear some hard numbers, apparently based on a careful, peer-reviewed study, regarding human visual bandwidth.

So all I needed to do was track down Tufte's assertion on the web, follow that to the original study and write it up: Our optic nerves can handle approximately X, so a display purporting to handle more than X may not be that useful (and maybe that's why Blu-Ray doesn't seem to be taking the world by storm). Granted, this is just a crude measure of bandwidth and leaves aside many, many details of human visual perception, but it's still a useful number for sanity checking and ballpark estimates.

Alas, I'm stuck at step 1. I'm only mostly sure the number was 20 and the units were megapixels per second, and I'm assuming that a pixel is more or less three bytes, based on fairly well-known results in color perception. So instead, here are some facts and factoids that turned up:

• The human eye has about 100 million receptors. This is sometimes quoted as "the eye has 100 megapixels," but trying to compare rods and cones to camera pixels is really apples and oranges.
  
• Unlike the uniform grid of digital cameras and video displays, the eye instead has about 100 million light/dark-sensitive rods and 5 million color-sensitive cones. The cones are clustered around the focus point of the lens. Peripheral vision is much less color-sensitive.
• Most people can't really tell the difference between a 6"x4" photograph printed at 150 dpi and one printed at 300 dpi when both are viewed at normal distance. 6"x4"x(150dpi)² is about half a megapixel. Half a megapixel times 20 frames per second is about 10 megapixels per second; that's low of Tufte's figure, but then a 6"x4" photo at normal distance doesn't completely fill the field of vision -- just the most acute portion.
  
• The optic nerve contains about 1.2 million fibers. That's a bit more than one for every hundred receptors, so either some aggregation is done on the retina, or the neurons are able to multiplex information from multiple receptors, or both.
• 1.2 million fibers times 20 frames per second is close to Tufte's 20 million per second.

All this suggests that, to a rough approximation, we can process still images of about a megapixel and moving images with around 20 megapixels per second of useful information. 20 megapixels per second at three bytes per pixel is 60MB/s or about 500Mb/s, so we have something close to a gigabit network right behind our eyes. This sort of thing is one reason I tend to put "broadband" in quotes.

If we can only process a megapixel or so, why have a bigger display than that? Good typographic resolution is more like 1200dpi. On an 8 1/2" x 11" page that's over 100 megapixels. Isn't that overkill?

Not really. You don't look at the entire page at once. You scan it, focusing on on piece, then the next. Each of those pieces needs to be sharp. A large, finely-printed page will give you about a hundred high-resolution patches to focus on. Similarly, you can't take in all of an IMAX image at once. Rather, you have a huge image that looks sharp no matter where you look at it.

A sharp display with only a megapixel of resolution would have to cover the entire field of view, and it would have to track eye movements so that which megapixel you got depended on where you were looking. Maybe some sort of really high-tech contact lens?

Another data point on immersion

A while ago I estimated the bandwidth required for 3-D Imax at 13GB/s, uncompressed. Today I got the latest newsletter from the California Academy of Sciences bragging about the new and improved Morrison Planetarium. They say this bad boy can blast 300 million pixels per second onto its hemispherical screen. At 32 bits per pixel, that's about 1.2GB/s, or about a tenth of my IMAX estimate (I think I used 32 bits per pixel to ensure a conservative estimate of the bandwidth required. 24 ought to be good enough).

Take out a factor of two since 3-D requires two images and assume that the frame rate is 24 frames/s in both cases. The remaining difference is down to spatial resolution. IMAX is about 10K by 7k, or 70 megapixels, so the Morrison is more like 14 megapixels. In the earlier article I guessed that the 13GB/s could probably be compressed down to 1GB/s, partly because the two 3-D images would be largely redundant. Planetarium-level 2-D video would also compress, but not quite as much. Bottom line, you're still looking at gigabits per second to be able to handle this stuff.

The Academy also claims that the panetarium would hold 2,402,494,700 M&Ms, but I'm skeptical about either the volume of an M&M or the planetarium, much less both, being known to a part in 10 million.

IMAX 3D vs. broadband

Watching a concert film on IMAX 3D is an interesting experience. You get a much better view, for a stadium show, than anyone actually in the house -- sort of like you're somewhere close to the stage and really tall and able to float up level with or above the performers at random intervals. The sound is great, and perfectly synchronized, unlike sitting up in the nosebleeds with a slight-but-noticeable lag between the big screen and the speakers. You can hear the virtual crowd roar around you, even if the actual crowd around you is quiet.

The screen is big enough and close enough that you can't quite take it all in at once. If something darts forward off to one side, you have to turn (slightly) to see it in focus. You don't have to turn as far as you would in real life, another slight but unavoidable disconnect between being fully immersed and sitting in a theater with a huge screen and great speakers.

Note to directors: The 3-D cameras love the drum kit -- all those cylinders poised at various angles -- but don't overdo it. There's also another slight disconnect here. The drumsticks strobe noticeably since even IMAX is still 24 frames per second.

Now for the interesting question: How many bits?

IMAX film has a resolution of approximately 10,000 by 7,000. Assuming 32-bit color, 24 frames per second and 2 cameras, that comes out to about 13 gigabytes per second, uncompressed. There's ample room for compression, particularly in 3D since the two images are largely identical, but you're still talking on the order of a gigabyte per second. Picture throwing two blu-ray DVDs into the maw of the beast every minute and you're in the ballpark.

Leaving aside the small matter of installing an IMAX home theater, could you at least stream the bits into your house? If you happen to have 10-gigabit ethernet or better coming in, you're good to go. 75-year-old Sigbritt Löthberg of Karlstad, Sweden does, thanks to her son, Peter (see here for slightly more details). I don't, and you probably don't, either.

On the other hand, gigabytes are getting cheaper every day. Last I looked, hard drives were running around $0.30/GB. A 90-minute movie would require about 5TB of disk (5400GB at 1GB/second), over $1000 retail. That's probably viable for theaters now -- IMAX film reels are massive -- but not quite ready for home use.