tag:blogger.com,1999:blog-21299291829185998482024-02-26T12:34:39.916-05:00Field notes on the WebFiguring out the web as I go along ...David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.comBlogger695125tag:blogger.com,1999:blog-2129929182918599848.post-6667087830103538702024-02-05T11:30:00.002-05:002024-02-05T11:30:22.205-05:00Do what I say, not what I mean<p>While watching a miniseries on ancient history, I got to wondering how quickly people could move around in those days. The scriptwriters mostly glossed over this, except when it was important to the overall picture, which seems fine, but it still seemed odd to see someone back in their capital city discussing a battle they'd taken part in a thousand kilometers away as though it had happened yesterday.</p><p>So I did a search for "How far can a horse travel in a day?". The answer was on the order of 40 kilometers for most horses, and closer to 150 for specially-bred endurance horses. That would make it about a week to cover 1000km, assuming conditions were good, except that a horse, even a specially-bred one, needs to rest.</p><p>What if you could set up a relay and change horses, say, every hour? At this point we're well off into speculation, and it's probably best to go to historical sources and see how long it actually took, or just keep in mind that it probably took a small number of weeks to cross that kind of distance and leave it at that. But speculation is fun, so I searched for "How far can a horse travel in an hour?"</p><p>It may not surprise you that I didn't get the answer I was looking for, at least not without digging, but I did get answers to a different question: What is the top speed of a horse in km/hr? (full disclosure, I actually got an answer in miles per hour, because US, but I try to write for a broader audience here). How fast a person or animal can sprint is not the same as how far can the same person or animal go in an hour.</p><p>This seems to be the pattern now that we have LLMs involved in web search. I don't know what the actual algorithms are (and couldn't tell you if I did), but it seems very much like:</p><p></p><ul style="text-align: left;"><li>Look at the query and create a model of what the user really wants, based on a Large Language Model (LLM)</li><li>Do text-based searches based on that model</li><li>Aggregate the results according to the model</li></ul><div>It's not hard to see how an approach like this would (in some sense) infer that I'm asking "How many kilometers per hour can a horse run?", which is very similar in form to the original question, even though it's not the <i>same</i> question at all. There are probably lots of examples in the training data of asking how fast something can go in some unit per hour and not very many of asking how <i>far</i> something can go in an hour. My guess is that this goes on at both ends: the search is influenced by an LLM-driven estimate of what you're likely to be asking, and the results are prioritized by the same model's estimate of what kind of answers you want.</div><div><br /></div><div>It's reasonable that questions like "How fast can a horse go?" or even "How fast is a horse?" would be treated the same as "How many km/hr can a horse run?". That's good to the extent that it makes the system more flexible and easier to communicate with in natural language. The problem is that the model doesn't seem good enough to realize that "How far can a horse travel in an hour?" is a distinct question and not just another way to phrase the more common question of a horse's top speed at a sprint.</div><div><br /></div><div>I wish I could say that this was a one-off occurrence, but it doesn't seem to be. Search-with-LLM's estimate of what you're asking for is driven by the LLM, which doesn't really understand anything, because it's an LLM. It's just going off of what-tends-to-be-associated-with-what. LLMs are great at recognizing overall patterns, but not so good at fine distinctions. On the question side, "How far in an hour?" associates well with "How fast?" and on the answer side, "in an hour" associates strongly with "per hour," and there you go.</div><div><br /></div><div>That's great if you're looking for a likely answer to a likely question, but it's actively in the way if you're asking a much-less-likely question that happens to closely resemble a likely question, which is something I seem to be doing a lot of lately. This doesn't just apply to one company's particular search engine. I've seen the same failure to catch subtle but important distinctions with AI-enhanced interfaces across the board.</div><div><br /></div><div>Before all this happened, I had pretty good luck fine-tuning queries to pick up the distinctions I was trying to make. This doesn't seem to work as well in a world where the AI will notice that your new carefully-reworded query looks a lot like your previous not-so-carefully-worded query, or maybe more accurately, it maps to something in the same neighborhood as whatever the original query mapped to, despite your careful changes.</div><div><br /></div><div>Again, I'm probably wrong on the details of how things actually work, but there's no mystery about what the underlying technology is: a machine learning (ML) model based on networks with backpropagation. This variety of ML is good at finding patterns and similarities, in a particular mathematical sense, which is why there are plenty of specialized models finding useful results in areas like chemistry, medicine and astronomy by picking out patterns that humans miss.</div><div><br /></div><div>But these MLs aren't even trying to form an explicit model of what any of it means, and the results I'm seeing from dealing with LLM-enhanced systems are consistent with that. There's a deeper philosophical question of to what extent "understanding" is purely formal, that is, can be obtained by looking only at how formal objects like segments of text relate to each other, but for my money the empirical answer is "not to any significant extent, at least not with this kind of processing".</div><div><br /></div><div><br /></div><div>Back in the olden days, "Do What I Mean", DWIM for short, was shorthand for any ability for a system to catch minor errors like spelling mistakes and infer what you were actually trying to do. For example, the UNIX/GNU/Linux family of command-line tools includes a command <span style="font-family: courier;">ls</span> (list files) and a command <span style="font-family: courier;">less</span> (show text a page at a time, with a couple of other conveniences). If you type <span style="font-family: courier;">les</span>, you'll get an error, because that's not a command, and nothing will ask you, or try to figure out from context, if you meant <span style="font-family: courier;">ls</span> or <span style="font-family: courier;">less</span>.</div><div><br /></div><div>A DWIM capability would help you figure that out. In practice, this generally ended up as error messages with a "Did you mean ...?" based on what valid possibilities were close in spelling to what you typed. These are still around, of course, because they're useful enough to keep around, crude though they are.</div><div><br /></div><div>There are now coding aids that will suggest corrections to compiler errors and offer to add pieces of code based on context. In my experience, these are a mixed bag. They work great in some contexts, but they are also good at suggesting plausible-but-wrong code, sometimes so plausible that you don't realize it's wrong until after you've tried it in a larger context, at which point you get to go back and undo it.</div><div><br /></div><div>There's always been a tension between the literal way that computers operate and the much less literal way human brains think. For a computer, each instruction means exactly the same thing each time it executes and each bit pattern in memory stays exactly the same until it's explicitly changed (rare random failures due to cosmic rays and such can and do happen, but that doesn't really affect the argument here). This carries over into the way things like computer languages are defined. A <span style="font-family: courier;">while</span> loop always executes the code in its body as long as its condition is true, <span style="font-family: courier;">ls</span> always means "list files" and so forth.</div><div><br /></div><div>Human brains deal in similarities and approximations. The current generation of ML represents a major advance in enabling computers to deal in similarities and approximations as well. We're currently in the early stages of figuring out what that's good for. One early result, I think, is that sometimes it's best just to talk to a computer like a computer.</div><p></p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com1tag:blogger.com,1999:blog-2129929182918599848.post-21538955927430943102024-02-03T15:07:00.000-05:002024-02-03T15:07:02.216-05:00What's in a headline? Find out here<p>Goodness, it looks like 2023 was an all-time low for this blog, with one (1) post. Not sure how that happened. I honestly thought I'd posted at least one more. On the other hand, I suppose it's consistent with the overall handwringing about whether there's even anything to post here. But this post won't be that.</p><p>When I was in journalism class in high school, which was more than a few years ago to be sure, I was taught the "inverted pyramid": put the most important information, the who, what, where, when, why and how at the top of the article, then the important detail, then other background information. The headline should concisely sum up the most important facts at the top.</p><p>Some typical headlines might be</p><p></p><ul style="text-align: left;"><li><i>Pat's Diner closing after 30 years</i></li><li><i>New ordinance bans parking on Thursdays</i></li><li><i>Midtown high senior wins Journalism award</i></li></ul><p></p><p>If you've noticed that the titles (that is, headlines) of posts here don't exactly follow that rule, that's because I'm writing opinion here, not news. That's my story, and I'm sticking with it even as I go on to complain about other people's headlines.</p><p>One of the worst sins in old-school journalism was to "bury the lede", that is, to put the most important facts late in the story (<i>lead</i> as in <i>lead paragraph</i> is spelled <i>lede, </i>probably going back to the days of lead type where the usual spelling might invite confusion). If Pat's diner is closing, you don't start with a headline of <i>Local diner closing</i> and a paragraph about how much people love their local diners and only later mention that it's Pat's diner that's closing.</p><p>Except, of course, that's exactly what happens a lot of the time. Here are some examples from the articles currently on my phone:</p><p></p><ul style="text-align: left;"><li><i>Windows 11 looks to be getting a key Linux tool added in the future</i></li><li><i>Nearly 1 in 5 eligible taxpayers don't claim this 'valuable credit', IRS says</i></li><li><i>46-year old early retiree who had $X in passive income heads back to work -- here's why</i></li></ul><div>I've tried to get out of the habit of clicking on articles like these, not because I think it will change the world (though if everybody did the same ...), but because I almost always find it irritating to click through on something to find out that they could have just put the important part in the headline:</div><div><ul style="text-align: left;"><li><i>Linux sudo command may be added to Windows 11</i></li><li><i>Nearly 1 in 5 eligible taxpayers don't claim earned income credit, IRS says</i></li><li><i>Early retiree with $X in passive income back to work after house purchase and child</i></li></ul><div>One of these rewrites is noticeably shorter than the original and the other two are about the same length, but they all include important information that the original leaves out: <i>which Linux tool?; which tax credit?; why go back to work?</i></div></div><div><i><br /></i></div><div>The lack of information in the originals isn't an oversight, of course. The information is missing so you'll click through on the article and read the accompanying ads. The headlines aren't pure clickbait, but they do live in a sort of twilight zone between clickbait and real headline. If you do get to the end of the article, you'll probably see several more links worth of pure clickbait, which is an art form in itself.</div><div><br /></div><div>Real headlines aren't dead, though. Actual news outlets that use a subscription model tend to have traditional headlines above traditional inverted-pyramid articles. They probably do this for the same reason that newspapers did: Subscribers appreciate being able to skim the headline and maybe the lede and then read the rest of the article if they're interested, and that sells subscriptions.</div><div><br /></div><div>I'm pretty sure half-clickbait headlines aren't even new. The newspaper "feature story" has been around considerably longer than the web. Its whole purpose is to draw the reader in for longer and tempt them to browse around -- and either subscribe for the features or spend more time on the same page as ads, or both. For that matter, I'm pretty sure a brief survey of tabloid publications in the last couple of centuries would confirm that lede-burying clickbait isn't exactly new.</div><div><br /></div><div>I started out writing this with the idea that the ad-driven model of most web-based media has driven out old-fashioned informative journalism, and also those kids need to <i>get off my lawn</i>, but I think I'm now back to my not-so-disruptive technology take: Clickbait and semi-clickbait aren't new, and the inverted pyramid with an informative headline isn't dead. In fact, when I checked, most of the articles in my feed did have informative headlines.</div><div><br /></div><div>In part, that's probably because I've stopped clicking on semi-clickbait so much, which is probably changing the mix in my feed. But it's probably also because the web hasn't changed things as much as we might like to think. All three kinds of headline/article (informative, semi-clickbait, pure clickbait) are older than the web, and so are both the subscription and ad-based business models (though subscription print publications often had ads as well). It's not too surprising that all of these would carry through.</div><div><br /></div><p></p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-44880082284040343852023-01-09T18:28:00.004-05:002023-06-17T20:10:03.322-04:00On web standards and social networking<p>I started this blog, um, a few minutes ago, back when I was involved with a couple of the committees involved in developing standards for the web, and it seemed like one thing a person in my position was supposed to do was to blog about it, whether to try to make one's expertise generally available, or to promote one's brand or consulting services, or to become visible as part of a group of similar individuals (the term "blogroll" comes to mind), or to help set the future direction of the web by presenting brilliant analyses or ... something of that nature.</p><p>Fortunately, I soon separated the blog from my professional life and settled into just putting out whatever thoughts I had as I wandered the web.world, without a lot of concern for what, if anything, it might all lead to. In the event, it hasn't led to all that much ... a few dozen followers, the occasional comment, and readership well below even thinking about trying to monetize anything ... but it has been a satisfying experience nonetheless. I've learned all sorts of things, and writing about it has given me the opportunity to think things through, organize my thoughts and maybe even get better at writing. The whole standards-committee thing seems long ago and far away.</p><p>Imagine my surprise, then, when <a href="https://arstechnica.com/gadgets/2023/01/mastodon-highlights-pros-and-cons-of-moving-beyond-big-tech-gatekeepers/">an <i>Ars Technica</i> article on Mastodon</a> popped up in my newsfeed and brought it all back.</p><p>When we <a href="https://blog.fieldnotesontheweb.com/2022/11/is-it-end-of-web-as-we-know-it.html">last left our story</a>, I was musing about how recent turbulence concerning cryptocurrencies and social media, and Twitter in particular, had gotten me thinking about what "web" even meant anymore and concluding that at least the core of the web, namely its interconnections, was alive and well, and also that it didn't have all that much to do with the turbulence. Before that, I had <a href="https://blog.fieldnotesontheweb.com/2021/10/why-so-quiet.html">said</a> that there didn't really seem to be all that much to blog about. Things were happening, but not a lot of things that I had much to say about.</p><p>But Mastodon is interesting, not just because of its role as a potential "giant killer" for Twitter -- I'm still not inclined to speculate on how that might shake out -- but because of <i>what</i> it is and how it was put together, both in its structure and in the process that led to it. Allow me to take a walk down memory lane:</p><p>When I was involved in the standards process, the typical web standard went through a life cycle something like this:</p><p></p><ul style="text-align: left;"><li>Someone, typically one of the Major Players, would create some new protocol or class of application to try to take advantage of the opportunities to be had on the rapidly expanding web. This was after the major protocols like TCP, FTP, HTTP and the various email protocols were widespread. Those tended to be less blatantly commercial, even though major corporations were generally involved.</li><li>If the idea looked useful, other players, large and small, would try to work with it, using whatever documentation was available.</li><li>This would often lead to a mess. It's hard to write clear, complete documentation and it's almost guaranteed that once other people start using a system they'll try things that the original authors hadn't thought of. When it's not clear what should happen in a particular situation, people will take their best guess, and different people will guess differently.</li><li>People will also want to add new features to cover cases that the original authors hadn't thought of, or did think of but didn't have time to implement, or implemented in what seemed like a less-than-optimal way, or whatever.</li></ul><div>In some ways, this is a good problem to have, since it means people are actually using the system, probably because it meets some need that hadn't been addressed before. The flip side, though, is that in a networked world a system only works, or at least only works well, if people agree on how to use it. If my version of AwesomeNet thinks that the command for "make everything awesome" is <span style="font-family: courier; font-size: x-small;">MKAWSM</span> and yours thinks it's <span style="font-family: courier; font-size: x-small;">make-awesome</span>, then chances are things won't be so awesome.</div><div><br /></div><div>There are plenty of other minor disagreements that can happen, either because different people made an arbitrary choice differently, or because there's a genuine disagreement as to the best way to do something, or for any number of other reasons. If this happens enough to be a real problem, there's usually a call for everyone to get together and create an agreement on which way to do things. A standard, in other words.</div><div><br /></div><div>Writing standards is a tricky thing. You want to capture what people are currently doing, but in an abstract enough way to avoid just having a laundry list of things people currently do. You want to establish boundaries that are tight enough to rein in the chaos, but not so tight to prevent people from finding new and better ways of doing things. There are often several ways to do this. For example, for the difference in command names you might</div><div><ul style="text-align: left;"><li>Just pick one, say <span style="font-family: courier; font-size: x-small;">make-awesome</span>, and "deprecate" the other, meaning that implementations of AwesomeNet aren't required to support it and if your code does use it, you should update your code.</li><li>Pick one as preferred, require implementations to understand the other one, but forbid them from using it. That means that a standard implementation of AwesomeNet will never use <span style="font-family: courier; font-size: small;">MKAWSM</span>, but if someone talking to it does, it will understand (this is an example of <a href="https://en.wikipedia.org/wiki/Robustness_principle">Postel's principle</a>).</li><li>Allow both, usually by defining one as an "alias" for the other, so that most of the standard text can just talk about <span style="font-family: courier;"><span style="font-size: x-small;">make-awesome</span></span>, except for the part that says "you can use <span style="font-family: courier; font-size: x-small;">MKAWSM</span> anywhere you can use <span style="font-family: courier; font-size: x-small;">make-awesome</span>"</li><li><span style="font-family: inherit;">Define an "abstract syntax" and talk about <i><make awesome></i> as a placeholder for "whatever name the implementation chooses to use for the <i>make everything awesome</i> command". This means that there will need to be some sort of protocol for two implementations to tell each other which names they actually use. That's probably not worth the trouble in most cases, but it's an option nonetheless.</span></li><li><span style="font-family: inherit;">If "make everything awesome" is just a convenient shorthand for a bunch of other commands, define a general way of defining shorthands like that (often called "macros") and leave it out of the "core standard" entirely. There are lots of ways to implement macros, since a macro language is essentially a programming language (even if it's not functionally complete), so this is probably only a good idea if at least some of the existing implementations already support it.</span></li><li>There are probably several other approaches I'm not thinking of.</li></ul><div>There are similar issues with features that only some of the existing implementations support. Do you require everyone to support everyone's features? Do you define a "core set" and make the rest optional? Do you say that some existing features <i>aren't</i> allowed in the standard, so implementations that have them have to drop them? Or maybe something else?</div><div><br /></div><div>Another important question is "what happens in this particular special case?". Sometimes the answer is "it's this particular behavior that you might not have thought of, and here's why". Sometimes the answer is "that's an error, and the implementation should report it like this". Sometimes, though, the answer is "the behavior is unspecified", either to leave room for new features or for the practical reason that different existing implementations do different things, or for whatever other reason. This is more useful than it might seem. It tells implementers that it's OK to do whatever's easiest, and not to expect any particular behavior in certain situations (and maybe try to avoid those situations altogether).</div></div><div><br /></div><div>There are lots of other principles and techniques that experienced standards writers employ (learning some of these as a not-so-experienced standards committee member was one of the best parts of the experience). One that sticks in mind is the principle that implementations should be able to safely ignore things they don't understand, so that it's safe to interact with something that implements a newer version of the standard.</div><div><br /></div><div>In a typical standards process, there are dozens of questions to hammer out, and the answers are often interconnected. It's a tricky technical problem and, of course, there are political considerations as well. In my experience the politics are typically pretty civil, but most participants come in with certain "red lines" they won't cross, usually because it would cost more to implement them than the standard is worth to them since there are already customers out there using the not-yet-standardized products. The usual approach is to stand firm on those and be prepared to be flexible on anything that doesn't cross a red line. If two or more participant's red lines overlap, things can get tricky.</div><div><br /></div><div>After all the meetings, if all goes well, the end result of this is a carefully crafted standards document full of MUST [NOT] and SHOULD [NOT], which are so important that there's actually <a href="https://www.ietf.org/rfc/rfc2119.txt">a standard defining what they mean</a>.</div><div><br /></div><div>The new standard captures everything the committee could capture about what it means to implement and use whatever's being standardized. With luck, the standard is at least self-consistent, but there will be parts that are deliberately left unspecified for reasons like those given above. There will also be plain old mistakes, ideally small ones that can be corrected by errata (corrections, ideally short and specific, published in a separate document) but sometimes larger ones that will require more meetings and a new version of the standard.</div><div><br /></div><div>Since standards need to be flexible, a standard doesn't always answer every question you'll need answered when coding to it. Because of this, it's common for a separate "interoperation committee" to form once a standard is out. These committees essentially fill in the blanks to create "profiles" that say how to use a particular standard in particular situations. There might be a lightweight profile that's meant to provide only the core features so it's easier to implement and run. There might be a security-oriented profile that specifies features to enable and parameters to use ("digital signatures are required for each message and private keys must be at least 2048 bits"). There might be a profile aimed particularly at business use, or whatever.</div><div><br /></div><div>I've gone through all this in detail because, judging by the <a href="https://arstechnica.com/gadgets/2023/01/mastodon-highlights-pros-and-cons-of-moving-beyond-big-tech-gatekeepers/"><i>Ars</i> article</a>, all of these things either have happened with Mastodon, or may happen at some point because the same basic issues have come up. More than that, Mastodon and similar services are, in fact, based on the <a href="https://www.w3.org/TR/activitypub/">ActivityPub standard</a>, which specifies two protocols, one for clients to talk to instances to interact with an inbox and outbox, and one for instances to send traffic to each other.</div><div><br /></div><div>(An instance is a server in the sense of the client-server model, but not in the sense of a particular process or a server machine -- at least in theory, an instance could run on multiple physical servers and a single physical server could host multiple instances)</div><div><br /></div><div>ActivityPub is meant to be very general-purpose, but in the context of social networking only some of the features are really important, so implementations in the Fediverse (the world of Mastodon and Mastodon-like services that talk to each other) will naturally tend to pay more attention to those features, which is probably fine until someone tries to take advantage of other features. The standard for data encryption and signatures is not officially part of the system, so implementations that support it still have to deal with unencrypted data. ActivityPub relies on a standard called WebFinger to discover what instances are out there, so implementing a real ActivityPub instance also means dealing with WebFinger.</div><div><br /></div><div>And so on.</div><div><br /></div><div>All of this is perfectly normal in the world of web standards. A standard is meant to reduce uncertainty and make it clear where the remaining uncertainty is, and establish common vocabulary and practices. It isn't meant to solve all problems with interoperation -- it's great if it can, but that's not always possible. It isn't meant to answer all questions an implementer might have, and in fact it deliberately <i>doesn't</i> answer some questions so that implementers will be free to come up with better answers. In general a standard is more about <i>what</i> than <i>how</i> (though a <i>what</i> in one context can be part of the <i>how</i> in another and vice versa).</div><div><br /></div><div>In any case, ActivityPub only talks about how instances talk to each other and how clients interact with instances. It doesn't, and shouldn't, talk about how people using it for social networking should interact. That's a matter of community guidelines, that is, what the people running the actual instances will or won't allow on them. Since the instances are deliberately federated, rather than owned by one entity as with Twitter, these decisions will be, too.</div><div><br /></div><div>One particularly interesting point is whether posts to one instance can be kept private to that instance. This proved contentions, so <i>Hometown</i> was forked off of Mastodon. Hometown allows posts to be private to a particular instance. Mastodon doesn't.</div><div><br /></div><div>The good news, I think, is that federated services built on web standards, while messy in real life, can and do work. Mastodon's success or failure depends more on how community guidelines shake out and whether people prefer to spend their time there than it does on the strength of the standards. I only spent a lot of time on standards in this post because it was striking to me how much of what I learned back in those days is still relevant.</div><div><br /></div><div><br /></div><div>One other thing struck me while pondering the article. This all seems a whole lot like Usenet, just with different underlying technology.</div><div><br /></div><div>Usenet used an email-based model where a particular user could subscribe to various newsgroups, receiving emails as updates came in, and publish messages on those groups, either new messages or replies to a thread that was already going. The main technical difference is that early usenet relied on a old protocol called UUCP (Unix-to-Unix Copy) to transfer files of messages point-to-point between sites (often but not always universities or corporations). UUCP isn't used much any more, but, like many old standards, it <i>is</i> still in use here and there.</div><div><br /></div><div>The main externally visible difference, besides the use of email, was that the newsgroups were owned by administrators (these might not be the server administrators, though server administrators could effectively veto a group by filtering it out). Rather than tag a post with <span style="font-family: courier; font-size: x-small;">#this</span> or <span style="font-family: courier; font-size: x-small;">#that</span>, you'd post to <span style="font-family: courier; font-size: x-small;">comp.sci.this</span> or <span style="font-family: courier; font-size: x-small;">alt.that.discuss</span> or whatever.</div><div><br /></div><div>From a social point of view, though, the similarities seem significant. Just as site administrators could establish rules, such as by blocking people from posting or filtering out newsgroups, Mastodon instance owners can establish their own standards. Just as different people would have different ideas as to what sort of discussion was acceptable and different sites and different newsgroups could have different standards, different instances have different rules of conduct.</div><div><br /></div><div>Newsgroups allowed people with similar interests to meet up and discuss things they were interested in. This included not only technical topics like scientific specialties or programming languages, but social topics like music and art and a fair bit of quite gamy content. They also allowed otherwise marginalized people to gather together, and for the curious to find out a bit about an unfamiliar group of people, but also for trolls and worse to barge in if they could get past the admins.</div><div><br /></div><div>In other words, from a social point of view, they had pretty much all the features, good and bad, of modern social networking communities. Whether you want to see that as a glass half full -- people keep finding ways to get together -- or half empty -- even after decades we're still up against the same problems -- or a bit of both -- my general inclination -- is up to you. If nothing else it might provide a counterweight to claims that any of what's going on now is unprecedented.</div><div><br /></div><div><br /></div><div>There's one other technical point that jumped out, though I doubt it will interest many people. Two patterns of communication come up over and over again in distributed services. One is the request/response pattern: I ask you a question or to do something, you respond with an answer or "I did it (and this happened)"/"I couldn't do that (for this reason)"/.... The other is the publish/subscribe pattern (pub/sub for short), where any number of parties can publish a message on a topic and any of those (including the publisher) can subscribe to messages on that topic. Those aren't the only possibilities, but they account for a lot of traffic. ActivityPub, as the name might suggest, is a pub/sub protocol.</div><div><br /></div><div>The whole point of pub/sub is that publishers and subscribers don't know about each other, or whether any of them even exist. I can publish to a topic that no one's listening to, and, depending on the type of service, either that message will be dropped on the floor, or it will be saved for delivery to anyone who does eventually subscribe. In either case, each subscriber will see its own copy of the message. The only difference is whether "each subscriber" means "each subscriber that was there when the message was published (more or less)" or "each subscriber that ever subscribes to this topic".</div><div><br /></div><div>Since publishers and subscribers don't know about each other, there has to be some sort of intermediary. For a large system, there will actually be many intermediaries communicating with each other in order to make sure published messages are delivered to subscribers. Publishers and subscribers talk point-to-point with those intermediaries. I tend to think of this as "I publish to the cloud, whatever's in the cloud does whatever it does, and the subscriber gets the message from the cloud".</div><div><br /></div><div>Mastodon and company fit this perfectly: You send a message to the instance you're using, it talks to other instances, and people see your message.<br /><br />The old Usenet followed the exact same pattern, just that the mechanism for the servers to talk to each other was quite a bit different.</div><div><br /></div><p></p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-86040336216542677452022-12-28T16:19:00.009-05:002023-06-17T20:13:26.088-04:00Goblin Mode McGoblin Modeface<p> Each year, <a href="https://languages.oup.com/">Oxford Languages</a>, which produces the <a href="https://www.oed.com/">Oxford English Dictionary</a> among other things, selects a <a href="https://languages.oup.com/word-of-the-year/2022/">word of the year</a>, "a word or expression reflecting the ethos, mood, or preoccupations of the past twelve months, one that has potential as a term of lasting cultural significance." This year, the choice was opened up to online voting. Over 300,000 people cast their votes over the course of two weeks and the winner was <i>goblin mode</i></p><blockquote><p>a slang term, often used in the expressions ‘in goblin mode’ or ‘to go goblin mode’ – is ‘a type of behaviour which is unapologetically self-indulgent, lazy, slovenly, or greedy, typically in a way that rejects social norms or expectations.’</p></blockquote><p>The runners up were <i>metaverse</i> and <i>#IStandWith.</i></p><p>The press I've seen about this tends to emphasize the online voting aspect of the selection, with the suggestion that those rowdy internet folks got one over on the stodgy old OED, but I think that misses a couple of important points.</p><p>First, the OED as an institution isn't particularly stodgy. While <i>Oxford</i> might suggest the British power structure or <a href="https://www.google.com/search?q=bright+college+days+lyrics&oq=bright+college+days+lyrics&aqs=chrome..69i57j69i60.606j0j1&sourceid=chrome&ie=UTF-8">Tom Lehrer's</a> indelible image of "ivy-covered professors in ivy-covered halls" (leaving aside that that's more a US reference), the dictionary itself has historically been concerned with documenting how people actually use the English language, rather than trying to dictate what "proper English usage" might be. It is <i>descriptive</i> rather than <i>prescriptive</i>.</p><p>The dictionary styles itself "the definitive record of the English language". This is meant to include everything, including dialects from all over the world, terms of art for all kinds of trades and professions, archaic words from a thousand years ago and all manner of other English usage, including today's internet slang.</p><p>From the OED's point of view, <i>goblin mode</i> is a perfectly good term to research and define, as is anything else that people actually use. If a bunch of internet trolls had decided to vote for <i>glurglebyte</i> or some other made-up word, and the OED actually went with it, that would have been a different matter, but there are plenty of examples of people using <i>goblin mode</i> prior to the online vote. The word of the year page even gives a couple of examples from <i>The Grauniad</i> and <i>The Times.</i></p><p>One might argue that people weren't using <i>goblin mode</i> all that much, and some other term, whether <i>metaverse, #IStandWith</i> or something else, might have made a better word of the year, but the fact that hundreds of thousands of people voted for it suggests that, even if the votes were meant ironically, there's <i>something</i> there that led people to coalesce around that particular word. You could even argue that the online vote gives an otherwise ordinary bit of internet slang a much better chance of becoming "a term of lasting cultural significance".</p><p>The word of the year page goes further and argues that <i>goblin mode</i> is indeed a good word for a year in which people are finding their way out of a world of lockdowns and overflowing hospitals and questioning just which pre-pandemic norms are really worth keeping. Sure, the Oxford folks may just be trying to put a brave face on being pwned, but to me it seems more like they saw the results and weren't particularly bothered.</p><p>I think there's another important point to note here. While there have been plenty examples of internet-driven crowds doing bad things, or even horrible things, it's worth remembering that this particular crowd of net.denizens was operating from a completely different mindset: As with <a href="https://en.wikipedia.org/wiki/Boaty_McBoatface">Boaty McBoatface</a>, they did it <i>because it was fun</i>, for sheer hack value.</p><p>While it would be a mistake to ignore bad behavior, it would also be a mistake to over-focus on it. Like anywhere else, bad things can happen on the web without making the whole place a cesspit. There's lots of questionable content out there and a certain amount of outright lies and hate, but there's also a lot of good information and not a little outright goofiness. However much there are people out there trying to steer us toward conflict and anger, we still have choices about what we browse and what we post.</p><p>A few hundred thousand people upvoting a random bit of slang may be a drop in the bucket, but there are a lot more drops like it. That says something about who's out there and what they want, just as surely as the nastiness elsewhere does.</p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-78099351127627341622022-11-25T13:59:00.003-05:002024-02-03T14:07:29.623-05:00Is it the end of the Web as we know it?<p>Or maybe a better question is "What is this <i>Web</i> we speak of, anyway?" My default answer: dunno, I'm figuring it out as I go along.</p><p>I think the last time I mulled that second question over, in the context of "Web 2.0" (Remember Web 2.0? I think it was one of the exits on the Information Superhighway), my opinion was that the big division was between everything that came before and "the Web", or "Web 1.0" as I don't recall anyone calling it very much. In other words, that first time someone chased a link from one web page to another using a graphical browser was an epochal event, even if hardly anyone noticed at the time, and what's come after has been a steady stream of technical improvements and services founded on that base.</p><p>Two types of service in particular have been prominent over the last decade or so: social media and cryptocurrencies, and both seem to be in questionable shape at the moment. I've cast a somewhat skeptical eye on both over the years, but that hasn't stopped them from intersecting with the lives of billions of people.</p><p>Billions in the case of social media, at least. I don't actually know how many people own cryptocurrencies, directly or indirectly, but who among us hasn't seen an ad for one or another, or read about the latest crash/rugpull, not to mention the millions of people living in countries that have made cryptocurrencies a significant part of their monetary system, so I'd say billions there, too, depending on how you count.</p><p>But the past year has not been particularly kind to either. This is all over the news at the moment, but just for later reference, let me list a few items of note</p><p></p><ul style="text-align: left;"><li>Elon Musk's takeover of Twitter is off to a rocky start. My guess is that the new ownership will find some way to keep the servers running and reach some sort of new equilibrium, but with a sizable majority of the workforce either forcibly terminated or choosing "take the severance and get on with my life" over hardcore intensity, it's safe to say there will be a period of adjustment. Major advertisers seem to be sitting on the sidelines in the meantime and, thanks to the billions in debt that came with the leveraged buyout, the burn rate has increased from "we'll be out of cash on hand in a couple of years if nothing changes" to "we'll owe more in interest this year than we have in the bank"</li><li>Facebook seems to have wandered off into the Metaverse. This seems to me to be a classic case of optimistic extrapolation run amok. Virtual reality is interesting technology. It clearly at least has good potential for useful applications in areas like design and education. Getting from there to a world where people spend comparable amounts of time in the virtual world to what they currently spend on scrolling through their feeds seems like a stretch. Personally, I've tried out an Oculus, and there were definitely some cool things on offer, from <a href="https://techcrunch.com/2019/05/03/the-key-tribeca-immersive-storyscapes/">a deeply moving immersive art piece on refugees</a> to <a href="https://www.oculus.com/experiences/media/142884360499053/281415019514101/?intern_source=blog&intern_content=the-slow-mo-guys-vr-brings-scientific-showmanship-to-oculus-tv-on-quest">super slow-mo of a couple of guys making showers of sparks</a> that you can walk around in. But the age of those links should tell you how long ago that was.</li><li>No less than Ian Bogost, of <i><a href="https://blog.fieldnotesontheweb.com/2011/11/in-which-theorist-discovers-something.html">Cow Clicker</a></i> fame among many other things, has written an article entitled <i><a href="https://www.theatlantic.com/technology/archive/2022/11/twitter-facebook-social-media-decline/672074/">The age of social media is ending. It should never have begun.</a> </i>I'm incorrigibly skeptical about <a href="https://blog.fieldnotesontheweb.com/2007/09/information-age-not-dead-yet.html">proclamations of the End of an Age,</a> or the beginning of one for that matter, but Bogost makes some good points about the crucial distinction between social <i>networking</i> (good, and computers can be very helpful) and social <i>media</i> (the never-ending pursuit of clicks, shares, followers, content and so forth, not so good in Bogost's estimation).</li><li>Crypto exchange FTX has imploded, taking SBF (its colorful founder Sam Bankman-Fried) down with it, the latest of many crypto plays that turned out, shockingly, to have been built atop a house of cards.</li><li>Bitcoin, the grandaddy of them all, has fallen from its all-time high of close to $69,000 to, at this writing, around $16,000, down over 75%. Interestingly, the price of BTC had pretty closely tracked the price of the S&P 500, leveraged about 3:1, until the recent FTX fiasco sent it further down. What it didn't do was rise as reserve currencies hit a round of inflation, which as I dimly understand it was what was supposed to happen.</li><li>The whole advent of crypto exchanges has only emphasized the disconnect between cryptocurrency in theory -- decentralized, anonymous, free from government interference -- and practice -- centralized by exchanges and mining pools, generally tied to bank accounts in reserve currencies and subject to government regulation from several directions.</li></ul><div>Plenty of cold water to be thrown on social media and cryptocurrency enthusiasts, but does this mean the whole thing is coming to an end?</div><div><br /></div><div>Social media doesn't seem to be going away. There's even been a rush of activity on Twitter, speculating about the demise of Twitter and what to do next, and if you want to use that as a jumping-off point for a rant about modern culture eating itself, be my guest.</div><div><br /></div><div>Even if cryptocurrency is dead as an alternative to reserve currencies and more conventional payment systems -- I'm not saying it is or isn't, but <i>even if</i> -- I doubt it's going to stop trading anytime soon. My personal benchmark for "crypto is dead" would be something on the order of "I can personally mine and take ownership of 1 BTC using my phone at a nominal cost". We're quite a ways from that, but on the other hand there's still plenty of time left before the mining reward rounds down to zero sometime around the year 2140 at current rates.</div><div><br /></div><div>In short, there are certainly some major disruptions going on in some of the major features of the Web landscape, but, in answer to the question in the title, they seem more like the kind of shakeup or reining in of excess that seems to happen fairly regularly, rather than some sort of deathblow to the Web itself. Webvan, anyone?</div><div><br /></div><div><br /></div><div>But then, as I asked at the top of the post, what <i>is</i> this <i>Web</i> we speak of, anyway?</div><div><br /></div><div>Apart from the time constraints of a busy life, I've been a less apt to post here, and in fact started a<a href="https://intermittent-conjecture.blogspot.com/"> whole other blog</a> (which I also don't post on very frequently), because I had come to the conclusion that a lot of things I wanted to post about weren't really related to the Web. Even here, one of my <a href="https://blog.fieldnotesontheweb.com/2021/10/why-so-quiet.html">more recent posts</a> was me fretting about what even <i>is</i> the Web any more and why am I not writing about it?</div><div><br /></div><div>That post, though, mainly talked about what the Web means day to day. For better or worse, a lot of that has to do with social media, and I have no interest in devoting a large chunk of my time to what's going on in social media. Plenty of other people do want to do that and do a better job than I would. But what is it that makes the Web webby, and how does that relate to the Web as it impacts our lives?</div><div><br /></div><div>If you peel back all the layers, all the way back to that first link chased on that first graphical browser, the Web is about <i>links.</i> If you've ever meandered from one Wikipedia article to the next, following links in the page or the "see also", you've been using the Web at its webbiest. Likewise, I think, if you've browsed your favorite magazine and followed the links from one article to the next, within that publication or outside. The web of interconnections is what makes the Web.</div><div><br /></div><div>That primordial web is still around and likely isn't going anywhere, because this sort of browsing from one topic to the next is probably pretty tightly wired in to the way our brains work. What <i>has</i> happened is that a couple of layers have grown on top of it.</div><div><br /></div><div>One is <i>search</i>. You can find all sorts of interesting things by browsing, but often you just want to know where to find, say, a replacement battery for your cordless vacuum. Browsing would be a horrible way to go about that, but you don't have to. Just type some likely terms into your search bar and there you are. This is useful enough that companies can make quite a bit of money by running ads on a search platform, and I doubt this business model is going away, whatever the fortunes of the particular companies providing it.</div><div><br /></div><div>Social media constitutes a different layer on top of the web. As I've mentioned before, I'm not active on social media, but it seems to me that while you can certainly browse the links of your social network to find people that people you know know, and you can follow links from a post/tweet/story/whatever to more things that you might be interested in, the main innovation in social media is the <i>feed,</i> which brings content to you without your having to search for it or stumble onto it.</div><div><br /></div><div>This isn't limited to social media. I spend quite a bit of time reading my news feed, anti-social though that may be. In any case, I think there is a distinction to be made between information you actively seek out and information that some person you're following, or some algorithm, or some combination of the two, brings to you. I doubt that this is going anywhere either, but it looks like there is some rethinking going on about how to control the feed of incoming information, and, to some extent, how much attention to pay to it at all.</div><div><br /></div><div>Interestingly there was a lot of interest a while back in <i>social search</i>, where you could ask questions of the crowd and people would dig up answers, and people would get paid, and various companies would take various cuts, one way or another. I think that fell by the wayside because automated search does a better job in many cases, and when it doesn't, asking someone you know without anyone in the middle generally works fine, or at least no worse than trying to ask a pool of random people.</div><div><br /></div><div>Also interesting: Nothing in those last few paragraphs involves cryptocurrencies, even though I implied earlier that upheaval in that world might have something to do with "the end of the Web as we know it". I think that's because, even if <i>stories</i> about cryptocurrency have been all over the web, cryptocurrency itself doesn't have much to do with the Web, because it just isn't webby in that primordial sense. Following some sort of network of transactions, link to link, is not exactly played up as a major use case.</div><div><br /></div><div><br /></div><div>I've actually found working through this pretty encouraging. A few posts ago (that is, over a year ago), I was ruminating on whether there was anything webby left that I might want to talk about. Going back to first principles about what makes the Web the Web immediately revealed a view in which the very basis for the Web is alive and well, and aspects of it that are prominent now, like search and feeds, can at least be understood in relation to it.</div><div><br /></div><p></p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-36221625740962142822022-07-30T13:08:00.006-04:002022-07-30T20:49:32.450-04:00Dear screenwriters: Bits can be copied<p>There's a new thriller movie out on one of the major streaming services. I don't think it matters which movie or which service. If you're reading this years from now, that statement will still probably true, at least to the extent there are still streaming services. If you're pretty sure you know which 2022 movie this is referring to, but haven't seen it yet and want to, be warned. There are mild spoilers ahead.</p><p>As with many such films, the plot revolves around a <i>MacGuffin</i>, a term apparently coined by Angus MacPhail, which Alfred Hitchcock famously glossed as "the thing that the spies are after, but the audience doesn't care." In other words, it doesn't really matter what the MacGuffin actually is, only that the characters <i>do</i> care who gets it and so spend the whole film trying to make sure it ends up in the right place and doesn't fall into the wrong hands.</p><p>The plot device of a MacGuffin is much older than the term itself, of course. The Holy Grail of Arthurian legend is one, and the oldest recorded story known so far, <i>The Epic of Gilgamesh</i>, sends its protagonist to the Underworld in search of one.</p><p>Clearly there's something in the human brain that likes stories about finding a magic item and keeping it away from the baddies, and in that sense the MacGuffin in the big streaming service movie is a perfectly good MacGuffin. The protagonists and antagonists vie over it, it changes hands a few times, lots of things explode and eventually the MacGuffin is destroyed, ending its magic powers.</p><p>Except ...</p><p>The MacGuffin in this case is basically a gussied-up thumb drive containing information certain people do not want to become known. Our protagonist receives the item early in the film (with suitable explosions all around) and promptly sends it off to a trusted colleague for safekeeping and decipherment. Later we learn that the trusted colleague has, in fact, received the drive and cracked its encryption, revealing the damning information.</p><p>In real life, this is when you would make a backup copy. Or a few. Maybe hidden in the insignificant bits of JPEGs of cute kittens on fake cloud accounts with several different services. Maybe on some confederate's anonymous server somewhere on the dark web. Or at least on a couple more thumb drives. For bonus points, swap out contents of the original thumb drive for a clip of the <a href="https://en.wikipedia.org/wiki/Dancing_baby">Dancing Baby</a> or some similar slice of cheese.</p><p>(As I understand it, there are some encrypted devices that are tamper-resistant and designed not to be readable without some sort of key, so you can't easily copy the encrypted bits and try to crack the encryption offline, but here we're told that the encryption has already been cracked, so they have the plaintext and can copy it at will.)</p><p>The problem with that, of course, is that the drive would then cease to be a MacGuffin. Why send teams of mercenaries and a few truckloads of explosives after something that might, at best, be one copy of the damning information? The only real reason is that it makes for an entertaining way to spend an hour or two and screenwriters know all about writing MacGuffin-driven thriller plots.</p><p>Which is fine, except ...</p><p>If you think about the practicalities, there's still plenty of tension to be had even if the bits are copied. Our protagonist has reason to want the secret information to remain secret except in case of a dire emergency, but they also want to be able to preserve it so that it can be released even if something happens to them. How to do this?</p><p>If you've uploaded the bits to one of the major services, then who gets access to them? Do you keep the information in a private file, memorize the account password and hope for the best? What if you're captured and coerced into giving up the password? On the other hand, if you die without revealing the information, it will just sit there until the account is closed, unless someone can figure out enough to subpoena the major service into handing over access to a bunch of cat pictures hiding the real information. Which you encrypted, of course, so who has the key?</p><p>Maybe you share the encrypted bits with a journalist (or two, or three ...) with an "in case of my death" cover letter saying where to get the encryption key. But what if they decide to go public with it anyway? The more journalists, the better the chance one of them will publish if something happens to you, but also the better the chance that one of them will publish anyway.</p><p>Maybe you put the encrypted bits someplace public but write the encryption key on a piece of paper and lock it away in a safe deposit box in a Swiss bank. Now you've traded one MacGuffin for another. But maybe someone at a <i>different</i> spy agency has a backdoor into your encryption. The baddies at your own agency are going to keep the contents to themselves, but maybe one of them has a change of heart, or gets double-crossed and decides to go public as revenge, and they need your copy since they no longer have access to the original bits and didn't make their own copy.</p><p>And so forth. The point is that information doesn't really act like a physical object, even if you have a copy in physical form, but even so there are lots of ways to go, each with its own dramatic possibilities depending on the abilities and motivations of the various characters. Most of these possibilities are pretty well-used themselves. Plots driven by who has access to what information have been around forever, though some have paid more attention to the current technology than others -- "Did you destroy the negatives?" "Yes, but I didn't realize they'd left another copy of the photographs in a locker at the bus station ..."</p><p>Opting for a bit more realism here gives up the possibility of a "destroy the magic item, destroy the magic" plot, but it opens up a host of other ones that could have been just as interesting. On the other hand, the movie in question doesn't seem to blink at the possibility of a full-on gun battle and massive explosions in the middle of a European capital in broad daylight. Maybe realism was never the point to begin with, since that seems pretty unlikely.</p><p>Oh, wait ...</p><p><br /></p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-68276118670442451102022-06-02T12:23:00.001-04:002024-02-25T16:03:51.233-05:00Check out this new kitchen hack!<p>In case that title somehow clickbaited you to this quiet backwater, no this isn't really about cooking, but for your trouble: The easiest and least tearful way I know to slice onions is to cut them in half lengthwise, so each half has a little piece of the roots holding it together. If you think of the roots as the South Pole and the stem end as the North Pole, the first slice is from pole to pole.</p><p>Chop off the stalk end and peel off the outer layers, under cold running water if that seems to help (I think this is a little easier than slicing the stem off first, but your mileage may vary). Put the halves down on the flat side and slice vertically with the slices parallel, also running north-south. Julia Child recommends another pass, horizontally, still slicing north-south, and who am I to argue? At this point, the root and the shape of the onion layers are still holding everything together. Finally, slice vertically, but with the slices running east-west. Each cut slices off a little pile of nicely diced pieces.</p><p>This isn't new -- I first heard about it on a <a href="https://en.wikipedia.org/wiki/Friedman_Paul_Erhardt">Chef Tell</a> segment many years ago, <i>Mastering the Art of French Cooking</i> came out in 1961 and I'm sure it's been around much longer -- but it works a charm. <i>Bon Apetit, </i>and remember that a dull kitchen knife is more dangerous than a sharp one.</p><p><br /></p><p>So it's not new, but is it a <i>hack</i>? And what's with all these "life hack" articles that have nothing to do with writing clever code?</p><p>For my money, the onion-dicing method is absolutely a nice hack. A hack, really, is an unexpected way of using something to solve a problem. The usual way to dice something is to slice it, then cut the slices crosswise into strips, then cut the strips crosswise into little dice. If you try that with an onion, the root is in the way of the north-south slices described above, and the easy way to start is to slice it east-west, into rings. You then have to dice up the rings, which are hard to stack since they're already separated, and like to slide around and separate into individual rings, and have a lot of exposed surface area to give off tear-producing onion fumes. In short, you have a mess.</p><p>The chef's method takes advantage of the two things that otherwise cause problems: It uses the root end to hold things in place and keep the exposed area to a minimum, and it uses the layering of the onion to save on cutting (if you omit the horizontal slices, as I usually do, you still get decently-diced pieces, good for most purposes, just a bit coarser). This is the essence of a hack: using something in a non-obvious way to get the result you want. It's particularly hackish to take advantage of something that seems to be an obstacle.</p><p>Not every hack is nice, of course. The other popular meaning of <i>hacking</i>, that many geeks including myself find annoying, the computing analog of breaking and entering or vandalizing someone's property, stems from a particular type of hacking: finding unexpected vulnerabilities in a system and taking advantage of them to break the system's security. As I've discussed at length elsewhere, this isn't necessarily bad. <i>White hat</i> hackers do just this in order to find and patch vulnerabilities and make systems more secure. The annoying part isn't so much that <i>hack</i> is associated with breaking and entering, but that it's associated with <i>any kind</i> of breaking and entering, regardless of whether there's any skill or actual hacking -- in the sense of making unexpected use of something -- involved.</p><p>I should note somewhere that <i>hack</i> often has negative connotations in software engineering for a completely different reason: If you take advantage of some undocumented feature of a system just to get something working, you have a fragile solution that is liable to break if the system you're hacking around changes in a future update. In widely-used systems this leads to <a href="https://en.wikipedia.org/wiki/API#Hyrums">Hyrum's law</a>, which basically says that people will write to what your system <i>does</i>, regardless of what you say it does, and with enough people using it, any externally visible change in behavior will break <i>someone's</i> code, even if it's not supposed to.</p><p>Hacking lives in gray areas, where behavior isn't clearly specified. "Dice this onion with this knife" doesn't say exactly <i>how</i> to dice the onion. Someone taking advantage of a quirk in an API can usually say "nothing said I <i>couldn't</i> do this". There's nothing wrong with unspecified behavior in and of itself. It's actively helpful if it gives people latitude to implement something in a new and better way. The trick is to be very specific about <i>what</i> can happen, but put as few restrictions as possible on <i>how</i>.</p><p>There's an art to this. If you're writing a sorting library, you could say "It's an error to try to sort an empty collection of things". Then you have to make sure to check that, and raise an error if the input is empty, and whoever's using your library has to be careful never to give it an empty collection. But why should it be an error? A collection with only one thing in it is always sorted, since there's nothing else for it to get out of order with. By that reasoning, so is an empty collection. If you define <i>sorted</i> as "everything in order", that raises the question "but what if there isn't anything?".</p><p>If you define <i>sorted</i> as "nothing out of order -- no places where a bigger thing comes before a smaller thing", then the question goes away. If there isn't anything in the collection, nothing's out of order and it's already sorted. In math, something is <i>vacuously true</i> if there's no way to make it false. "Nothing out of order" is vacuously true for an empty collection. Often, allowing things to be vacuously true makes life easier by sidestepping special cases.</p><p>As a general rule, the fewer special cases you need to specify what happens, the easier a system is to write and maintain, the more secure it is against unwanted forms of hacking like security exploits and Hyrum's law, and the friendlier it is to good kinds of hacking, like people finding clever new ways to improve the implementation or to use the system.</p><p><br /></p><p>So what about all this "life hacking"? Should people use computing jargon for things that have nothing to do with computing? I have two answers.</p><p>First, the term <i>hack</i> isn't really about computing. It's about problem solving. The first definition in the <a href="http://www.catb.org/jargon/html/">Jargon File</a> (aka <i>Hacker's Dictionary</i>) is "Originally, a quick job that produces what is needed, but not well.", with no mention of computing, and elsewhere it attributes early use of the term to ham radio hobbyists. As it happens, the actual definitions of <i>hack</i> in the Jargon File don't really include "using something in a non-obvious way to get the result you want", but I'd argue that the definition I gave is consistent with the <i><a href="http://www.catb.org/jargon/html/meaning-of-hack.html">The Meaning of 'Hack'</a></i> section.</p><p>Second, though, even if <i>hack</i> was originally only applied to coding hacks, so what? Language evolves and adapts. Extending <i>hack</i> to other clever tricks reveals something new about what people are trying to get at by using the word, and in my view it's a lot better than restricting it to security exploits, clever or not. Sure, not every "kitchen hack" or "life hack" is really that hackish, and headline writers are notoriously pressed for time (or lazy, if you're feeling less generous, or more apt to make money with clickbait, if you're feeling cynical), but there are plenty of non-computing hacks floating around now that are just as hackish as anything I've ever done with code.</p><p><br /></p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-60976008641256647302021-10-28T13:26:00.007-04:002022-06-02T10:46:35.338-04:00Why so quiet?<p>I hadn't meant for things to go so quiet here, and it's not just a matter of being busy. I've also been finding it harder to write about "the web", not because I don't want to, but because I'm just not running across as many webby things to write about.</p><p>That got me thinking, just what <i>is</i> the web these days? And that in turn got me thinking that the web is, in a way, receding from view, even as it becomes more and more a part of daily life, or, in fact, <i>because</i> it's more and more a part of daily life.</p><p>There is still plenty of ongoing work on the technical side. HTML5 is now a thing, and Adobe Flash is officially "end of life" (though there's a bit of a mixed message in that <a href="https://get.adobe.com/flashplayer/about/">Adobe's site for it</a> still says "Adobe Flash Player is the standard for delivering high-impact, rich Web content." right below the banner that says "Flash Player’s end of life is December 31st, 2020"). Microsoft has replaced Internet Explorer with Edge, built on the Chromium engine. Google is working to replace cookies. I realize those are all fairly Google-centric examples, and I don't want to imply that no one else is doing important work. Those were just the first examples that came to mind, for some strange reason.</p><p>On the one hand, those are all big developments. Adobe Flash was everywhere. It's hard to say how many web pages used it, but at the peak, there would be on the order of billions of downloads when Adobe pushed a release, because it was in every browser. Internet Explorer was the most-used browser for over a decade, and the standard browser on Windows, which would put its user base in the billions as well (even if some of us only used it to download Chrome). Somewhere around 20% of web sites, however many that is, use cookies.</p><p>On the other hand, they are all nearly invisible. I can remember a few times, early in the process a couple of years ago, when Chrome wouldn't load some particular website because Flash was disabled, but not enough to cause any real disruption. I'm sure that the shift from Explorer to Edge was disruptive to some, but when I set up a laptop for a relative a little while ago, they were much more concerned with being able to check email, write docs or play particular games than which browser was making that happen. As for cookies, I haven't looked into exactly how they're being replaced, because I don't have to and I haven't made time to look it up.</p><p>Because the web is everywhere, the huge number of websites and people browsing means that it's most important to keep everything running smoothly. Unless you're introducing some really amazing new feature, it's usually bad news if anyone knows that you made some change behind the scenes (whatever you think of Facebook as a company, please spare a thought for the people who had to deal with that outage -- even with a highly-skilled, dedicated team keeping the wheels turning, these things can happen, and it can be devastating to those involved when it does).</p><p>The upshot here is that I don't really have much interesting to say about much of the technical infrastructure behind everyday web experience. Besides not having been close to the standards process for several years, I figured out very early that I didn't want to write about the standards and protocols themselves -- there are plenty of people who can do that better than I can -- but how they appear in the wild. Thus the <i>field notes</i> conceit.</p><p>It was interesting to write about, say, <a href="https://blog.fieldnotesontheweb.com/2007/10/and-speaking-of-trusting-dns.html">Paul Vixie's concerns about DNS security</a> or what <a href="https://blog.fieldnotesontheweb.com/search/label/copyrights">copyrights</a> mean in the digital age, but topics like that seem less interesting today. Regardless of the particular threats, the real benchmark of computer security is whether people are willing to put their money on the web -- buy, sell, send money to friends, check their bank statements or retirement accounts, and so forth. That's been the case for a while now, through a combination of security technology and legal protections. Importantly, the technology doesn't have to be perfect, and a good thing, that.</p><p>The question of how creators get paid on the web is still shaking out, but one the one hand, I think this is one of those problems that is <i>always</i> shaking out without ever getting definitively resolved, and on the other hand, I'm not sure I have anything significant to add to the discussion.</p><p><br /></p><p>As much as I don't want to write a purely technical blog, I also don't want to lose sight of the technical end entirely. I'm a geek by training and by nature. The technical side is interesting to me, and it's also where I'm most likely to know something that isn't known to a general audience.</p><p>Obviously, a lot of the important discussion about the web currently is about social media, but I don't want to jump too deeply into that pool. Not only is it inhabited by a variety of strange and not-always-friendly creatures, but if I were commenting on it extensively, I'd be commenting on sociology, psychology and similar fields. I muse about those on <a href="https://intermittent-conjecture.blogspot.com/">the other blog</a>, but intermittently conjecturing about what consciousness is or how language works is an entirely different thing from analyzing social media.</p><p>Even so, <a href="https://blog.fieldnotesontheweb.com/search/label/Twitter">Twitter</a> is one of the top tags here, ironic since I don't have a Twitter account (or at least not one that I use).</p><p>My main point on social media was that some of the more utopian ideas about the wisdom of crowds and the self-correcting nature of the web don't tend to hold up in practice. I made that point in the context of Twitter a while ago, in <a href="https://blog.fieldnotesontheweb.com/2011/12/rumours-and-tweets-of-rumours.html">this post</a> in particular. I wasn't the first and I won't be the last. I think it's pretty widely understood today that the web is not the idyllic place some said it would be a few decades ago (not that that kept me from commenting on that very topic in the <a href="https://blog.fieldnotesontheweb.com/2021/05/please-leave-us-5-star-review.html">most recent post before this one</a>).</p><p>On the other hand, it might be interesting to look into why the web <i>can be</i> self-correcting, if still not idyllic, under the right circumstances. Wikipedia comes to mind ...</p><p><br /></p><p>Finally, I've really been trying to keep the <i><a href="https://blog.fieldnotesontheweb.com/search/label/annoyances">annoyances</a></i> tag down to a dull roar. That might seem a bit implausible, since it's generally the top tag on the list (48 posts and counting), but in my defense it's fairly easy to tell if something's annoying or not, as opposed to whether its related to, say, <i>copyrights, publishing,</i> both or neither, so it doesn't take a lot of deliberation to decide to apply that label. Also, with the web a part of everyday life, there's always something to be annoyed about.</p><p><br /></p><p>So if you take out "technical stuff that no one notices unless it breaks", "social media critiques", "annoying stuff, unless maybe it's particularly annoying, funny or interesting", along with recusing myself from "hmm ... what's Google up to these days?", what's left?</p><p>Certainly <i>something</i>. I haven't stopped posting entirely and I don't plan to. On the other hand, there doesn't seem to be as much low-hanging fruit as there used to be, at least not in the particular orchard I'm wandering through. Some of this, I think, is because the web has changed, as I said up top. Some of it is because my focus has changed. I've been finding the topics on <a href="https://intermittent-conjecture.blogspot.com/">the other blog</a> more interesting, not that I've been exactly prolific there either. Some of it is probably the old adage that if you write every day, there's always something to say, while if you write infrequently, it's hard to get started.</p><p>A little while ago, I went through the whole blog from the beginning and made several notes to myself to follow up, so I may come back to that. In any case new topics will certainly come up (one just did, after all, about why Wikipedia seems to do much better at self-correcting). I think it's a safe bet, though, that it will continue to be a while between posts. Writing this has helped me to understand why, at least.</p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-39200666498354954152021-05-01T19:58:00.002-04:002021-10-28T13:37:29.982-04:00Please leave us a 5-star review<p>It's been long enough that I can't really say I remember for sure, and I can't be bothered to look it up, but as I recall, reviews were supposed to be one of the main ways for the web to correct itself. I might advertise my business as the best ever, even if it's actually not so good, but not to worry. The reviewers will keep me honest. If you're searching for a business, you'll know to trust your friends, or you'll learn which reviewers are worth paying attention to, good information will drive out bad and everyone will be able to make well-informed decisions.</p><p>This is actually true, to an extent, but I think it's about the same extent as always. Major publications try to develop a reputation for objective, reliable reviews, as do some personalities, but then, some also develop a reputation for less-than-objective reviews. Some, even, may be so reliably un-objective that there's a bit of useful information in what they say after all. And you can always just ask people you know.</p><p>But this is all outside the system of customer reviews that you find on web sites all over the place, whether provided by the business itself, or companies that specialize in reviews. These, I personally don't find particularly useful or, if I were feeling geekly, I'd say the signal/noise ratio is pretty low. It turns out there are a couple of built-in problems with online reviews, that were not only predictable, but were predicted at the time.</p><p>First, there's the whole question of identity on the internet. In some contexts, identity is an easy problem: an identity is an email address or a credit or debit account with a bank, or ownership of a particular phone, or something similar that's important to a person in the real world. Email providers and banks take quite a bit of care to prevent those kind of identities from being stolen, though of course it does still happen. </p><p>However, for the same reason, we tend to be a bit stingy with this kind of identity. I try hard not give out my credit card details unless I'm making an actual purchase from a reputable merchant, and if my credit card details do get stolen, that card will get closed and a new one opened for the same account. Likewise, I try not to hand out my personal email or phone number to just anyone, for whatever good that does.</p><p>When it comes to reviews, though, there's no good way to know who's writing. They might be an actual customer, or an employee of the business in question, or they might be several time zones away writing reviews for money, or they might even be a bot. Platforms are aware of this, and many seem to do a good job of filtering out bogus reviews, but there's always that lingering doubt. As with identities in general, the stakes matter. If you're looking at a local business, the chances are probably good that everyone who's left a review has actually been there, though even then they might still have an axe to grind. In other contexts, though, there's a lot more reason to try to game the system.</p><p>But even if everyone is on the up-and-up and leaving the most honest feedback they can, there are still a few pitfalls. One is <i>selection bias</i>. If I've had a reasonably good experience with a business, I'll try to thank the people involved and keep them in mind for future work, or mention them if someone asks, but I generally don't take time to write a glowing review -- and companies that do that kind of work often seem to get plenty of business anyway.</p><p>If someone does a really horrible job, or deals dishonestly, though, I might well be in much more of a mood to share my story. Full disclosure: personally I actually don't tend to leave reviews at all, but it's human nature to be more likely to complain in the heat of the moment than to leave a thoughtful note about a decent experience, or even an excellent experience. In other words, you're only seeing the opinions of a small portion of people. That wouldn't be so bad if the portion was chosen randomly, but it's anything but. You're mostly seeing the opinions of people with <i>strong</i> opinions, and particularly, strong negative opinions.</p><p>The result is that reviews tend to cluster toward one end or the other. There are one-star "THIS PLACE IS TERRIBLE!!!" reviews, there are five-star "THIS PLACE IS THE MOST AWESOME EVER!!!" reviews, and not a lot in between. A five-point scale with most of the action at the endpoints is really more of a two-point scale. In effect, the overall rating is the weighted average of the two: the number of one-star reviews plus five times the number of five-star reviews, divided by the total number of reviews. If the overall rating is close to five, then most of the reviews were 5-star. If it's 3, it's much more likely that the good and the bad are half-and-half than most of the reviews being 3-star.</p><p>The reader is left to try to decide <i>why</i> the reviewers have such strong opinions. Did the car wash do a bad job, or was the reviewer somehow expecting them to change the oil and rotate the tires as well and then get angry when they didn't? Is the person praising a consultant's integrity actually just their cousin? Does the person saying that a carpenter did a great job with their shelves actually know much about carpentry or did they just happen to like the carpenter's personality? If the shelves collapse after a year and a half, are they really going to go back and update their review? Should they, or should they maybe not store their collection of lead ingots from around the world on a set of wooden shelves?</p><p>Specifics can help, but people often don't provide much specific detail, particularly for positive reviews, and when they do, it's not always useful. If all I see is three five-star reviews saying "So and so was courteous, professional and did great work", I'm not much better off than when I started. If I see something that starts out with "Their representative was very rude. They parked their truck in a place everyone in the neighborhood knows not to park. The paint on the truck was chipped. Very unprofessional!" I might take what follows with a grain of salt.</p><p><br /></p><p>There's a difference, I think, between an opinion and a true review. A true review is aimed at laying out the information that someone else might need to make a decision. An opinion is just someone's general feeling about something. If you just ask people to "leave a review", you're going to get a lot more personal impressions than carefully constructed analyses. Carefully constructing an analysis is work, and no one's getting paid here.</p><p>Under the "wisdom of crowds" theory, enough general impressions will aggregate into a complete and accurate assessment. A cynic would say that this is like hoping that if you put together enough raw eggs, you'll end up with a soufflé, but there are situations where it can actually work (for a crowd, that is, not for eggs). The problem is that in many cases you don't even have a crowd. You have a handful of people with their various experiences and opinions.</p><p><br /></p><p>This all reaches its logical conclusion in the gig economy. When ride share services first started, I used to think for a bit about what number to give a driver. "They were pretty good, but I wish they had driven a bit less (or in some cases maybe more) aggressively". "The car was pretty clean, but there was a bit of a funny smell" or whatever.</p><p>Then I started noticing that almost all drivers had 5-star ratings, or close. The number before the decimal point doesn't really mean anything. You're either looking at 5.0 or 4.something. A 4.9 is still a pretty good rating, but a 4.0 rating is actually conspicuously low. I don't know the exact mechanics behind this, but the numbers speak for themselves.</p><p>It's a separate question to what extent we should all be in the business of rating each other to begin with, but I'll let <i><a href="https://en.wikipedia.org/wiki/Nosedive_(Black_Mirror)">Black Mirror</a></i> speak to that.</p><p>Following all this through, if I give someone a 4-star review for being perfectly fine but not outstanding, I may actually be putting a noticeable dent in their livelihood, and if I give someone 3 stars for being pretty much in the middle, that's probably equivalent to their getting a D on a test. So anyone who's reasonably good gets five stars, and if they're not that good, well, maybe they were just having a bad day and I'll just skip the rating. If someone actively put my life in danger, sure, they would get an actual bad rating and I'd see if I could talk to the company, but beyond that ... everyone is awesome.</p><p>Whatever the reasons, I think this is a fairly widespread phenomenon. Reviews are either raves or pans, and anyone or anything with reviews much short of pure raves is operating at a real disadvantage. Which leads me back to the title.</p><p>Podcasts that I listen to, if they mention reviews at all, don't ask "Please leave a review so we can tell what's working and what we might want to improve". They ask "Please leave a 5-star review". The implication is that anything less is going to be harmful to their chances of staying in business. Or at least that's my guess, because I've heard this from science-oriented podcasts and general-interest shows that clearly take care to present their stories as objectively as they can, the kind of folks who might genuinely appreciate a four-star review with a short list of things to work on.</p><p>This is a shame. A five-point scale is pretty crude to begin with, but when it devolves to a two-point scale of horrible/awesome, it's not providing much information at all, pretty much the opposite of the model that I'm still pretty sure people were talking about when the whole ratings thing first started.</p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com1tag:blogger.com,1999:blog-2129929182918599848.post-65238450752668594702020-09-05T18:24:00.003-04:002020-10-03T15:12:56.686-04:00One thing at a time<p> As much as I gripe about UX annoyances (and all manner of other annoyances), I really do try to look out for specific ways to improve. I don't come up with many, most likely because UX is hard and lots of people who are better at it than I am have spent a lot of time on the problem and come up with a lot of good ideas. Much of the low-hanging fruit has been picked, and so has a lot of the not-so-low-hanging fruit.</p><p>However, while grumbling at a particular web page today, I think I hit upon a good rule. I doubt it's new, because a lot of sites follow it (and see above regarding fruit), but a lot don't, so I'll put it out here anyway, for my vast audience, just in case.</p><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;">Changing one setting on a page should only change the corresponding thing(s) on that page</p></blockquote><p>For example, say I'm looking at statistics on farm production in US states. I can rank output by, say, yield per acre, dollar value, amount per capita and dollar value per capita. I can pick a specific list of states or crops. I pick corn and soybeans for crops and North and South Dakota, Nebraska, Kansas and Oklahoma for states. Up comes a nice table, initially sorted alphabetically by state. I change the sorting order to dollars per capita, from high to low. So far so good.</p><p>Now I decide to add wheat to the set of crops. On a well-designed page, I will now see the same data, for the new set of crops, <i>sorted the same way as before</i>. On all too many sites, I see the data for corn, beans and wheat, but sorted alphabetically by state, because that's how all tables start life. I changed one thing -- which crops I'm interested in -- but two things changed, namely the data being shown and the sort order. I only wanted one thing to change, namely the set of crops.</p><p>This is a small example, but I'd be surprised if you haven't run across something similar. As described, it's a minor annoyance, but as the options get more sophisticated, annoyance turns into unusability. If I've spent five minutes setting up a graph or chart of, say, crop distribution as a function of latitude, I don't want that all to go away if I decide to include Colorado or Iowa in my set of states.</p><p>This is not to say you can't have settings with wider-ranging effects. If there's a tab on the page for, say, trends in agricultural veterinary medicine, I wouldn't expect my graph of crop production to stick around (though I would very much like it to still be there if I go back to its tab). That's fine. I changed one setting, but it's a big setting and the "corresponding things" that need changed are correspondingly big.</p><p>Again, this is nothing new. For example, it fits nicely into <i><a href="https://blog.fieldnotesontheweb.com/2011/08/considerate-software.html">considerate software remembers</a></i>. Still, it's often useful to find specific examples of more general principles.</p><p></p>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-2053811950378259982020-07-25T13:11:00.035-04:002020-08-08T15:55:45.932-04:00Still here, still annoyed with the internetLooks like it's been several months since the last post, which has happened before but probably not for quite this long. I've been meaning to put something up, first about working from home (or WFH as we like to call it), then more about machine learning (or ML as we like to call it), which seems to be going interesting places but probably not as far and fast as some might say. I probably will get back to at least one of those topics, but so far, having settled into a new routine, I just haven't worked in much time for the blogs.<div><br /></div><div>I have been reading quite a bit, on various topics, a lot of it on my phone. I've managed to train my news feed to deliver a reasonable mix of nerdy stuff, light entertainment and what's-going-on-in-the-world. I'm often happy to read the light entertainment in particular, since I get to use my analytical brain plenty between work and writing the occasional analytical blog post. The only problem with the light reading is the actual <i>reading</i>.</div><div><br /></div><div>I've always said that writers, and "content creators" in general, need to get paid, and I don't mind having to look at the occasional ad or buy the occasional subscription to support that. It's just that the actual mechanics of this are getting a bit out of hand.</div><div><br /></div><div>Generally one of three things happens. For major outlets, or most of the nerdy stuff, or publications for which I do have a subscription, I click through and read. Great.</div><div><br /></div><div>If there's a paywall, I usually see the first paragraph or so, enough to confirm what the article is about, and then a button asking me to join in order to see more. I pretty much never do, even though I'm fine with the concept and subscriptions are generally pretty cheap, because</div><div><ul style="text-align: left;"><li>Dude, I just wanted to read the article and it sure would have been nice to have seen a paywall notice <i>before</i> I clicked through (sometimes they're there, but usually not).</li><li>I'm leery of introductory rates that quietly start charging you more because you forgot to go back and cancel.</li><li>And combining the previous two items, I don't really want to dig through the subscription terms and find out how much I'm really paying and what I'm actually paying for.</li></ul><div>I'm a bit more amenable to the "You have N free articles left this month" approach, because I get to read the particular article I was interested in and figure out the subscription stuff at my leisure. I seldom get around to that last part, but nonetheless I think all the subscriptions I've actually bought have been on that basis. I'm sure there have been theses written about the psychology behind that.</div></div><div><br /></div><div>Having re-read the whole blog a while ago, I recall that <a href="https://blog.fieldnotesontheweb.com/2011/04/xanadu-vs-web-part-iii-xanadu-business.html">Xanadu</a> advocated for a similar pay-as-you-go approach. As far as I could tell from the demo I saw, it would have led to a sort of taxicab-like "meter is running" experience. This seems even slightly less pleasant than paywalls and subscriptions, but Xanadu could probably have supported either model, in theory.</div><div><br /></div><div>The more common experience, of course, is ads, particularly in the light entertainment department. What happens is interesting: You see the ads so much you don't see them, and depending on your level of patience, you might not bother to see the light entertainment either.</div><div><br /></div><div>Suppose you run across a suitably light-looking title. Some popular ones are "Learn something new about <i><your favorite movie, album, artist etc.></i>" and "N best/worst/most surprising/... Xs". In either case, there are always two or three paragraphs of things you already know. "<i>My Cousin the Vampire Chauffeur</i> [not a real movie that I know of] was one of the great hits of the 1980s, starring Moviestar McMoviestarface as the vampire and That One Actor as their best friend. At first, the friend only thinks it's a little odd that the Chauffeur only drives at night and has removed the rearview mirror from the car, but events take an unexpected turn when ..." Yep, knew that. I clicked through on this because I liked that movie so yes, I've seen it.</div><div><br /></div><div>About that time the whole screen starts to rearrange itself as various ad-things jostle for position. Often, it all settles back down with the text you were reading still in roughly the same place, but sometimes you have to scroll. About the same time, a video starts playing across the bottom of the screen. There's generally a tiny "x" box at the corner to make it go away, but that's a fool's errand. Another hydra head will regrow to take its place, and there's always the chance you'll accidentally click through instead of dismissing. Instead, stare steadfastly at the text on the screen and nothing else, secure in the knowledge that the whole "subliminal advertising" thing was most likely overblown.</div><div><br /></div><div>Finish the paragraph you're on and scroll past the display ad between it and the next paragraph. With a fair wind and a favorable moon phase, you'll get to the next paragraph. If not, the game of musical chairs will resume until the new batch of ads have all found places, at which point I generally head for the exit. But you persevere. You quickly realize that this paragraph as well is more filler, so you try to scroll to the bottom for the nugget of information you were really after. You scroll too far, too fast, and land in a column of photos and links for similar articles, some of which you've already read because, well, we're all human, right?</div><div><br /></div><div>Scroll back up and you find the object of your quest, that last paragraph, derive whatever edification you can from it and hit the back button. Rather than going back to the news feed, you quite likely go back to a previous version of the page you were reading, and maybe another after that, before ending up back in civilization. I could write a whole other rant about "Why doesn't the back button just take me back?" but I'm not sure that would improve either my life or yours.</div><div><br /></div><div>I mean, in the grand scheme of things this is all pretty trivial, but then, in the grand scheme of things so is this blog, so I guess we're even.</div><div><br /></div><div>Except for ads in the middle of lists-of-N-things that disguise their click-through buttons as "next item" buttons. Those are pure evil.</div><div><br /></div><div>So, still here, still annoyed with the internet.</div><div><br /></div>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com1tag:blogger.com,1999:blog-2129929182918599848.post-21232858017368339652019-10-29T15:18:00.002-04:002021-04-30T19:15:23.611-04:00Did the internet kill the radio tower?<i>"Half your advertising budget is wasted. You just don't know which half."</i><br />
<br />
The other day I turned on my radio on the drive home from work. There was a breaking news story I was interested in ("breaking" as in, it was actually happening at the time, not "someone told the Chyron writer to make something look important"). I hadn't done that in months. Years ago, listening to the radio was <a href="https://blog.fieldnotesontheweb.com/2011/11/voices-from-dashboard.html">an integral part of making a cross-country trip</a>, just as reading the Sunday funnies and (lightly) browsing the rest of the newspaper used to be a regular habit. Even not so long ago, listening to the news on the way home was the default option.<br />
<br />
Then came podcasts. I was a bit late to the game, mostly because I'm somewhat deliberately lazy about adopting new technology, but once I got a suitable app set up to my liking, podcasts rapidly took over. I could pick out whatever information or entertainment I wanted streamed into my brain, rewind or fast forward as needed and never have to worry about reception. The only downside is needing to get everything set up nicely before actually starting the car moving. I'm sure there are apps for that, but as I said, I'm a bit lazy about apps and such.<br />
<br />
I know that people still listen to the radio. Somehow my switching over didn't magically cause everyone else to stop tuning in to their favorite on-air personalities and call-in shows. But for a certain kind of listener, there's little reason to fiddle with a radio. Chances are you can livestream your favorite sports events if you like, though then you <i>do</i> have to worry about reception.<br />
<br />
Podcasts, livestreams and other online content don't just change things for listeners. There's a crucial difference for the people creating and distributing the content. Even if "podcast" deliberately sounds like "broadcast", it's actually a classic example of <i>narrowcasting</i> -- delivering content directly to the "content consumers" based on their particular preferences.<br />
<br />
Broadcasting is anonymous. I send a signal into the ether and whoever picks it up picks it up. I have no direct way of knowing how many people are listening, much less who. Obviously this is <a href="https://blog.fieldnotesontheweb.com/2010/03/anonymity-at-source.html">much more anonymous than the internet</a>. It also has economic implications.<br />
<br />
There are two main ways of paying for content: subscription and advertising. In either case, it's valuable to know exactly who's on the other end. Narrowcasting gives very find-grained information about that, while broadcasting provides only indirect, aggregated information based on surveys or, failing that, the raw data of who's buying advertising and how much they're paying. Between that and satellite radio's subscription model, is there any room left for broadcast radio?<br />
<br />
Probably. I mean, it hasn't gone away yet, any more than printed books have.<br />
<br />
Sure, broadcasters and the people who advertise on broadcast radio don't have detailed information about who's listening to the ads, but that may not matter. The advertiser just needs to know that spending $X on old-fashioned radio advertising brings in more than $X in business. The tools for figuring that out have been around since the early days of radio.<br />
<br />
If people still find radio advertising effective, the broadcaster just has to know that enough people are still buying it to keep the lights on and the staff paid. In a lot of cases that staff is shared across multiple physical radio stations anyway (and the shows, I would expect, are sent to those stations over the internet). In other words, it may be valuable to know in detail who's listening to what, but it's not essential.<br />
<br />
On the other hand, if broadcast radio does go away, I probably won't find out about it until I happen to switch my car audio over to it and nothing's there.<br />
<br />David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com3tag:blogger.com,1999:blog-2129929182918599848.post-34492555988303012792019-07-16T02:54:00.002-04:002020-03-08T19:19:21.069-04:00Space Reliability EngineeringIn a <a href="https://blog.fieldnotesontheweb.com/2015/08/margaret-hamilton-1-new-horizons-0.html">previous post on the <i>Apollo 11</i> mission</a>, I emphasized the role of software architecture, and the architect Margaret Hamilton in particular, in ensuring the success of the <i>Apollo 11</i> lunar landing. I stand by that, including the assessment of the whole thing as "awesome" in the literal sense, but as usual there's more to the story.<br />
<br />
Since that non-particularly-webby post was on <i>Field Notes</i>, so is this one. What follows is mostly taken from the BBC's excellent if majestically paced podcast <i><a href="https://www.bbc.co.uk/programmes/w13xttx2">13 Minutes to the Moon</a></i> <i>[I hope to go back and recheck the details directly at some point, but searching through a dozen or so hours of podcast is time-consuming and I don't know if there's a transcript available -- D.H.], </i>which in turn draws heavily on NASA's <i><a href="https://historycollection.jsc.nasa.gov/JSCHistoryPortal/history/oral_histories/oral_histories.htm">Johnson Space Center Oral History Project</a></i>.<br />
<br />
I've also had a look at Ars Technica's <a href="https://arstechnica.com/science/2019/07/no-a-checklist-error-did-not-almost-derail-the-first-moon-landing/"><i>No, a "checklist error" did </i>not<i> almost derail the </i>Apollo 11</a><i><a href="https://arstechnica.com/science/2019/07/no-a-checklist-error-did-not-almost-derail-the-first-moon-landing/"> mission</a>, </i>which takes issue with Hamilton's characterization of the incident and also credits Hal Laning as a co-author of the Executive portion of the guidance software which ultimately saved the day (to me, the main point Hamilton was making was that the executive saved the day, regardless of the exact cause of the 1202 code).<br />
<br />
Before getting too far into this, it's worth reiterating just how new computing was at the time. The term "software engineer" didn't exist (Hamilton coined it during the project -- Paul Niquette <a href="http://niquette.com/books/softword/part0.htm">claims to have coined the term "software" itself</a> and I see no reason to doubt him). There wasn't any established job title for what we now call software engineers. The purchase order for the navigation computer, which was the very first order in the whole Apollo project, didn't mention software, programming or anything of the sort. The computer was another piece of equipment to be made to work just like an engine, window, gyroscope or whatever. Like them it would have to be installed and have whatever other things done to it to make it functional. Like "programming" (whatever that was).<br />
<br />
In a way, this was a feature rather than a bug. The Apollo spacecraft have been referred to, with some justification, as the first fly-by-wire vehicles. The navigational computer was an unknown quantity. At least one astronaut promised to turn the thing off at the first opportunity. Flying was for pilots, not computers.<br />
<br />
This didn't happen, of course. Instead, as the podcast describes so well, control shifted back and forth between human and computer depending on the needs of the mission at the time, but it was far from obvious at the beginning that this would be the case.<br />
<br />
Because the computer wasn't trusted implicitly, but treated as just another unknown to be dealt with, -- in other words, another risk to be mitigated -- ensuring its successful operation was seen as a matter of engineering, just like making sure that the engines were efficient and reliable, and not a matter of computer science. This goes a long way toward explaining the self-monitoring design of the software.<br />
<br />
Mitigating the risk of using the computer included figuring out how to make it as foolproof as possible for the astronauts to operate. The astronauts would be wearing spacesuits with bulky gloves, so they wouldn't exactly be swiping left or right, even if the hardware of the time could have supported it. Basically you had a numeric display and a bunch of buttons. The solution was to break the commands down to a verb and a noun (or perhaps more accurately a predicate and argument), each expressed numerically. It would be a ridiculous interface today. At the time it was a highly effective use of limited resources <i>[I don't recall the name of the designer who came up with this. It's in the podcast --D.H.]</i>.<br />
<br />
But the only way to really know if an interface will work is to try it out with real users. Both the astronauts and the mission control staff needed to practice the whole operation as realistically as possible, including the operation of the computer. This was for a number of reasons, particularly to learn how the controls and indicators worked, to be prepared for as many contingencies as possible and to try to flush out unforeseen potential problems. The crew and mission control conducted many of these simulations and they were generally regarded as just as demanding and draining as the real thing, perhaps moreso.<br />
<br />
It was during one of the simulations that the computer displayed a status code that no one had ever seen before and therefore didn't know how to react to. After the session was over, flight director Gene Kranz instructed guidance software expert Jack Garman to look up and memorize every possible code and determine what course of action to take when it came up. This would take a lot of time searching through the source code, with the launch date imminent, but it had to be done and it was. Garmin produced a handwritten list of every code and what to do about it.<br />
<br />
As a result, when the code 1202 came up with the final opportunity to turn back fast approaching, capsule communicator (CAPCOM) Charlie Duke was able to turn to guidance controller Steve Bales, who could turn to Garman and determine that the code was OK if it didn't happen continuously. There's a bit of wiggle room in what constitutes "continuously", but knowing that the code wasn't critical was enough to keep the mission on track. Eventually, Buzz Aldrin noticed that the code only seemed to happen when a particular radar unit was being monitored. Mission Control took over the monitoring and the code stopped happening.<br />
<br />
<br />
I now work for a company that has to keep large fleets of computers running to support services that billions of people use daily. If a major Google service is down for five minutes, it's headline news, often on multiple continents. It's not the same as making sure a plane or a spaceship lands safely or a hospital doesn't lose power during a hurricane, but it's still high-stakes engineering.<br />
<br />
There is a whole profession, Site Reliability Engineer, or SRE for short, dedicated to keeping the wheels turning. These are highly-skilled people who would have little problem doing my job instead of theirs if they preferred to. Many of their tools -- monitoring, redundancy, contingency planning, risk analysis, and so on -- can trace their lineage through the Apollo program. I say "through" because the concepts themselves are considerably older than space travel, but it's remarkable how many of them were not just employed, but significantly advanced, as a consequence of the effort to send people to the moon and bring them back.<br />
<br />
One tool in particular, Garman's list of codes, played a key role at a that critical juncture. Today we would call it a playbook. Anyone who's been on call for a service has used one (I know I have).<br />
<br />
<br />
<br />
In the end, due to a bit of extra velocity imparted during the maneuver to extract the lunar module and dock it to the command module, the lunar module ended up overshooting its intended landing place. In order to avoid large boulders and steep slopes in the area they were now approaching, Neil Armstrong ended up flying the module by hand in order to find a good landing spot, aided by a switch to increase or decrease the rate of descent.<br />
<br />
The controls were similar to those of a helicopter, except the helicopter was flying sideways through (essentially) a vacuum over the surface of the moon, steered by precisely aimed rocket thrusts while continuing to descend, and was made of material approximately the thickness of a soda can which could have been punctured by a good jab with a ball-point pen. So not really like a helicopter at all.<br />
<br />
The Eagle landed with eighteen seconds of fuel to spare. It helps to have a really, really good pilot.David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com1tag:blogger.com,1999:blog-2129929182918599848.post-23887980513570372092019-04-16T10:48:00.000-04:002019-07-17T21:56:47.842-04:00Distributed astronomyRecently, news sources all over the place have been reporting on the imaging of a black hole, or more precisely, the immediate vicinity of a black hole. The black hole itself, more or less by definition, can't be imaged (as far as we know so far). Confusing things a bit more, any image of a black hole will <i>look</i> like a black disc surrounded by a distorted image of what's actually in the vicinity, but this is because the black hole distorts space-time due to its gravitational field, not because you're looking at something black. It's the most natural thing in the world to look at the image and think "Oh, that round black area in the middle is the black hole", but it's not.<br />
<br />
Full disclosure: I don't completely understand what's going on here. Katie Bouman has done <a href="https://youtu.be/UGL_OL3OrCE">a really good lecture</a> on how the images were captured, and Matt Strassler has <a href="https://profmattstrassler.com/2019/04/09/a-non-experts-guide-to-a-black-holes-silhouette/">an also really good, though somewhat long overview</a> of how to interpret all this. I'm relying heavily on both.<br />
<br />
Imaging a black hole in a nearby galaxy has been likened to "spotting a bagel on the moon". A supermassive black hole at the middle of a galaxy is big, but even a "nearby" galaxy is far, far away.<br />
<br />
To do such a thing you don't just need a telescope with a high degree of magnification. The laws of optics place a limit on how detailed an image you can get from a telescope or similar instrument, regardless of the magnification. The larger the telescope, the higher the resolution, that is, the sharper the image. This applies equally well to ordinary optical telescopes, X-ray telescopes, radio telescopes and so forth. For purposes of astronomy these are all considered "light", since they're all forms of electromagnetic radiation and so all follow the same laws.<br />
<br />
Actual telescopes can only be built so big, so in order to get sharper images astronomers use interferometry to combine images from multiple telescopes. If you have a telescope at the South Pole and one in the Atacama desert in Chile, you can combine their images to get the same resolution you would with a giant telescope that spanned from Atacama to the pole. The drawback is that since you're only sampling a tiny fraction of the light falling on that area, you have to reconstruct the rest of the image using highly sophisticated image processing techniques. It helps to have more than two telescopes. The Event Horizon Telescope project that produced the image used eight, across six sites.<br />
<br />
Even putting together images from several telescopes, you don't have enough information to precisely know what the full image really would be and you have to be really careful to make sure that the image you reconstruct shows things that are actually there and not artifacts of the processing itself (again, <a href="https://youtu.be/UGL_OL3OrCE">Bouman's lecture</a> goes into detail). In this case, four teams worked with the raw data independently for seven weeks, using two fundamentally different techniques, to produce the images that were combined into the image sent to the press. In preparation for that, the image processing techniques themselves were thoroughly tested for their ability to recover images accurately from test data. All in all, a whole lot of good, careful work by a large number of people went into that (deliberately) somewhat blurry picture.<br />
<br />
All of this requires very precise synchronization among the individual telescopes, because interferometry only works for images taken at the same time, or at least to within very small tolerances (once again, the details are ... more detailed). The limiting factor is the frequency of the light used in the image, which for radio telescopes is on the order of gigahertz. This means that images from the telescopes have to be recorded on the order of a billion times a second. The total image data ran into the petabytes (quadrillions of bytes), with the eight telescopes producing hundreds of terabytes (that is, hundreds of trillions of bytes) each.<br />
<br />
That's a lot of data, which brings us back to the web (as in "Field notes on the ..."). I haven't dug up the exact numbers, but accounts in the popular press say that the telescopes used to produce the black hole images produced "as much data as the LHC produces in a year", which in approximate terms is a staggering amount of data. A radio interferometer comprising multiple radio telescopes at distant points on the globe is essentially an extremely data-heavy distributed computing system.<br />
<br />
Bear in mind that one of the telescopes in question is at the south pole. Laying cable there isn't a practical option, nor is setting up and maintaining a set of radio relays. Even satellite communication is spotty. According to the <a href="https://en.wikipedia.org/wiki/Amundsen%E2%80%93Scott_South_Pole_Station#Communication">Wikipedia article</a>, the total bandwidth available is under 10MB/s (consisting mostly of a 50 mega<i>bit</i>/second link), which is nowhere near enough for the telescope images, even if stretched out over days or weeks. Instead, the data was recorded on physical media and flown back to the site where it was actually processed.<br />
<br />
I'd initially thought that this only applied to the south pole station, but in fact all six sites flew their data back rather than try to send it over the internet (just to throw numbers out, receiving a petabyte of data over a 10GB/s link would take about a day). The south pole data just took longer because they had to wait for the antarctic summer.<br />
<br />
Not sure if any <a href="https://blog.fieldnotesontheweb.com/2009/09/classic-software-engineering-and-mighty.html">carrier pigeons</a> were involved.David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com1tag:blogger.com,1999:blog-2129929182918599848.post-50475557492703956302019-04-04T10:46:00.000-04:002019-05-15T22:47:47.267-04:00Martian talkThis morning I was on the phone with a customer service representative about emails I was getting from an insurance company and which were clearly meant for someone else with a similar name (fortunately nothing earth-shaking, but still something this person would probably like to know about). As is usually the case, the reply address was a bit bucket, but there were a couple of options in the body of the email: a phone number and a link. I'd gone with the phone number.<br />
<br />
The customer service rep politely suggested that I use the link instead. I chased the link, which took me to a landing page for the insurance company. Crucially, it was just a plain link, with nothing to identify where it had come from*. I wasn't sure how best to try to get that across to the rep, but I tried to explain that usually there are a bunch of magic numbers or "hexadecimal gibberish" on a link like that to tie it back to where it came from.<br />
<br />
"Oh yeah ... I call that 'Martian talk'," the rep said.<br />
<br />
"Exactly. There's no Martian talk on the link. By the way, I think I'm going to start using that."<br />
<br />
We had a good laugh and from that point on we were on the same page. The rep took all the relevant information I could come up with and promised to follow up with IT.<br />
<br />
What I love about the term 'Martian talk' is that it implies that there's communication going on, but not in a way that will be meaningful to the average human, and that's exactly what's happening.<br />
<br />
And it's fun.<br />
<br />
I'd like to follow up at some point and pull together some of the earlier posts on Martian talk -- magic numbers, hexadecimal gibberish and such -- but that will take more attention than I have at the moment.<br />
<br />
<br />
* From a strict privacy point of view there would be plenty of clues, but there was nothing to tie the link to a particular account for that insurance company, which was what we needed.David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-65759103047001617922019-01-03T19:56:00.000-05:002019-07-17T22:02:02.509-04:00Hats off to New HorizonsA few years ago, around the time of the <i>New Horizons</i> encounter with Pluto (or if you're really serious about the demotion thing, minor planet 134340 Pluto), I gave the team <a href="https://blog.fieldnotesontheweb.com/2015/08/margaret-hamilton-1-new-horizons-0.html">a bit of grief</a> over the probe having to go into "safe mode" with only days left before the flyby, though I also tried to make clear that this was still engineering of a very high order.<br />
<br />
Early on New Year's Day (US Eastern time), <i>New Horizons</i> flew by a Kuiper Belt object nicknamed Ultima Thule (two syllables in Thule: <i>THOO-lay</i>). I'm posting to recognize the accomplishment, and this post will be grief-free.<br />
<br />
The Ultima Thule encounter was much like the Pluto encounter with a few minor differences:<br />
<ul>
<li>Ultima Thule is much smaller. Its long axis is about 1-2% of Pluto's diameter</li>
<li>Ultima Thule is darker, reflecting about 10% of light that reaches, compared to around 50% for Pluto. Ultima Thule is about as dark as potting soil. Pluto is more like old snow.</li>
<li>Ultima Thule is considerably further away (about 43 AU from the sun as opposed to about 33 AU for Pluto at the time of encounter -- an AU is the average distance from the Sun to the Earth)</li>
<li><i>New Horizons</i> passed much closer to Ultima Thule than it did to Pluto (3,500 km vs. 12,500 km). This requires more accurate navigation and to some extent increased the chances of a disastrous collision with either Ultima Thule or, more likely, something near it that there was no way to know about. At 50,000 km/h, even a gravel-sized chunk would cause major if not fatal damage.</li>
<li>Because Ultima Thule is further away, radio signals take proportionally longer to travel between Earth and the probe, about six hours vs. about four hours.</li>
<li>Because Ultima Thule is much smaller, much darker and significantly further away, it's much harder to spot from Earth. Before <i>New Horizons</i>, Pluto itself was basically a tiny dot, with a little bit of surface light/dark variation inferred by taking measurements as it rotated. Ultima Thule was nothing more than a tinier dot, and a hard-to-spot dot at that.</li>
<li>We've had decades to work out exactly where Pluto's orbit goes and where its moons are. Ultima Thule wasn't even discovered until after <i>New Horizons</i> was launched. Until a couple of days ago we didn't even know whether it had moons, rings or an atmosphere (it appears to have none). <i>[Neither Pluto nor Ultima Thule is a stationary object, just to add that little additional degree of difficulty. The Pluto flyby might be considered a bit more difficult in that respect, though. Pluto's orbital speed at the time of the flyby was around 20,000 km/h, while Ultima Thule's is closer to 16,500 km/h. I'd think this would mainly affect the calculations for rotating to keep the cameras pointed, so it probably doesn't make much practical difference.]</i></li>
</ul>
<div>
In both cases, <i>New Horizons</i> had to shift from pointing its radio antenna at Earth to pointing its cameras at the target. As it passes by the target at around 50,000 km/h, it has to rotate to keep the cameras pointed correctly, while still out of contact with Earth (which is light-hours away in any case). It then needs to rotate its antenna back toward Earth, "phone home" and start downloading data at around 1,000 bits per second. Using a 15-watt transmitter slightly more powerful than a <a href="https://en.wikipedia.org/wiki/Citizens_band_radio">CB radio</a>. Since this is in space, rotating means firing small rockets attached to the probe in a precise sequence (there are also gyroscopes on <i>New Horizons, </i>but they're not useful for attitude changes).</div>
<div>
<br /></div>
<div>
So, a piece of cake, really.<br />
<br />
Seriously, though, this is amazing engineering and it just gets more amazing the more you look at it. The Pluto encounter was a major achievement, and this was significantly more difficult in nearly every possible way.</div>
<div>
<br /></div>
<div>
So far there don't seem to be any close-range images of Ultima Thule on the <a href="http://pluto.jhuapl.edu/">mission's web site</a> (see, this post is actually about the web after all), but the team seems satisfied that the flyby went as planned and more detailed images will be forthcoming over the next 20 months or so. As I write this, <i>New Horizons</i> is out of communication, behind the Sun from Earth's point of view for a few days, but downloads are set to resume after that. <i>[The images started coming in not long after this was posted, of course --D.H. Jul 2019]</i></div>
David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-18660367436410487592018-12-13T16:05:00.001-05:002020-02-29T16:07:49.281-05:00Common passwords are bad ... by definitionIt's that time of the year again, time for the annual lists of worst passwords. Top of at least one list: <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">123456</span>, followed by <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">password</span>. It just goes to show how people never change. Silly people!<br />
<br />
Except ...<br />
<br />
A good password has a very high chance of being unique, because a good password is selected randomly from a very large space of possible passwords. If you pick your password at random from a trillion possibilities*, then the odds that a particular person who did the same also picked your password are one in a trillion, the odds that one of a million other such people picked your password are about one in a million, as are the odds that any particular two people picked the same password. If a million people used the same scheme as you did, there's a good chance that <i>some</i> pair of them accidentally share a password, but almost certainly almost all of those passwords are unique.<br />
<br />
If you count up the most popular passwords in this idealized scenario of everyone picking a random password out of a trillion possibilities, you'll get a fairly tedious list:<br />
<ul>
<li>1: some string of random gibberish, shared by two people</li>
<li>2 - 999,999: Other strings of random gibberish, 999,998 in all</li>
</ul>
<div>
Now suppose that seven people didn't get the memo. Four of them choose <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">123456</span> and three of them choose <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">password</span>. The list now looks like</div>
<div>
<ul>
<li>1: <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">123456</span>, shared by four people</li>
<li>2: <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">password</span>, shared by three people</li>
<li>3: some string of random gibberish, shared by two people</li>
<li>4-999,994: Other strings of random gibberish, 999,991 in all</li>
</ul>
<div>
Those seven people are pretty likely to have their passwords hacked, but overall password hygiene is still quite good -- 99.9993% of people picked a good password. It's certainly better than if 499,999<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span>people picked <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">123456 </span>and 499,998 picked <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">password</span>, two happened to pick the same strong password and the other person picked a different strong password, even though the resulting rankings are the same as above.</div>
</div>
<div>
<br /></div>
<div>
Likewise, if you see a list of 20 worst passwords taken from 5 million leaked passwords, that could mean anything from a few hundred people having picked bad passwords to everyone having done so. It would be more interesting to report <i>how many</i> people picked popular passwords as opposed to unique ones, but that doesn't seem to make its way into the "wow, everyone's still picking bad passwords" stories.</div>
<div>
<br /></div>
<div>
From what I was able to dig up, that portion is probably around 10%. Not great, but not horrible, and probably less than it was ten years ago. But as long as <i>some</i> people are picking bad passwords, the lists will stay around and the headlines will be the same, regardless of whether most people are doing a better job.</div>
<div>
<br /></div>
<div>
(I would have provided a link for that 10%, but the site I found it on had a bunch of broken links and didn't seem to have a nice tabular summary of bad passwords vs other passwords from year to year, so I didn't bother)<br />
<br />
*A password space of a trillion possibilities is actually pretty small. Cracking passwords is roughly the same problem as the hash-based proof-of-work that cyrptocurrencies use. Bitcoin is currently doing around 100 million trillion hashes per second, or a trillion trillion hashes every two or three hours. The Bitcoin network isn't trying to break your password, but it'll do for estimating purposes. If you have around <a href="http://blog.fieldnotesontheweb.com/2011/08/building-better-password.html"><span style="color: black;">100 bits of entropy</span></a>, for example if you choose a random sequence of fifteen capital and lowercase letters, digits and 30 special characters, it would take a password-cracking network comparable to the Bitcoin network around 400 years to guess your password. That's probably good enough. By that time, password cracking will probably have advanced far beyond where we are and, who knows, maybe we'll have stopped using passwords by then.</div>
David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com2tag:blogger.com,1999:blog-2129929182918599848.post-60787912784868628132018-12-08T23:22:00.003-05:002021-10-29T23:29:27.766-04:00Software citiesIn the previous post I stumbled on the idea that software projects are like cities. The more I thought about it, I said, the more I liked the idea. Now that I've had some more time to think about it, I like the idea even more, so I'd like to try to draw the analogy out a little bit further, ideally not past the breaking point.<br />
<br />
What first drew me to the concept was realizing that software projects, like cities, are neither completely planned nor completely unplanned. Leaving aside the question of what level of planning is best -- which surely varies -- neither of the extremes is likely to actually happen in real life.<br />
<br />
If you try to plan every last detail, inevitably you run across something you didn't anticipate and you'll have to adjust. Maybe it turns out that the place you wanted to put the city park is prone to flooding, or maybe you discover that the new release of some platform your depending doesn't actually support what you thought it did, or at least not as well as you need it to.<br />
<br />
Even if you could plan out every last detail of a city, once people start living in it, they're going to make changes and deviate from your assumptions. No one actually uses that beautiful new footbridge, or if they do, they cut across a field to get to it and create a "social trail" thereby bypassing the carefully designed walkways. People start using an obscure feature of one of the protocols to support a use case the designers never thought of. Cities develop and evolve over time, with or without oversight, and in software there's always a version 2.0 ... and 2.1, and 2.2, and 2.2b (see <a href="https://blog.fieldnotesontheweb.com/2011/10/so-what-version-are-we-on.html">this post</a> for the whole story).<br />
<br />
On the other hand, even if you try to avoid planning and let everything "just grow", planning happens anyway. If nothing else, we codify patterns that seem to work -- even if they arose organically with no explicit planning -- as customs and traditions.<br />
<br />
In a distant time in the Valley, I used to hear the phrase "paving the cow paths" quite a bit. It puzzled me at first. Why pave a perfectly good cow path? Cattle are probably going to have a better time on dirt, and that pavement probably isn't going to hold up too well if you're marching cattle on it all the time ... Eventually I came to understand that it wasn't about the cows. It was about taking something that people had been doing already and upgrading the infrastructure for it. Plenty of modern-day highways (or at least significant sections of them) started out as smaller roads which in turn used to be dirt roads for animals, foot traffic and various animal-drawn vehicles.<br />
<br />
Upgrading a road is a conscious act requiring coordination across communities all along the roadway. Once it's done, it has a significant impact on communities on the road, which expect to benefit from increased trade and decreased effort of travel, but also communities off the road, which may lose out, or may alter their habits now that the best way to get to some important place is by way of the main road and not the old route. This sort of thing happens both inside and outside cities, but for the sake of the analogy think of ordinary streets turning into arterials or bypasses and ring roads diverting traffic around areas people used to have to cross through.<br />
<br />
One analogue of this is in software is standards. Successful standards tend to arise when people get together to codify existing practice, with the aim of improving support for things people were doing before the standard, just in a variety of similar but still needlessly different ways. Basically pick a route and make it as smooth and accessible as possible. This is a conscious act requiring coordination across communities, and once it's done it has a significant impact on the communities involved, and on communities not directly involved.<br />
<br />
This kind of thing isn't always easy. A business district thrives and grows, and more and more people want to get to it. Traffic becomes intolerable and the city decides to develop a new thoroughfare to carry traffic more efficiently (thereby, if it all works, accelerating growth in the business district and increasing traffic congestion ...). Unfortunately, there's no clear space for building this new thoroughfare. An ugly political fight ensues over whose houses should get condemned to make way and eventually the new road is built, cutting through existing communities and forever changing the lives of those nearby.<br />
<br />
One analog of this in software is the rewrite. A rewrite almost never supports exactly the same features as the system being rewritten. The reasons for this are probably material for a separate post, but the upshot is that some people's favorite features are probably going to break with the rewrite, and/or be replaced by something different that the developers believe will solve the same problem in a way compatible with the new system. Even if the developers are right about this, which they often are, there's still going to be significant disruption (albeit nowhere near the magnitude of having one's house condemned).<br />
<br />
<br />
Behind all this, and tying the two worlds of city development and software develop together, is culture. Cities have culture, and so do major software projects. Each has its own unique culture, but, whether because the same challenges recur over and over again, leading to similar solutions, or because some people are drawn to large communities while others prefer smaller, the cultures of different cities tend to have a fair bit in common, perhaps more in common with each other than with life outside them. Likewise with major software projects.<br />
<br />
Cities require a certain level of infrastructure -- power plants, coordinated traffic lights, parking garages, public transport, etc. -- that smaller communities can mostly do without. Likewise, a major software project requires some sort of code repository with version control, some form of code review to control what gets into that repository, a bug tracking system and so forth. This infrastructure comes at a price, but also with significant benefits. In a large project as in a large city, you don't have to do everything yourself, and at a certain point you <i>can't</i> do everything yourself. That means people can specialize, and to some extent have to specialize. This both requires a certain kind of culture and tends to foster that same sort of culture.<br />
<br />
<br />
It's worth noting that even large software projects are pretty small by the standards of actual cities. Somewhere around 15,000 people have contributed to the git repository for the Linux kernel. There appear to be a comparable (but probably smaller) number of Apache committers. As with anything else, some of these are more active in the community than others. On the corporate side, large software companies have tens of thousands of engineers, all sharing more or less the same culture.<br />
<br />
Nonetheless, major software projects somehow seem to have more of the character of large cities than one might think based on population. I'm not sure why that might be, or even if it's really true once you start to look more closely, but it's interesting that the question makes sense at all.David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com1tag:blogger.com,1999:blog-2129929182918599848.post-13471048483386758552018-11-04T14:08:00.003-05:002021-10-29T23:23:45.683-04:00Waterfall vs. agileNear where I work is a construction project for a seven-floor building involving dozens of people on site and suppliers from all over the place, supplying items ranging from local materials to an impressively tall crane from a company on another continent. There are literally thousands of things to keep track of, from the concrete in the foundation to the location of all the light switches to the weatherproofing for the roof. The project will take over a year, and there are significant restrictions on what can happen when. Obviously you can't put windows on before there's a wall in place to put them in, but less obviously there are things you can't do during the several weeks that the structural concrete needs to cure, and so forth.<br />
<br />
Even digging the hole for the parking levels took months and not a little planning. You have to have some place to put all that dirt and the last part takes longer since you no longer have a ramp to drive things in and out with, so whatever you use for that last part has to be small enough you can lift it out.<br />
<br />
I generally try to keep in mind that no one else's job is as simple as it looks when you don't have to actually do it, but this is a particularly good example. Building a building this size, to say nothing of an actual skyscraper, is real engineering. Add into the mix that lives are literally on the line -- badly designed or built structures do fail and kill people, not to mention the abundance of hazards on a construction site -- and you have a real challenge on your hands.<br />
<br />
And yet, the building in question has been proceeding steadily and there's every reason to expect that it will be finished within a few weeks of the scheduled date and at a cost reasonably close to the initial estimate.<br />
<br />
We can't do that in my line of work.<br />
<br />
Not to say it's never happened, but it's not the norm. For example, I'm trying to think of a major software provider that still gives dates and feature lists for upcoming releases. Usually you have some idea of what month the next release might be in, and maybe a general idea of the major features that the marketing is based around, but beyond that, it comes out when it comes out and whatever's in it is in it. That fancy new feature might be way cooler and better than anyone expected, or it might be a half-baked collection of somewhat-related upgrades that only looks like the marketing if you squint hard enough.<br />
<br />
The firmer the date, the vaguer the promised features and vice versa ("schedule, scope, staff: pick two"). This isn't isolated to any one provider (I say "provider" rather than "company" so as to include open source). Everyone does it in their own way.<br />
<br />
In the construction world, this would be like saying "The new building will open on November 1st, but we can't say how many floors it will have or whether there will be knobs on the doors" or "This building will be completely finished somewhere between 12 and 30 months from now." It's not that construction projects never overrun or go over budget, just that the normal outcomes are in a much tighter range and people's expectations are set accordingly.<br />
<br />
<i>[Re-reading this, I realize I didn't mention small consultants doing projects like putting up a website and social media presence for a local store. I haven't been close to that end of the business for quite a while, but my guess is that delivering essentially on time and within the budget is more common. However, I'm more interested here in larger projects, like, say, upgrading trade settlement for a major bank. I don't have a lot of data points for large consultants in such situations, but what I have seen tends to bear out my main points here]</i><br />
<br />
Construction is a classic waterfall process. In fact, the use of roles like "architect" and "designer" in a waterfall software methodology gives a pretty good hint where the ideas came from. In construction, you spend a lot of time up front working with an architect and designer to develop plans for the building. These then get turned into more detailed plans and drawings for the people actually doing the construction. Once that's done and construction is underway, you pretty much know what you're supposed to be getting.<br />
<br />
In between design and construction there's a fair bit of planning that the customer doesn't usually see. For example, if your building will have steel beams, as many do, someone has to produce the drawing that says exactly what size and grade of beam to use, how long to cut it and (often) exactly where and what size to drill the holes so it can be bolted together with the other steel pieces. Much of this process is now automated with CAD software, and for that matter more and more of the actual cutting and drilling is automated, but the measurements still have to specified and communicated.<br />
<br />
Even if there's a little bit of leeway for changes later in the game -- you don't necessarily have select all the paint colors before they pour concrete for the underground levels -- for the most part you're locked in once the plans are finalized. You're not going to decide that your seven-level building needs to be a ten-level building while the concrete is curing, or if you do, you'll need to be ready to shell out a lot of money and throw the schedule out the window (if there's one to throw it out of).<br />
<br />
Interwoven with all this is a system of zoning, permitting and inspections designed to ensure that your building is safe and usable, and fits in well with the neighborhood and the local infrastructure. Do you have enough sewer capacity? Is the building about the same height as the buildings around it (or is the local government on board with a conspicuously tall or short one)? Will the local electrical grid handle your power demand, and is your wiring sufficient? This will typically involve multiple checks: The larger-scale questions like how much power you expect to use are addressed during permitting, the plans will be inspected before construction, the actual wiring will be inspected after it's in, and the contractor will need to be able to show that all the electricians working on the job are properly licensed.<br />
<br />
This may seem like a lot of hassle, and it is, but most regulations are in place because people learned the hard way. Wiring from the early 1900s would send most of today's licensed electricians running to the fuse box (if there is one) to shut off the power, or maybe just running out of the immediate area. There's a reason you Don't Do Things That Way any more: buildings burned down and people got electrocuted.<br />
<br />
Putting all this together, large-scale construction uses a waterfall process for two reasons: First, you can't get around it. It's effectively required by law. Second, and more interesting here, is that it works.<br />
<br />
Having a standard process for designing and constructing a building, standard materials and parts and a standard regime of permits, licenses and inspections gives everyone involved a good idea of what to expect and what to do. Having the plans finalized allows the builder to order exactly the needed materials (or more realistically, to within a small factor) and precisely track the cost of the project. Standards and processes allow people to specialize. A person putting up drywall knows that the studs will be placed with a standard spacing compatible with a standard sheet of drywall without having to know or care exactly how those studs got there.<br />
<br />
Sure, there are plenty of cases of construction projects going horribly off the rails, taking several times as long as promised and coming in shockingly over budget, or turning out to be unusable or having major safety issues requiring expensive retrofitting. It happens, and when it does lots of people tend to hear about it. But drive into a major city some time. All around you will be buildings that went up without significant incident, built more or less as I described above. On your way in will be suburbs full of cookie-cutter houses built the same way on a smaller scale. The vast majority of them will stand for generations.<br />
<br />
Before returning to the field of software, it's worth noting that this isn't the only way to build. Plenty of houses are built by a small number of people without a lot of formal planning. Plenty of buildings, particularly houses, have been added onto in increments over time. People rip out and replace cabinetry or take out walls (ideally non-structural ones) with little more than general knowledge of how houses are put together. At a larger scale, towns and cities range from carefully planned (Brasilia, Washington D.C) to agglomerations of ancient villages and newer developments with the occasional Grand Plan forced through (London comes to mind).<br />
<br />
<br />
So why does a process that works fine for projects like buildings, roads and bridges or, with appropriate variations, Big Science projects like CERN, the Square Kilometer Array or the Apollo missions, not seem to carry over to software? I've been pondering this from time to time for decades now, but I can't say that I've arrived at any definitive answer. Some possibilities:<br />
<ul>
<li>Software is soft. Software is called "software" because, unlike the hardware, which is wired together and typically shipped off to a customer site, you can change any part of the software whenever you want. Just restart the server with a new binary, or upgrade the operating system and reboot. You can even do this with the "firmware" -- code and data that aren't meant to be changed often -- though it will generally require a special procedure. Buildings and bridges are quintessential hardware.</li>
</ul>
<div>
Yes, but ... in practice you <i>can't</i> change anything you want at any time. Real software consists of a number of separate components acting together. When you push a button on a web site, the browser interprets the code for the web page and, after a series of steps, makes a call to the operating system to send a message to the server on the other end. The server on the other end is typically a "front-end" whose job is to dispatch the message to another server, which may then talk to several others, or to a database (probably running on its own set of servers), or possibly to a server elsewhere on the web or on the internal network, or most likely a combination of some or all of these, in order to do the real work. The response comes back, the operating system notifies the browser and the browser interprets more of the web page's code in order to update the image on the screen.</div>
<div>
<br /></div>
<div>
This is actually a highly simplified view. The point here is that all these pieces have to know how to talk to each other and that puts fairly strict limits on what you can change how fast. Some things are easy. If I figure out a way to make a component do its current job faster, I can generally just roll that out and the rest of the world is fine, and ideally happier since things that depend on it should get faster for free (otherwise, why make the change?). If I want to add a new feature, I can generally do that easily too, since no one has to use it until it's ready.</div>
<div>
<br /></div>
<div>
But if I try to change something that lots of people are already depending on, that's going to require some planning. If it's a small change, I might be able to get by with a "flag" or "mode switch" that says whether to do things the old way or the new way. I then try to get everyone using the service to adapt to the new way and set the switch to "new". When everyone's using the "new" mode, I can turn off the "old" mode and brace for an onslaught of "hey, why doesn't this work any more?" from whoever I somehow missed.</div>
<div>
<br /></div>
<div>
Larger changes require much the same thing on a larger scale. If you hear terms like "backward compatibility" and "lock-in", this is what they're talking about. There are practices that try to make this easier and even to anticipate future changes ("forward compatibility" or "future-proofing"). Nonetheless, software is not as soft as one might think.</div>
<div>
<ul>
<li>Software projects are smaller. Just as it's a lot easier to build at least some kind of house than a seven-level office building, it's easy for someone to download a batch of open source tools and put together an app or an open source tool or any of a number of other things (my <a href="https://blog.fieldnotesontheweb.com/2018/09/one-csv-30-stories-for-small-values-of.html">post</a> on Paul Downey's hacking on a CSV of government data shows how far you can get with not-even-big-enough-to-call-a-project). There are projects just like this all over GitHub.</li>
</ul>
<div>
Yes, but ... there are lots of small projects around, some of which even see a large audience, but if this theory were true you'd expect to see a more rigid process as projects get larger, as you do in moving from small DIY projects to major construction. That's not necessarily the case. Linux grew from Linus's two processes printing "A" and "B" on the screen to millions of lines of kernel code and millions more of tools and applications. The development process may have gradually become more structured over time, but it's not anything like a classic waterfall. The same could be said of Apache and any number of similar efforts. Projects do tend to add some amount of process as they go ("So-and-so will now be in charge of the Frobulator component and will review all future changes"), but not nearly to the point of a full-fledged waterfall.</div>
</div>
<div>
<br /></div>
<div>
For that matter, it's not clear that Linux or Apache should be considered single projects. They're more like a collection of dozens or hundreds of projects, each with its own specific standards and practices, but nonetheless they fit together into a more or less coherent whole. The point being, it can be hard to say how big a particular project is or isn't.</div>
<div>
<br /></div>
<div>
I think this is more or less by design. Software engineering has a strong drive to "decouple" so that different parts can be developed largely independently. This generally requires being pretty strict about the boundaries and interfaces between the various components, but that's a consequence of the desire to decouple, not a driving principle in and of itself. To the extent that decoupling is successful, it allows multiple efforts to go on in parallel without a strictly-defined architecture or overall plan. The architecture, such as it is, is more an emergent property of the smaller-scale decisions about what the various components do and how they interact.</div>
<div>
<br /></div>
<div>
The analogy here is more with city planning than building architecture. While it's generally good to have someone taking the long view in order to help keep things working smoothly overall, this doesn't mean planning out every piece in advance, or even having a master plan. You can get surprisingly well with the occasional "Wait, how is this going to work with that" or "There are already two groups working on this problem -- maybe you should talk to them before starting a third one".</div>
<div>
<br /></div>
<div>
Rather than a single construction project, a project like Linux or Apache is much like a city growing in stages over time. In fact, the more I think about it, the more I like that analogy. I'd like to develop it further.</div><div><br /></div><div>I had wanted to add a few more bullet points, particularly "Software is innovative" -- claiming that people generally write software in order to do something new, while a new skyscraper, however innovative the design, is still a skyscraper (and more innovative buildings tend to cost more, take longer and have a higher risk of going over budget) -- but I think I'll leave that for now. This post is already on the longish side and developing the city analogy seems more interesting at the moment.</div>
David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-88168306193595777852018-10-07T15:54:00.000-04:002018-10-07T15:54:34.769-04:00Economics of anonymity -- but whose economics?Re-reading the posts I've tagged with the <i><a href="https://blog.fieldnotesontheweb.com/search/label/anonymity">anonymity</a></i> label, I think I finally put my finger on something that had been bothering me pretty much from the beginning. I'd argued (<a href="https://blog.fieldnotesontheweb.com/2007/11/anonymous-three-card-monte.html">here</a>, for example) that economic considerations should make it hard to successfully run an anonymizer -- a service that allows you to connect with sites on the internet without giving away who or where you are. And yet they exist.<br />
<br />
The argument was that the whole point of an anonymizer is that whoever's connecting to a given site could be anyone using the anonymizer, that is, if you're using the anonymizer, you could be associated with anything that anyone on the anonymizer was doing. Since some things that people want to do anonymously are risky, an anonymizer is, in some sense, distributing risk. People doing risky things should be happy to participate, but people doing less risky things may not be so willing. As a result, there may not be enough people participating to provide adequate cover.<br />
<br />
However, this assumes that anyone can be (mis)taken for anyone else. At the technical level of IP addresses, this is true, but at the level of who's actually doing what, which is what really matters if The Man comes knocking, it's less applicable.<br />
<br />
There are lots of reasons to want anonymity -- the principle of the thing, the ability to do stuff you're not supposed to be able to do at work, wanting to hide embarrassing activity from the people around you, wanting to blow the whistle on an organization, communicating to the outside world from a repressive regime, dealing in illicit trade, planning acts of violence and many others. The fact of using an anonymizer says little about <i>why</i> you might be doing it.<br />
<br />
If I'm anonymously connecting to FaceSpace at work, there's little chance that the authorities in whatever repressive regime will come after me for plotting to blow up their government buildings, and vice versa (mutatis mutandis et cetera.). In other words, there's probably not much added risk for people doing relatively innocuous things in places where using an anonymizer is not itself illegal.<br />
<br />
On the other hand, this is little comfort to someone trying to, say, send information out of a place where use of the internet, and probably anonymizers in particular, is restricted. The local authorities will probably know exactly which hosts are connecting with the anonymizer's servers and make it their business to find out who's associated with those hosts -- a much smaller task than tracking down all users of the anonymizer.<br />
<br />
This is much the same situation as, say, spies in WWII trying to send radio messages out before the local authorities can triangulate their position. Many of the same techniques should apply -- never setting up in the same place twice, limiting the number of people you communicate with and how much you know about them, limiting the amount of information <i>you</i> know, and so forth.<br />
<br />
So I suppose I'll be filing this under <i>not-so-disruptive technology</i> as well as <i>anonymity.</i>David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-40500620096786177812018-09-29T19:36:00.002-04:002019-04-04T12:44:23.531-04:00Agile vs. waterfallAnother comment reply that outgrew the comment box. Earl comments<br />
<blockquote class="tr_bq">
<span style="background-color: white; color: #333333; font-family: "georgia" , serif; font-size: 13px;">This speaks to my prejudice in favor of technique over technology. And the concept of agility seems to be just the attitude of any good designer that your best weapon is your critical sense, and your compulsion to discard anything that isn't going to work.</span></blockquote>
To which I would reply:<br />
<br />
Sort of ... "agile" is a term of art that refers to a collection of practices aimed at reducing the lag between finding out people want something and giving it to them. Arguably the core of it is "launch and iterate", meaning "put your best guess out there, find out what it still needs, fix the most important stuff and try again".<br />
<br />
This is more process than design, but there are definitely some design rules that tend to go with agile development, particularly "YAGNI", short for "You ain't gonna need it", which discourages trying to anticipate a need you don't yet know that you have. In more technical terms, this means not trying to build a general framework for every possible use case, but being prepared to "refactor" later on if you find out that you need to do more than you thought you did. Or, better, designing in such a way that later functionality can be added with minimum disruption to what's already there, often by having less to disrupt to begin with, because ... YAGNI.<br />
<br />
Downey refers to "agile" both generally and in the more specific context of "agile vs. waterfall". The "waterfall" design process called for exhaustively gathering all requirements up front, then producing a design to meet those requirements, then implementing the design, then independently testing the implementation against the requirements, fixing any bugs, retesting and eventually delivering a product to the customer. Each step of the process flows into the next, and you only go forward, much like water flowing over a series of cascades. Only the test/fix/retest/... cycle was meant to be iterative, and ideally with as few iterations as possible. Waterfall projects can take months at the least and more typically years to get through all the steps, at which point there's a significant chance that the customer's understanding of what they want has evolved -- but don't worry, we can always gather more requirements, produce an improved design ...<br />
<br />
(As an aside, Downey alludes to discussion over whether "customer" is an appropriate term for someone, say, accessing a public data website. A fair point. I'm using "customer" here because in this case this is someone paying money for a the service of producing software. The concept of open source cuts against this, but that's a whole other discussion.)<br />
<br />
The waterfall approach can be useful in situations like space missions and avionics. In the first case, when you launch, you're literally launched and there is no "iterate". In the second, the cost of an incomplete or not-fully-vetted implementation is too high to risk. However, there's a strong argument to be made that "launch and iterate" works in more cases than one might think.<br />
<br />
In contrast to waterfall approaches, agile methodologies think more in terms of weeks. A series of two-week "sprints", each producing some number of improvements from a list, is a fairly common approach. Some web services go further and use a "push on green" process where anything that passes the tests (generally indicated by a green bar on a test console) goes live immediately. Naturally, part of adding a new feature is adding tests that it has to pass, but that should generally be the case anyway.<br />
<br />
Superficially, a series of two-week sprints may seem like a waterfall process on a shorter time scale, but I don't think that's a useful comparison. In a classic waterfall, you talk to your customer up front and then go dark for months, or even a year or more while the magic happens, though the development managers may produce a series of progress reports with an aggregate number of requirements implemented or such. Part of the idea or short sprints, on the other hand, is to stay in contact with your customer in order to get frequent feedback on whether you're doing the right thing. Continuous feedback is one of the hallmarks of a robust control system, whether in software or steam engines.<br />
<br />
There are also significant differences in the details of the processes. In an agile process, the list of things to do (often organized by "stories") can and does get updated at any time. The team will generally pick a set of things to implement at the the beginning of a sprint in order to coordinate their efforts, but this is more a tactical decision, and "requirements gathering" is not blocked while the developers are implementing.<br />
<br />
Work in agile shops tends to be estimated in relative terms like "small", "medium" or "large", since people are much better at estimating relative sizes, and there's generally an effort to break "large" items into smaller pieces since people are better at estimating them. Since this is done frequently, everyone ends up doing a bunch of fairly small-scale estimates on a regular basis, and hopefully skills improve.<br />
<br />
Waterfall estimates are generally done up front by specialists. By the end of the design phase, you should have a firm estimate of how long the rest will take (and, a cynic might add, a firm expectation of putting in serious overtime as the schedule begins to slip).<br />
<br />
It's not clear how common a true waterfall process is in practice. I've personally only seen it once up close, and the result was a slow-motion trainwreck the likes of which I hope never to see again. Among other things, the process called for designers to reduce their designs to "pseudocode", which is basically a detailed description of an algorithm using words instead of a formal computer language.<br />
<br />
This was to be done in such detail that the actual coder hired to produce the code would not have to make any decisions in translating the pseudocode to actual code. This was explicitly stated in the (extensive) process documentation. But if you can explain something in that much detail, you've essentially coded it and you're just using the coder as an expensive human typewriter, not a good proposition for anyone involved. You've also put a layer of scheduling and paperwork between designing an algorithm and finding out whether it works.<br />
<br />
We did, however, produce an impressive volume of paper binders full of documentation. I may still have a couple somewhere. I'm not sure I or anyone else has ever needed to read them.<br />
<br />
This is an extreme case, but the mindset behind it is pervasive enough to make "agile vs. waterfall" a real controversy. As with all such controversy at least some of the waterfallish practices actually out there have more merit than the extreme case. The extreme case, even though it does exist in places, functions more as a strawman. Nonetheless, I tend to favor the sort of "admirable impatience" that Downey exemplifies. Like anything else it can be taken too far, but not in the case at hand.David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-14760253582748920042018-09-28T18:21:00.000-04:002019-07-16T00:56:06.751-04:00One CSV, 30 stories (for small values of 30)While re-reading some <a href="https://blog.fieldnotesontheweb.com/search/label/anonymity">older posts on anonymity</a> (of which more later, probably), and updating the occasional broken link, I happened to click through on the <a href="http://blog.whatfettle.com/">credit link</a> on my profile picture. Said link is still in fine fettle and, while it hasn't been updated in a while, and one of the more recent posts is Paul Downey chastising himself for just that, there's still plenty of interesting material there, including the (current) last post, now nearly three years old, with a <a href="http://blog.whatfettle.com/2015/10/13/scope-creep/">brilliant observation</a> on "scope creep".<br />
<br />
What caught my attention in particular was the series <i><a href="http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/">One CSV, thirty stories</a></i>, which took on the "do 30 X<something>s in 30 days" kind of challenge in an effort to kickstart the blog. Taken literally, it wasn't a great success -- there only ended up being 21 stories, and there hasn't been much on the blog since -- but purely from a blogging point of view I'd say the experiment was indeed a success.</something><br />
<br />
Downey takes a single, fairly large, CSV file containing records of land sales transactions from the UK and proceeds to turn this raw data into useful and interesting information. The analysis starts with basic statistics such as how many transactions there are (about 19 million), how many years they cover (20) and how much money changed hands (about £3 trillion) and ends up with some nifty visualizations showing changes in activity from day to day within the week, over the course of the year and over decades.<br />
<br />
This is all done with off-the-shelf tools, starting with old-school Unix commands that date back to the 70s and then pulling together various free-source from off the web. Two of Downey's recurring themes, which were very much evident to me when we worked together on standards committees, um, a few years ago, are also very much in evidence here: A deep commitment to open data and software, and an equally strong conviction that one can and should be able to do significant things with data using basic and widely available tools.<br />
<br />
A slogan that pops up a couple of times in the stories is "Making things open makes them better". In this spirit, all the code and data used is publicly available. Even better, though, the last story, <i><a href="http://blog.whatfettle.com/2014/12/02/mistakes-were-made/">Mistakes were made</a></i>, catches the system in the act of improving itself due to its openness. On a smaller scale, reader suggestions are incorporated in real time and several visualizations benefit from collaboration with colleagues.<br />
<br />
There's even a "<a href="http://blog.whatfettle.com/2014/11/07/one-csv-thirty-stories-14-hackday/">hack day</a>" in the middle. If anything sums up Downey's ideal of how technical collaboration should work, it's this: "My two favourite hacks had multidisciplinary teams build something, try it with users, realise it was the wrong thing, so built something better as a result. All in a single day!" It's one thing to believe in open source, agile development and teamwork in the abstract. The stories show them in action.<br />
<br />
As to the second theme, the whole series, from the frenetic "30 things in 30 days" pace through to the actual results, shows an admirable sort of impatience: Let's not spend a lot of time spinning up the shiniest tools on a Big Data server farm. I've got a laptop. It's got some built-in commands. I've got some data. Let's see what we can find out.<br />
<br />
Probably my favorite example is the use of geolocation in <i><a href="http://blog.whatfettle.com/2014/11/06/one-csv-thirty-stories-13-postcodes/">Postcodes</a>. </i>It would be nice to see sales transactions plotted on a map of the UK. Unfortunately, we don't have one of those handy, and they're surprisingly hard to come by and integrate with, but never mind. Every transaction is tagged with a "northing" and "easting", basically latitude and longitude, and there are millions of them. Just plot them spatially and, voila, a map of England and Wales, incidentally showing clearly that the data set doesn't cover Scotland or Northern Ireland.<br />
<br />
I wouldn't say that just anyone could do the same analyses in 30 days, but neither is there any deep wizardry going on. If you've taken a couple of courses in computing, or done a moderate amount of self-study, you could almost certainly figure out how the code in the stories works and do some hacking on it yourself (in which case, please contribute anything interesting back to the repository). And then go forth and hack on other interesting public data sets, or, if you're in a position to do so, make some interesting data public yourself (but please consult with your local privacy expert first).<br />
<br />
In short, these stories are an excellent model of what the web was meant to be: open, collaborative, lightweight and fast.<br />
<br />
Technical content aside, there are also several small treasures in the prose, from Wikipedia links on a variety of subjects to a bit on the connection between the cover of <i>Joy Division</i>'s <i>Unknown Pleasures</i> and the discovery of pulsars by <a href="https://www.scientificamerican.com/article/pulsar-discoverer-jocelyn-bell-burnell-wins-3-million-breakthrough-prize1/">Jocelyn Bell Burnell</a> et. al..<br />
<br />
Finally, one of the pleasures of reading the stories was their sheer <i>Englishness</i> (and, if I understand correctly, their Northeast Englishness in particular). The name of the blog is whatfettle. I've already mentioned postcodes, eastings and northings, but the whole series is full of Anglicisms -- <i>whilst</i>, <i>a spot of breakfast</i>, <i>cock-a-hoop, if you are minded</i>,<i> splodgy</i> ... Not all of these may be unique to the British Isles, but the aggregate effect is unmistakeable.<br />
<br />
I hesitate to even mention this for fear of seeming to make fun of someone else's way of speaking, but that's not what I'm after at all. This isn't cute or quaint, it's just someone speaking in their natural manner. The result is <i>located</i> or even <i>embodied</i>. On the internet, anyone could be anywhere, and we all tend to pick up each other's mannerisms. But one fundamental aspect of the web is bringing people together from all sorts of different backgrounds. If you buy that, then what's the point if no one's background shows through?David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com2tag:blogger.com,1999:blog-2129929182918599848.post-28516743213398040912018-05-31T15:16:00.002-04:002018-05-31T15:17:57.503-04:00Cookies, https and OpenIdI finally got around to looking at the various notices that have accumulated on the admin pages for this blog. As a result:<br />
<ul>
<li>This blog is supposed to display a notice regarding cookies if you access it from the EU. I'm not sure that this notice is actually appearing when it should (I've sent feedback to try to clarify), but as far as I can tell blogspot is handling cookies for this blog just like any other. I have not tried to explicitly change that behavior.</li>
<li>This blog has for some time used "redirect to https". This means that if you try to access this blog via http://, it will be automatically changed to https://. This shouldn't make any difference. On the one hand, https has been around for many years, all browsers I know of handle it just fine and in any case this blog has been redirecting to https for a long time without incident. On the other hand, this is a public blog, so there's no sensitive private information here. It might maybe make a difference if you have to do some sort of login to leave comments, but I doubt it.</li>
<li>Blogger no longer supports OpenID. I think this would only matter if I'd set up "trust these web sites" under the OpenId settings, but I didn't.</li>
</ul>
<div>
In other words, this should all be a whole lot of nothing, but I thought I'd let people know.</div>
David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-60487297312301990292018-05-18T02:19:00.000-04:002020-02-29T15:45:03.050-05:00Bombshell revelations about Hedy Lamarr (sorry, couldn't resist)A while ago I posted about <a href="http://blog.fieldnotesontheweb.com/2013/02/hedwig-and-mobile-web.html">the role of actor/inventor Hedy Lamarr</a> in the development of the "frequency hopping" communication technology now used in cell phones, WiFi, Bluetooth and various military applications, among others. Now (well, since last year) there's a whole movie about it: <i><a href="https://www.imdb.com/title/tt6752848/">Bombshell: The Hedy Lamarr Story</a>.</i> In fact, there's actually something of a cottage industry in Lamarr-related productions, including books, plays, magazine articles and not a few blog posts.<br />
<br />
The movie goes into detail on Lamarr's life, from her childhood in Vienna to her Hollywood career, her relationship with aviation magnate Howard Hughes (who provided access to his company's chemists and engineers for her personal projects), her now-famous patent with George Antheil, her reclusive later days and her eventual recognition for her technical contributions. There are fresh interviews with friends, admirers, children and grandchildren along with archival interviews of Lamarr herself, altogether providing a well-rounded view of a lively mind.<br />
<i><br /></i>
<i>Bombshell </i>and the Hedy Lamarr revival in general have given long-overdue appreciation to someone who until recently was most recognized for her work in an industry concerned nearly exclusively with her looks, despite her efforts to find more challenging roles and her ultimately unsuccessful attempts to break into producing. My one quibble with the film concerns the impact of her patent. Because geek.<br />
<br />
It is clear that US Navy contractors used the patent directly in the development of a communication system among ships, planes and sonobuoys and that a related system was used during the naval blockade in the Cuban missile crisis. At least some of that work was done before the patent expired, lending support to Lamarr's claim that she was due the customary payment for the use of her and Antheil's invention.<br />
<br />
It's also clear that Antheil contributed significantly to the patent, particularly because he had experience in synchronizing player pianos. The scheme used 88 frequencies, one for each key on a piano, and needed the ship's radio and the torpedo's to stay in sync. Three guesses <a href="https://en.wikipedia.org/wiki/Piano_roll">how this was accomplished</a>. Nonetheless, the initial idea of frequency hopping was Lamarr's and it was she who actively recruited Antheil for assistance with the practical design.<br />
<br />
This was a real, significant contribution to communications, and one that was put to practical use, albeit not as originally designed, including in one of the most critical moments of the 20th century. Neither was it a fluke. Lamarr had a lifelong interest in what we now call STEM and this was not her only invention, just the most successful. From what I've seen I have no doubt at all of her geek cred.<br />
<br />
However, we should be careful not to go too far, as least not without providing a bit of perspective. The film itself doesn't go so far as to say Lamarr invented WiFi. That would simply not be accurate, though that hasn't stopped the occasional web site from making just that claim. However, it does heavily imply that frequency hopping plays a major role in, for example, the security of military satellites, and it explicitly calls out WiFi, bluetooth and cell phones to claim that Lamarr's invention touches a huge swath of modern-day humanity.<br />
<br />
Yes, but ... No one person invented WiFi, which includes everything from the use of the frequency spectrum to the handshaking and security protocols, an implementation of IP (internet protocol) and the specification of metadata such as the SSID. Likewise, Lamarr's is not the only patent or invention concerning frequency hopping. None other than Nikola Tesla filed an early patent describing it. Wikipedia has a <a href="https://en.wikipedia.org/wiki/Frequency-hopping_spread_spectrum#Multiple_inventors">summary</a> of other developments of the concept, including use by the German military in World War I, well before Lamarr and Antheil's patent.<br />
<br />
WiFi certainly doesn't use piano rolls to select frequencies. Technology has marched on in the decades since then. If we're going to credit the general concept of frequency hopping, we should recognize its full history, not just Lamarr's contribution. Likewise for frequency hopping in the context of Bluetooth and CDMA in cell phones.<br />
<br />
As to security, while the patent was originally filed for a "secret communication system", frequency hopping doesn't provide meaningful security against modern attackers. Any real security is provided by encryption systems built on usual suspects like DES and RSA, which operate on completely different principles and were of course developed completely separately. To see this, look no further than WiFi, whose original security was notoriously weak, even though it included both frequency hopping and an encryption layer on top of that. Neither that layer nor the later improvements have anything to do with frequency hopping.<br />
<br />
On the other hand, another of Lamarr's original motivations was to circumvent jamming -- flooding a frequency with enough noise to drown out any real communication. Frequency hopping with a shared key -- the gist of the patent -- is still effective for that. If you don't know which of several dozen frequencies the actual signal is on, you have jam all of them, which requires proportionately more power and might not even be practical in some situations. That's very much alive, and important in military applications, but it's not a form of secrecy and it has little to do with WiFi or bluetooth, except that switching WiFi or Bluetooth frequencies can avoid accidental interference from other devices.<br />
<br />
<br />
By all accounts Hedy Lamarr was an impressive person: talented, highly intelligent, diligent, creative and in many respects ahead of her times. <i>Bombshell</i> does a fine job conveying this, and, probably more importantly, the often tragic tension between who she was and who most of the world wanted her to be. It is also forthright about her flaws and failings and it even does a decent job with the technical details, even considering the critiques above. All this is well worth recognizing and celebrating. Putting Lamarr and Antheil's patent in proper perspective does nothing to diminish this, even if it takes away the easy narrative of Lamarr the unsung hero of the mobile revolution.David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0tag:blogger.com,1999:blog-2129929182918599848.post-55570564285845572042018-04-07T22:30:00.004-04:002018-05-18T12:10:39.174-04:00Bitcoin and the B-wordIt's not hard these days to find a chart comparing the price of Bitcoin to the NASDAQ from the dotcom bubble. They line up pretty well, provided you stretch the time scale of Bitcoin out considerably on the way up and somewhat less considerably on the way down. Whether this means Bitcoin is actually in a bubble is something we don't know just yet. Bubbles are only referred to with certainty in past tense: "That was a bubble".<br />
<br />
There are arguments on either side. One that I don't buy is "The last time Bitcoin fell by X amount, it later rose by way-more-than-X amount." That's nice, but I'd kinda like to see a mechanism, not just numbers. There aren't that many 10X jumps left before Bitcoin is literally all the money in the world (I do realize that that's the endgame if you're a hardcore cryptocurrency advocate). There are interesting questions about just exactly how that would happen, but what happens after that? What does it mean for Bitcoin to appreciate another 10X over the dollar if there are no more dollars?<br />
<br />
Presumably at some point BTC rising against the dollar no longer means that Bitcoin is becoming more valuable in terms of the goods and services a Bitcoin can purchase, but that the dollar is becoming less valuable in real terms until it finally goes away. How we get hyperinflation in a currency whose supply is going to zero is an interesting question. I'm not sure there are any historical precedents here. When the major currencies went off the gold standard in the 20th century, gold was still there and still had value. If dollars go away entirely then at some point we're dividing zero by zero, which can come out to any number you like.<br />
<br />
But, for the Nth time in drafting this post, I digress down a rabbit hole of speculation over Just What Does It All Mean. The question I wanted to address here is, "Is Bitcoin in a bubble?"<br />
<br />
But I already answered that. One of the defining properties of a bubble is you don't know for sure if you're in it. It's possible, at least in principle, that the price of Bitcoin could stabilize in some fairly tight range, and if that range isn't too far off the top (somewhere north of $19K) then we had some wild fluctuations, but not a bubble. If it keeps falling toward zero, even if it eventually stabilizes around $1000, or $100, or $10 or whatever, then I think you'll have to say we had a bubble.<br />
<br />
Does it matter?<br />
<br />
I'd say not really. If you're trading BTC, the important part is how you came out. You might have made a lot of money, or lost it, or even both, but you can do that trading blue chips on the NYSE (How do you make a small fortune on Wall Street? Start with a large one.) If you're interested in Bitcoin as a store of value or a payment system or whatever, then what matters is whether that comes to pass, not the exact price of BTC when that may happen.<br />
<br />
People throw around terms like "correction", "crash" or "bubble collapse" mainly, I think, to express an opinion about what's going to happen, not what has. A <i>correction</i> implies that things got a bit out of hand but everything will be fine in the long run. A <i>crash</i> means prices fell far and fast, but doesn't really say anything about the future. A <i>bubble collapse</i> says that not only did prices fall, but there was much less ever there than people thought at the time. If prices ever do recover, it won't be for a good long time, and for much different reasons than drove the runup to the bubble bursting. Which of those is happening now depends on who you ask (though it's hard to call a 60%+ drop off the top a correction with a straight face).<br />
<br />
I think that, along with the previous post, is about all I want to say about Bitcoin for a while. I note that it's been a prevalent topic here for a while -- not that I've been posting that much on anything here -- and there's got to be other web.stuff to talk about.<br />
<br />David Hullhttp://www.blogger.com/profile/07602323703256325141noreply@blogger.com0