On web standards and social networking

I started this blog, um, a few minutes ago, back when I was involved with a couple of the committees involved in developing standards for the web, and it seemed like one thing a person in my position was supposed to do was to blog about it, whether to try to make one's expertise generally available, or to promote one's brand or consulting services, or to become visible as part of a group of similar individuals (the term "blogroll" comes to mind), or to help set the future direction of the web by presenting brilliant analyses or ... something of that nature.

Fortunately, I soon separated the blog from my professional life and settled into just putting out whatever thoughts I had as I wandered the web.world, without a lot of concern for what, if anything, it might all lead to.  In the event, it hasn't led to all that much ... a few dozen followers, the occasional comment, and readership well below even thinking about trying to monetize anything ... but it has been a satisfying experience nonetheless.  I've learned all sorts of things, and writing about it has given me the opportunity to think things through, organize my thoughts and maybe even get better at writing.  The whole standards-committee thing seems long ago and far away.

Imagine my surprise, then, when an Ars Technica article on Mastodon popped up in my newsfeed and brought it all back.

When we last left our story, I was musing about how recent turbulence concerning cryptocurrencies and social media, and Twitter in particular, had gotten me thinking about what "web" even meant anymore and concluding that at least the core of the web, namely its interconnections, was alive and well, and also that it didn't have all that much to do with the turbulence.  Before that, I had said that there didn't really seem to be all that much to blog about.  Things were happening, but not a lot of things that I had much to say about.

But Mastodon is interesting, not just because of its role as a potential "giant killer" for Twitter -- I'm still not inclined to speculate on how that might shake out -- but because of what it is and how it was put together, both in its structure and in the process that led to it.  Allow me to take a walk down memory lane:

When I was involved in the standards process, the typical web standard went through a life cycle something like this:

• Someone, typically one of the Major Players, would create some new protocol or class of application to try to take advantage of the opportunities to be had on the rapidly expanding web.  This was after the major protocols like TCP, FTP, HTTP and the various email protocols were widespread.  Those tended to be less blatantly commercial, even though major corporations were generally involved.
• If the idea looked useful, other players, large and small, would try to work with it, using whatever documentation was available.
• This would often lead to a mess.  It's hard to write clear, complete documentation and it's almost guaranteed that once other people start using a system they'll try things that the original authors hadn't thought of.  When it's not clear what should happen in a particular situation, people will take their best guess, and different people will guess differently.
• People will also want to add new features to cover cases that the original authors hadn't thought of, or did think of but didn't have time to implement, or implemented in what seemed like a less-than-optimal way, or whatever.

In some ways, this is a good problem to have, since it means people are actually using the system, probably because it meets some need that hadn't been addressed before.  The flip side, though, is that in a networked world a system only works, or at least only works well, if people agree on how to use it.  If my version of AwesomeNet thinks that the command for "make everything awesome" is MKAWSM and yours thinks it's make-awesome, then chances are things won't be so awesome.

There are plenty of other minor disagreements that can happen, either because different people made an arbitrary choice differently, or because there's a genuine disagreement as to the best way to do something, or for any number of other reasons.  If this happens enough to be a real problem, there's usually a call for everyone to get together and create an agreement on which way to do things.  A standard, in other words.

Writing standards is a tricky thing.  You want to capture what people are currently doing, but in an abstract enough way to avoid just having a laundry list of things people currently do.  You want to establish boundaries that are tight enough to rein in the chaos, but not so tight to prevent people from finding new and better ways of doing things.  There are often several ways to do this.  For example, for the difference in command names you might

• Just pick one, say make-awesome, and "deprecate" the other, meaning that implementations of AwesomeNet aren't required to support it and if your code does use it, you should update your code.
• Pick one as preferred, require implementations to understand the other one, but forbid them from using it.  That means that a standard implementation of AwesomeNet will never use MKAWSM, but if someone talking to it does, it will understand (this is an example of Postel's principle).
• Allow both, usually by defining one as an "alias" for the other, so that most of the standard text can just talk about make-awesome, except for the part that says "you can use MKAWSM anywhere you can use make-awesome"
• Define an "abstract syntax" and talk about <make awesome> as a placeholder for "whatever name the implementation chooses to use for the make everything awesome command".  This means that there will need to be some sort of protocol for two implementations to tell each other which names they actually use.  That's probably not worth the trouble in most cases, but it's an option nonetheless.
• If "make everything awesome" is just a convenient shorthand for a bunch of other commands, define a general way of defining shorthands like that (often called "macros") and leave it out of the "core standard" entirely.  There are lots of ways to implement macros, since a macro language is essentially a programming language (even if it's not functionally complete), so this is probably only a good idea if at least some of the existing implementations already support it.
• There are probably several other approaches I'm not thinking of.

There are similar issues with features that only some of the existing implementations support.  Do you require everyone to support everyone's features?  Do you define a "core set" and make the rest optional?  Do you say that some existing features aren't allowed in the standard, so implementations that have them have to drop them?  Or maybe something else?

Another important question is "what happens in this particular special case?".  Sometimes the answer is "it's this particular behavior that you might not have thought of, and here's why".  Sometimes the answer is "that's an error, and the implementation should report it like this".  Sometimes, though, the answer is "the behavior is unspecified", either to leave room for new features or for the practical reason that different existing implementations do different things, or for whatever other reason.  This is more useful than it might seem.  It tells implementers that it's OK to do whatever's easiest, and not to expect any particular behavior in certain situations (and maybe try to avoid those situations altogether).

There are lots of other principles and techniques that experienced standards writers employ (learning some of these as a not-so-experienced standards committee member was one of the best parts of the experience).  One that sticks in mind is the principle that implementations should be able to safely ignore things they don't understand, so that it's safe to interact with something that implements a newer version of the standard.

In a typical standards process, there are dozens of questions to hammer out, and the answers are often interconnected.  It's a tricky technical problem and, of course, there are political considerations as well.  In my experience the politics are typically pretty civil, but most participants come in with certain "red lines" they won't cross, usually because it would cost more to implement them than the standard is worth to them since there are already customers out there using the not-yet-standardized products.  The usual approach is to stand firm on those and be prepared to be flexible on anything that doesn't cross a red line.  If two or more participant's red lines overlap, things can get tricky.

After all the meetings, if all goes well, the end result of this is a carefully crafted standards document full of MUST [NOT] and SHOULD [NOT], which are so important that there's actually a standard defining what they mean.

The new standard captures everything the committee could capture about what it means to implement and use whatever's being standardized.  With luck, the standard is at least self-consistent, but there will be parts that are deliberately left unspecified for reasons like those given above.  There will also be plain old mistakes, ideally small ones that can be corrected by errata (corrections, ideally short and specific, published in a separate section or document) but sometimes larger ones that will require more meetings and a new version of the standard.

Since standards need to be flexible, a standard doesn't always answer every question you'll need answered when coding to it.  Because of this, it's common for a separate "interoperation committee" to form once a standard is out.  These committees essentially fill in the blanks to create "profiles" that say how to use a particular standard in particular situations.  There might be a lightweight profile that's meant to provide only the core features so it's easier to implement and run.  There might be a security-oriented profile that specifies features to enable and parameters to use ("digital signatures are required for each message and private keys must be at least 2048 bits").  There might be a profile aimed particularly at business use, or whatever.

I've gone through all this in detail because, judging by the Ars article, all of these things either have happened with Mastodon, or may happen at some point because the same basic issues have come up.  More than that, Mastodon and similar services are, in fact, based on the ActivityPub standard, which specifies two protocols, one for clients to talk to instances  to interact with an inbox and outbox, and one for instances to send traffic to each other.

(An instance is a server in the sense of the client-server model, but not in the sense of a particular process or a server machine -- at least in theory, an instance could run on multiple physical servers and a single physical server could host multiple instances)

ActivityPub is meant to be very general-purpose, but in the context of social networking only some of the features are really important, so implementations in the Fediverse (the world of Mastodon and Mastodon-like services that talk to each other) will naturally tend to pay more attention to those features, which is probably fine until someone tries to take advantage of other features.  The standard for data encryption and signatures is not officially part of the system, so implementations that support it still have to deal with unencrypted data.  ActivityPub relies on a standard called WebFinger to discover what instances are out there, so implementing a real ActivityPub instance also means dealing with WebFinger.

And so on.

All of this is perfectly normal in the world of web standards.  A standard is meant to reduce uncertainty and make it clear where the remaining uncertainty is, and establish common vocabulary and practices.  It isn't meant to solve all problems with interoperation -- it's great if it can, but that's not always possible.  It isn't meant to answer all questions an implementer might have, and in fact it deliberately doesn't answer some questions so that implementers will be free to come up with better answers.  In general a standard is more about what than how (though a what in one context can be part of the how in another and vice versa).

In any case, ActivityPub only talks about how instances talk to each other and how clients interact with instances.  It doesn't, and shouldn't, talk about how people using it for social networking should interact.  That's a matter of community guidelines, that is, what the people running the actual instances will or won't allow on them.  Since the instances are deliberately federated, rather than owned by one entity as with Twitter, these decisions will be, too.

One particularly interesting point is whether posts to one instance can be kept private to that instance.  This proved contentions, so Hometown was forked off of Mastodon.  Hometown allows posts to be private to a particular instance.  Mastodon doesn't.

The good news, I think, is that federated services built on web standards, while messy in real life, can and do work.  Mastodon's success or failure depends more on how community guidelines shake out and whether people prefer to spend their time there than it does on the strength of the standards.  I only spent a lot of time on standards in this post because it was striking to me how much of what I learned back in those days is still relevant.

One other thing struck me while pondering the article.  This all seems a whole lot like Usenet, just with different underlying technology.

Usenet used an email-based model where a particular user could subscribe to various newsgroups, receiving emails as updates came in, and publish messages on those groups, either new messages or replies to a thread that was already going.  The main technical difference is that early usenet relied on a old protocol called UUCP (Unix-to-Unix Copy) to transfer files of messages point-to-point between sites (often but not always universities or corporations).  UUCP isn't used much any more, but, like many old standards, it is still in use here and there.

The main externally visible difference, besides the use of email, was that the newsgroups were owned by administrators (these might not be the server administrators, though server administrators could effectively veto a group by filtering it out).  Rather than tag a post with #this or #that, you'd post to comp.sci.this or alt.that.discuss or whatever.

From a social point of view, though, the similarities seem significant.  Just as site administrators could establish rules, such as by blocking people from posting or filtering out newsgroups, Mastodon instance owners can establish their own standards.  Just as different people would have different ideas as to what sort of discussion was acceptable and different sites and different newsgroups could have different standards, different instances have different rules of conduct.

Newsgroups allowed people with similar interests to meet up and discuss things they were interested in.  This included not only technical topics like scientific specialties or programming languages, but social topics like music and art and a fair bit of quite gamy content.  They also allowed otherwise marginalized people to gather together, and for the curious to find out a bit about an unfamiliar group of people, but also for trolls and worse to barge in if they could get past the admins.

In other words, from a social point of view, they had pretty much all the features, good and bad, of modern social networking communities.  Whether you want to see that as a glass half full -- people keep finding ways to get together -- or half empty -- even after decades we're still up against the same problems -- or a bit of both -- my general inclination -- is up to you.  If nothing else it might provide a counterweight to claims that any of what's going on now is unprecedented.

There's one other technical point that jumped out, though I doubt it will interest many people.  Two patterns of communication come up over and over again in distributed services.  One is the request/response pattern: I ask you a question or to do something, you respond with an answer or "I did it (and this happened)"/"I couldn't do that (for this reason)"/....  The other is the publish/subscribe pattern (pub/sub for short), where any number of parties can publish a message on a topic and any of those (including the publisher) can subscribe to messages on that topic.  Those aren't the only possibilities, but they account for a lot of traffic.  ActivityPub, as the name might suggest, is a pub/sub protocol.

The whole point of pub/sub is that publishers and subscribers don't know about each other, or whether any of them even exist.  I can publish to a topic that no one's listening to, and, depending on the type of service, either that message will be dropped on the floor, or it will be saved for delivery to anyone who does eventually subscribe.  In either case, each subscriber will see its own copy of the message.  The only difference is whether "each subscriber" means "each subscriber that was there when the message was published (more or less)" or "each subscriber that ever subscribes to this topic".

Since publishers and subscribers don't know about each other, there has to be some sort of intermediary.  For a large system, there will actually be many intermediaries communicating with each other in order to make sure published messages are delivered to subscribers.   Publishers and subscribers talk point-to-point with those intermediaries.  I tend to think of this as "I publish to the cloud, whatever's in the cloud does whatever it does, and the subscriber gets the message from the cloud".

Mastodon and company fit this perfectly: You send a message to the instance you're using, it talks to other instances, and people see your message.

The old Usenet followed the exact same pattern, just that the mechanism for the servers to talk to each other was quite a bit different.