Please leave us a 5-star review

It's been long enough that I can't really say I remember for sure, and I can't be bothered to look it up, but as I recall, reviews were supposed to be one of the main ways for the web to correct itself.  I might advertise my business as the best ever, even if it's actually not so good, but not to worry.  The reviewers will keep me honest.  If you're searching for a business, you'll know to trust your friends, or you'll learn which reviewers are worth paying attention to, good information will drive out bad and everyone will be able to make well-informed decisions.

This is actually true, to an extent, but I think it's about the same extent as always.  Major publications try to develop a reputation for objective, reliable reviews, as do some personalities, but then, some also develop a reputation for less-than-objective reviews.  Some, even, may be so reliably un-objective that there's a bit of useful information in what they say after all.  And you can always just ask people you know.

But this is all outside the system of customer reviews that you find on web sites all over the place, whether provided by the business itself, or companies that specialize in reviews.  These, I personally don't find particularly useful or, if I were feeling geekly, I'd say the signal/noise ratio is pretty low.  It turns out there are a couple of built-in problems with online reviews, that were not only predictable, but were predicted at the time.

First, there's the whole question of identity on the internet.  In some contexts, identity is an easy problem: an identity is an email address or a credit or debit account with a bank, or ownership of a particular phone, or something similar that's important to a person in the real world.  Email providers and banks take quite a bit of care to prevent those kind of identities from being stolen, though of course it does still happen.  

However, for the same reason, we tend to be a bit stingy with this kind of identity.  I try hard not give out my credit card details unless I'm making an actual purchase from a reputable merchant, and if my credit card details do get stolen, that card will get closed and a new one opened for the same account.  Likewise, I try not to hand out my personal email or phone number to just anyone, for whatever good that does.

When it comes to reviews, though, there's no good way to know who's writing.  They might be an actual customer, or an employee of the business in question, or they might be several time zones away writing reviews for money, or they might even be a bot.   Platforms are aware of this, and many seem to do a good job of filtering out bogus reviews, but there's always that lingering doubt.  As with identities in general, the stakes matter.  If you're looking at a local business, the chances are probably good that everyone who's left a review has actually been there, though even then they might still have an axe to grind.  In other contexts, though, there's a lot more reason to try to game the system.

But even if everyone is on the up-and-up and leaving the most honest feedback they can, there are still a few pitfalls.  One is selection bias.  If I've had a reasonably good experience with a business, I'll try to thank the people involved and keep them in mind for future work, or mention them if someone asks, but I generally don't take time to write a glowing review -- and companies that do that kind of work often seem to get plenty of business anyway.

If someone does a really horrible job, or deals dishonestly, though, I might well be in much more of a mood to share my story.  Full disclosure: personally I actually don't tend to leave reviews at all, but it's human nature to be more likely to complain in the heat of the moment than to leave a thoughtful note about a decent experience, or even an excellent experience.  In other words, you're only seeing the opinions of a small portion of people.  That wouldn't be so bad if the portion was chosen randomly, but it's anything but.  You're mostly seeing the opinions of people with strong opinions, and particularly, strong negative opinions.

The result is that reviews tend to cluster toward one end or the other.  There are one-star "THIS PLACE IS TERRIBLE!!!" reviews, there are five-star "THIS PLACE IS THE MOST AWESOME EVER!!!" reviews, and not a lot in between.  A five-point scale with most of the action at the endpoints is really more of a two-point scale.  In effect, the overall rating is the weighted average of the two: the number of one-star reviews plus five times the number of five-star reviews, divided by the total number of reviews.  If the overall rating is close to five, then most of the reviews were 5-star.  If it's 3, it's much more likely that the good and the bad are half-and-half than most of the reviews being 3-star.

The reader is left to try to decide why the reviewers have such strong opinions.  Did the car wash do a bad job, or was the reviewer somehow expecting them to change the oil and rotate the tires as well and then get angry when they didn't?  Is the person praising a consultant's integrity actually just their cousin?  Does the person saying that a carpenter did a great job with their shelves actually know much about carpentry or did they just happen to like the carpenter's personality?  If the shelves collapse after a year and a half, are they really going to go back and update their review?  Should they, or should they maybe not store their collection of lead ingots from around the world on a set of wooden shelves?

Specifics can help, but people often don't provide much specific detail, particularly for positive reviews, and when they do, it's not always useful.  If all I see is three five-star reviews saying "So and so was courteous, professional and did great work", I'm not much better off than when I started.  If I see something that starts out with "Their representative was very rude.  They parked their truck in a place everyone in the neighborhood knows not to park.  The paint on the truck was chipped.  Very unprofessional!" I might take what follows with a grain of salt.

There's a difference, I think, between an opinion and a true review.  A true review is aimed at laying out the information that someone else might need to make a decision.  An opinion is just someone's general feeling about something.  If you just ask people to "leave a review", you're going to get a lot more personal impressions than carefully constructed analyses.  Carefully constructing an analysis is work, and no one's getting paid here.

Under the "wisdom of crowds" theory, enough general impressions will aggregate into a complete and accurate assessment.  A cynic would say that this is like hoping that if you put together enough raw eggs, you'll end up with a soufflé, but there are situations where it can actually work (for a crowd, that is, not for eggs).  The problem is that in many cases you don't even have a crowd.  You have a handful of people with their various experiences and opinions.

This all reaches its logical conclusion in the gig economy.  When ride share services first started, I used to think for a bit about what number to give a driver.  "They were pretty good, but I wish they had driven a bit less (or in some cases maybe more) aggressively".  "The car was pretty clean, but there was a bit of a funny smell" or whatever.

Then I started noticing that almost all drivers had 5-star ratings, or close.  The number before the decimal point doesn't really mean anything.  You're either looking at 5.0 or 4.something.  A 4.9 is still a pretty good rating, but a 4.0 rating is actually conspicuously low.  I don't know the exact mechanics behind this, but the numbers speak for themselves.

It's a separate question to what extent we should all be in the business of rating each other to begin with, but I'll let Black Mirror speak to that.

Following all this through, if I give someone a 4-star review for being perfectly fine but not outstanding, I may actually be putting a noticeable dent in their livelihood, and if I give someone 3 stars for being pretty much in the middle, that's probably equivalent to their getting a D on a test.  So anyone who's reasonably good gets five stars, and if they're not that good, well, maybe they were just having a bad day and I'll just skip the rating.  If someone actively put my life in danger, sure, they would get an actual bad rating and I'd see if I could talk to the company, but beyond that ... everyone is awesome.

Whatever the reasons, I think this is a fairly widespread phenomenon.  Reviews are either raves or pans, and anyone or anything with reviews much short of pure raves is operating at a real disadvantage.  Which leads me back to the title.

Podcasts that I listen to, if they mention reviews at all, don't ask "Please leave a review so we can tell what's working and what we might want to improve".  They ask "Please leave a 5-star review".  The implication is that anything less is going to be harmful to their chances of staying in business.  Or at least that's my guess, because I've heard this from science-oriented podcasts and general-interest shows that clearly take care to present their stories as objectively as they can, the kind of folks who might genuinely appreciate a four-star review with a short list of things to work on.

This is a shame.  A five-point scale is pretty crude to begin with, but when it devolves to a two-point scale of horrible/awesome, it's not providing much information at all, pretty much the opposite of the model that I'm still pretty sure people were talking about when the whole ratings thing first started.

Defending your reputation (for a small fee)

Some time back, when I had a somewhat different vision of this blog, I ruminated about how one might model reputation.  Whether or not the model is any good, taking some time to think about what reputation might be was a useful exercise.  Re-reading the posts in that thread, one of the more useful observations was:

We try to control our reputations (at least)

• through our actions
• by controlling access to information about us
• by influencing people's interpretation of the information (we think) we know

Of those, we have the most control over the first, though perhaps more effort is devoted to the third.  The second has its own special quirk:  It's possible for information to disappear from the web, if all permanent copies can be removed, but the safe assumption is that information only accumulates.

Nonetheless, there are companies in the business of helping people control access to information, and thereby their reputations.  A fool's errand?  Probably not.  There are several services that reputation protection services can and do provide:

• Monitoring what you look like on the web.  If someone posts something slanderous about you, you may not find out until it's too late, unless you're constantly monitoring the web -- or have someone doing it for you.
• There are various online lists and databases that you can sometimes have your personal details purged from, but who has the time?
• You can't erase information from the web, particularly if it's a rumor that's already spread far and wide, but you can respond and try to counter it.  In this sense, protecting a reputation is just old-fashioned PR.
• If you choose to, say, put all your purchases and reading selections and reading up for your friends to peruse, you might want to use a different identity to mention that you're reading World Domination in Six Easy Evil Steps or to purchase that 1.21 gigawatt laser.  But if you don't already know that, a service may not be of much help.

What's less clear to me is how much any of this is worth to private individuals.  If your ex has just posted those embarrassing videos of you from the last christmas party, it's not going to help much to learn about it in a report form your reputation service.  It would seem it's the PR function that's most useful in such cases, but unless you're directly in the public eye you probably don't have call for that.   If you do need it, you're not going to get it online for a small monthly fee.

I'd liken it to search engine optimization.  If you're doing serious business online, you definitely want it, along with real marketing expertise.  If you're blogging in a web.backwater, probably not so much.

Or so I hope.

Some kinds of reputation

I've argued that reputation is an inherently subjective assessment of a set of assertions associated with a persona (I need a less-jawbreaking way to say that). That doesn't mean that there's no point in trying to model it, just that there's no single or objective model.

On the other hand, there are any number of real-world examples of reputations, some of which already have well-known models. Some that come to mind:

eBay: Michael O'Connor Clarke talks about this in his piece on personal reputation management. In eBay, your reputation is a summary of the positive and negative feedback you've received over the course of your transactions. There are various checks and balances that make the system work (e.g., if you bash someone needlessly, it will catch up to you).

Clarke states that "It's your reputation, but you don't even own it. If you ever chose to leave eBay, you can't take it with you."

I'd basically agree but I might put it differently. Your eBay reputation is a summary of the history of your eBay persona. If you can convince someone that, say, your Blogger persona and your eBay persona are both you, then you've convinced them that, say, the person who posted "I just bought a new camera on eBay" has a 98% positive rating on eBay. If you leave eBay, you lose a means of convincing someone that you made those particular transactions and got those particular feedback votes.

If you can carry those assertions with you, you can carry your reputation with you. For example, a subculture of eBay users could all agree to send appropriately signed records of their transactions and feedback to a trusted third party as they make them. This slice of your eBay reputation would then be portable (assuming eBay doesn't take this sort of thing amiss).

Blogrolls and social networking sites: Clarke mentions these as well. Blogrolls are just a less formal version of the same basic idea. Again, the problem in carrying over reputations from one service to the next is mostly a matter of linking up the personae.

I could imagine a sort of "meta-network" service where I can tell the meta-service that I've joined a new service, giving it my login, password etc. The meta-service can then look up who else it knows that has already joined that service and who is connected to me in other services I belong to and, subject to rules I specify, invite them on my behalf.

Similarly, it can accept invitations on my behalf from people it knows I've accepted invitations from in other services. It can monitor my various services and propagate changes across them. E.g., if I blogroll someone on Blogger, it could automatically invite the other person on LinkedIn, and if they also subscribe, accept the invitation on their behalf.

This seems somewhat different from OpenID, in that the service stores connections as well as identities, and from FOAF, which appears more concerned with marking up existing pages to make them more easily machine-readable. As always, though, I may be slow on the uptake here.

The various public-key webs of trust: This is probably what I was thinking about here, when I said "It occurs to me that there is a different, more Web 2.0-ish notion of reputation as a network of people's ratings, other people's ratings of their ratings, and so forth." Each key is a persona, and the trust scheme of the system you're using accumulates the various assertions known about a persona into an assessment of the persona -- its reputation, in other words.

Poker: This is a world where you literally buy and sell your reputation. Each bet you make says something about how you play. Do you bluff a lot? Are you likely to call someone else's bluff? Do you play every hand or do you fold all but the best? Every time you act, you provide information for the other players' answers to those questions, and that in turn affects how they will play against you and thus affects your winning chances.

Are there other poker-like situations on the web, outside gaming? I'm sure there must be, but they're not coming to mind at the moment.

Recommendation lists on e-commerce sites: These aren't so much reputations in the usual sense of trustworthiness, but they are assessments of assertions about a persona. The user ID is the persona and the assertions record the purchasing history and other actions of that user. The site then uses its own secret sauce to cook these up, often together with the histories of other users, into a list of recommendations.

Refining the persona/reputation model

Previously I asked whether it was worth breaking up the M:N relation between personae and attributes. My personal bias is to break these up whenever there's a natural name for the resulting link or (and this is closely correlated) whenever the link might naturally have relations outside the two linked entities. My further bias is that this is generally true.

I also asked whether opinions are different from other assertions.

I think these can both be answered by dropping attribute, action and opinion, adding

• An assertion is a claim made by a persona

and reworking the relations as follows:

• A person (1) assumes an identity (N)
• An identity (1) is assumed at a time (N)
  
• An identity (N) is as a persona (1)
• A persona (1) makes an assertion (N)
• An assertion (N) is made at a time (1)
  
• An assertion (1) is about a persona (0 .. N)
  

Instead of attributes, actions and opinions, we have various assertions, e.g.,

• I assert (at time t) that the sky is blue (plain assertion, not about anyone)
• I assert that I am handsome (one way of bestowing an attribute)
• I assert that my friend is homely (another way)
• My friend asserts that my opinion can't be trusted
• My bank asserts that I made a deposit this morning (assertion of an action)
• I assert (at time t+1) that the sky is not blue (changing my opinion)

This model is probably missing a key ingredient or two, particularly a means of granting access to assertions, but it seems enough to support reputations. E.g., if I assert that I have $1,000,000 in the bank but my bank asserts I have $0, that will influence my reputation. Reputations can be built both by tying assertions to the outside world (You're convinced that my bank is trustworthy) or by studying the relationships between them (X has a history of lying about his bank account).

Persona, reputation and VRM

One thing that should jump out from the previous post on modeling reputation is that there's nothing called "reputation" in the proposed model. This is not a complete accident. Reputation is a very subjective thing, both in that one's reputation will vary depending on whom one asks, and in that it can be reckoned in any number of different ways. What the model models is not reputation itself, but the elements from which one might derive a reputation.

There is, however, a very prominent piece called "persona". From a VRM standpoint, as I understand VRM, this is the piece we want to break out and re-use. In the stock example of movie preferences, I might like to share my "movie buff" persona amongst the IMDB and the various online vendors, so that whenever I notice something anywhere and comment on it, or rent a movie, everyone knows about it. Renting or reviewing a movie is an action on the part of my "movie buff" persona.

Under this model, I will have different reputations with different vendors, even if they see the exact same information about my persona, because they will draw different conclusions from the same facts.

It occurs to me that there is a different, more Web 2.0-ish notion of reputation as a network of people's ratings, other people's ratings of their ratings, and so forth. There is probably a useful analogue to page rank in web sites to be had here, for example.

That's cool stuff, but for my money it's one particular way of deriving a reputation from the public acts of a persona.

Modeling reputation

Joe Andrieu has some interesting points about reputation on the web, among them that no one seems to have a concrete model for this crucial concept.

Going on that basis, and probably re-inventing a few wheels along the way, here are some thoughts:

• You have a reputation with someone as something
• A reputation is another person's interpretation of what they associate with you, including
  
• Attributes ("Statutory")
  
• Actions ("Case-based")
• Your reputation with third parties and their reputations as judges.
  
• Everyone will have a different view of those things, and even if two people somehow had the same view, they will almost certainly interpret it differently.
• We try to control our reputations (at least)
• through our actions
• by controlling access to information about us
  
• by influencing people's interpretation of the information (we think) we know
  
• One way to control this access is to take on a persona. The most obvious way is via a pseudonym, but subtler forms are also common. For example, we tend to separate, say, our business correspondence from our personal correspondence.
• This control is very incomplete.

A bit more on personae:

• A given person will take on many personae
• Some personae are taken on by many people, either sequentially (The Dread Pirate Roberts) or concurrently (Bourbaki)
• The association between a person and persona is an independent entity. If nothing else, it has a life cycle (on this day, Peter Parker became Spiderman)
• The knowledge (or insinuation) that two personae are the same person can make a drastic difference in certain cases (X is just a sock puppet for Y; Someone on a company's message boards is actually the CEO)

Translating this into old high ERM, here are some entities that appear to fall out:

• A person is a flesh-and-blood person.
• A persona is a public image of a person.
• An identity is the association between a person and a persona.
• An attribute is something the owner of a persona asserts to be true about it.
• An action is something a persona is publicly seen to have done.
• An opinion is something one persona asserts to be true about another persona (rendering an opinion is an action).

and some relations:

• A person (1) assumes an identity (N)
• An identity (1) is assumed at a time (N)
  
• An identity (N) is as a persona (1)
• A persona (M) carries attributes (N)
• A persona (1) takes an action (N)
• An action (1) occurs at a time (N)
  
• A persona (1) renders an opinion (N)
• An opinion(N) concerns a persona (1)
• A persona (1) may change an opinion (N)

This is only a very basic model, of course, and there may well be other models that deal with reputation as well or better. Some questions:

• Is it worth breaking up the M:N relation between personae and attributes?
• Is rendering an opinion any different from making any other assertion? Probably not.
• What's the best way to handle group reputations. In a case like Bourbaki, there is only one persona and the present model will handle it. In a case like "the legislature is incompetent", both the reputations of the legislators and of the legislature as a whole are involved.