Common passwords are bad ... by definition

It's that time of the year again, time for the annual lists of worst passwords.  Top of at least one list: 123456, followed by password.  It just goes to show how people never change.  Silly people!

Except ...

A good password has a very high chance of being unique, because a good password is selected randomly from a very large space of possible passwords.  If you pick your password at random from a trillion possibilities*, then the odds that a particular person who did the same also picked your password are one in a trillion, the odds that one of a million other such people picked your password are about one in a million, as are the odds that any particular two people picked the same password.  If a million people used the same scheme as you did, there's a good chance that some pair of them accidentally share a password, but almost certainly almost all of those passwords are unique.

If you count up the most popular passwords in this idealized scenario of everyone picking a random password out of a trillion possibilities, you'll get a fairly tedious list:

• 1: some string of random gibberish, shared by two people
• 2 - 999,999: Other strings of random gibberish, 999,998 in all

Now suppose that seven people didn't get the memo.  Four of them choose 123456 and three of them choose password.  The list now looks like

• 1: 123456,  shared by four people
• 2: password,  shared by three people
• 3: some string of random gibberish, shared by two people
• 4-999,994:  Other strings of random gibberish, 999,991 in all

Those seven people are pretty likely to have their passwords hacked, but overall password hygiene is still quite good -- 99.9993% of people picked a good password.  It's certainly better than if 499,999 people picked 123456 and 499,998 picked password, two happened to pick the same strong password and the other person picked a different strong password, even though the resulting rankings are the same as above.

Likewise, if you see a list of 20 worst passwords taken from 5 million leaked passwords, that could mean anything from a few hundred people having picked bad passwords to everyone having done so.  It would be more interesting to report how many people picked popular passwords as opposed to unique ones, but that doesn't seem to make its way into the "wow, everyone's still picking bad passwords" stories.

From what I was able to dig up, that portion is probably around 10%.  Not great, but not horrible, and probably less than it was ten years ago.  But as long as some people are picking bad passwords, the lists will stay around and the headlines will be the same, regardless of whether most people are doing a better job.

(I would have provided a link for that 10%, but the site I found it on had a bunch of broken links and didn't seem to have a nice tabular summary of bad passwords vs other passwords from year to year, so I didn't bother)

*A password space of a trillion possibilities is actually pretty small.  Cracking passwords is roughly the same problem as the hash-based proof-of-work that cyrptocurrencies use.  Bitcoin is currently doing around 100 million trillion hashes per second, or a trillion trillion hashes every two or three hours.  The Bitcoin network isn't trying to break your password, but it'll do for estimating purposes.  If you have around 100 bits of entropy, for example if you choose a random sequence of fifteen capital and lowercase letters, digits and 30 special characters, it would take a password-cracking network comparable to the Bitcoin network around 400 years to guess your password.  That's probably good enough.  By that time, password cracking will probably have advanced far beyond where we are and, who knows, maybe we'll have stopped using passwords by then.

Now can we stop the password madness?

I've ranted about this plenty of times, and now it seems like the world has come around to my point of view.

Um yeah, right.

I think pretty much anyone who's had to deal with restrictions like "This password must be eight characters long, contain at least one number, one uppercase letter, one lowercase letter, one special character and the characters Pa$$w0rd in order" has recoiled in disgust.  So maybe it wasn't my vast influence.

In any case, headlines are now circulating that the person who promulgated those rules (one Bill Burr of NIST) has said "Sorry, it was all a horrible mistake."  So the person responsible has fessed up and the annoying rules should be history in, oh, let's say ten or twenty years.

As usual, I think the real story is a bit more nuanced, as they say, but it looks like the Naked Security blog at Sophos has already done a better piece on it than I will.  Basically, the advice in the original guidelines in 2003 wasn't bad at the time and it's not Bill Burr's fault that people cargo-culted it into the annoying mess we see today.

Now if we can just get rid of "security questions" ...

That CAPTCHA moved!

While recovering a password for a site -- that is, my real password was whatever information the recovery page needed -- I noticed a new wrinkle on CAPTCHA: Moving CAPTCHA.  Instead of the usual smeared-out or obscured letters, three plainly readable letters, somewhat tilted, on a clearly contrasting background, but wiggling slightly back and forth.

Seems like an interesting step in the whole OCR arms race, except ...

The problem for an attacker to solve here isn't recognizing a moving character, which might or might not be harder than recognizing a still one.  It's grabbing a frame of the animation to examine.  If you can do that at all, then recognizing one particular arrangement of the letters is no harder than recognizing any other CAPTCHA.  Easier, in fact, since you have nice, legible letters, and you can re-run the OCR on each frame and go with the consensus.

Again, I haven't looked at this in detail, but there would seem to be two main ways of putting the moving image up in the first place: A .gif or other animated image format, which is no problem to decode into its images, or some sort of JavaScript animation.  That might be harder to grab, but not because of the animation.  You can just as well use JavaScript to put up a still image, and in either case the answer is to render the JavaScript and then grab the pixels.

In other words, it seems unlikely that the moving image adds any real difficulty for an attacker.  It does look harder, intuitively, to the human eye, but the attacker isn't using a human eye -- that's the whole point of the exercise to begin with.

Password reductio ad absurdum

I was just now logging into a site I hadn't logged into in several months, one for which I wanted to be sure I had a unique password.  Naturally, I'd forgotten the password.  So I clicked on the forgotten password link and chose the email option.  There was also a security questions option.  I should remember to make up some random lies for that, since I'm not going to use it and would prefer no one else did either.

Before too long, an email arrived with a clearly randomly-generated sequence of twelve upper- and lowercase letters.  That's about 68 bits of entropy.  If you could guess a trillion passwords a second [which, scarily enough, is not at all far-fetched], it would still take about 12 years to guess all the possibilities.  I'm not a great fan of passwords in general, except when used locally to unlock something that's actually secure, but that's a pretty reasonable password generation scheme.

So I log in with my new password.  Before I actually get in to the site, I'm told I need to change my password.

Because it's too weak.

Because it doesn't have a letter and a number.

But I'm free to make up any seven-character or longer sequence that does contain a number and a letter,  which does at least filter out all but two of SplashData's top 25 list of weak passwords (all but trustno1 and password1).  Let's just say it's 92% effective at improving password security and leave it at that.

Yet another wacky security scheme

Passwords are easy to get wrong.  Trying to make people come up with "stronger" passwords just makes it worse.  Security questions just provide another avenue of attack, probably an easier one.  So, ladies and gentlemen, may I introduce to you: The security word.

"What is it?", you may later regret asking.

You give the site a "security word".  Later, they will ask you not for the word, but a few randomly selected letters, for example the second, fifth and eighth, and next time it might be the first, fifth and sixth (note to self -- lopado­temacho­selacho­galeo­kranio­leipsano­drim­hypo­trimmato­silphio­parao­melito­katakechy­meno­kichl­epi­kossypho­phatto­perister­alektryon­opte­kephallio­kigklo­peleio­lagoio­siraio­baphe­tragano­pterygon may not be the best choice for this exercise).

If you picked, say, security, and the system asks for the second fifth and eighth letters, you would give 'e', 'r' and 'y'.  If someone's looking over your shoulder, how much information do they have?  Let's fire up the old UNIX shell

$ grep '^.e..r..y.*' /usr/share/dict/words | wc -l
   84

What this means is that there are 84 words in the dictionary on my system that have 'e', 'r' and 'y' in those positions, or about six bits of entropy.  Most of them are words like ventrohysteropexy and dextrogyratory that people are unlikely to pick.  The person who helped me set up the account in question recommended something "easy to remember".  Odds are it's "security".

If not, all an attacker has to do is guess the letters that the site asks for next time.  There's a good chance that at least one will be one the attacker has already seen.  There won't be a lot of choices for the unknown letters.  Without looking at the list, I'd bet that 'q' isn't on it and 'e', 't' and a few others cover most of the possibilities.  Even without having looked over your shoulder, an attacker would know just from the security word being English that certain letters are better to try in certain positions.

So basically you have another hoop to jump through that adds minimal actual security, but tries to create the illusion of strong security, while really just making the system harder to use.  Huzzah.

Banking on web security

People do care about web security.  There are highly competent full-time professionals in the field.  There are conferences on the subject on a regular basis.  You'll see them in the press -- Experts Meet to Fix Security on the Web.

And yet, in large part because the problems to be solved are hard and involve significant non-techical factors, there is no shortage of things that could stand to be fixed.

• Authentication is a mess.  For the most part, we have passwords and security questions.  I've griped about this before, multiple times, and I'm sure I'll gripe about it again.
• Identity is a mess.  Everyone has scads and scads of identities -- logins here, there and everywhere. They can easily get confused ("That wasn't me, that was some other David Hull!").  There's no good way to say two random identities are or aren't the same.  I've griped and speculated about this before, too, and I expect I'll have more to say on that, too.
• Anonymity is problematic.  Everything you do on the web leaves traces, but unless you're paying extremely close attention you generally don't know exactly what kind, or whether they can be tied to your identity (whatever that is).
• Network infrastructure is scary.  Https with certificates is widely deployed, and most people probably at least know that some sites are "secured" and some aren't, but many fewer understand (or should need to understand) details like signatures, secure hashes and certificate authorities, or what can fail and what's less likely to.  Did I mention DNS?
• PCs are scary.  Viruses, rootkits, system crashes ... some platforms are better designed than others, but nothing's perfect.
• The cloud has its own problems.  Who owns what you put there?  Who's liable if data is lost or compromised?  Who can see what?  Who can see who sees what?
• Spam is a perennial problem, not helped by any of the above.

I could go on, but if it's so bad -- and it is -- how does it work at all?  People continue to be able to use credit cards both online and in person, people continue to email and text each other all sorts of sensitive information, people continue to turn to the web for all sorts of vital information.  Clearly Bad Things can happen to a person on the web, but just as clearly it's not bad enough often enough to put people off the web entirely.  Far from it.

My guess is that banks have a lot to do with it, at least in the US.  In particular

• Banks handle liability.  If someone steals your credit or debit card, whether physically or online, you can tell your bank and generally they will make sure you don't have to pay for things you didn't buy.  That's oversimplified, and there are certainly cases where that simple process has turned into a nightmare, but it's still a vital part of getting people to do business confidently online.
• Bank cards provide a de facto stable identity.  If you're buying something from my web site, I do care who you are (well, I would, and stores in general do seem to care what their customers are up to), but I certainly also care that your payment is going to go through.  To some extent I'm talking to you, but I'm also talking to your bank account.

On the first point, you're not responsible for keeping your bank accounts absolutely safe.  You're responsible for taking reasonable precautions, so that if someone does get hold of your account number and misuses it, they're clearly at fault (the usual "I'm not a lawyer" disclaimer applies here).  Putting the rest of the burden on the banks and legal system is a large part of what keeps the wheels turning.

On the second point, if I shop at store A and store B, it's important that my bank knows that those purchases both come out of my account, and I know that I'm the same person in both cases (at least on a good day).  It's less important that store A and store B know I'm the same person.  There may even be cases where I'd rather they didn't know.

In short, security and identity matter when money is at stake, in which case your accounts serve as your identity and you have legal protections that predate the web.

Security and identity also matter where reputation is at stake, that is in the social realm, be it email, social networks, Twitter or whatever.  The landscape is different there, but it's worth noting that most accounts and identities, including your bank accounts, don't play into that much.  If someone compromises my account at widgetco.com, they might be able to have a truckload of widgets sent to my address at my expense, but they won't be able to say embarrassing things about me on this blog.  Likewise if they compromise my bank account, though that would of course be bad for other reasons.

If you buy that, then you should make sure to use strong unique passwords and unique security questions for your bank accounts, your email accounts and your major social accounts, and use better security than that when it's available.  How much to worry about other accounts depends on how closely they're tied to the accounts that matter.  For example, if your city's online parking ticket paying site doesn't remember credit card numbers or your nefarious history of overparking, you probably don't care as much about security there.

Forbes on passwords

Wandering through the web (but not necessarily figuring it out as I went along) I ran across a slide show in Forbes on the subject of passwords, with what seems to me mostly reasonable advice.  Some highlights, mostly common-sense stuff that bears repeating:

• Change important passwords frequently and don't reuse them
• Use different passwords for different purposes.
• Important passwords (e.g., for bank accounts) should be unique. 
• Less important accounts can share passwords, but be aware that if one account is compromised you should consider all of them compromised.
• Don't choose a password that's ever appeared elsewhere.  This rules out memorable phrases like "We the people of the United States of America".
• Passwords should contain nothing personally associated with you (basically a version of the previous item).
• Password managers may be useful.  The advantage is you can use random gibberish and the manager will remember it for you.  The disadvantage is that if the master password is ever cracked, you're completely hosed.
• Use HTTPS when logging in.  HTTPS encrypts all connections and uses digital certificates to ensure that you're really talking to whom you think you are (just exactly how secure this system is is a whole other can of worms, but for now let's assume it's basically OK). You can tell if you are because web sites with it start with "https://" instead of "http://" and browsers now indicate whether you have a secure connection
• Don't type your password into anyone else's machine.
• Assume that a public WiFi access point is just that, public (the actual slide says to avoid it entirely).  If you're not using an encrypted connection of some sort (HTTPS, SSH, a VPN or such) assume that anyone can see your network traffic, including passwords you type when you log in.  Also assume that any random person can see anything that's publicly shared on your computer (another fine can-o-worms).
• Don't depend on passwords generated by web sites or random software.  Even if everything's on the up-and-up, it's very easy to get password generation wrong, typically by using a weak random number generator (see this post for more on generating passwords).
• Archive your important passwords in case of catastrophe, for example by writing them down on a piece of paper and storing it in a safe deposit box that can be opened in an emergency.
• In general, if you're going to record a password somewhere, do it on a physical medium separate from your computer (see disadvantage of password managers, above).

There are also a few items that don't seem actively harmful, but probably don't help greatly either

• When replacing letters with numbers and such, use non-obvious numbers, e.g., r7place instead of r3place.  This will add a few bits of entropy, which is good, but not really good enough on its own.  If your base word is in a dictionary of 500,000 words and you replace up to three characters with one of 15 replacements, you have about 30 bits of entropy, which is not that much.
• Add a number to the end of sentence-based passwords "for extra uniqueness".  Adding a number adds about three bits of entropy.  Meh.
• Scramble a password when writing it down.  This will make it harder, but not impossible, for someone who finds your written password to figure out the actual password, but it will also make it harder for you to come up with the actual password at two in the morning when you discover you don't quite remember how you scrambled it and the Very Important Site locks out accounts with more than three login failures.  Of course, you could write down how you scrambled it ...
• Deliberately misspelling words can make passwords more secure.  Yes, but not very much more secure.
• Use a sentence with lots of words, and include punctuation.  In theory this can work, but in practice people come up with much-less-than-random-words, particularly if the sentence actually makes sense.  Also, surprisingly many systems get indigestion if you try to use a long password.

Building a better password

[I've updated this post slightly to reflect the back-of-the-envelope calculation in this post suggesting that 100 bits of entropy is probably more reasonable than my original statement that 48 bits was "not bad".  Under the assumptions in that post, a 48-bit password would take on the order of microseconds to crack --D.H. Feb 2020]

I've recently complained about the irritating nature of the password strength checkers that have been popping up everywhere, so I feel obliged at least to try to analyze the problem and offer solutions.  This is leaving aside the question of whether password authentication is a useful approach at all.

Fundamentally the real measure of password strength is how many passwords you'd expect to have to guess in order to get the right one.  A more formal version of this is the notion of bits of entropy.  If you had a list of all possible passwords in your scheme, I could identify any particular one so long as I could get answers to a series of yes/no questions, for example:  "Is it in the first half of the list or the last?",   "Is it in the first half of that half or the last?" and so forth.  The number of such questions I need is the number of bits of entropy.  Twenty questions means twenty bits, etc..

If I know that your password is either "0" or "1", you have exactly one bit of entropy.  If I know it's an uppercase letter, lowercase letter, digit, "$" or "%", there are 64 possibilities, so you have 6 bits of entropy.  If I know it's two such characters, you have 12 bits, and if it's seventeen such characters you have 102 bits, which is not too bad.  Someone trying to guess your password would have to guess about two thousand billion billion billion passwords, on average, before stumbling on yours.  That may seem like a lot, but keep in mind that the current network of Bitcoin miners can try on the order of a hundred thousand billion billion hashes -- roughly the same problem as guessing a password -- every second.

[Don't assume that guessing a password requires typing it in to the same text box you have to use.  If someone steals the right data from your service provider, they can throw as much computing power as they've got at guessing the passwords.  Quite possibly they'll be happy enough just to try a few thousand weak passwords for each account, since that will crack depressingly many, but attacks like running through the OED with simple substitutions of letters for numbers are absolutely feasible as well, even on fairly ordinary hardware.]

This is assuming that you picked eight characters at random.  If I knew instead that your password was either "F1%ldN0t3$" or "sasssafras" (maybe I'd watched you read your password off a piece of paper with only those two words on it but couldn't quite see which you were typing), then you have only a single bit of entropy, even though both passwords are not just eight but ten characters long and one has plenty of non-letters.

More realistically, if I knew you'd picked an uncommon English word and maybe changed some of the letters to numbers, you'd have somewhere around two dozen bits of entropy.  That's not nothing, but keeping in mind that each added bit doubles the number of passwords a cracker has to try, it's nearly a billion billion billion times weaker than the 102-bit scheme above.

The fundamental flaw of password strength checkers is that they can only look at the password you gave them.  They have no idea what other possible passwords you might have chosen.  The assumption is that if you're forced to jump through enough hoops you'll be forced to expand your parameters, but in fact it's possible to generate passwords in a secure manner using only letters, and or to generate them insecurely in a way that will still satisfy any strength checker out there.  Which is why I half-grimace, half-laugh when I see the "password strength indicator" jump from "poor" to "great" as soon as I type a number.

Now, it's perfectly possible to generate completely random 17-character passwords.  The problem is that something like "qcrQf1x2" or "u%js%hPQ" is a pain to try to memorize, so most people will fall back to picking a "hard" word and maybe altering it a bit.  However, as xkcd points out, it's possible to do a lot better by using random short words.

For example, here's a kind of clunky way of producing a random, memorable password:

BIG HONKING DISCLAIMER: This is just for demo purposes.  The second site I mention uses http, not https, so in theory anyone could be looking in on your session.  Even with https, the sites might be logging all your traffic and recording the results you come up with.  I personally seriously doubt they would, and it's hard to imagine they would be able to connect the dots and figure out what you were using the generated password for, but if you really want to be on solid ground, get the source, look it over, run it locally and use something like /dev/urandom or D&D dice to generate the random input (23d20 will give you close to 100 bits ... not that I would have any idea at all what "23d20" means).  There are also smartphone apps that do more or less the same thing, I believe.

[I last checked that the recipe below worked on 28 Feb 2020]

With that out of the way:

• Go to this site and copy the random string you see there (e.g., 60990FFC250C).  If for some reason you don't like what you see, just reload.
• Go to this site.
• Type some short number and a space into the Challenge box and paste the random string from the first step in after it (e.g. 123 60990FFC250C)
• Type anything at all into the Secret box (e.g., "secret").  This doesn't have to be hard to guess.  The real entropy is coming from the random string (alternatively, put any number you like, a space, and anything else into the "challenge" box and paste the random bytes into the "secret" box).
• Press the Compute with SHA-1 button.  Again, the cryptographic details of how strong SHA-1 is don't matter here.  You're just converting a random number to short words.  A simple table lookup would do just as well.

In the Response box you will see six short words followed by some hexadecimal gibberish (in this case, WOVE COOT SLEW WIT SIGH I (FE2D 5F7B 22CD BC39)).  Each of those words represents just over 10 bits of entropy.  We'll need ten words in all, so repeat the procedure but this time just take the first four words (I got FIRE CUFF GALA MINK from A4B455FEFFE7BFAD).

You can play around with this formula to get words that are easier to memorize, or type, or are just more to your liking.  If you reorder your words or try typing in several different things instead of 123 or secret and then picking what you like, you're decreasing your entropy, depending on what criteria you're using to filter out passphrases you don't like and whether your attacker knows what kinds of phrases you like.  If you just try a few different secrets until you see something that seems memorable, that should be fine.  If you do something like sort the words (and your attacker knows only to try sorted lists of words), you've lost almost 22 bits of entropy, which wouldn't be good.

Once you've selected your words add a random punctuation character, number, capital letters or whatever makes your site's password strength checker happy.  Voila!  Your password is now Wovecootslewwitsighifirecuffgalamink5? or whatever.  This isn't great to have to type, but it's pretty secure as passwords go, and probably better than trying to remember something like C;cTbfThoO4ePFTt or 67EE386A205C4563DB8908A6C4.

If your site's password checker imposes an 8-character limit (and, incredibly enough, some do), cry.

This password madness has got to stop

It's well known that people like to choose bad passwords, and for years other people have suggested rules for making passwords more secure.  I'm not really sure why it should be happening now in particular, but it seems that every site that has a password must now jump on the bandwagon and have a password policy enforcer.

And of course, they're all a little different.

Fortunately there are plenty of possibilities.  Here's a do-it-yourself guide in case you think your site needs one.  First, pick any two of

• The password must contain at least one number
• The password must contain at least one lowercase letter
• The password must contain at least one uppercase letter
• The password must contain at least one special character

Next flip a coin to pick one of the remaining two to disallow.

Now pick a minimum length.  Back in the day, when computers were much slower than they are now and it wasn't fairly easy to get a gazillion computers to cooperate (with or without the owners' consent), the recommended minimum length was eight characters.  Today it should probably be more like 12 or 14.  So make sure the minimum length is at least six.

Now set a maximum length of 8.  Why a maximum, given that all other things equal, longer passwords are stronger, and the whole point of the exercise is to encourage strong passwords?  Don't know.  Probably whoever put the database together remembered the old eight-character rule and decided that should be the maximum.  But 8 is a magic number for passwords and everyone else does it.

Finally, add an arbitrary hidden restriction.  For example, if the password has to have a number, make sure it can't be the first character (yes, I ran into that one).  If it has to be a special character, quietly disallow '$' and '!'.  Something like that, just to reduce the strength and make people work a little harder.

Voila.  You now have a password policy.  If I did that math right, there are three dozen basic policies, times however many arbitrary rules there are, so there are easily hundreds of possibilities.  Chances are fair that your poor user will never have encountered your exact policy before and never will again.

Chances are also fair that once they jump through all your hoops (bonus points if this is all happening on a smartphone or tablet), your poor user will have never come up with that particular password on the spot before.  That's good, since sharing passwords can be dangerous.  The only drawback is that poor user is liable to forget this ad-hoc password within five minutes of logging in.

So urge them to write it down "some place safe."

Then have them pick three or four secret questions and answers for when they have to reset the password next time they log on.  But that's a different rant.


If you feel you need further security advice, you can always consult a real expert.

What's my mother's first dog's maiden name?

I can remember my login, but I can't remember my password. "No problem," says the site, "Just tell me that secret you told me when you set up the account. What was your first dog's name?" So I type in ... oh wait a minute, I can't tell you that ... and the site sets me up with a new password. Pretty slick, yes?

Well, not quite. Since lots and lots of sites are doing this, I've got two main choices:

• Use the same small set of questions and answers everywhere.
• Use different questions and/or answers.

Using the same everywhere means not having to remember as much. If I make up a bunch of answers and/or use different ones everywhere, then I have to remember what I made up. Basically, I'm up against the exact same problem as with passwords themselves, except now there are two weak spots for attackers to exploit: My actual password, and the questions guarding my password.

Of the two the password is probably a bit more secure, assuming I haven't used one of the 500 worst passwords of all time (unfortunately, some people seem to confuse scatology with security). City of birth? There's a good chance it's one of the top 100. Mother's maiden name? Not exactly classified information. First dog's name? There are a lot of Maxes and Buddies out there.

Now I'd say most sites don't let you reset a password directly. Typically they'll email some gibberish to the address you registered with and you use that to log in and reset the password for real. But in that case, why bother with the rigmarole? Whatever real security there is comes from putting email in the loop.

All in all, it's a classic example of a more complex system looking more secure than a simple one, but actually being less secure.