Even though I wasn’t able to attend the eighth Internet Identity Workshop this week in Mountain View (check out the latest episode of TheSocialWeb.tv for a glimpse), I wanted to do my part to contribute so I’m sharing the results of a study that Brynn Evans and I performed on Mechanical Turk a short while ago.
I’ll cut to the chase and then go into some background detail.
Of the 302 responses we received, we only rejected one, leaving us with 301 valid data points to work with. Of those 301:
- 19.3% had heard of OpenID (58 people)
- 9.0% knew what OpenID was used for (27) and 8.0% were unsure (24)
- 1.3% used OpenID (4) and 18.3% were unsure if they used it (55).
- 5.3% recognized the OpenID icon (16) and 7.0% were unsure (21).
Combining some of the results, we found that:
- of those who know what OpenID is, 14.81% use it.
- of those who have merely heard of it, 6.9% use it.
That’s what the data show.
Background
Several weeks ago, Yahoo released usability research and best practices for OpenID (PDF). This research was performed by Beverly Freeman in the Yahoo! Customer Insights division in July of this year and involved 9 female Yahoo! users age 32-39 with self-declared medium-to-high level of Internet savvy
.
This research, along with Eric Sachs’ later contributions (Google), have taken us from virtually zero research on the usability of OpenID to having a much more robust pool of information to pull from. And though I’m sure many would agree that this research only points to opportunities for improvement, many people interpreted the results as an indication that “OpenID is too confusing” or that it “befuddles users“.
A lot of people also took cheap shots, using the Yahoo! results to bolster their long-held arguments against the protocol and its unfamiliar interaction flow. The problem with such criticism, as far as I’m concerned, is that generalizing from the experiences of nine female Yahoo! users in their thirties is not necessarily representative of the web at large, nor are the conditions favorable to such research. Y’know, Ford got a lot of flack too when he introduced the Model T because everyone loved their horse and carriages. Good thing Ford was right.
Now, some of the criticism of OpenID is valid, especially if it can be turned into productive outcomes, like making OpenID easier to use, or less awkward.
And it serves no one’s interests to make grandiose claims on the basis of minimal data, so given Brynn’s work using Mechanical Turk (with Ed Chi from PARC), I thought I’d ask her to help me set up a study to discover just what awareness of OpenID might be among a wider segment of the population, especially with Japanese awareness of OpenID topping out around 28% (with usage of OpenID at 15%, more than ten times what we saw with Turkers).
Mechanical Turk Demographics
First, it’s important to point out something about Turker demographics. Because Turkers must have either a US bank account or be willing to be paid in Amazon gift certificates, the quality of participants you get (especially if you design your HIT well) will actually be pretty good (compared with, say, a blog-based survey). Now, Mechanical Turk actually has rules against asking for demographic or personally identifying information, but some information has been gathered by Panos Ipeirotis to shed some light on who the Turkers are and why they participate. I’ll leave the bulk of the analysis up to him, but it’s worth noting that a survey put out on Mechanical Turk about OpenID will likely hit a fairly average segment of the internet-using population (or at least one that doesn’t differ greatly from college undergraduates).
Method
Over the course of a week (October 19 – 26), we fielded 302 responses to our survey, paying $0.02 for each valid reply (yes, we were essentially asking people for their “two cents”). We only rejected one response out of the batch, leaving us with 301 valid data points at a whooping cost of $6.02.
Findings
As I reported above, contrary to the 0% awareness demonstrated in the Yahoo! study of nine participants, we found that nearly 20% of respondents had at least heard of OpenID, though a much smaller percentage (1.3%) actually used it (or at least were consciously aware of using it — nearly everyone (18%) who’d heard of OpenID didn’t know if they used it or not).
There was also at least some familiarity with the OpenID logo/icon (5.3%).
What’s also interesting is that many respondents, upon hearing about “OpenID”, expressed an interest in finding out more: “What is it? LOL.”; “I’ve gotta look it up!”; “This survey has sparked my interest”; “Heading to Google to find out”. I can’t say that this shows clear interest in the concept, but at least some folks showed a curious disposition, as such:
How can I tell for sure whether I’ve used OpenID or not when I don’t know what it is? I’ve surely heard of it. That confuses me mainly in Magnolia {bookmarking service} where I want to sign up, but I can’t as it asks for OpenID. And until you mentioned above, it simply didn’t occur to me to just search it up. Hell, after submitting this hit, I’m going to do that first and foremost. Anyways, thanks a lot for indirectly suggesting a move!!!
Now, I won’t repeat the other findings, as they’ve already been reported above.
Thoughts and next steps
The results of this survey are interesting to me, but not unexpected. They’re not reassuring either, and they tell me that we’re doing well considering that we’ve only just begun.
Consider that 20% of a random sampling of 300 people on the internet had at least heard of OpenID, before Google, MySpace or Microsoft turned on their support for the protocol (MySpace announced their intention to support OpenID in July).
Consider that nearly a year ago Marshall Kirkpatrick sounded the deathknell of what seemed like the forgone conclusion about OpenID:
Big Players are Dragging Their Feet … Sharing User Info is a Whole Other Matter … Public Facing Profiles are Anemic … Ease of Use and Marketing Clarity Remain Low Priorities
Consider that no concerted effort has been made to date to inform or educate the general web population about OpenID, or about the problems with sharing your user credentials all over the web, and that many of the large providers have yet to turn on their OpenID support (despite all coming to the table and agreeing that it’s the way forward for identity on the web (save, as usual, Facebook, looking more Microsoftian by the day).
Consider also that momentum to rev the protocol to accommodate email addresses in OpenID is just now gaining traction.
In other words, with areas of user education becoming obvious, with provider adoption starting to happen (vis-a-via MySpace demonstrating the value and prevalence of URL-based identifiers) and necessary usability improvements starting to take shape (both in terms of the OpenID and OAuth flows being combined, and in terms of email addresses becoming valid in OpenID flows), we’re truly just getting started with making OpenID ready for mainstream audiences. It’s been a hard slog so far, and it’s bound to continue to be challenging, but the shared vision for where we’re going gets clearer every time there’s an Internet Identity Workshop.
I plan to re-run this study every 3-6 months from this point forward to keep track of our progress. I hope that these numbers will shed some much-needed balanced light on the subject of OpenID awareness and adoption — both to demonstrate how far we have to go, and how far we’ve come.
Factory Joe 
thanks to you and brynn for doing this work! i wish the foundation was doing more to contribute tangible progress like this. i think it is critical that there is an increasing analytical focus on the open ecosystem, especially if we are ever to convey concrete benefits.
i want to share an interesting statistic I learned at MySpace:
% of active app users with a vanity url = 93.7%
this means that we have about 30 million users who not only understand the basic concept of social applications, but also see their identity as being represented by a url.
it is this statistic that really excites me about OpenID at MySpace, and makes me hopeful that when you run this study a year from now, that the results will be very different.
What was the basis for the rejection of the one bad data point? Merely curious.
I’m actually shocked by how many people knew what openID was. Maybe i live in a sheltered area, but most folks here have never heard of it. shrugs..
i will def check out the social web episode
thank guys
@Max: I agree and wish there were more leadership coming from the foundation, but I think it’s also a matter of demanding more from it. Nat from the Japan OIDF is going gangbusters — we should take a hint!
It will also be very interesting to rerun this study once people start seeing “MySpace OpenID” or preferrably “OpenID” as a consistent sign in option.
@Scott: good question. I believe it was a nonsensicle response. Something like they had never heard of OpenID, didn’t know what it was, but used it and had seen the logo. I think one of their their comments also suggested they weren’t taking the survey seriously.
This 19.3% is lacking a baseline, so it’s statistically irrelevant, sadly.
As an example, you could have asked this question: Which of the following technologies have you heard of?
* Bluetooth
* World Wide Web
* OpenID
* Vericrypt
* SMS
Note the non-existent option. Seeing how many people selected something that sounded A LITTLE BIT like something they MIGHT have heard of would allow you to subtract that percentage from the OpenID line, to give you a more realistic answer.
(As I recall from my time at RhymesWithNose, the baseling of people claiming to have heard of a non-existent technology was usually around 12%, so that always got subtracted from the percentage who claimed to have heard of Blu-Ray or whatever we were really surveying about)
@Rod: you make a good point. Perhaps when we rerun the survey next time we’ll insert a question like that.
Thanks!
“…18.3% were unsure if they used it…”
Ouch! That’s like the worst one. Its so hard to understand, use, that many people don’t even know.
I am supposed to be a cheerleader for OpenID and even I can make it work half the time. I try to comment here, on this very blog, using my Yahoo ID and I get spun off to the Yahoo page that offers me a couple thousand characters of text and 124 character long string that is supposedly my “Open ID”.
Good grief Charlie Brown we have got to all get together and figure this out!
@Todd: Of course not all OpenID providers are created equal. It’ll be curious to see how competition in the space will heat up and improve this situation.
I’m sure email was similarly tricky and unreliable in the beginning too before people started really running their businesses on top of it. Yahoo went out of their way to ensure users’ security and ability to be “non-correlatable”… but to ill effect it seems.
You should try the OpenID provider built into the WordPress OpenID plugin, if only to see a slightly different user experience. 😉
Here’s a representaion of what Yahoo provides me as an OpenID
https://me.yahoo.com/ab/aJE6r472qdf6ud3yRjrw3yrq5rfwqt6f.0add
So, liek iz sippoadz to commit dat teh memory?
@Todd: you know you can change that right?
Just go to:
http://openid.yahoo.com/
And then pick a new identifier:
Great survey and findings Chris. I like the notion of doing longitudinal studies so we can see how awareness and usage is trending over time. Nat Sakimura of NRI has been doing this for over a year so far in Japan, we should be doing this across all major geographies as well.
One request I have for your readership is what are the best ways for the OpenID Foundation and its members to get the word out on OpenID. As we know, there has been some press and analyst coverage on OpenID including the recent Wall St. Journal article. What can be done to get the word out beyond the blogging community. Clearly more RP adoption by mainstream websites is one of the most important objectives, but what else can be done?
Looking forward to suggestions and recommendations.
Cheers, Brian Kissel, OpenID Foundation Marketing Committee
@Todd you’re not supposed to remember that… Yahoo users should use their yahoo email address (or just yahoo.com) as their OpenID.
It’s great to hear people’s feedback to this survey. I wanted to note that this was not meant to be the best solution to learning about OpenID adoption. Chris & I were just trying to make some effort to complement previous OpenID usability studies.
The Mechanical Turk “database” will surely have some bias (as will all convenient samples)—this is a common (yet valid) criticism. @Rod’s point is even more valid: I actually hadn’t thought of testing OpenID awareness against a baseline. My other concern going forward is if we continue to use MTurk for testing, our sample may become increasingly biased after having seen this survey posted in the first place. On the other hand, a number of users reported wanting to learn more…so in the end, this may be a great way to educate the community about OpenID!
A final word on Mechanical Turk demographics: I’ve run several other studies on MTurk and can attest to the diversity (and general non-tech savvy) of the community. Although they tend to be slightly betted educated than your average American, they are definitely a different cohort than you people reading this post 🙂
@Stephen Paul Weber,
Re “Yahoo users should use their yahoo email address (or just yahoo.com) as their OpenID.”
That’s an interesting point. Although the OpenID login process now allows me to put in the shorthand ‘yahoo.com’ instead of remembering my flickr URL, … should I really consider ‘yahoo.com’ to be my OpenID? Isn’t it your OpenID too? and the OpenID of every other Yahoo user. I can see this causing confusion…
Hey Chris!
“I plan to re-run this study every 3-6 months”
Did you? Any more plans in this direction?
Ah, wow. I dropped the ball on this one, eh? Thanks for following up but no, I never did re-run this survey.
It might be worth it to revisit this idea in 2011, but I think the motivation for this exercise largely depends on where the OpenID Foundation wants to take the brand — and whether consumer awareness matters as much to its proponents as it had the potential to, say, in 2008.