Independent study on OpenID awareness using Mechanical Turk

Even though I wasn’t able to attend the eighth Internet Identity Workshop this week in Mountain View (check out the latest episode of for a glimpse), I wanted to do my part to contribute so I’m sharing the results of a study that Brynn Evans and I performed on Mechanical Turk a short while ago.

I’ll cut to the chase and then go into some background detail.

Heard of OpenID?Of the 302 responses we received, we only rejected one, leaving us with 301 valid data points to work with. Of those 301:

  • 19.3% had heard of OpenID (58 people)
  • 9.0% knew what OpenID was used for (27) and 8.0% were unsure (24)
  • 1.3% used OpenID (4) and 18.3% were unsure if they used it (55).
  • 5.3% recognized the OpenID icon (16) and 7.0% were unsure (21).

Combining some of the results, we found that:

  • of those who know what OpenID is, 14.81% use it.
  • of those who have merely heard of it, 6.9% use it.

That’s what the data show.


Several weeks ago, Yahoo released usability research and best practices for OpenID (PDF). This research was performed by Beverly Freeman in the Yahoo! Customer Insights division in July of this year and involved 9 female Yahoo! users age 32-39 with self-declared medium-to-high level of Internet savvy.

This research, along with Eric Sachs’ later contributions (Google), have taken us from virtually zero research on the usability of OpenID to having a much more robust pool of information to pull from. And though I’m sure many would agree that this research only points to opportunities for improvement, many people interpreted the results as an indication that “OpenID is too confusing” or that it “befuddles users“.

A lot of people also took cheap shots, using the Yahoo! results to bolster their long-held arguments against the protocol and its unfamiliar interaction flow. The problem with such criticism, as far as I’m concerned, is that generalizing from the experiences of nine female Yahoo! users in their thirties is not necessarily representative of the web at large, nor are the conditions favorable to such research. Y’know, Ford got a lot of flack too when he introduced the Model T because everyone loved their horse and carriages. Good thing Ford was right.

Now, some of the criticism of OpenID is valid, especially if it can be turned into productive outcomes, like making OpenID easier to use, or less awkward.

And it serves no one’s interests to make grandiose claims on the basis of minimal data, so given Brynn’s work using Mechanical Turk (with Ed Chi from PARC), I thought I’d ask her to help me set up a study to discover just what awareness of OpenID might be among a wider segment of the population, especially with Japanese awareness of OpenID topping out around 28% (with usage of OpenID at 15%, more than ten times what we saw with Turkers).

Mechanical Turk Demographics

First, it’s important to point out something about Turker demographics. Because Turkers must have either a US bank account or be willing to be paid in Amazon gift certificates, the quality of participants you get (especially if you design your HIT well) will actually be pretty good (compared with, say, a blog-based survey). Now, Mechanical Turk actually has rules against asking for demographic or personally identifying information, but some information has been gathered by Panos Ipeirotis to shed some light on who the Turkers are and why they participate. I’ll leave the bulk of the analysis up to him, but it’s worth noting that a survey put out on Mechanical Turk about OpenID will likely hit a fairly average segment of the internet-using population (or at least one that doesn’t differ greatly from college undergraduates).


Over the course of a week (October 19 – 26), we fielded 302 responses to our survey, paying $0.02 for each valid reply (yes, we were essentially asking people for their “two cents”). We only rejected one response out of the batch, leaving us with 301 valid data points at a whooping cost of $6.02.


As I reported above, contrary to the 0% awareness demonstrated in the Yahoo! study of nine participants, we found that nearly 20% of respondents had at least heard of OpenID, though a much smaller percentage (1.3%) actually used it (or at least were consciously aware of using it — nearly everyone (18%) who’d heard of OpenID didn’t know if they used it or not).

There was also at least some familiarity with the OpenID logo/icon (5.3%).

What’s also interesting is that many respondents, upon hearing about “OpenID”, expressed an interest in finding out more: “What is it? LOL.”; “I’ve gotta look it up!”; “This survey has sparked my interest”; “Heading to Google to find out”. I can’t say that this shows clear interest in the concept, but at least some folks showed a curious disposition, as such:

How can I tell for sure whether I’ve used OpenID or not when I don’t know what it is? I’ve surely heard of it. That confuses me mainly in Magnolia {bookmarking service} where I want to sign up, but I can’t as it asks for OpenID. And until you mentioned above, it simply didn’t occur to me to just search it up. Hell, after submitting this hit, I’m going to do that first and foremost. Anyways, thanks a lot for indirectly suggesting a move!!!

Now, I won’t repeat the other findings, as they’ve already been reported above.

Thoughts and next steps

The results of this survey are interesting to me, but not unexpected. They’re not reassuring either, and they tell me that we’re doing well considering that we’ve only just begun.

Consider that 20% of a random sampling of 300 people on the internet had at least heard of OpenID, before Google, MySpace or Microsoft turned on their support for the protocol (MySpace announced their intention to support OpenID in July).

Consider that nearly a year ago Marshall Kirkpatrick sounded the deathknell of what seemed like the forgone conclusion about OpenID:

Big Players are Dragging Their Feet … Sharing User Info is a Whole Other Matter … Public Facing Profiles are Anemic … Ease of Use and Marketing Clarity Remain Low Priorities

Consider that no concerted effort has been made to date to inform or educate the general web population about OpenID, or about the problems with sharing your user credentials all over the web, and that many of the large providers have yet to turn on their OpenID support (despite all coming to the table and agreeing that it’s the way forward for identity on the web (save, as usual, Facebook, looking more Microsoftian by the day).

Consider also that momentum to rev the protocol to accommodate email addresses in OpenID is just now gaining traction.

In other words, with areas of user education becoming obvious, with provider adoption starting to happen (vis-a-via MySpace demonstrating the value and prevalence of URL-based identifiers) and necessary usability improvements starting to take shape (both in terms of the OpenID and OAuth flows being combined, and in terms of email addresses becoming valid in OpenID flows), we’re truly just getting started with making OpenID ready for mainstream audiences. It’s been a hard slog so far, and it’s bound to continue to be challenging, but the shared vision for where we’re going gets clearer every time there’s an Internet Identity Workshop.

I plan to re-run this study every 3-6 months from this point forward to keep track of our progress. I hope that these numbers will shed some much-needed balanced light on the subject of OpenID awareness and adoption — both to demonstrate how far we have to go, and how far we’ve come.


Inventing contact schemas for fun and profit! (Ugh)

And then there were three.

Today, Yahoo! announced the public availability of their own Address Book API. Though Plaxo and LinkedIn have been using this API behind the scenes for a short while, today marks the first time the API is available for anyone who registers for an App ID to make use of the bi-directional protocol.

The API is shielded behind Yahoo! proprietary BBAuth protocol, which obviates the need to request Yahoo! member credentials at the time of import initiation, as seen in this screenshot from LinkedIn (from April):

LinkedIn: Expand your network

Now, like Joseph, I applaud the release of this API, as it provides one more means for individuals to have utter control and access to their friends, colleagues and contacts using a robust protocol.

However, I have to lament yet more needless reinvention of contact schema. Why is this a problem? Well, as I pointed out about Facebook’s approach to developing their own platform methods and formats, having to write and debug against yet another contact schema makes the “tax” of adding support for contact syncing and export increasingly onerous for sites and web services that want to better serve their customers by letting them host and maintain their address book elsewhere.

This isn’t just a problem that I have with Yahoo!. It’s something that I encountered last November with the SREG and proposed Attribute Exchange profile definition. And yet again when Google announced their Contacts API. And then again when Microsoft released theirs! Over and over again we’re seeing better ways of fighting the password anti-pattern flow of inviting friends to new social services, but having to implement support for countless contact schemas. What we need is one common contacts interchange format and I strongly suggest that it inherit from vcard with allowances or extension points for contemporary trends in social networking profile data.

I’ve gone ahead and whipped up a comparison matrix between the primary contact schemas to demonstrate the mess we’re in.

Below, I have a subset of the complete matrix to give you a sense for where we’re at with OpenSocial (né GData), Yahoo Address Book API and Microsoft’s Windows Live Contacts API, and include vcard (RFC 2426) as the cardinal format towards which subsequent schemas should converge:

vcard OpenSocial 0.8 Windows Live Contacts API Yahoo Address Book API
UID uid url id cid cid
Nickname nickname nickname NickName nickname
Full Name n or fn name NameTitle, FirstName, MiddleName, LastName, Suffix name
First name n (given-name) given_name FirstName name (first)
Last name n (family-name) family_name LastName name (last)
Birthday bday date_of_birth Birthdate birthday (day, month, year)
Anniversary Anniversary anniversary (day, month, year)
Gender gender gender gender
Email email email Email (ID, EmailType, Address, IsIMEnabled, IsDefault) email
Street street-address street-address StreetLine street
Postal Code postal-code postal-code PostalCode zip
City locality locality
State region region PrimaryCity state
Country country-name country CountryRegion country
Latitutude geo (latitude) latitude latitude
Longitude geo (longitude) longitude longitude
Language N/A N/A
Phone tel (type, value) phone (number, type) Phone (ID, PhoneType, Number, IsIMEnabled, IsDefault) phone
Timezone tz time_zone TimeZone N/A
Photo photo thumbnail_url N/A
Company org CompanyName company
Job Title title, role organization.title JobTitle jobtitle
Biography note about_me notes
URL url url URI (ID, URIType, Name, Address) link
Category category, rel-tag tags Tag (ID, Name, ContactIDs)

An update to my my OpenID Shitlist, Hitlist and Wishlist

OpenID hitlist

Back in December, I posted my OpenID Shitlist, Hitlist and Wishlist. I listed 7 companies on my shitlist, 14 on hitlist and 12 on my wishlist. Given that it’s been all of two months but we’ve made some solid progress, I thought I’d go ahead and give a quick update on recent developments.

First, the biggest news is probably that Yahoo has come online as an OpenID 2.0 identity provider. This is old news for anyone who’s been watching the space, but given that I called them out on my wishlist (and that their coming online tripled the number of OpenIDs) they get serious props, especially since Flickr profile URLs can now be used as identity URLs. MyBlogLog (called out on my shitlist) gets a pass here since they’re owned by Yahoo, but I’d still like to see them specifically support OpenID consumption

Second biggest news is the fact that, via Blogger, Google has become both an OpenID provider (with delegation) and consumer. Separately, Brad Fitzpatrick released the Social Graph API and declared that URLs are People Too.

Next, I’ll give big ups to PBWiki for today releasing support for OpenID consumption. This is a big win considering they were also on my shitlist and I’d previously received assurances that OpenID for PBWiki would be coming. Well, today they delivered, and while there are opportunities to improve their specific implementation, I’d say that Joel Franusic did a great job.

And, in other good news, Drupal 6.0 came out this week, with support for OpenID in core (thanks to Walkah!), so there’s another one to take off my hitlist.

I’d really like to take Satisfaction off my list, since they’ve released their API with support for OAuth, but they’ve still not added support for OpenID, so they’re not out of the woods just yet… even though their implementation of OAuth makes me considerably happy.

So, that’s about it for now. I hear rumblings from Digg that they want to support OpenID, but I’ve got no hard dates from them yet, which is fine. There’re plenty more folks who still need to adopt OpenID, and given the support the foundation has recently received from big guys like Microsoft, Google, Yahoo, Verisign and IBM, my job advocating for this stuff is only getting easier.