Our ongoing UX bench-marking research with Cisco continues to generate fascinating UX data and insights.
Just recently Cisco tested and implemented some improvements to the online registration form which users must complete to create an account. As this task is included in our continuing studies, we were able to look at the impact in some detail and show how the UX improvements were cumulative.
There were 4 main improvements to the form:
- The form was reduced in length from 22 mandatory fields/selectors to 12
- Mandatory fields were more clearly indicated
- Real-time password guidance was improved
- An ‘address mis-match’ problem was resolved
The shorter form was trialed, then removed, then implemented. The other 3 improvements (Nos. 2-4 above) were implemented once. The chart below shows the data from 4 rounds of our Task Performance Indicator series of tests, showing which changes were implemented when, and what their impact was on task success rates (in green).
From the data, we can see that reducing the length of the form contributed 21% improvement to the success rate, and the other 3 design improvements contributed 29%, taking the success rate from 0% to 50%.
But the story isn’t quite as simple as that. Almost all participants in the Mar ’14 testing failed to complete the task within the target 5-minute time limit (coded as Time Out). From our qualitative analysis of the participants’ behaviour in the tests, we could see that the improvements were cumulative:
From observing tests, we knew that on the long form participants would try to work out which fields were mandatory and which were not. They would often make mistakes and only discover they’d missed a mandatory field when they submitted the form. Improved indication of which fields were mandatory certainly helped. But, more importantly, what we observed with the short form was that participants weren’t even trying to work out which fields were mandatory and which were not. They simply started at the top of the form and worked their way down.
This fits in nicely with Steve Krug’s design principle ‘Don’t make me think’:
When participants saw the short form, they didn’t even bother to think about mandatory vs optional fields; they simply completed each field, in order, reducing their cognitive effort.
This suggests there can be ‘tipping points’ in design, where users’ behavioural strategies actually change. On a long form, people look for ways to reduce their effort. On a short form, they simply complete the form.
But the shorter form had other cumulative impacts:
- Fewer fields meant fewer opportunities for the participants to make mistakes (and fewer opportunities for poor design!)
- Less time on the form meant people had more time to resolve problems they came across elsewhere in the task – e.g. mis-typing passwords, generating a password that was sufficiently strong, delays in receipt of confirmation email, etc.
Apart from reminding us how valuable it is to design to reduce users’ cognitive load (the ‘Don’t Make Me Think’ principle), there are a couple of other key insights we took away from this case study:
- Quantitative PLUS Qualitative: Quantitative research identified WHAT impact the improvements had. Qualitative observations told us WHY, and gave us design principles to apply elsewhere.
- User Experience improvements can be cumulative: one improvement can have a positive impact on other UX issues, creating a ‘snowball’ effect.
We would also hypothesize that, with the shorter form, more people might consider registering for an online account. There might also be fewer abandons of the registration process, fewer support calls about Registration difficulties, and Cisco may get a higher number of registrations – but that’s another case study with metrics we would love to be able to write up at some point.
See also: A presentation by Mike Atyeo and Bill Skeet of Cisco on Cisco’s UX measurement strategy: http://www.slideshare.net/neoinsight/cisco-measures-tasks-digital-strategy-conference-may2015 (Digital Strategy conference 2015, Vancouver).
