One of my current responsibilities is to find ways to automate as much as practical of the ‘testing’ of the user experience (UX) of complex web-based applications. In my view, full test automation of UX is impractical and probably unwise, however we can use automation to find potential problems in UX even of rich, complex applications. I, and others, are working to find ways to use automation to discover various types of these potential problems. Here’s an overview of some of the points I made. I intend to extend and expand on my work in future posts.
In my experience, heuristics are useful in helping identify potential issues. Various people have managed to create test automation that essentially automates various heuristics.
Examples of pushing the boundaries
- Dynamic Usability / Accessibility Testing. See the following article I wrote that describes some of my work in the area. http://queue.acm.org/detail.cfm?id=1925091 The code is available here, http://code.google.com/p/web-accessibility-testing/ you’re welcome to use and experiment with it
- Fighting Layout Bugs. This is by Michael Tamm. He described the work in a public ‘tech talk’ at Google’s Test Automation Conference (GTAC) in 2009, the link is available on his project’s homepage at http://code.google.com/p/fighting-layout-bugs/
- Crawljax. Another opensource project which applies actions. It uses patterns to determine when to apply the actions. I’ve seen it used for significant, global, web applications. There is a video online which describes some of that work at http://www.youtube.com/watch?v=rYAO94GnBlY
- BiDi Checker. This software helps to identify problems related to bi-directional content on web sites and web applications. It successfully finds and identifies a wide range of potential issues. http://googletesting.blogspot.com/2010/11/test-your-app-from-right-to-left.html
You might notice that all the examples I’ve provided are available as free opensource software (FOSS). I’ve learnt to value opensource because it reduces the cost of experimentation and allows us to extend and modify the code e.g. to add new heuristics relatively easily (you still need to be able to write code, however the code is freely and immediately available).
Automation is (often) necessary, but not sufficient
Automation and automated tests can be beguiling, and paradoxically increase the chances of missing critical problems if we chose to rely mainly or even solely on the automated tests. Even with state of the art (the best we can do across the industry) automated tests I still believe we need to ask additional questions about the software being tested. Sadly, in my experience, most automated tests are poorly designed and implemented, which increases the likelihood of problems eluding the automated tests.
Here are 2 articles which describe some of the key concerns.
The first describes how people can be biased into over-reliance on automation. It is called “Beware of Automation Bias” by M.L. Cummings, in 2004. The article is available online at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2634&rep=rep1&type=pdf
The second helped me understand where testing helps us work out which questions to ask (of the software), and that we need to use a process to identify the relevant questions. The article is called 5 Orders of Ignorance, by Phillip G Armour, CACM 2000 http://www-plan.cs.colorado.edu/diwan/3308-s10/p17-armour.pdf
Note: the essence of this material was presented as a lightning keynote at the Belgium Testing Days conference on 15th February 2011
 potential problems is one term I use to avoid getting into arguments about whether a problem is a bug or not. I prefer to use the term ‘undesirable effects’ since software (and things in general) may meet the requirements but still have undesirable effects. Here I’m happy to focus on potential problems; perhaps I’ll write a post on the topic of undesirable effects soon…