UX Test Automation for QAandTEST 2011 (27 Oct 2011)

The main content is identical to the presentation planned for EuroSTAR 2011 21st to 24th November 2011. I may revise the main content again by the time of EuroSTAR, if so, I’ll post the updated material online.

Update: I received the best presentation award at the conference for this presentation

Designs and Need adding perspective to our testing (11 Oct 2011) This presentation was given at Microsoft’s Redmond office. The material is not specific to any company or web site, rather I present concepts, ideas, and tools which should be generally relevant to people who want to create software which suits the needs of a wide range of users.

The aim is to encourage additional perspective to our software, as developers, designers and testers, etc. rather than focusing purely on ‘functionality’.

The material was presented at SASQAG in Seattle on 14th October 2011

If you’d like to come to the conference, the web site is http://www.sqe.com/starwest/

I have made the materials available online and you are welcome to download and use them. The material on ‘pushing the boundaries’ is on this site at UX Test Automation for StarWest 2011 The material for the tutorial is hosted at http://code.google.com/p/mwta/downloads/list

I’m continuing to revise the material for various conferences so expect to see updates published on this site from time to time. The next time I present on this topic is at StarWest 2011 Track W9 on Wednesday October 5th. http://www.sqe.com/StarWest/Concurrent/Default.aspx

In my experience, heuristics are useful in helping identify potential issues. Various people have managed to create test automation that essentially automates various heuristics.

Examples of pushing the boundaries

You might notice that all the examples I’ve provided are available as free opensource software (FOSS). I’ve learnt to value opensource because it reduces the cost of experimentation and allows us to extend and modify the code e.g. to add new heuristics relatively easily (you still need to be able to write code, however the code is freely and immediately available).

Automation is (often) necessary, but not sufficient

Automation and automated tests can be beguiling, and paradoxically increase the chances of missing critical problems if we chose to rely mainly or even solely on the automated tests. Even with state of the art (the best we can do across the industry) automated tests I still believe we need to ask additional questions about the software being tested. Sadly, in my experience, most automated tests are poorly designed and implemented, which increases the likelihood of problems eluding the automated tests.

Here are 2 articles which describe some of the key concerns.

The first describes how people can be biased into over-reliance on automation. It is called “Beware of Automation Bias” by M.L. Cummings, in 2004. The article is available online at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2634&rep=rep1&type=pdf

The second helped me understand where testing helps us work out which questions to ask (of the software), and that we need to use a process to identify the relevant questions. The article is called 5 Orders of Ignorance, by Phillip G Armour, CACM 2000 http://www-plan.cs.colorado.edu/diwan/3308-s10/p17-armour.pdf

Note: the essence of this material was presented as a lightning keynote at the Belgium Testing Days conference on 15th February 2011

[1] potential problems is one term I use to avoid getting into arguments about whether a problem is a bug or not. I prefer to use the term ‘undesirable effects’ since software (and things in general) may meet the requirements but still have undesirable effects. Here I’m happy to focus on potential problems; perhaps I’ll write a post on the topic of undesirable effects soon…

Lots of companies, and teams have a burden of unreliable, problematic, automated tests that are troublesome and time-consuming to maintain. We need ways to address these poor-quality tests, ways to recover from the current unhealthy situation to a healthy environment where the tests are reliable, trustworthy, and easy to maintain as the underlying software changes.

For new test automation projects and teams, with the ‘luxury’ of starting fresh, you might also find these topics salutary, advance warning of a situation you might end up in if you don’t apply good test automation practices from the outset. Don’t say you weren’t warned

Here are the initial topics I’m going to cover

• Re-engineering and Refactoring automated tests
• Understanding Critical Success Factors for test automation
• Applying Design Patterns
• Design and Structure your test automation code
• Coping with large volumes of ‘legacy’ and ‘broken’ tests (including record & playback)
• Remove boiler-plating or dumbed-down interfaces
• Making the interaction with web pages resilient and robust (using IDs, working with developers, etc).
• Don’t be fooled (again) avoid being beguiled by automated tests
• Sunk by dependencies (e.g. on live back-end servers). Coping with Environmental Issues.
• Slow tests
• Modelling Techniques
• The intersection of automation and in-person testing
• Three possible outcomes of a test
• Writing readable code
• Stringing tools together
• Patterns for: Data creation, reuse, sharing and cleanup
• Designing tests to safely run in parallel

One of my aims is to create a useful, succinct guide to help you, and others, create and establish useful, readable and maintainable automated tests for your software projects. I’m drawing on the work, experience and expertise of various people in the software testing communities, and welcome your input and ideas.