Automated software tests are topical where they seem to be replacing much of the testing done by humans. Automated tests are faster, provide early feedback and cost little to run many times. Agile projects need automated tests to keep up with the frequent builds which may arrive tens or hundreds of times a day and need testing.

So human testing seems to be gathering cobwebs, even despised as unproductive, low-skilled work done by testers who don’t have the ‘skills’ to write automated tests. However, as an industry we ignore testing by humans at our peril. There’s so much testing that’s beyond practical reach of automated tests. It’s time to revive interactive testing performed by motivated and interested humans. This talk will help you to find a new impetus and focus for your interactive testing to complement automated tests.

Feelings and emotions are what users will judge your apps on, so let’s test and explore how users may feel about the mobile apps. Michael Bolton published an insightful article called: “I’ve Got a Feeling: Emotions in Testing by Michael Bolton”

Fast, efficient testing can augment the repetitive automated testing. BugFests, where a group of people meet to test the same piece of software together for up to an hour can be extremely productive at finding problems the automated tests haven’t.

Another technique is moving both you (from place to place) and the phone (by rotating it from portrait to landscape modes, etc.) may help find and expose bugs which are hard for your automated tests to discover.

I will be giving a keynote at VistaCon 2013 in April 2013 on this topic. Please email me if you would like to get involved in the discussion, share ideas, criticize, etc.

Over the last week I have spent about a day of effort getting to grips with the recently launched UIAutomator test automation framework for Android. It was launched with version 16 of Android (Android 4.1) however on 4.1 devices the framework doesn’t even have all the documented methods available. With version 17 of Android (Android 4.2), support has improved to the point that the examples can work acceptably. Here is the official example http://developer.android.com/tools/testing/testing_ui.html

However in the minor update between Android 4.2.1 and Android 4.2.2 someone seems to have broken the support for automatic scrolling through pages of results.  I have reported the problem on the adt-dev forum, https://groups.google.com/forum/?fromgroups=#!topic/adt-dev/TjeewtpNWf8 which seems to be where the Android development team monitor comments. I have implemented a workaround, using a helper method, below:

    /**
     * Launches an app by it's name. 
     * 
     * @param nameOfAppToLaunch the localized name, an exact match is required to launch it.
     */
    protected static void launchAppCalled(String nameOfAppToLaunch) throws UiObjectNotFoundException {
        UiScrollable appViews = new UiScrollable(new UiSelector().scrollable(true));
          // Set the swiping mode to horizontal (the default is vertical)
          appViews.setAsHorizontalList();
          appViews.scrollToBeginning(10);  // Otherwise the Apps may be on a later page of apps.
          int maxSearchSwipes = appViews.getMaxSearchSwipes();

          UiSelector selector;
          selector = new UiSelector().className(android.widget.TextView.class.getName());
          
          UiObject appToLaunch;
          
          // The following loop is to workaround a bug in Android 4.2.2 which
          // fails to scroll more than once into view.
          for (int i = 0; i < maxSearchSwipes; i++) {

              try {
                  appToLaunch = appViews.getChildByText(selector, nameOfAppToLaunch);
                  if (appToLaunch != null) {
                      // Create a UiSelector to find the Settings app and simulate      
                      // a user click to launch the app.
                      appToLaunch.clickAndWaitForNewWindow();
                      break;
                  }
              } catch (UiObjectNotFoundException e) {
                  System.out.println("Did not find match for " + e.getLocalizedMessage());
              }

              for (int j = 0; j < i; j++) {
                  appViews.scrollForward();
                  System.out.println("scrolling forward 1 page of apps.");
              }
          }
    }

I ended up writing several skeletal demo Android apps to help me explore the capabilities of UI Automator. In each case I was working through publicly reported problems on http://stackoverflow.com where I’ve posted answers and feedback to several reported problems.

Here are the links to my comments:

http://stackoverflow.com/questions/13991977/how-to-switch-on-wifi-in-uiautomator-test-case-in-android-device

http://stackoverflow.com/questions/15204154/uiautomator-failing-on-4-1-2-device

http://stackoverflow.com/questions/15111001/uiautomator-getlasttraversedtext

Strengths of UI Automator

The key strengths include:

• We can test most applications, including Google’s installed apps such as Settings. Thankfully the example from the Android site does just that, albeit at a perfunctory level. However the example to change the Wi-Fi setting on stackoverflow provides a better example of what we can now do. Because the tests interact with the objects, they have a direct connection to the app being tested, rather than crude interactions by clicking at locations, OCR, etc.
• Using UI Automator relies on the underlying support for Accessibility in the platform and therefore may help to encourage improved support for Accessible Android apps as developers refine their apps to make them testable by Ui Automator.
• We can test apps on several devices from one computer, through related changes to the Android build tools.
• There are debug and exploration tools available on both the device (using adb shell uiautomator) and from my computer, using uiautomationviewer.

Weaknesses

• Text based matching makes testing localized apps much harder than using the older Android Instrumentation which could easily share resource files with the app being tested.
• There is virtually no documentation or examples, and the documentation that does exist doesn’t provide enough clues to address key challenges e.g. obtaining the text from WebViews.
• UI Automation cannot be used when the Accessibility features e.g. Explore-By-Touch is enabled on the device.
• There are bugs in the current version of Android and there’s no easy way to revert devices to 4.2.1
• Automation is very slow e.g. paging through the set of apps takes several seconds to go to the next page.

Other characteristics

• All the tests are bundled into a single jar file, deployed to the device. This risks one set of tests overwriting the bundle of tests.

Further reading

• Russell Collins’ site http://everybodytests.blogspot.co.uk/search/label/UI%20Automation had various useful examples.

I recently read a well written and helpful paper written by Doug Hoffman titled: Test Automation Architectures: Planning for Test Automation. You can find it online at http://softwarequalitymethods.com/papers/autoarch.pdf

It covers many key points that need to be considered if you want to have effective and useful automated tests. Thank you Doug for writing it so many years ago and for sharing it.

Over the last seven months I have been talking to various people about how test automation ‘works’ and how the working affect the viability of their test automation. In December 2012, LogiGear published an abridged version of an article I have written on the topic http://www.logigear.com/magazine/mobile-testing/test-automation-interfaces-for-mobile-apps/ I hope you find the article informative and helpful.

I sometimes find analogies help people to grasp concepts and ideas which I otherwise might struggle to communicate effectively. So here are a couple of analogies for test automation interfaces:

1. They are the glue between your automated tests and the app you want to test. By picking the most appropriate glue for the job, your tests are more likely to stick around and work effectively.
2. The interface is similar to the way Velcro works, the hooks bind with the eyes to establish an effective connection.

I have some ideas and plans to expand the initial article into a small book on effective software test automation. e-mail me if you’d like to encourage that work. My email address is my name (julianharty) at Google’s fine email service: gmail.com I assume a human will be able to create the correct email address from this information

One of my current responsibilities is to find ways to automate as much as practical of the ‘testing’ of the user experience (UX) of complex web-based applications. In my view, full test automation of UX is impractical and probably unwise, however we can use automation to find potential problems[1] in UX even of rich, complex applications. I, and others, are working to find ways to use automation to discover various types of these potential problems. Here’s an overview of some of the points I made. I intend to extend and expand on my work in future posts.

In my experience, heuristics are useful in helping identify potential issues. Various people have managed to create test automation that essentially automates various heuristics.

Examples of pushing the boundaries

You might notice that all the examples I’ve provided are available as free opensource software (FOSS). I’ve learnt to value opensource because it reduces the cost of experimentation and allows us to extend and modify the code e.g. to add new heuristics relatively easily (you still need to be able to write code, however the code is freely and immediately available).

Automation is (often) necessary, but not sufficient

Automation and automated tests can be beguiling, and paradoxically increase the chances of missing critical problems if we chose to rely mainly or even solely on the automated tests. Even with state of the art (the best we can do across the industry) automated tests I still believe we need to ask additional questions about the software being tested. Sadly, in my experience, most automated tests are poorly designed and implemented, which increases the likelihood of problems eluding the automated tests.

Here are 2 articles which describe some of the key concerns.

The first describes how people can be biased into over-reliance on automation. It is called “Beware of Automation Bias” by M.L. Cummings, in 2004. The article is available online at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2634&rep=rep1&type=pdf

The second helped me understand where testing helps us work out which questions to ask (of the software), and that we need to use a process to identify the relevant questions. The article is called 5 Orders of Ignorance, by Phillip G Armour, CACM 2000 http://www-plan.cs.colorado.edu/diwan/3308-s10/p17-armour.pdf

Note: the essence of this material was presented as a lightning keynote at the Belgium Testing Days conference on 15th February 2011

[1] potential problems is one term I use to avoid getting into arguments about whether a problem is a bug or not. I prefer to use the term ‘undesirable effects’ since software (and things in general) may meet the requirements but still have undesirable effects. Here I’m happy to focus on potential problems; perhaps I’ll write a post on the topic of undesirable effects soon…

Introduction

Lots of companies, and teams have a burden of unreliable, problematic, automated tests that are troublesome and time-consuming to maintain. We need ways to address these poor-quality tests, ways to recover from the current unhealthy situation to a healthy environment where the tests are reliable, trustworthy, and easy to maintain as the underlying software changes.

For new test automation projects and teams, with the ‘luxury’ of starting fresh, you might also find these topics salutary, advance warning of a situation you might end up in if you don’t apply good test automation practices from the outset. Don’t say you weren’t warned

Here are the initial topics I’m going to cover

• Re-engineering and Refactoring automated tests
• Understanding Critical Success Factors for test automation
• Applying Design Patterns
• Design and Structure your test automation code
• Coping with large volumes of ‘legacy’ and ‘broken’ tests (including record & playback)
• Remove boiler-plating or dumbed-down interfaces
• Making the interaction with web pages resilient and robust (using IDs, working with developers, etc).
• Don’t be fooled (again) avoid being beguiled by automated tests
• Sunk by dependencies (e.g. on live back-end servers). Coping with Environmental Issues.
• Slow tests
• Modelling Techniques
• The intersection of automation and in-person testing
• Three possible outcomes of a test
• Writing readable code
• Stringing tools together
• Patterns for: Data creation, reuse, sharing and cleanup
• Designing tests to safely run in parallel

One of my aims is to create a useful, succinct guide to help you, and others, create and establish useful, readable and maintainable automated tests for your software projects. I’m drawing on the work, experience and expertise of various people in the software testing communities, and welcome your input and ideas.