- “readily available for the owner’s use as required”
- “intended to be thrown away after use”
For many years test environments were hard to obtain, maintain, and update. Technologies including Virtual Machines and Containers reduce the effort, cost, and potentially the resources needed to provide test environments as and when required. Picking the most appropriate for a particular testing need is key.
For a recent project, to test Kafka, we needed a range of test environments, from lightweight ephemeral self-contained environments to those that involved 10’s of machines distributed at least 100 km apart. Most of our testing for Kafka used AWS to provide the computer instances and connectivity where environments were useful for days to weeks. However we also used ESXi and Docker images. We used ESXi when we wanted to test on particular hardware and networks. Docker, conversely, enabled extremely lightweight experiments, for instance to experiment with self-contained Kafka nodes where the focus was on interactive learning rather than longer-lived evaluations.
Some, not all, of the contents of a test environment has a life beyond that of the environment. Test scripts, the ability to reproduce the test data and context, key results and lab notes tend to be worth preserving.
- what to keep and what to discard: We want ways to identify what’s important to preserve and what can be usefully and hygienically purged and freed.
- timings: how soon do we need the environment, what’s the duration, and when do we expect it to be life-expired?
- fidelity: how faithfully and completely does the test environment need to be?
- count: how many nodes are needed?
- tool support: do the tools work effectively in the proposed runtime environment?