In recent decades, as software development in general has evolved from following a classic waterfall process to one that is more agile and adaptable, the QA discipline has lagged behind. Increasingly, that has meant bottlenecks and added cost to projects. Recognizing the need to change, several new models have emerged over the last decade. We’ve been experimenting with blending some of the workable components of these with our own trial and error to develop a faster, more flexible QA process for our clients.
An Antiquated Notion.
QA as a discipline started with the Industrial Revolution. New manufacturing techniques required testing physical objects and processes to ensure quality. New bureaucracies, safety regulations and good old-fashioned CYA demanded extensive documenting of that testing. Traditional manufacturing was a natural fit for a waterfall process—concepts, designs, implementation and testing all had to be done in a specific order to keep the assembly line moving.
In its infancy, software development borrowed many of these principles from the manufacturing model. QA stayed at the bottom of the waterfall. The process worked because the assembly line process meant the product didn’t change too much (though many more advanced manufacturers became leaner). Digital products are different.
Realizing this, software developers altered their approach with new approaches like Agile, Lean and Kanban. Perpetual change and iteration became the norm, with designs and interactions allowed to evolve throughout the project instead of being locked down at the top of the waterfall. Developers broke new products into chunks and built them in parallel instead of trying to complete the entire project at once.
But not QA. We still wrote scripted test cases as if everything was locked down. Existing documentation had taken time and money to create, and nobody wanted to throw it out and do something new. QA became an expensive aspect of a given development project, and testing only began once everything was “completed.” Projects lost momentum as the team waited for QA to test an entire product. Even then, things were never truly “done,” resulting in new updates by QA to the long-form test cases.
The QA community has recognized the need to adapt and better integrate testing with the other parts of a more agile development process. Testing documents do not need to be exhaustively detailed, step-by-step instructions. After all, documentation for how things should work already exists, and a decent tester should be able to derive what to test from that. Further, QA should be able test as pieces are done and continue as changes are made. Testing should happen early and often, with QA active throughout a project, instead of only at the very end.
A prominent response to these challenges—and one that seemed particularly suited for agencies—is Session Based Testing, developed by Jonathan Bach in 2000. We liked it because it took a technique we knew worked, Exploratory Testing, and provided a way for it to be managed and measured which would be important for our clients.
Session Based Testing has three primary elements:
- Charters: A charter is defined as a goal or agenda for a test session. Information for charters can be found in any project documentation such as the SOW, Test Plan, Annotations or just come out of an ad hoc conversation with a team member. Charters should be created as soon as there is enough information to do so, and they can be added during the entire testing lifecycle. A charter should be constructed so that it can be tested within one session and no more. For what to include, see this presentation by Anders Claesson.
- Sessions: A session is 60 - 90 minutes of uninterrupted exploratory testing. Each session should concentrate on a single charter. Charters should be reviewed prior to the start of the session to ensure coverage and understanding.
- Test Management: Successful Session Based Testing requires a Test Lead who can effectively set up charters, manage schedules, remain vigilant in tracking progress, and be able to adapt as things change.
Towards an Agile QA Model for Agencies.
For us, Session Based Testing didn’t work completely. We wanted to go farther in integrating QA leads into project teams that included Solution Architects, Developers, Business Analysts and Designers. The entire team should work together in order to close User Stories. But the agency setting is also a great environment for testing new ways of operating, with its diversity of project scopes, sizes and timelines. Failure is fine as long as we’re fast enough to recognize problems and fix them.
So we took the elements of Session Based testing that we liked and adjusted them to fit our unique needs. We chose a project for testing our new approach. We converted Charters into Jira sub-tasks of a User Story and we removed timeboxing as a requirement. We changed Session notes into comments, and we attached Defects to the User Story. Metrics from Jira measured what the project team as a whole was doing instead of just QA.
With a testing method in place that works for us, we took steps to educate colleagues from other disciplines about how we could help solve problems, not just find them. Each QA team member adopted a specific specialty so we could be more than manual testers: Accessibility, Automation, Performance, Mobile, Security, and Non-Traditional Devices. Each specialty meets as a group and works towards mastery, supported by attending webinars, classes, and conferences. Each team produces a monthly report and presentation in order to share what they have learned.
We also evangelized, stressing how important it is to include QA early in the agile process. We’ve begun to participate in reviewing SOWs and other early planning sessions. We’ve been able to get ahead of potential issues and propose alternative solutions. Our co-workers have begun to see us as valued experts and not just people who find problems too late to fix them. We’re testing early and often, and quality is being more assured than ever.