On Tuesday August 4, 2015, I presented my experience report on moving from test cases to context-driven testing at the 10th annual Conference for the Association for Software Testing 2015 in Grand Rapids Michigan. This was my inaugural speaking engagement and my second software conference.
Thank you to all who participated by asking questions. The conversation doesn’t stop just because the conference is over. I welcome you to comment here on my blog or contact me on Twitter @1030omlette for further discussion.
I’m looking for a couple of software testers to join my team in San Francisco, where we work on a global mobile payments platform that enables people to buy things easily and pay through their mobile provider. We practice testing using context-driven methods and heuristic test strategies. We are budding exploratory testers. If our style of testing sounds like something you’d like to get involved in, read the job description below and we welcome your application if you think you’re a fit.
View our open positions online and if you are what we’re looking for, what is stopping you from applying?
Trying this out after being inspired by @katrina_tester … So far it's working … I think I'm doing it rightish http://t.co/OsyScxlsN1—
Sean (@seancresswell) August 13, 2014
Each role on the team was assigned a colored sticky note, one for Development, one for Testing, and one for Other (which includes the product owner). Each person on the team was asked to identify how they contribute to testing on this project today and to place the sticky against the two axis. When that was done, I would ask the team to consider the testing activities that are missing from the strategy and to plot their importance in the strategy. I also put up a list of common test techniques and approaches, so that the team has some ideas to work with. This added a little complexity to the session because the list had both approaches and techniques. One question that arose was with respect to a unit test. A unit test may cover a function or a feature or a boundary check and there was doubt about how to reflect that on the chart. Is that a unit test or a boundary test? I asked the team to keep it simple and if they felt they needed to identify a certain technique within an approach, that it was fine and we would discuss it during the session.
In the last few weeks of 2014, I have felt the need to identify what I want to do with my testing career in the year 2015. This desire came from both within my self and later from the leadership at work asking all staff what we want to be doing and where we want to be career-wise in 2015. Having already committed thought to this topic, I had a few ideas for next year, but I need to think more deeply and holistically about my career-related desires so that I set up the right path for next year. I have some time before I have to submit my response and I am using this writing opportunity to reflect on the past year, see what I have done, and continue the thought process of answering those questions.
Note: I’m sharing an abridged version of my year to keep the article length reasonable. Also, this is my first real blog post.
At the start of 2014 I knew the basics of exploratory testing, gained from reading blog posts and articles such as Exploratory Testing Explained by James Bach. I lacked any structured or mentored training and was unexperienced with exploratory testing. I wanted to learn more to gain the confidence that I was missing to give this a try in the workplace (context: I am the test manager and don’t do a lot of testing). I eagerly waited for and purchased my own copy of Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing by Elisabeth Hendrickson. I read right through chapter 1 and once I got to chapter 2 and read about charters, I could feel the missing gaps begin to fill. The story about Lewis and Clark and the simple charter template helped me better understand how to frame my exploration and get value out of the session. Having read through most of the book, I bought a copy of Explore It! for work and shared it with another tester who was interested in exploratory testing. She read through the book and wanted to give it a try. We decided to use James Bach’s Session-Based Test Management (SBTM) for structuring exploration into time-boxed sessions and its sample session sheet to record the charter and outcome of the session. We also included the post-session debrief so that we could share feedback about the session. She performed a handful of sessions and found a few issues, and we were both encouraged by the approach. However, some of the forces working against us included available time and available testers. Our small test team is occupied constantly with feature testing for weekly releases and due to a lack of automation, exploratory testing would at best be a hope-to-have. At present, exploratory testing is something I would like our test team to practice, and pending a change of the conditions of time and automated checks, its on my parking lot list.
A Heuristic Test Strategy
Looking back, I waited a long time before I felt ready to implement any context-driven or heuristic test strategies. I’ve read lots of material on the subjects, but I also had a confidence issue that prevented me from feeling ready to practice it at work. In 2014, my interest level in software testing was rising, likely due to exposure to more blogs, being more active as a Twitter reader, and reading about the positive experiences on the conference circuit, particularly CAST. One day in April I stumbled upon a post from Karen Johnson describing a mnemonic of her design, RCRCRC, used in regression testing. Her post caused me to think about our regression strategy, which was to execute a suite of test cases each iteration. We rarely found bugs. I started to question the value of our strategy. I queried the test case management system and found that it had been at least 6 iterations since we last found an issue. I drew up a mind map of the RCRCRC heuristic and showed it casually to my boss one day. After stating my observations of test case results, he emphatically tasked me with the job of giving it a try. So I tossed the test cases out, as it were, and trained the team to use RCRCRC instead. Part of my argument to move away from test cases was that nobody was interested in the metrics, and I didn’t see value in them enough to even gather them. I needed a replacement and became aware of the idea of reporting dashboards. I learned of James Bach’s Low-Tech Testing Dashboard and implemented a version of that to report our results. By May, we had officially replaced our regression strategy with one based on the context of each release. This was a big win for me both personally and professionally, as I was fulfilling a long time desire to formally practice context-driven testing in the workplace and testing was getting better for the company as well. This practice also fulfilled the testers in a meaningful way, giving them a task that they felt was intellectual and one they truly believed would engender the intended results of quickly finding bugs. Did it work? Well, we did find some bugs but its hard to say if that is the result of our new strategy. We’d like to believe so. We’ll monitor the rate of new bugs post-release as an indicator of the success of our strategy.
Feeling positively on the heels of launching RCRCRC, I wanted to go all out and try to implement a heuristic test strategy and get traction on exploratory testing. In order to do that, I had to guy buy-in from senior leadership, what one of my bosses termed as socializing the idea. I had less success in the past with implementing some ideas, which I didn’t socialize – they didn’t get traction. I was going to take my bosses advice and I knew exactly the resource to go back to – a video I had watched not long before on how to talk to C-Level management about testing, by Keith Klain.
I must have watched that video no less than 10 times to prepare for three conversations I was about to have. One with the SVP of (Partner) Operations, a significant stakeholder in product development, one with the SVP of Product & Marketing and the VP of Product, and lastly the Chief Business Officer who was imminently taking on the CEO role. I decided I would take them all out to lunch where I’d put forth my proposal to rebuild our test strategy. This was important in my presentation strategy because I wanted to build additional rapport with these colleagues, and I needed their support for my plan to be a success.
The first lunch was with someone who has a fondness for my passion and manner of testing, and knew that I did good testing that brought value from our working together when she in a former role, so it was a breeze to elicit her blessing. I made sure to tell her what support I needed from her to make this a success and she pledged that support. My next meeting was with two people who I had less regular interaction with, so I felt I had to work a little harder to obtain their understanding and blessing. To my surprise, they picked up on the ideas very quickly and spoke back to me the ways they could see this bring more value and how this would be better testing that addressed risk. We had a fulfilling conversation, but they put me to the test and gave me some things to thing about which I hadn’t considered before, such as measuring the success of the strategy (How do we know its working?), as well as how we would make it work with some our small testing team. I also told them what support I needed from them, and this was very important as these were the top two leaders of the product department, overseeing all product ownership and management. I needed their buy-in before I could get the product owners and managers to buy in. It worked. My last meeting ended up flipping to the then-imminent CEO taking me out to breakfast. This was the second time I was having a chat with this person and he saw my passion for testing in our first chat and was ready to hear what I had to say. I told him about some of the problems I had observed and how we could solve those problems by implementing a different test strategy, one that was context driven using heuristics and better integrated teams. Very quickly into the conversation, he asked me what I wanted him to do (Keith Klain was right!). I told him that I needed his support and the ways in which he could support me. He was on board and was very excited to see the fruits of these labors. I had the buy in, and it wasn’t so tough, but it was a great learning experience. One thing I learned is that its near impossible to eat your lunch when you are doing a proposal at a restaurant.
Where are things now? Quite similar to exploratory testing. I’m positive for change in the new year with a change in leadership in the workplace. I look forward to following up on this topic.
I enjoyed watching CAST, the annual Conference of the Association of Software Testing, in 2013 from Twitter and YouTube, and I knew I had to attend 2014’s conference. The joy of the attendee’s as seen through Twitter really pushed me to attend, and having just gone through my experiences with with exploratory testing and heuristic strategy implementations, it was a perfect time to attend. And so I did. I had a wonderful time. I met several people who I knew from Twitter or their blogs. I networked, made new friends, learned quite a lot from tutorials, workshops, talks, and games ,and walked away from CAST on a high. I was ready to conquer context-driven testing at work and nothing could stop me. That feeling was one of the interesting effects I felt being amongst like-minded people. It was a great rejuvenator to take back home and keep with me while I do the good work. While at the conference, I realized that my theme for the year had been something along the lines of ‘Don’t wait to be invited.”, meaning don’t wait for someone to invite you to change your test strategy, approach, technique, or process. If you want change, you need to be that change. That is something I think is an important theme for every year, so I’ll be carrying that with me into 2105. I also look forward to an equally fulfilling experience at CAST 2015.
Rapid Testing Intensive
To execute on implementing a heuristic strategy, I felt some training was important, but I had to work within budget constraints. James Bach’s Rapid Testing Intensive was my choice and I got approval for the whole team to attend RTI online. This was an enriching course for us, introducing us to the concepts of product coverage outlines and risk lists, as well as other concepts and ideas. By the end of the second day of the webinar, the feedback from the team was that each of them wanted to use PCOs at work, and that wasn’t it – they wanted to do session-based testing and peer reviews of the work. After the webinar, the testing team was quite excited about trying out some of newly learned skills, and we began to draft PCOs for the products that we were testing and we did peer reviews of the work. Presently, we have about 6 PCOs going, but they are all works in progress, and the review process is working with some success, inhibited by working with testers in remote time zones. I have shown the PCOs to a few folks around the office and there is some interest in them being used more widely, and not just for testing. We also suffer a lack of time to spend on these PCOs and I’ll be working with the new leadership in 2015 to see what we can do about changing that. I think the skills we learned in RTI can bring a lot of positive change in the workplace, and I’ll be working on implementing them next year.
I am happy with the accomplishments I’ve made this year and I am excited about continuing the work next year. New opportunities are presenting themselves as the new year approaches and I am confident that I’ll be successful in driving our new test strategy further into success. I look forward to sharing those experiences with other testers.
Thank you for reading.
I’ve been a follower of numerous blogs and twitter accounts that publish content about software testing, especially context-driven. Up til now, I’ve been only a reader, waiting for the time that I could apply what I had learned and share those experiences with the testing community; my way of giving back.
The catalyst that drove me to start this blog was a plethora of events, starting with my endeavor to implement context-driven testing at my workplace, which I’ll be blogging about in upcoming posts. A few days ago, I was watching the 99 second talks from this years’ Test Bash and there was one particular talk that gave me the push I needed – Richard Bradshaw (10:42):
And that is how I start my blog, Software Test Kitchen, in which I plan to share my experiences in software testing, starting with how I’m implementing context-driven testing in my workplace.