From Test Cases to Context-Driven Testing

Gary Miller speaking at CAST conference.
Gary Miller presenting “From Test Cases to Context-Driven Testing: A Start-up Story” at CAST 2015

On Tuesday August 4, 2015, I presented my experience report on moving from test cases to context-driven testing at the 10th annual Conference for the Association for Software Testing 2015 in Grand Rapids Michigan.  This was my inaugural speaking engagement and my second software conference.

Thank you to all of those who attended and watched online. My talk and the open season questions were recorded and are available online from the AST thanks to Ben Yaroch.

Thank you to Katrina Clokie who encouraged me to speak, as well as Michael Larsen, both who helped me write the proposal.  Your help was invaluable.

My presentation slides are available here  and on the CAST 2015 website.

Thank you to all who participated by asking questions.  The conversation doesn’t stop just because the conference is over.  I welcome you to comment here on my blog or contact me on Twitter @1030omlette for further discussion.

Hiring Software Testers in the SF Bay Area

I’m looking for a couple of software testers to join my team in San Francisco, where we work on a global mobile payments platform that enables people to buy things easily and pay through their mobile provider.  We practice testing using context-driven methods and heuristic test strategies. We are budding exploratory testers. If our style of testing sounds like something you’d like to get involved in, read the job description below and we welcome your application if you think you’re a fit.

View our open positions online and if you are what we’re looking for, what is stopping you from applying?

A Test Strategy Retrospective Experience

At my workplace, we hadn’t ever documented what our test strategy was, let alone worked to identify one.  We worked in an environment that had a single tester and because that tester was always just getting by, there wasn’t much room for techniques beyond feature and function testing.  The environment has changed for the better and we now have a team of testers working with development, taking on more testing activities.  Yes, we could do more, but was it the more of the right testing?  To see where we are and where we want to go, I felt it was time to assemble a test strategy.  I decided to approach a single scrum team to start and used ideas from two of Katrina Clokie’s blog posts to get me started.

 

In Katrina’s experience, she facilitated two types of discussions.  One of them focused on testers on different scrum teams who had drifted from a once unified test strategy.  The other experience focused on a cross-functional agile team engaging in a test strategy retrospective.  I also found out I wasn’t the only one interested in Katrina’s ideas, @seancresswell already gave it a try and I liked his quadrant approach:

 

 

I decided to blend all of this and make it work for my context and share that journey with all of you.

 

Here is what we started with – a blank test strategy based off of Sean and Katrina’s model.

Blank

Each role on the team was assigned a colored sticky note, one for Development, one for Testing, and one for Other (which includes the product owner).  Each person on the team was asked to identify how they contribute to testing on this project today and to place the sticky against the two axis.  When that was done, I would ask the team to consider the testing activities that are missing from the strategy and to plot their importance in the strategy.  I also put up a list of common test techniques and approaches, so that the team has some ideas to work with.  This added a little complexity to the session because the list had both approaches and techniques.  One question that arose was with respect to a unit test.  A unit test may cover a function or a feature or a boundary check and there was doubt about how to reflect that on the chart.  Is that a unit test or a boundary test?  I asked the team to keep it simple and if they felt they needed to identify a certain technique within an approach, that it was fine and we would discuss it during the session.

 

The team then spent about 10 minutes writing down all of their activities on their respective sticky notes and I asked them to plot them on the chart.  When everyone had their ideas up, this is what we saw:
Filled In
Immediately I saw something that I didn’t expect to see – a sea of orange colored stickies (the color assigned to development).  I was surprised to see the high level of contribution to testing by the developers and that was challenging my assumptions in a good way.  I made sure to highlight and acknowledge their contribution to testing on the team, as I felt this was something they don’t get enough credit for.  If they hadn’t plotted their contibutions, I might have never been aware of them.

 

I also saw a lot of overlap.  I took a moment to collate all of the common sticky notes while the team had a discussion about each others placement of sticky notes.  In cases where multiple roles identified a technique or approach, I did not collate them.  This resulted in the tester having their approach and the developers have their approach, even if there as a common technique being used.  While I did that, an interesting conversation arose about what makes a good unit test and what makes a good integration test.  There was also disucssion about overlapping efforts between the two.  I used the opportunity to remind the team that transparency and awareness about testing is important for all team members because it was clear we all had different ideas about testing.  Should we have the same questions asked in a unit test and integration test?  These were all good questions that came up, but it wasn’t the right forum to complete the discussion, so we carried on with reviewing our test strategy.  Once the stickies were collated, I worked with the team to replace them on the chart.  Here is what that looked like.
Final

 

We then discussed each quadrant and talked about the activities that we are doing which we want to reinforce and the activities that we aren’t doing enough of or not at all, which we want to incorporate.  We then talked about how the teams could bring these activities into realization, which would occur as part of the typical agile/scrum ceremonies such as sprint planning and the Three Amigos meeting.

 

To end the meeting, I asked the team to own the test strategy document and make it a living document – meaning that it always reflects what the strategy is.  Since I had seen great contributions from the team, I encouraged them to place the strategy in a visible place so that stakeholders and other teams could see the great things that are being done.  We now have the test strategy document hanging on the wall close to the scrum team who owns it.

 

Now that I have a successful test strategy session under my belt and it is public, it will be a conversation starter.  I’ve already had a product manager from another team ask when we are doing it for them.

 

I want to thank Katrina for sharing her ideas in a way that makes it easy for others to bring into their context, and I also want to thank Sean for sharing his own experience.  From all of that I was able to create my own and have it be something I’m quite proud of.

 

When are you going to have your own test strategy retrospective?  I look forward to hearing about it when you do.

My 2014: A Tester’s Retrospective

In the last few weeks of 2014, I have felt the need to identify what I want to do with my testing career in the year 2015.  This desire came from both within my self and later from the leadership at work asking all staff what we want to be doing and where we want to be career-wise in 2015.  Having already committed thought to this topic, I had a few ideas for next year, but I need to think more deeply and holistically about my career-related desires so that I set up the right path for next year.  I have some time before I have to submit my response and I am using this writing opportunity to reflect on the past year, see what I have done, and continue the thought process of answering those questions.

Note: I’m sharing an abridged version of my year to keep the article length reasonable.  Also, this is my first real blog post.

Exploratory Testing

At the start of 2014 I knew the basics of exploratory testing, gained from reading blog posts and articles such as Exploratory Testing Explained by James Bach.  I lacked any structured or mentored training and was unexperienced with exploratory testing.  I wanted to learn more to gain the confidence that I was missing to give this a try in the workplace (context: I am the test manager and don’t do a lot of testing).  I eagerly waited for and purchased my own copy of Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing by Elisabeth Hendrickson.  I read right through chapter 1 and once I got to chapter 2 and read about charters, I could feel the missing gaps begin to fill.  The story about Lewis and Clark and the simple charter template helped me better understand how to frame my exploration and get value out of the session.  Having read through most of the book, I bought a copy of Explore It! for work and shared it with another tester who was interested in exploratory testing.  She read through the book and wanted to give it a try.  We decided to use James Bach’s Session-Based Test Management (SBTM) for structuring exploration into time-boxed sessions and its sample session sheet to record the charter and outcome of the session.  We also included the post-session debrief so that we could share feedback about the session.  She performed a handful of sessions and found a few issues, and we were both encouraged by the approach.  However, some of the forces working against us included available time and available testers.  Our small test team is occupied constantly with feature testing for weekly releases and due to a lack of automation, exploratory testing would at best be a hope-to-have.  At present, exploratory testing is something I would like our test team to practice, and pending a change of the conditions of time and automated checks, its on my parking lot list.

A Heuristic Test Strategy

Looking back, I waited a long time before I felt ready to implement any context-driven or heuristic test strategies.  I’ve read lots of material on the subjects, but I also had a confidence issue that prevented me from feeling ready to practice it at work.  In 2014, my interest level in software testing was rising, likely due to exposure to more blogs, being more active as a Twitter reader, and reading about the positive experiences on the conference circuit, particularly CAST.  One day in April I stumbled upon a post from Karen Johnson describing a mnemonic of her design, RCRCRC, used in regression testing.  Her post caused me to think about our regression strategy, which was to execute a suite of test cases each iteration.  We rarely found bugs.  I started to question the value of our strategy.  I queried the test case management system and found that it had been at least 6 iterations since we last found an issue.  I drew up a mind map of the RCRCRC heuristic and showed it casually to my boss one day.  After stating my observations of test case results, he emphatically tasked me with the job of giving it a try.  So I tossed the test cases out, as it were, and trained the team to use RCRCRC instead.  Part of my argument to move away from test cases was that nobody was interested in the metrics, and I didn’t see value in them enough to even gather them.  I needed a replacement and became aware of the idea of reporting dashboards.  I learned of James Bach’s Low-Tech Testing Dashboard and implemented a version of that to report our results.  By May, we had officially replaced our regression strategy with one based on the context of each release.  This was a big win for me both personally and professionally, as I was fulfilling a long time desire to formally practice context-driven testing in the workplace and testing was getting better for the company as well.  This practice also fulfilled the testers in a meaningful way, giving them a task that they felt was intellectual and one they truly believed would engender the intended results of quickly finding bugs.  Did it work?  Well, we did find some bugs but its hard to say if that is the result of our new strategy.  We’d like to believe so.  We’ll monitor the rate of new bugs post-release as an indicator of the success of our strategy.

Feeling positively on the heels of launching RCRCRC, I wanted to go all out and try to implement a heuristic test strategy and get traction on exploratory testing.  In order to do that, I had to guy buy-in from senior leadership, what one of my bosses termed as socializing the idea.  I had less success in the past with implementing some ideas, which I didn’t socialize – they didn’t get traction. I was going to take my bosses advice and I knew exactly the resource to go back to – a video I had watched not long before on how to talk to C-Level management about testing, by Keith Klain.

I must have watched that video no less than 10 times to prepare for three conversations I was about to have.  One with the SVP of (Partner) Operations, a significant stakeholder in product development, one with the SVP of Product & Marketing and the VP of Product, and lastly the Chief Business Officer who was imminently taking on the CEO role.  I decided I would take them all out to lunch where I’d put forth my proposal to rebuild our test strategy.  This was important in my presentation strategy because I wanted to build additional rapport with these colleagues, and I needed their support for my plan to be a success.

The first lunch was with someone who has a fondness for my passion and manner of testing, and knew that I did good testing that brought value from our working together when she in a former role, so it was a breeze to elicit her blessing.  I made sure to tell her what support I needed from her to make this a success and she pledged that support.  My next meeting was with two people who I had less regular interaction with, so I felt I had to work a little harder to obtain their understanding and blessing.  To my surprise, they picked up on the ideas very quickly and spoke back to me the ways they could see this bring more value and how this would be better testing that addressed risk.  We had a fulfilling conversation, but they put me to the test and gave me some things to thing about which I hadn’t considered before, such as measuring the success of the strategy (How do we know its working?), as well as how we would make it work with some our small testing team.  I also told them what support I needed from them, and this was very important as these were the top two leaders of the product department, overseeing all product ownership and management.  I needed their buy-in before I could get the product owners and managers to buy in.  It worked.  My last meeting ended up flipping to the then-imminent CEO taking me out to breakfast.  This was the second time I was having a chat with this person and he saw my passion for testing in our first chat and was ready to hear what I had to say. I told him about some of the problems I had observed and how we could solve those problems by implementing a different test strategy, one that was context driven using heuristics and better integrated teams.  Very quickly into the conversation, he asked me what I wanted him to do (Keith Klain was right!).  I told him that I needed his support and the ways in which he could support me.  He was on board and was very excited to see the fruits of these labors.  I had the buy in, and it wasn’t so tough, but it was a great learning experience.  One thing I learned is that its near impossible to eat your lunch when you are doing a proposal at a restaurant.

Where are things now?  Quite similar to exploratory testing.  I’m positive for change in the new year with a change in leadership in the workplace.  I look forward to following up on this topic.

CAST

I enjoyed watching CAST, the annual Conference of the Association of Software Testing, in 2013 from Twitter and YouTube, and I knew I had to attend 2014’s conference.  The joy of the attendee’s as seen through Twitter really pushed me to attend, and having just gone through my experiences with with exploratory testing and heuristic strategy implementations, it was a perfect time to attend.  And so I did.  I had a wonderful time.  I met several people who I knew from Twitter or their blogs.  I networked, made new friends, learned quite a lot from tutorials, workshops, talks, and games ,and walked away from CAST on a high.  I was ready to conquer context-driven testing at work and nothing could stop me.  That feeling was one of the interesting effects I felt being amongst like-minded people.  It was a great rejuvenator to take back home and keep with me while I do the good work.  While at the conference, I realized that my theme for the year had been something along the lines of ‘Don’t wait to be invited.”, meaning don’t wait for someone to invite you to change your test strategy, approach, technique, or process.  If you want change, you need to be that change.  That is something I think is an important theme for every year, so I’ll be carrying that with me into 2105.  I also look forward to an equally fulfilling experience at CAST 2015.

Rapid Testing Intensive

To execute on implementing a heuristic strategy, I felt some training was important, but I had to work within budget constraints.  James Bach’s Rapid Testing Intensive was my choice and I got approval for the whole team to attend RTI online.  This was an enriching course for us, introducing us to the concepts of product coverage outlines and risk lists, as well as other concepts and ideas.  By the end of the second day of the webinar, the feedback from the team was that each of them wanted to use PCOs at work, and that wasn’t it – they wanted to do session-based testing and peer reviews of the work.  After the webinar, the testing team was quite excited about trying out some of newly learned skills, and we began to draft PCOs for the products that we were testing and we did peer reviews of the work.  Presently, we have about 6 PCOs going, but they are all works in progress, and the review process is working with some success, inhibited by working with testers in remote time zones.  I have shown the PCOs to a few folks around the office and there is some interest in them being used more widely, and not just for testing.  We also suffer a lack of time to spend on these PCOs and I’ll be working with the new leadership in 2015 to see what we can do about changing that.  I think the skills we learned in RTI can bring a lot of positive change in the workplace, and I’ll be working on implementing them next year.

I am happy with the accomplishments I’ve made this year and I am excited about continuing the work next year.  New opportunities are presenting themselves as the new year approaches and I am confident that I’ll be successful in driving our new test strategy further into success.  I look forward to sharing those experiences with other testers.

Thank you for reading.

Why I decided to start a software testing blog

I’ve been a follower of numerous blogs and twitter accounts that publish content about software testing, especially context-driven.  Up til now, I’ve been only a reader, waiting for the time that I could apply what I had learned and share those experiences with the testing community; my way of giving back.

The catalyst that drove me to start this blog was a plethora of events, starting with my endeavor to implement context-driven testing at my workplace, which I’ll be blogging about in upcoming posts.  A few days ago, I was watching the 99 second talks from this years’ Test Bash and there was one particular talk that gave me the push I needed – Richard Bradshaw (10:42):

And that is how I start my blog, Software Test Kitchen, in which I plan to share my experiences in software testing, starting with how I’m implementing context-driven testing in my workplace.

-Gary