Great Example of Exploratory Testing Session notes

Exploratory Testing (ET) can be done well or it can be done poorly. How do you know how well you're doing ET? I believe you need 2 things: (1) you need to keep notes as you perform your testing, and (2) you need good feedback from a competent, experienced tester.

Taking good notes is not easy. It takes practice. A lot of it. In the process, you will learn something about how you organise your thoughts. You will also learn about biases, assumptions, techniques, critical thinking and communication of important facts.

Let's return to organisation of thoughts for a minute. One analogy that I often use is for a tester to think of your test notes like a Science Report. You know.. one of those reports that you likely had to do in elementary or high school for a science project, assignment or fair. There are basic and important sections/elements in a Science Report, and I believe those elements are also key for good test session notes.

I'm not going go into much detail about the comparisons here (for that, see my thoughts on my web site at http://www.staqs.com/sbtm/), rather I thought I'd share a link to a news story that I just came across on the Time.com web site. One journalist decided to perform a test of the new Google "Mail Goggles" feature.

Read the article here: http://www.time.com/time/business/article/0%2C8599%2C1849897%2C00.html?imw=Y

What do you think?

There are a number of things I like about that article. One of the things that struck me the most was how much I liked the "test notes" in that article. I think it's a great example of test notes that you should keep during an ET session. I've reviewed more session notes over the last 5 years than I can remember. The test notes in this Time article are very good.

The "science report" structure is both amusing and helps to efficiently communicate what the author did. I doubt the journalist knows anything about ET or session-based test management. The flow and clarity of the report is very good because the journalist has a skill that I often find lacking in the "poor" ET session reports I've reviewed over the years. What's that skill? A journalist knows how to focus on and communicate the facts. The science report structure helped the author organise her thoughts and communicate the facts efficiently. I liked it. I think it worked. And it was funny too! :)

Which brings me to another important point. Just because you add structure to a report doesn't mean you lose your personality. You can be both clear and funny. Emotion and impressions are important in good notes too because they help raise awareness of the "qualitative" aspects of testing. Too many people put too much value on the "quantitative" aspects of testing.

I like Albert Einstein's quote in regards to this point:

"Not everything that can be counted counts and not everything that counts can be counted."

A good ET session report/test notes should reflect both qualitative and quantitative elements. The science report structure might provide you with a good framework for organising your thoughts. Oh, and if you're looking to improve your technical writing in the Software Testing profession, perhaps you might consider a journalism course.

Testers Create Bugs, Not Bug Reports!

I've been thinking a lot about this phrase lately: "Your thoughts shape your reality." I think it's from a Buddhist teaching but I'm not sure because I read it as a second-hand quote somewhere that I can't recall right now.

I'm also familiar with the tester's lament: "hey, I didn't put the bug in the code, I just report it."

But what about the phrase: "if a tree falls in the forest and no one is around to hear it, does it make a sound?" That one has always annoyed me. The way I see it, the answer to the question depends mostly on how you define what a sound is. If a sound requires a listener to be real, then, no, no sound was made.

What about a bug? If there is a bug in the software but no one encounters it, is there really a bug in the software?

Now, I report bugs. A lot of bugs. I employ many creative and imaginative tricks to find them. They love me. They come to me even when I don't expect it. One of the tricks I use is to change my perspective on how I look at software. I also exploit vague areas, or areas of doubt within the development team, development processes or methods of communication. For example, if it looks like there might be some vague statement in a specification that could be confusing or interpreted in different ways, I'm pretty confident there will be lots of bugs there.

But hold on. What if I don't go looking for the bugs? What if I don't report them and no customer ever sees the problem or complains about it? Does the bug really exist?

Or does the mere act of identifying this discrepancy between intention and execution create the entity that I call a bug? Did I create it or was it already there?

Scenario: A spec-writer wrote a document that was interpreted in some way by one or more programmers. According to their interpretation, the implementation (the software) is working as intended. They check that their code measures up to their understanding (unit testing?) and then build the software into a deliverable product.

Then along comes a tester. A tester looks at it in a completely different way and suddenly the software is full of bugs. But wait a minute! The bugs weren't there until someone looked and said they were there.

This reminds me of Schrödinger's cat. Is the cat alive? Is it dead? Is it both alive and dead, or neither? You have to look to find out, but once you've looked you've changed everything. What a psych trip!

So, what if finding bugs falls into the same category? In that case, the mere act of observing has changed the reality of what's in the box. By looking inside and asking the question, we have shaped the reality by our thoughts.

Doesn't that mean that we have therefore created the bugs that we report? The programmers didn't put the bugs there. They interpreted the specs and design documents according to their thoughts and shaped the software according to their reality. They built no bugs.

The bugs were called into existence when we, the testers, said they were there with our mind tricks and clever tools.

The bug report, therefore, is just a record of how we have reshaped reality with our thoughts in a way that is different from how reality looked before we started testing. We created those bugs in the bug reports.

One might argue, then, that if you make the bug reports go away you make the bugs go away. If no one (customer) sees the bug, does it really exist?


That's a change in perspective from what I was taught about testing. I don't report bugs. I create them!

I Am The Arrow

One of my favourite television series was called Babylon 5. Good stuff. From time to time I recall particularly well-written scenes and moments in that series that seem to apply well to some situation happening in my life. That happened again today.

I had a meeting today that changed everything. I've spent the last decade searching for answers that apparently weren't there. Today I found a well, no a sea, of good answers. It's going to take me a while to assimilate everything and find a way to communicate it back to people. It's not the right piece of information, it's the right way of looking at things.

I said to someone this afternoon that this is the first time in 9 years that I've really learned something NEW. Something that makes me think. Something that makes me think in a new way.

For the first time in almost a decade, the phrase that comes to mind is "I am the arrow." I know where I'm going and what I need to do.

The context for that phrase is from an episode in Babylon 5 - Season 3, episode 61: "War Without End (Part Two)" There was this scene where Sinclair is absolutely certain he knows what is going to happen (in his future which is really everyone else's past - Time paradox stuff) and what he needs to do. That scene has always haunted me. It's not every day that you know with such certainty exactly what you need to do.

In all fairness, that has happened to me once before. It was the day that I started the relationship with my wife -- that was almost 20 years ago. I had no doubt. I was absolutely certain.

Unfortunately, moments like that don't come around very often. I needed to put a little note here to mark the occasion that it happened to me again today. It feels good for a change - to know the solution to a decade-long problem.


Happy St. Patrick's Day! =)

Ross Collard's Tea-Test (Technique)

I was reviewing the test notes for one of my testers today when I came across an interesting note. I decided to look up the problem report in the bug tracking system to read the details. The bug report said that if you go to particular new page in a web app (currently in dev't) and enter some information, wait 35 minutes, and then press a button to continue, you get an error. If you wait less than 30 minutes there is no error, and if you wait over 60 minutes then the application will timeout (as expected), so you have to wait just the right amount of time.

Suddenly I started laughing out loud saying: "Hey, it's the Tea Test!"

I called over the tester whose notes I was reviewing to thank him for his good work in finding, isolating and reporting the bug and to tell him this story. You see, there are many kinds of test techniques out there. At the very least, most programmers and testers have heard about BVA and Equivalence Classes, and some testers who take an interest in their profession learn about other techniques as well. In our test team, we can rattle off at least a dozen techniques at any given time and we are usually selecting from among a pool of 30 or more techniques on any given project, but that's not important right now.

What was important at this moment was that the technique that I could apply to the bug found wasn't on any list I had ever seen, but it was a technique that I knew about.

Back in the summer of 2003, I drove down to Virginia to attend a special 5-day "Black-Box Software Testing" workshop offered by both Cem Kaner and James Bach. It was a great opportunity and I didn't want to miss it. Much to my surprise and delight, Ross Collard had also come to attend the course. Ross had taught me my first courses in Test Case Design and Test Management some 5 years prior, and it is information I still use to this day.

One day during the BBST workshop Cem and James asked the participants to name some test techniques. Ross offered two that I hadn't heard before. The first was the "shoe test". That's where you take off your shoe and put it on the keyboard. Then you wait to see how the app handles the non-stop input.

I've seen this technique happen in real life. I've seen someone lean back against their desk while talking to others not realising that they were leaning on the keyboard. Another time, I saw someone put a magazine on their desk which accidentally landed partly on the keyboard and proceeded to cause a beeping noise from the computer as the keyboard input cache filled up.

This can be an interesting technique as you ponder which key on the keyboard to place your 'shoe' for maximum effect in the App Under Test. For example, how well do you think your web app can handle you pressing [F5] to refresh the page non-stop? Can you find a web page that makes several database calls and then try [F5] repeatedly again? Think you can bring down a database server by doing this? [evil grin]

The second technique Ross mentioned was the "tea test". I hadn't heard of that one before, so I asked if the "T-test" was related to the Statistics t-test. He said "no, it wasn't." He said that what he would do here is enter some input in an app, get up from the desk, walk over to make a cup of tea, have the tea, walk back and enter the next input. And he punctuated this by saying that since he doesn't walk very fast these days, this process could take anywhere from 30-40 minutes. Ha! That was funny. I hadn't heard of any test technique like that before.

Fast forward to today. That was exactly the amount of time (30-40 mins) that my tester had to wait for the bug to appear! The tester told me how he had spent over an hour trying to reproduce that bug in a background VMWare session, so that he could continue with other testing while waiting for the right amount of time to pass. We both laughed at how the "Tea test" applied here.

The developer assigned to fix the bug (who sits on the other side of the desk partition from me) must have overheard me telling this story. He piped up and said: "I hate that bug! I have to wait a long time to try and reproduce it!" We laughed harder. =D

This was the first time I've seen Ross' "Tea test" actually work -- i.e. actually find a bug. I thought it was just a joke at the time. I now know there's truth in that technique. It's not that I really doubted Ross, it's just that he's a funny guy and sometimes you can't tell when he's pulling your leg. =)

Ross, you really are the Test Master. Next time I see you, the tea is on me. Cheers!