Great Example of Exploratory Testing Session notes

Exploratory Testing (ET) can be done well or it can be done poorly. How do you know how well you're doing ET? I believe you need 2 things: (1) you need to keep notes as you perform your testing, and (2) you need good feedback from a competent, experienced tester.

Taking good notes is not easy. It takes practice. A lot of it. In the process, you will learn something about how you organise your thoughts. You will also learn about biases, assumptions, techniques, critical thinking and communication of important facts.

Let's return to organisation of thoughts for a minute. One analogy that I often use is for a tester to think of your test notes like a Science Report. You know.. one of those reports that you likely had to do in elementary or high school for a science project, assignment or fair. There are basic and important sections/elements in a Science Report, and I believe those elements are also key for good test session notes.

I'm not going go into much detail about the comparisons here (for that, see my thoughts on my web site at http://www.staqs.com/sbtm/), rather I thought I'd share a link to a news story that I just came across on the Time.com web site. One journalist decided to perform a test of the new Google "Mail Goggles" feature.

Read the article here: http://www.time.com/time/business/article/0%2C8599%2C1849897%2C00.html?imw=Y

What do you think?

There are a number of things I like about that article. One of the things that struck me the most was how much I liked the "test notes" in that article. I think it's a great example of test notes that you should keep during an ET session. I've reviewed more session notes over the last 5 years than I can remember. The test notes in this Time article are very good.

The "science report" structure is both amusing and helps to efficiently communicate what the author did. I doubt the journalist knows anything about ET or session-based test management. The flow and clarity of the report is very good because the journalist has a skill that I often find lacking in the "poor" ET session reports I've reviewed over the years. What's that skill? A journalist knows how to focus on and communicate the facts. The science report structure helped the author organise her thoughts and communicate the facts efficiently. I liked it. I think it worked. And it was funny too! :)

Which brings me to another important point. Just because you add structure to a report doesn't mean you lose your personality. You can be both clear and funny. Emotion and impressions are important in good notes too because they help raise awareness of the "qualitative" aspects of testing. Too many people put too much value on the "quantitative" aspects of testing.

I like Albert Einstein's quote in regards to this point:
"Not everything that can be counted counts and not everything that counts can be counted."

A good ET session report/test notes should reflect both qualitative and quantitative elements. The science report structure might provide you with a good framework for organising your thoughts. Oh, and if you're looking to improve your technical writing in the Software Testing profession, perhaps you might consider a journalism course.

4 comments:

  1. I laughed. Neat link.

    An additional observation: Good session notes often stand up to different approaches to reading. In this case, the timestamps let the reader judge the rate of consumption, and the absence of quantity information after midnight might indicate that the experimenter had started to loose clear recall.

    I frequently find that looking over old session reports, especially for something that's currently playing up, reveals things that I had noted, but had not noticed, as I tested.

    Cheers - James

    ReplyDelete
  2. >>>Exploratory Testing (ET) can be done well or it can be done poorly. How do you know how well you're doing ET? I believe you need 2 things:

    Paul, You started this post with a bang !!! (first three sentences point to that aspect). I think you did not connect these 3 sentences to the 4th one here.

    I am still at loss to understand "how well am I doing exploratory testing"

    Points on taking notes and presence/guidance are great but they are still enablers for good ET and I am not sure if they tell you the relative goodness of your ET?

    Am I making sense?

    Shrini Kulkarni
    http://shrinik.blogspot.com

    ReplyDelete
  3. Shrini, as you know, the real testing takes place in your head.

    So, when you are learning how to test, how do you measure your progress? How do you get what's in your head *out* so that someone else can help you to do it better? Communication is the key.

    There are about a dozen or more things that a good exploratory tester needs to keep in mind while testing. Note-taking is not one of them.

    HOWEVER, if you wish to communicate your testing progress to someone else, for some purpose, THEN note-taking is important.

    I think that note-taking is an important *complementary* skill for good exploratory testing. In the process of taking notes, you need to organise your thoughts. You will expose your biases, assumptions, blind spots and weaknesses, as well as identify testable elements, risks, oracles, techniques, observations, tools and inferences.

    Gosh. That's a mouthful for just one sentence.. just imaging what it's like to look for those things all the time in someone else's testing notes. I can't look for them in someone's head because I am not always around to watch a tester actually test.

    I can, however, *see* all those things in someone's test notes. It is my music. I *hear* the rhythm of someone's thoughts in the words they say and don't say in their notes. In the process, I can provide them with effective feedback to improve their testing ability.

    Without the notes, I would be unable to understand how well they are thinking about the testing. Without the notes, the "session debrief" is pretty much meaningless.

    So, if you are going to take notes, there needs to be a structure to it to facilitate the communication. The SBTM session sheet template is a good start by providing some high-level sections. The "TEST NOTES" section is the real key to the ET session though. In conjunction with the Charter (or "Hypothesis"), it tells you what the tester was thinking (i.e. *testing*) during that time period.

    I believe that a science report structure is a pretty good framework to help a tester organise their thoughts. It's not the only thing, but I think it is one of the good starting points.

    Does that help? Did I answer your question? If not, please let me know.

    Take care & all the best! Thanks for the note.

    Paul.

    ReplyDelete
  4. Paul,

    You make a convincing point about creating well-structured test notes during an ET session. I especially like notes created when the author is not holding him/ her back. Not only would such notes help another tester provide feedback on the session, they would also help the author remember the session in vivid detail.

    Thanks,
    Inder

    ReplyDelete