I've just been challenged with a sobering reality.
I've heard the term "Exploratory Testing" used many times over the last few years by developers and testers at various gatherings. I've practiced it myself for over 6 years in various black-box system testing efforts. When training new testers on my team, I provided them with foundational concepts in context, risk, scientific method, test techniques and communication. Then over the course of several weeks, I reviewed their test sessions and provided feedback during debrief sessions to improve their understanding and application of the various testing skills required to be efficient and effective.
People have told me that I have really high standards, and perhaps I do. To me, testing is a passion and fun, and quality is an ideal achieved through effective communication and interactions with all the stakeholders on a project.
But that's all besides the point. If the question is "what is Exploratory Testing and how do you do it?" then my standards and expectations from team members are irrelevant.
ET is simply an approach to testing software where the tests are not predefined and the focus is on iterative learning, test design and execution (to paraphrase a simplified definition).
How someone learns, how someone designs tests, how someone executes those tests - these things are not defined by any standard; they are applied differently by different people. ET can be performed by anyone. There aren't any requirements for how well or thoroughly someone should perform it.
To quote from the animated movie "Ratatouille": "Anyone can cook. But I realize, only now do I truly understand what he meant. Not everyone can become a great artist, but a great artist can come from anywhere."
So, when I hear the term ET thrown around, I have about as much understanding of how they're testing as I do from a development shop that uses the term "Agile". That is, I don't know anything about what it means to them, how they're applying it, how effective it is, or how it compares to my standards/expectations.
I've been reading articles and research lately comparing ET and Test Case-driven Testing (TCT) approaches, and it never ceases to amaze me how stats and research may be twisted to support everyone's beliefs about which is better than the other.
Developers and Product Managers who have worked with me understand the quality of the information and feedback that my testing style provides. They have said that it is a whole new level of testing feedback they've never seen before. It makes me feel good to hear that - that I'm providing a valuable service.
But when I read one of these comparison articles, I have to assume that the ET applied in the research studies aren't at the same level that I apply it. I have to accept that. I may not like it, but that's the reality. To me, the same research applied would likely show that Agile and Waterfall aren't really all that different in terms of produced output. Sigh.
Am I missing something?
Hi Paul Carvalho,
ReplyDeleteNice to know a tester who practices ET approach.
One point I wanted to clarify.
With the title of the post being: "What skill does Exploratory Testing require?" the rest of the article seems to be slightly out of sync with the post title.
Correct me if I've interpreted incorrectly.
Regards,
Ajay Balamurugadas
http://enjoytesting.blogspot.com
Nice post. I'm not sure I agree with your comparison to Waterfall/Agile. For me waterfall anf agile do deliver very different results. Especially if applied correctly.
ReplyDeleteAs for ET. I would kind of agree with you. ET is very subjective. It's a technique to apply to your testing and how people apply that technique is very personal.
I would also suggest that ET can be done by anyone, but I wouldn't expect "everyone" to be good at it.
I think ET relies on many personal traits and aspects, of which some people simply do not have. Enthusiasm, passion, domain knowledge, experience, confidence and communication skills are all essential factors I've noted "really good" E Testers possess.
And not everyone possesses these traits. And not everyone wants to. And not everyone can be bothered.
Great post. Nice to get a personal view on Exploratory Testing.
Rob..
Hello Paul, my thoughts exactly. I see ET as a lot more effective and interesting in so many ways. I assume you are referring to the study called Defect Detection Efficiency: Test Case Based vs. Exploratory Testing by Juha Itkonen, Mika Mäntylä and Casper Lassenius? Apparently, there are a lot of things that are not quite right about that study. Michael Bolton did a very interesting blogpost on that recently (http://www.developsense.com/2010/01/defect-detection-efficiency-evaluation.html").
ReplyDeleteMain problems:
- A lot of the testers involved in the study were novices, with no relevant experience.
- The participants received some training on on test techniques, but no training on exploratory approaches or heuristics.
- The study focuses mainly on quantitative results, rather than qualitative ones. There is no mentioning of valuable risk-related information that might come out of the test process.
I look forward to studies on the same subject that are a little less biased.
Hi, Paul...
ReplyDeleteI agree that there are many interpretations of exploratory testing, and I believe that some researchers seem to have an impoverished view of it. So I think there is at least one way in which you could be more helpful in advancing the discussion. In addition to saying, "the research guys don't seem to be talking about what I understand as E.T.", be explicit. Talk about the heuristics that you use, the supporting practices that you use, the problems that you've solved, and so forth. Link to them if you've done it already.
For example: you once showed your version of the testing dashboard. You included a column that indicated the ratio of sessions chartered to sessions completed. Some people might like that and use it; other people might disagree with you, might not like it, and might not use it. But that's okay; the discussion might be illuminating no matter what conclusions people draw.
When people talk about exploratory testing and include "just" or "simply", these days I like to ask them, "Does 'just' or 'simply' include this stuff?" Whatever the answer, it gives us a chance to start a richer conversation.
---Michael B.
Hello Ajay, thank you for the feedback. You are correct in your interpretation. I noticed that perspective myself after I hit [Publish].
ReplyDeleteThere was a nuance of disbelief in the way I said/wrote that title. Picture someone with wide eyes shaking their hands at the sky when you hear those words. ;)
The blog post wasn't intended to be a list of required skills, so I apologise if the title led you to think that's what it would be. In fact, it's intended to communicate the exact opposite - that people do it so differently, how can we say any skills are required at all?
- Paul.
Hello Zeger, thank you for the reply.
ReplyDeleteYes, that was the research/study that I came across in the last few days. There was also a blog and an article drawing "conclusions" from that research that I found particularly infuriating. And yet, that information has brought me a new perspective that people "dabble" in Exploratory Testing (in comparison to how myself and colleagues practice it) and that's just the way it is.
I didn't know about Michael Bolton's blog post until you sent along the link. Thank you! It was a terrific read.
Cheers! Paul.
Hello Michael, thank you for the feedback.
ReplyDeleteI agree that I have much more to share that I haven't yet. I haven't held back intentionally.. I plan to change that this year. (no more excuses.. time to do a major brain-dump!)
I love your resource page! (re-included here due to a typo in the link you sent)
When I look at ET it's usually in light of the bigger picture.. i.e. part of an integrated whole of the testing service provided. The work that you, James, Cem and Mike Kelly (and others!) have published make up the core of what I would consider "good practice" in this field.
Thank you!
I think the approaches like Exploritory Testing and Test Case Driven Testing can be likened to different tools that can be applied in different situations. Speaking personally I find it difficult to understand those that just sit on one side of the fence saying their approach is the only approach worth considering.
ReplyDeleteDepending on the software tesitng task you are faced with, and the environment you are working in, one or other approach may be more suitable. Maybe it is more a question of learning both approaches and having the experience to know in which situation which is the most appropriate approach to apply.
William Echlin
SoftwareTesting.net/blog
Hello William, you wrote:
ReplyDelete> "Depending on the software tesitng task you are faced with, and the environment you are working in, one or other approach may be more suitable."
Exploratory Testing isn't a tool. ET is an entire way of looking at the problem and complexity of testing software, and making the conscious decision to engage your brain in a way that opens you up to more information, insights, and problem identification - in a way that Test Case-driven Testing (TCT) never can or will.
> "Maybe it is more a question of learning both approaches and having the experience to know in which situation which is the most appropriate approach to apply."
I have learned and applied both approaches, and my experience and judgment tell me that ET (done well) is more powerful and effective than TCT.
Done poorly, ET can be no more effective than TCT, and in that case I would agree with you that it doesn't really matter. The reason I sit on one side of that fence is that I don't do ET poorly. I care about the quality of the work that I do, so I employ the best methods I can to do the job well.