I've been a big fan of the idea of "Good Enough" software testing over the last decade. Rather than thinking that the problem of doing good Software Testing is akin to "Digital" technology with it's complete, precise values, I've thought of it more like "Analog" technology with the big dials and reasonable, approximate (and cheaper) signals.
This past week, I've watched my seven year old son play with a new LEGO set that he got for Christmas. It's a neat mechanical lego set that lets him build a motorised helicopter, cool car, or attack crab thingy. (ASIDE: I can't begin to imagine what the Marketing team's conversation was like when they thought up that last one!) I noticed when he completed the helicopter and turned on the motor, that it didn't sound right to me. So I went over and took a close look at his creation. It looked correct. There didn't seem to be any missing pieces, but when he turned it on again, I noticed that not all of the gears turned together consistently. I picked it up and took a really good look at it. Not knowing much about how it was built, I just randomly squeezed together lego pieces that weren't tightly packed together whenever I came across them.
There was one set of lego pieces that had a gap of about a millimetre. When I squeezed them together, it made a (good) snap sound. I asked my son to turn on the motor again and this time it not only sounded correct, but the gears all worked together in perfect synch also. Voila!
I thought about this for a few moments afterwards. Up until then, my son had worked on the premise that if the lego pieces were reasonably attached, that it was "good enough". He didn't need to have a tight fit between every single piece to see the finished product. I mean, it looked like the complete picture of the helicopter in the instruction manual, so what difference would a small gap between a few pieces make?
In this case it made a big difference. If it needs to work like clockwork, then "good enough" is probably not enough.
So what's the tie in to Software Testing? Well, just how scalable is the "Good Enough" approach? For me, it's always been about testing to the most important Risks and using whatever tools and techniques seem appropriate to the situation at hand. It's always seemed kind of foolproof to me.
Maybe my Digital/Analog analogy is a flawed one. I mean, Analog technology has its limits and is not very scalable. Digital technology is more precise and can handle more information. Is there a point when a Digital solution gets so large that it requires an Analog approach again? (I think the answer here is 'yes.')
Is there a time when "good enough" needs to be replaced with a more complete, structured or methodical approach to software testing? I can't think of any situations like that right now, but that doesn't mean there aren't any. That is, I can't think of a time when I wouldn't want to say that good software testing has to strike a balance between the economics, quality and time to market for a product or system. Shipping with bugs is okay if you know that they aren't critical or life-threatening.
So perhaps "good enough" doesn't always apply when we're dealing with real-world objects like lego creations, automobiles, watches, et cetera. I think that it still holds pretty well to the virtual world of software testing. Until someone can give me a good example or two of when "good enough" wouldn't be good enough for testing software, I think I'll chalk this up to another distinction between testing software and testing hardware.
Ruby at Work - Internet access through a Proxy Server
While reviewing Brian Marick's book "Scripting for Testers" (due out later this year or early next), I discovered that some of the internet access RUBY commands and scripts always failed when I tried them from work. Here's the weird thing - the same commands and scripts worked from my home computer. So what gives?
I found the answer when I googled comp.lang.ruby and tinkered around in IRB from my work computer. It turned out to be the Proxy Server at work. For lack of a better place to store this nugget of knowledge, this seems like a good place for now.
THE PROBLEM:
Open a command prompt on a Windows machine, and launch IRB. Issue the following commands:
----
irb(main):001:0> require 'open-uri'
=> true
irb(main):002:0> url = 'http://www.amazon.com/gp/product/0974514055'
=> "http://www.amazon.com/gp/product/0974514055"
irb(main):003:0> page = open(url)
Errno::EBADF: Bad file descriptor - connect(2)
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `initialize'
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
from c:/ruby/lib/ruby/1.8/timeout.rb:48:in `timeout'
from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
from c:/ruby/lib/ruby/1.8/net/http.rb:555:in `do_start'
from c:/ruby/lib/ruby/1.8/net/http.rb:544:in `start'
from c:/ruby/lib/ruby/1.8/open-uri.rb:245:in `open_http'
from c:/ruby/lib/ruby/1.8/open-uri.rb:629:in `buffer_open'
from c:/ruby/lib/ruby/1.8/open-uri.rb:167:in `open_loop'
from c:/ruby/lib/ruby/1.8/open-uri.rb:165:in `open_loop'
from c:/ruby/lib/ruby/1.8/open-uri.rb:135:in `open_uri'
from c:/ruby/lib/ruby/1.8/open-uri.rb:531:in `open'
from c:/ruby/lib/ruby/1.8/open-uri.rb:86:in `open'
from (irb):3
----
THE SOLUTION:
You need to add the Proxy Server address (and user & password, if required) to the URL you are trying to access. Here's an example of something that worked for me from my work machine (replacing the relevant proxy address as required, of course):
----
require 'net/http'
host = 'www.amazon.com'
path = '/gp/product/0974514055'
proxy_addr = 'your.proxy.host'
proxy_port = 8080
response = Net::HTTP::Proxy(proxy_addr, proxy_port).get_response(host,
path)
----
=> response.body now returns the page source, just like it does from my home computer. (Response '200')
The second problem I had accessing a remote page was fixed with the following:
----
require 'open-uri'
url = 'http://www.amazon.com/gp/product/0974514055'
proxy_addr = 'http://your.proxy.host:'
proxy_port = 8080
page = open(url, :proxy => (proxy_addr + proxy_port.to_s))
text = page.read; nil
----
Note that the proxy_addr is slightly different here than with the 'net/http' commands above. (You could probably put it all in one variable here.)
If your Proxy server requires Username and Password, then the open command will be longer. (Luckily I don't need to enter that here at my work.)
I don't have any RUBY scripts that do this sort of thing yet, but I was happy to figure out the solution anyway. Nice change of pace. Ruby totally rocks. I have yet to find something that I can't do with it for testing automation.
I found the answer when I googled comp.lang.ruby and tinkered around in IRB from my work computer. It turned out to be the Proxy Server at work. For lack of a better place to store this nugget of knowledge, this seems like a good place for now.
THE PROBLEM:
Open a command prompt on a Windows machine, and launch IRB. Issue the following commands:
----
irb(main):001:0> require 'open-uri'
=> true
irb(main):002:0> url = 'http://www.amazon.com/gp/product/0974514055'
=> "http://www.amazon.com/gp/product/0974514055"
irb(main):003:0> page = open(url)
Errno::EBADF: Bad file descriptor - connect(2)
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `initialize'
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
from c:/ruby/lib/ruby/1.8/timeout.rb:48:in `timeout'
from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
from c:/ruby/lib/ruby/1.8/net/http.rb:555:in `do_start'
from c:/ruby/lib/ruby/1.8/net/http.rb:544:in `start'
from c:/ruby/lib/ruby/1.8/open-uri.rb:245:in `open_http'
from c:/ruby/lib/ruby/1.8/open-uri.rb:629:in `buffer_open'
from c:/ruby/lib/ruby/1.8/open-uri.rb:167:in `open_loop'
from c:/ruby/lib/ruby/1.8/open-uri.rb:165:in `open_loop'
from c:/ruby/lib/ruby/1.8/open-uri.rb:135:in `open_uri'
from c:/ruby/lib/ruby/1.8/open-uri.rb:531:in `open'
from c:/ruby/lib/ruby/1.8/open-uri.rb:86:in `open'
from (irb):3
----
THE SOLUTION:
You need to add the Proxy Server address (and user & password, if required) to the URL you are trying to access. Here's an example of something that worked for me from my work machine (replacing the relevant proxy address as required, of course):
----
require 'net/http'
host = 'www.amazon.com'
path = '/gp/product/0974514055'
proxy_addr = 'your.proxy.host'
proxy_port = 8080
response = Net::HTTP::Proxy(proxy_addr, proxy_port).get_response(host,
path)
----
=> response.body now returns the page source, just like it does from my home computer. (Response '200')
The second problem I had accessing a remote page was fixed with the following:
----
require 'open-uri'
url = 'http://www.amazon.com/gp/product/0974514055'
proxy_addr = 'http://your.proxy.host:'
proxy_port = 8080
page = open(url, :proxy => (proxy_addr + proxy_port.to_s))
text = page.read; nil
----
Note that the proxy_addr is slightly different here than with the 'net/http' commands above. (You could probably put it all in one variable here.)
If your Proxy server requires Username and Password, then the open command will be longer. (Luckily I don't need to enter that here at my work.)
I don't have any RUBY scripts that do this sort of thing yet, but I was happy to figure out the solution anyway. Nice change of pace. Ruby totally rocks. I have yet to find something that I can't do with it for testing automation.
"The Effects"
Patterns are funny things. Sometimes it is so easy to jump to silly conclusions because we notice the obvious even if we know there might be valid explanations for them.
Take for example patterns of behaviour. Here are a few things that I see on a regular basis:
Ha ha. Is that Murphy's Law, or what?
I wonder how many other "Effects" there are in this industry? Write back a Comment to this Blog entry to let me know if you've come across any other immediate patterns. I'd be very interested to hear them.
Cheers!
Take for example patterns of behaviour. Here are a few things that I see on a regular basis:
- The Developer Effect - The software application always works as expected on a Developer's computer.
- The IT Effect - Any weird or unexpected computer behaviour (e.g. unable to print) will suddenly start to work correctly when you call an IT Helpdesk person over and they are looking over your shoulder as you retry the steps.
- The QA Effect - Any normally working system will suddenly stop working as expected when a Tester starts to look at something.
Ha ha. Is that Murphy's Law, or what?
I wonder how many other "Effects" there are in this industry? Write back a Comment to this Blog entry to let me know if you've come across any other immediate patterns. I'd be very interested to hear them.
Cheers!
I Know What I'm Doing!
This will sound weird but I had a strange realisation the other day. I thought to myself: "I know what I'm doing!"
Okay, this didn't just come up out of the blue and I wasn't practicing some sort of confidence-building exercise. It all started when I happened to notice a bug in the software I was testing and I decided not to report it.
What?! But you're a tester, man! Report the bugs you find!
Ah, well, yes, I plan to. But you see, I was busy at that particular moment thinking about something else, and I didn't want to distract myself by following up on a different thread that might make me lose my train of thought. To justify not reporting it I had to reflect for a moment as to how likely it would be that I would find the bug again. I had absolute confidence that I would find the bug again, so I simply forgot about it and went back to what I was working on.
I think it was later that night that the event started to settle in my mind and it started to make synapse connections with other memories in my brain. One particular memory came to me from my teenage years, some 20 years ago.
I recall one time when I was at a party and was talking to some of my older sister's friends. Coincidentally enough I think the topic of conversation was (creepy, crawly) bugs. There was this one guy who said that he was playing a video game in his home when he saw a millipede scurry across the floor. He said that his reaction was to pause for a moment, smile, and then keep playing his game. Someone asked him why he didn't just drop what he was doing and catch the icky bug right away?
His reply? He said that he knew those bugs and that he would easily find it later and take care of it then. He wasn't worried, so he didn't let it interrupt his game since there was no pressing need to take care of it right away.
I remembered that story for a long time and I always thought that if I ever saw an undesired bug I would simply take care of it then and there. Too many variables in life to let opportunities pass you by.
And so I have. For many years, I have always taken care of the bugs that I've found right away - both the creepy crawly kind, and the software kind.
So then, what happened this time? Why did I let a software bug go and not report it right away?
I realised that I was absolutely confident that I would find that bug again, and since there was no immediate need to take care of it, I let it go for the time being.
Sure enough, I got around to testing that particular area of the application and I had the opportunity to properly focus on the bug and report it with sufficient detail (it was a tricky one). I felt really great about it too.
When you do Exploratory Testing you are faced with the reality that none of your tests are pre-scripted. Exploratory Testing is kind of like what improvisation is to acting: there's no script. It certainly doesn't mean that the Improv people don't know how to act. In fact, it's quite the opposite. They are very talented actors. They just happen to know how to react to the circumstances that arise and can adapt to it with style.
I must confess that I've still been a bit nervous about Exploratory Testing. It doesn't help that I am bombarded with doubt by every other tester in the profession still doing it the traditional way: What if you miss something? How can you be sure of coverage? How do you know when to stop testing? and so on.
Well, I now *know* that I know what I'm doing. If I can totally forget about a bug and then stumble upon it again at a later date, then there must be something to this "improvisation" stuff after all.
Okay, this didn't just come up out of the blue and I wasn't practicing some sort of confidence-building exercise. It all started when I happened to notice a bug in the software I was testing and I decided not to report it.
What?! But you're a tester, man! Report the bugs you find!
Ah, well, yes, I plan to. But you see, I was busy at that particular moment thinking about something else, and I didn't want to distract myself by following up on a different thread that might make me lose my train of thought. To justify not reporting it I had to reflect for a moment as to how likely it would be that I would find the bug again. I had absolute confidence that I would find the bug again, so I simply forgot about it and went back to what I was working on.
I think it was later that night that the event started to settle in my mind and it started to make synapse connections with other memories in my brain. One particular memory came to me from my teenage years, some 20 years ago.
I recall one time when I was at a party and was talking to some of my older sister's friends. Coincidentally enough I think the topic of conversation was (creepy, crawly) bugs. There was this one guy who said that he was playing a video game in his home when he saw a millipede scurry across the floor. He said that his reaction was to pause for a moment, smile, and then keep playing his game. Someone asked him why he didn't just drop what he was doing and catch the icky bug right away?
His reply? He said that he knew those bugs and that he would easily find it later and take care of it then. He wasn't worried, so he didn't let it interrupt his game since there was no pressing need to take care of it right away.
I remembered that story for a long time and I always thought that if I ever saw an undesired bug I would simply take care of it then and there. Too many variables in life to let opportunities pass you by.
And so I have. For many years, I have always taken care of the bugs that I've found right away - both the creepy crawly kind, and the software kind.
So then, what happened this time? Why did I let a software bug go and not report it right away?
I realised that I was absolutely confident that I would find that bug again, and since there was no immediate need to take care of it, I let it go for the time being.
Sure enough, I got around to testing that particular area of the application and I had the opportunity to properly focus on the bug and report it with sufficient detail (it was a tricky one). I felt really great about it too.
When you do Exploratory Testing you are faced with the reality that none of your tests are pre-scripted. Exploratory Testing is kind of like what improvisation is to acting: there's no script. It certainly doesn't mean that the Improv people don't know how to act. In fact, it's quite the opposite. They are very talented actors. They just happen to know how to react to the circumstances that arise and can adapt to it with style.
I must confess that I've still been a bit nervous about Exploratory Testing. It doesn't help that I am bombarded with doubt by every other tester in the profession still doing it the traditional way: What if you miss something? How can you be sure of coverage? How do you know when to stop testing? and so on.
Well, I now *know* that I know what I'm doing. If I can totally forget about a bug and then stumble upon it again at a later date, then there must be something to this "improvisation" stuff after all.
Do you feel like the Stanford Bear?
I was asked just now if I felt like the Stanford Bear. I had to laugh, because sometimes I do.
What's the Stanford Bear, you ask? That's a good question. I'll tell you what I know..
I heard a story recently of an introductory computer class at Stanford University where the Teaching Assistants were so tired of answering the same questions over and over again that they programmed a machine to answer all the commonly asked questions. They added voice-recognition software to it and packaged it up nicely in a large teddy bear that sat on a shelf at one end of the class. Now, when a student had a problem or question, the TA's wouldn't help the student unless they knew that they had gone to the bear first. This relieved much of the workload from the TA's so that they could focus on the problems that really required their attention.
I'll admit that I don't know how much truth there is in this story. I spent some time scanning the internet and I couldn't find any pages to corroborate the tale. I'd like to think that it's true though.
True or not, it got me to wondering as to whether or not you could program something to give out canned answers on Software Testing to commonly asked questions.
I don't think that you could do it so that it would recognise every possible context and provide you with an appropriate response for every situation. (While a cool idea, let's be realistic here!) However, within a particular context - say revolving around the particular testing procedures for a particular class or industry, I'd think this might be do-able.
Better yet, I think it might be cool to program a Testing Bear that's hooked up to a central database somewhere. That way, people could program the Bear to answer all the common questions within their context, but then people all over the world would have access to the same questions and answers.
Now that might be something! Hmm.
What's the Stanford Bear, you ask? That's a good question. I'll tell you what I know..
I heard a story recently of an introductory computer class at Stanford University where the Teaching Assistants were so tired of answering the same questions over and over again that they programmed a machine to answer all the commonly asked questions. They added voice-recognition software to it and packaged it up nicely in a large teddy bear that sat on a shelf at one end of the class. Now, when a student had a problem or question, the TA's wouldn't help the student unless they knew that they had gone to the bear first. This relieved much of the workload from the TA's so that they could focus on the problems that really required their attention.
I'll admit that I don't know how much truth there is in this story. I spent some time scanning the internet and I couldn't find any pages to corroborate the tale. I'd like to think that it's true though.
True or not, it got me to wondering as to whether or not you could program something to give out canned answers on Software Testing to commonly asked questions.
I don't think that you could do it so that it would recognise every possible context and provide you with an appropriate response for every situation. (While a cool idea, let's be realistic here!) However, within a particular context - say revolving around the particular testing procedures for a particular class or industry, I'd think this might be do-able.
Better yet, I think it might be cool to program a Testing Bear that's hooked up to a central database somewhere. That way, people could program the Bear to answer all the common questions within their context, but then people all over the world would have access to the same questions and answers.
Now that might be something! Hmm.
Subscribe to:
Posts (Atom)