Lessons Learned by a Software Tester

Quality Agile Metrics

2012-01-27T00:24:00.001-05:00

I was asked recently what metrics I would collect to assess how well an agile team is improving. I paused for a moment to scan through 12 years of research, discussion, memories and experiences with Metrics on various teams, projects and companies - mostly failed experiments. My answer to the question was to state that I presently only acknowledge one Metric as being meaningful: Customer Satisfaction.

We discussed the topic further and I elaborated some more on my experiences. Regarding specific "quality" metrics, I explained that things like counting Test Cases and bug fix rates are meaningless. I also referred to the book "Implementing Lean Software Development" by Mary and Tom Poppendieck (which I highly recommend BTW) which warns against "local optimizations" because they will eventually sabotage optimization of the whole system. In other words, if I put a metric in place to try and optimize the Testing function, it doesn't mean the whole [agile] development team's efficiency will improve.

It needs to be a whole team approach to quality and value. Specific measurements and metrics often lead to gaming of the system and focus on improving the metrics rather than putting the focus on delivering quality and value. If the [whole] team is measured on the customer satisfaction, then that is what they will focus on. I have long since stopped measuring individual performance on a team.

I haven't stopped thinking about this question though, so I put this question out on Twitter this morning:

Aside from Customer Satisfaction, are there any other Quality metrics you'd recommend in an #agile environment?

Here are the responses I received:

Churn or team turnover. (Real case: Product delivered on time, customer happy, whole team left.)
Escaped defects, inbound support calls/emails
Value created - Reference: "Lean Startup" by Eric Ries (I'm reading it right now)
Number of contributors to each story - not because exact count is meaningful but because it encourages collaboration & review.
Profitability of the project is one. Are the goals (whatever they may be) of the project met?
Lines of code changed during regression. That can expose some severe problems.
Production Defects in 15 days after 'Go Live'.
Cost of rework due to requirement changes.
Code churn as a measure of "quality" (e.g. System Defect Density) - Research, Code example, Discussion, and Sample Stats.

Hm, almost a Top 10 list. I am happy with the responses here and think many of them may be worth exploring further to see what insights they provide. Cautionary Note with all Metrics: Beware the impact they have on the team. If behaviour or performance starts to change in a negative way, STOP immediately!

One of the things I talked about in the conversation was the importance of Retrospectives to allow the team to own their improvement activities. If the team uses these opportunities, their improvements should be observable over several iterations. I think a Happiness Index reading during Retrospectives might be an interesting indicator of the overall effectiveness of the improvement strategies employed.

What do you think? Anything else I should consider?

Testers, Learn about Agile (and Lean)

2011-12-06T01:34:00.000-05:00

Let me tell you about something called Dramatic Irony. You see it in movies, television shows, plays and in many other places. It happens when you (as the audience or observer) see or understand something that the main characters don't. Often times this is funny, sometimes it's not. Personally, I am one of those that likes to laugh when I see this happen.

On my learning/education quest over a decade ago, I took many different positions and roles within various IT organisations so that I could learn different aspects of Quality. I went through various phases, and the one I am least proud of was the "Quality champion." This wasn't a job title so much as a belief that (mis-)guided my actions. The role/part/perspective came mainly from believing what my employer(s) told me at the time - namely that "the QA/Test team was responsible for quality."

If you have worked in Software Development for a while, and perhaps for a larger organisation, you have likely seen someone who believes they are a Quality Champion. They don't want to see any (known) bugs go out; they check up on everyone in the team to see that they have done their reviews or had someone else inspect their work before passing it onto the next person/team; they join committees to create, document, maintain or present processes that will increase the quality of the delivered products/solutions; and so on.

Ah, the poor misguided fools. Bless their hearts.

The first problem is in the company creating a scapegoat culture that puts the responsibility/blame of poor quality on the group of individuals who are least likely to help change the quality late in a development cycle - especially when the (test) team is under-informed, under-funded, under-staffed, under ridiculous time constraints, unappreciated and/or uneducated.

Quality is everyone's job. And I mean everyone. It starts with the president, moves through every person in the organisation and even includes the customers and users of your product/solution.

Returning to the naive tester who doesn't know or understand this, they do their part to motivate, inspire, nudge and, to some extent, manage the individuals affecting the quality of the released products. The effect of this is easy to predict. The nail that sticks out gets hammered down.

As it happened to me, I have seen it happen to other testers - they become disheartened, give up and withdraw back into their own work routine, ignoring everyone else and just focussing on their part.

I didn't give up entirely though - I can be persistent. I kept my eyes and ears open and looked in different places for ideas to help me understand how the organisation/system and "quality" fit together.

At the start of the 21st century, I stumbled upon something called the Agile Manifesto. I can't say that I completely understood it at the time but I was certainly excited about it. I mentioned it to my manager and he said it was a passing fad and that in 5 years no one would ever remember it. I felt that he was wrong and trusted my instincts on this one.

Over the next several years I learned about Agile and the different implementations. It all seemed very programmer/developer-centric to me as none of the models, articles, books or people ever seemed to talk about the testers. There was certainly a lot about devs taking responsibility for the quality of their work and incorporating testing practices into their regular routines. Hey, I'm all over that! Rah, rah, rah, sis-boom-bah, yaaaay Agile! :-D

Now, a decade later, I understand the Agile movement in a deeper, richer way. It is part of how I think and solve problems. It is part of how I encourage people to work together and to focus on the things that matter when delivering value to the customers. Just as I once described myself as someone who eats, sleeps and breathes 'Testing', I believe I would say the same thing about 'Agile'.

Here's the kick: the Agile movement is an ENTIRE COMMUNITY OF QUALITY CHAMPIONS! (oops, the caps lock got stuck for a moment there.)

That's right, listen up all of you testers out there who think you are alone and that no one is listening to your cries of "there must be a better way." There are people out there - Agile Coaches and Consultants - who are working to do just that. They find, create and teach better ways for development teams to work together to raise the quality/value of the delivered solutions.

If you don't know anything about Agile, start now. Read up on it, talk to others, attend a course, find online webinars, go to a conference - anything! Just get out there are start learning about Agile now! The same applies for Lean Software Development - learn about it.

Please keep in mind that there is a big difference between going through the motions of agile practices and actually being/thinking "agile". The agile mindset is more important to me than any particular set of practices.

This is especially important to keep in mind if someone tells you that testers have no place in Agile teams. Those people are what I like to call "wrong." (Get off my lawn.)

Testers can bring valuable insights to the agile software development process if the team works together and embraces the strengths of each team member. You, dear testers, must be open to change and adapt your role to work in new ways.

Warning, Warning, Danger, Danger: if someone tells you that you are "doing agile" and you (as a tester) should keep updating your manual regression test cases and test plans, please tell them to kindly "get off my lawn" for me. Thanks.

I still see this happen. I go into a client's office and look at how the software development team members are working. I see the testers off on their own, testing things in isolation and complaining about how no one seems to care about Quality because they only see and test the product at the end, just before it goes out the door.

Ooo, irony. I see something happening around you (within the community, industry, and sometimes within your own company) that you don't see.

Testers: if you feel like you are in this "Quality champion" role, be the hero and talk to an Agile Coach. Get one to come in and do an assessment. You aren't alone. You can help make a difference. Ask for the right help.

Be prepared to change yourself - to learn, to adapt to new ways of working with others, and to deliver a whole new level of quality and value that you didn't think was possible. Be a model team member and let the Agile coach help guide the rest of the team. Quality isn't your job - it's everyone's.

The Future(s) of Software Testing

2011-10-18T01:52:00.000-04:00

This topic keeps coming up in various discussions so here are some thoughts on what I think a future may hold for us one day. What does it mean to predict the future? Does it mean it is inevitable? If it is something appealing, maybe it is something we can work towards.

What does it mean to talk about the Future of Software Testing without talking about "Quality" (whatever that is)? I believe that Testing is a means to helping you attain the desired quality but that on its own it is not an indicator of what the delivered quality will be. I think it is fair to speculate on the practice of Testing in isolation of the idea of Quality. Just like it is fair to speculate on the kind of vehicles you use for travel without saying it is a clear indicator of the quality of the journey.

When it comes to predicting the future, how far ahead do we look? It used to bother me that H.G. Wells chose to go 800,000 years in the future in his famous work, The Time Machine. Some authors go 100 years or even just 5 or 10 to tell their tale. I will not be writing about what will happen in 5 years time and I really hope it doesn't take 100 years. I don't know how long some of these events will take to manifest. I do have some idea of 'markers' that we may see along the way though.

When it comes to Software Testing, two thoughts jump immediately to mind: 1) Software Testing is an inseparable part of the creative Software/Solution Development process; and 2) there will be many different possible futures depending on how you do testing today. Put another way, there is no one right way to solve a problem, and creating software is a complex problem-solving activity performed in shifting organisations of human interaction so there are many ways to do it. In my opinion, the technical 'fiddly bits' like code, tests and user documentation pale in comparison to the difficulty of those two primary challenges: (1) solving the real problem, and (2) working with others to get the job done.

When we ask what is the future of software testing, we are really asking what is the future of software development. So what will Testing look like in the future? Well, what does Testing look like today? What does Development look like today?

Scenario 1: No (internal formal) Testing
For some, testing is only done by the end users. This may be a single individual, a small group of people or a larger population depending on the application. For example, if you have a highly-specialised application, the best testers may be the domain experts themselves (e.g. researchers, scientists, and so on). Aside from some quick checks made during development, no formal testing is performed prior to release of the software to the end user(s) for evaluation.

In some cases, non-critical software applications that people can live without if they fail are sometimes also released without any special internal testing phases. I have found that this often happens when a company is in a new niche market and they have no competitors yet. These applications may be prototypes or ways to try to figure out user/market value. If the apps stop working, people can continue on without them.

I don't see the future of this kind of software development changing much. No (internal) Testers are affected in the present or future of this kind of development shop. I believe with the advances in development technologies, we will see the quality of deliverables improve over time although the activity of determining suitability or fitness for purpose will always remain. That is, specialised software will always require the approval of the expert customers. They are the testers here.

Scenario 2: Programmer-driven-testing
I believe that "testers" are "developers" by definition, so rather than saying "developer-testing" (which would be confusing) I will say the code jockeys are the ones who own and perform the testing here. This is different from the scenario above in that formal test frameworks are in place and testing happens prior to release to the customer or users. There are no separate testers or test teams within the company to take a second look at the system to make sure it "looks right" (big air quotes here). Many Agile Development shops operate this way, especially smaller ones.

Again, as there are no separate testers to speak of here, the future here won't seem very surprising. I expect things will pretty much look the same as they do today - only the tools will become more sophisticated. (more about that below, in the next scenario)

Scenario 3: Functional System Testing
Unfortunately, this makes up a large part of the software development companies out there with QA/Test teams - the "traditional" software testing. It depends on the context of course, however my experience has been that this activity is largely a waste of time and money and is performed more for show and to satisfy lawyers (i.e. copious test paperwork as "evidence" of due diligence) than it is to actually raise software quality.

Anyone today whose job is to create mountainous test documentation that slows the creative development process, creates division and mistrust between project team members, and only serves to check that what has already been built has been "built as specified" is completely wasting everyone's time and money. I believe the term here is WOMBAT - waste of money, brains and time.

The future here is easy to predict. This tester role will completely disappear in the future as it is "wasteful" (in "Lean" development terms) and provides no additional value to the development process or solution quality. Sorry kiddos - adapt or die.

What will trigger this future? Simple. Responsible development practices and smarter development tools. When you take a look at the value that these test cases (i.e. usually very narrow-focussed functional "checks") provide, they are generally standard, straightforward things like:

1. Is the feature element (button, field, widget, text, whatever) that you said is supposed to be there actually there?

Test-Driven Development (TDD) is a development practice (available today!) that completely eliminates the need to have separate testers do this kind of thing manually. There is simply no good reason for this testing activity to continue in a manual fashion into the future. It shouldn't even be done manually today!
(TDD-style) Automated functional "checks" directly linked to the code are way more efficient to maintain. They also facilitate good quality deliverables that don't degrade unexpectedly over time.

2. Do input fields allow unexpected inputs?

I believe that (future) advanced development tools (programming languages, compilers, etc.) will automatically include model-based testing (MBT) subroutines that will automatically scan for such trivial aspects of developed code.
There is really nothing magical in this kind of testing activity. Yes, it finds a lot of really good bugs. The only perceived "magic" here comes from inexperienced programmers who don't know how to develop better solutions.

Generally speaking, I don't know when programmers stopped taking responsibility for such basic testing and checking activities as part of their coding tasks. If you go back to the 1960's and 70's, there were no separate testers - programmers did it all. This is easy stuff. I really believe that if programmers had continued to "own" this kind of testing, it would have been part of the development tools by now.

We took one step forward in the 60's and 70's and then two steps backwards in the 90's and 2000's. It's no wonder that the "agile movement" is trying to move programmers back in the right direction. When this quality/value ownership in development becomes more widespread, I will be happy to report that we will have achieved 1960's-level of development craftsmanship. Again. sigh.

Scenario 4: Business Analysts, Personas and Suitability Testing
Testing in some companies happens with BA's or other specialists who act on the customer's behalf to check that the application or system developed (SUT) meets the expected fitness for purpose. That is, does the SUT meet the business needs (rules, SOP's, statutes, industry standards, and so on) of the customers or users? These types of internal testers generally have some kind of industry or domain experience or knowledge.

In testing jargon, this is more "validation" kind of testing or "did we build the right thing?" Sometimes this is done with the help of "typical user profiles" called "personas." In the absence of formal requirements or tests, one can ask the question "what would user X do in this situation?" This kind of testing has more of a basis in business and psychology and doesn't presently lend itself to automation very well.

I believe that in the future, these kinds of tests will be automated as well, using intelligent systems and algorithms that can calculate the percentage probability of a developed solution falling within the desired user parameters.

I believe that the Eureqa system provides us with a glimpse of what intelligent computer systems can do today. With advancements in hardware technology, I believe it will be practical for humans one day to interact with computers via natural-language voice control and have the computers do these kinds of checks for us. The "comparing system" or "oracle" will group business rules together into a model, run through the SUT and heuristically compare the data output with the desired business model. At this point it is simple mathematics to let you know that the SUT meets approximately 72% of the desired model, include confidence limits, and tell you which rules the SUT fails to meet.

We're not there yet. If this describes your job, I'm pretty sure your job is safe for the next 5-10 years (if you are good at what you do). Development of this kind of "oracle" really depends on a lot of things, including how quickly we get through scenario 3 above.

Scenario 5: Life-Critical Systems
This is more than simply a variation of scenario 4 above. These kinds of systems are things like medical devices, nuclear/energy management systems, aerospace and deep-sea technologies, and so on. Basically, any system that if it fails someone will very likely die.

When people's lives are on the line, I believe the responsible action will be to always have people involved in the evaluation and assessment of the SUT.

Yes, I believe that advanced development tools and technologies such as what I mentioned in scenarios 3 & 4 will greatly improve the foundational quality of all systems developed - including life-critical systems. However, the role of the responsible development organisation here will be to have good, smart, skilled people own the field trials in a way that determines "fitness for use" at a level well beyond what I have already described above.

If my word processor is unavailable, I may get annoyed but I will find another way to write a message. If a pacemaker stops running after two weeks of use, you can be sure that the patient involved cares about this problem in a really big way. As will his or her family.

Scenario 6: Black-Box System Testing, Para-Functional Testing, Exploratory Testing
This is a superset of scenarios 3 & 4 and complementary to scenario 5. In this role, the tester takes a look at more than just the functionality of the system and is trained to ask questions about the SUT and user expectations that sometimes challenges the current design of the developed solution.

A good Exploratory Tester identifies assumptions and asks open questions about them. For example, questions about the user experience, flow of data, security and privacy, internationalisation, reliability and many other facets that are often skipped or ignored in "traditional" software test teams.

I do this style of testing, and I know many good people who also do it around the world. Unfortunately, I also know that we represent a small percentage of all the test teams out there.

I believe that this testing role fills an important niche in the software development "creative problem-solving" activity that is currently lacking in many companies. That is, we go back to the important question: are we solving the right problem for everyone who matters?

I see two possible futures here. First, if people are still involved in the development process, I believe that this testing role will be split in two parts. The hands-on testing part will be automated using the advanced development tools I already described above. We can simply add new MBT subroutines to the tools to account for new personas, perspectives and potential problem types. The creative, investigative people-interaction part will still need a skilled person to go around, talk to people and ask the right questions. This will lead to creating the right tests for the tools to help us answer.

In the second possible future, if people are not involved in the software development process, this whole testing activity will be automated in a fashion similar to what I described in scenario 4 above.

Scenario 7: Specialised tests - e.g. Performance, Usability
This is a variation of scenario 6 above. A specialised test is one that answers a specific class of questions. My experience in Performance Testing tells me that how I do this kind of testing is different from other kinds of testing. At the end of the day, there are a set of rules and models that apply to do this kind of testing properly. We will be able to program or teach these rules to an "oracle" system at some point in the future.

Human research and expertise will go into the problem-solving models that we will program into these computer systems. Depending on the complexity of the development project, the oracle system may be able to ask the appropriate questions and execute the tests without further prompting. In some cases, I expect the initial questions will come from a person and the computer system will be able to perform the test and report the results as required.

To sum up, I see the future of software development as being much more plug-and-play than anything we have today. Software testing activities will be largely automated with the possibility that humans may still be involved in asking important questions that lead to suitability or fitness for purpose. In time, I think that learning computer systems will be able to anticipate those kinds of questions and they will free us up to do different, more interesting creative work.

Don't worry, testers. Your jobs are safe as long as the programmers' jobs are safe - provided you are contributing value to the development activities. I believe our "developer" roles will disappear in close tandem. Software development and engineering is still a relatively young field/industry. We still have a lot of growing up to do.

Hobbies and Interests

2011-08-23T01:29:00.000-04:00

Several years ago I wrote an article summarising some of the key points I keep in mind while interviewing candidates for a test team. The article is called "Hiring Software Testers in an Information Age" and is available as a PDF on my main site. The article was originally targetted to recruiters who kept asking me for advice on hiring software testers and they would always be surprised at the level of detail that I went through in describing what it takes to hire a good person for a testing position.

Conversations with recruiters over coffee would always start the same. I would say something like: if you are just trying to find a warm body to fill a position, then you don't need to hear what I have to say. If you want to hire someone who thinks and has a good chance of fitting in with the culture of the team and organisation to provide value, then it is a complex problem that requires insights into what the position actually involves.

There are about a dozen different checkpoints that I go through when considering and interviewing candidates, and the paper I wrote touched on some of the major points but not all of them. Actually, I even removed some of them from the article as early drafts had too much information. My intention was to get some of the important points across without writing a book.

Recently, a colleague and friend, Michael Mahlberg tweeted the following:

RT @NolanBushnell: At Atari we hired based on hobbies and not grades in school. We ended up with the best engineering group in the world.

I liked that comment and followed up with a supporting tweet:

On Hiring: if a résumé or cover letter doesn't describe Hobbies or other Interests, I usually skip it.

This sparked some conversation on twitter and I want to elaborate on my comment here.

First off, it is important to know that I place myself in the Context-Driven School of Testing. The principles of this 'school' puts the focus on the people working together to help deliver the right solution. One of the things I have noticed over the last 22 years in the software industry is that the best testers are the ones who care about testing. They have fun with it. They believe they are working to improve things for the customer. They have a drive and motivation that lets them ignore or put up with a lot of crap that is dumped on them by unhealthy organisations and ignorant individuals.

Unfortunately, that kind of motivation, passion or drive doesn't come across on a standard résumé. I can spot it straight away when I am talking with someone, but how do you communicate it in a standard functional résumé? In general, I don't see it in the "technical skills" or job description sections that focus on accomplishments and other task-oriented details. The majority of the time I notice evidence or hints of passion and motivation in the cover letter, if anywhere at all.

So, when I am hiring for a test team, a team that I want to integrate well with the rest of development and the organisation, a team that I want to focus on building human relationships as well as exercise systems and scientific thinking in their quality investigations, a team that I want to encourage fun and respect for their hard work in providing valuable information to help make timely decisions, where do I start when I am looking at a stack of résumés and job applications?

People who submit cover letters go to the top of the pile. People who include "Hobbies and Interests" in their résumés are next. People who don't submit a cover letter and don't tell me anything other than a bunch of dry technical information and job details are put at the bottom of the list and often fall right off the pile.

Having a cover letter or a "Hobbies and Interests" section doesn't guarantee an interview but your chances of having me give you a quick call are higher. So, what's the deal with this "Hobbies and Interests" section anyway?

Johanna Rothman wrote an awesome book called "Hiring the Best Knowledge Workers, Techies & Nerds." If you don't have a copy yet, and you are in the business of hiring technical people, I highly recommend you get a copy of this book. Johanna is an awesome person and a terrific writer. She writes a few blogs and I learn a lot from them.

Johanna wrote a good blog post back in 2004 titled "Tips for Reviewing Resumes". One of the points she wrote in that post says: "Hobbies or other personal information. This stuff isn’t relevant to the job and should not be part of how you select candidates."

I partly agree with this sentence. I really don't care about personal information in an application. By "personal information" I mean things like marital status, age, sex, or anything else that the government might use to classify you in their census demographics charts.

What about hobbies and other interests? Things like: playing music (e.g. piano or guitar), likes to read, play video games, do magic, go cycling, martial arts, improv, and so on.

This stuff I find both interesting and relevant! Why? Creativity, passion and (skill) transference.

I am looking to hire people -- intelligent, thinking, caring, fun people and not drones. I honestly don't know whether the stuff I'm reading on your application is true, embellished or fabricated. I want to get an idea of the whole picture of who you are. I will test you for your technical ability during the interview, so I'm not worried about that part. Finding someone who has the right mindset and will fit in with the rest of the team is harder to grasp from technical details and job accomplishments alone.

Based on how I currently understand, apply and teach Software Testing, I like to find candidates who exercise both their creative and analytical parts of their minds -- i.e. people who are right and left-brained (to use a dated, flawed model of the human mind). When I read through the candidate's hobbies and interests, I build a list of assumptions that I can check in a phone call or in-person interview.

Assumptions might be things like:

if I see someone who likes to do creative things like play music, sing, knit, et cetera, these are right-brained/creative activities. This is a good sign that the candidate might be good at Exploratory Testing as I do it.
if I see someone who participates in theatre, improv or role-playing games, then they may be good at user profiling and test design.
if someone likes to read, this may tell me that they are learners and continue to feed their imaginations. This is also a good sign for Exploratory Testing, and I am happy to ask them about the kinds of books they read.
sometimes their interests may align very well with the interests of the organisation and the product or solution in development. Things like playing video games or participating in sports are really good hobbies to have if the hiring company makes games or develops solutions for the Sports industry.

There are too many examples for me to list here. The point is that sometimes the Hobbies and Interests section can provide me with insights that I won't get from the Work Experience, Technical Skills or Education sections of a résumé alone. I look for evidence of creativity, motivation, passion, balance, feeding imagination, professional activities, community involvement, and so on. Sometimes, the skills they develop in their hobbies are directly transferable to the testing tasks required -- e.g. learning, observation, relationship-building, organisation, analysis, design, problem-solving, and so on. Sometimes they aren't - it depends on the individual.

These are all assumptions, I know, so I treat them like assumptions. I make note of them and ask about them in phone calls and interviews.

If I like the candidate and hire them, I use their hobbies/interests as examples when coaching them on testing theory, models and practices. Since the hobby is familiar to them, I find this is a really powerful teaching technique. I also encourage them to come up with their own testing analogies using knowledge and experiences they are familiar with. It raises their self-confidence and helps them to remember abstract ideas in their own terms.

The likelihood of me finding a Testing Superstar by skimming through résumés is pretty slim. The likelihood of me finding a candidate who I can help become a Testing Superstar is much higher if I can learn more about them in their job application; learn more about what makes them a unique, creative, passionate individual who cares about others, doing excellent work, providing value and enjoying life.

But that's just me. All interviewers are different. Your mileage may vary.

When in doubt, I suggest candidates be themselves. If you are penalised in a hiring process for it, you don't want to work there anyway. You are allowed to have a life. Work is just a part of it.

Quality Center Must Die

2011-07-28T02:56:00.001-04:00

It is not a matter of "if" -- it is a matter of "when" HP's Quality Center software will die. And you, my dear readers will help make that happen.

"How?" you may ask? Simple. There are two things you should do: (1) think, and (2) don't put up with crap that gets in the way of delivering value to the customer and interacting intelligently with other human beings.

But I am getting ahead of myself. Let's rewind the story a bit...

Several months ago I was hired by my client to help train one of the test teams on agile and exploratory testing methods. The department has followed a mostly Waterfall development model until now and wants to move in the Agile direction. (A smart choice for them, if you ask me.) Why am I still there after all this time? That's a good question.

After attending the Problem Solving Leadership course last year, and after attending a few AYE conferences, I changed my instructional style to be more the kind of consultant that empowers the client with whatever they need to help themselves learn and grow. It's a bit of a slower pace, but the results are more positive and long-lasting.

I am a part of a "pilot" agile/scrum team and am working closely with one of the testers (I will call him "Patient Zero") to coach him on good testing practices to complement the agile development processes. I have done this several times now at different clients, so this is nothing new to me. One of the unexpected surprises that cropped up this time was that this development team is not an end-to-end delivery team, so when they are "done" their work, the code moves into a Waterfall Release process and it all kind of falls apart. There are still some kinks to be solved here and I am happy to see some really bright, caring people trying to solve these problems. So that's okay.

Patient Zero and I are part of a larger test team, and the rest of the test team all work on Waterfall-style projects, use Waterfall-compatible tools, and they generally don't get how we work. :) Unfortunately, one of the tools mandated for our team's use is HP's Quality Center (HPQC). I hadn't seen that tool in about a decade and it looked very similar to how I last remember it.

To my agile coach/practitioner friends I should clarify that at no time during our sprint development work does anyone ever touch HPQC! However, once the code is deployed/falls into the Waterfall Release process, regression test cases are created in HPQC and it is used for defect tracking. It is mandated, and so shall it be done. I can live with that. It's just a tool at this point and the impact to our ability to deliver a good solution is eliminated by the fact that we don't touch it until after we are "done". (Communication and collaboration FTW!)

Two days ago.

Our whole test team took part in a 2-day HPQC training workshop on something HP calls "Business Process Testing" or BPT. Being naturally curious to learn something new, I wanted to know what BPT was and how it fits into the bigger testing picture. Here we go.

We were given a handout with some "test scenarios" to be used for training. The test scenarios fell into this pattern:

Scenario name/title
Requirement description
Test Situation (I am staring at this right now and I still don't know what this means)
Role (kind of system user this requirement/situation applies to)
Steps

That's okay information. The "Steps" are what you might typically expect to see if you have been testing for a while. Here is an example for working with a sample web app:

Login to the system with (a certain type of user)
Navigate to some module in the app
Click "Create" button from the tool-bar
Enter mandatory field values and save
Search for the information you created in the previous step
Logout of the system

And there were 6 of these scenarios.

I read through these tests and then I tried to follow them using the system. I quickly encountered a half-dozen bugs - some with the system, some with the test scenarios/cases, and some were open questions that I would follow-up with the Product Owner for requirement clarification.

But, woah-woah-woah-hey.. wait a minute. We only need to worry about *these* documented test scenarios! I struggled hard to keep my mouth shut about the value of time and the many different kinds of tests I would happily engage in at this point if I could leave the tool alone. But, I left that to my "inner voice" and we were now one hour into the first day's training.

At this point, we were given an overview of the HPQC modules and told to (1) enter the Requirements, (2) create BPT test scenarios, and (3) "componentize" the test scenarios into groupings of related steps. This last part required some explanation since this was new to me, and once we got it, we set to work on the task. Since we could see (1) and (2) on the handout sheets in front of us, we went straight to work on part (3). That was kind of fun working in small group of 4 looking at these scenarios and trying to come up with solutions.

And then someone went and spoiled the fun. We were "told" that we weren't supposed to do part (3) until after we had entered the information for (1) and (2) into the tool. I was all like "really?" and then a team lead came by and repeated the exact same thing. This may be summed up as: Enter data into the tool first, and think later.

I was kind of shocked with this comment and attitude and in retrospect it kind of foreshadowed the rest of the training experience -- i.e. Tool and process first; think later; maybe. Okay. I'll play along and see where this goes.

During this exercise, I began to grasp this idea of "components" as HP uses them. I think they are like Page Objects - chunks of code (or in this case, test steps) that perform a certain function, promote reuse, and reduce duplication. Although it was never described in this way, I believe the HP "BPT" module is a proprietary Do-It-Yourself DSL. Aha! I have experience with those. I get that stuff.

So, once I got it, I started to explain the concept of YAGNI to my group team members. That is, let's not overthink or over-engineer these components. Let's build/write them based on the needs of the requirements in front of us. We will modify the components in the future as new requirements appear. This idea was well received and we quickly came to an efficient solution for our small group.

When we took up the exercise, I found I had to explain the YAGNI concept to the instructor/trainer (and rest of the class/team) as he proposed that we try to abstract out these components to allow for compatibility with other features and system elements. What a waste of time! We cannot know what we will need beyond our immediate needs so that kind of abstraction is a pointless exercise that leads to more headaches than you need.

Eventually, I started to get that the point of writing the test scenarios using BPT was that these components form a basic vocabulary that may then be automated at some point in the future -- yes, BPT integrates with HP's QTP automation tools. Now, I'm all in favour of consistency, clarity, reuse, and automating tests that humans should never have to do more than once (and if there is value in re-running the test), so I struggled to understand why it was never explained to us this way. As long as I kept the DSL/automation model in my head, I understood what we were doing and can see the potential benefits of it.

I saw many testers struggle with the exercises and models we were presented with. End of day one.

Day two began with a quick recap and then we were introduced to the bureaucracy that is Waterfall and HPQC. That is, there are review processes and workflows to cover each requirement, BPT and component. Welcome to Wasteland. (I mean the Lean Development concept of "waste" here, although other interpretations of "wasteland" may be just as valid.) Easily a third of the 2-day training was spent on the processes surrounding the management and review of the various HPQC objects. sigh.

We then moved onto the concept of "parameters" for the components we created yesterday. Okay, I get method parameters when I am scripting with Ruby, so this was no sweat. Given the length of time I spent parametrizing my components compared to everyone else in the class, I think I may be one of the few who really got it.

I learned a few more things about the main instructor. One was that he didn't know how to identify web page elements using commonly available browser tools. Umm, aren't these tools supposed to interact with web pages? This never came up before? You have never wondered how to find the name/id for an element on a web page? really?

The other was that he had a really bad sense of humour. He made a reference to a "QA" joke that I shall not repeat here. Needless to say that I found it insulting, offensive, it made my skin crawl and my blood boil. Many unpleasant feelings and ideas arose in me and it took all my willpower and strength not to react to the blatant stupidity of insulting the profession of the students in your class and the market that the HPQC tool represents.

The final blow for the day came to me when we tried to "execute" these BPT test scenarios using the HPQC tool. THEN I discover that these "parameters" can have 2 kinds of values - fixed/hard-coded and run-time. Anyone who has done intelligent automation knows NOT to hard-code values in their scripts. Data-driven is way better, and as an exploratory tester, I may not know what value I will choose until moments before.

Here's where the HPQC tool gets stupid..er. Regardless which parameter type you choose, they may ONLY be defined BEFORE the test execution! You cannot leave empty parameters so that when you get to a particular step, the tester can decide what value to input and feed it back into the tool.

What does this mean? As a tester, let's say you want to create a new user profile in an app of some kind. The test parameters for things like Name, Email, Address, Country, and so on, must all be defined and set before the test is executed. You cannot decide while you are actually testing!

Why does this matter to me? It matters because it means that the humans who execute these BPT tests manually have no choice in what values they may input. Testing techniques like Equivalence Classes and BVA that help guide our choices to pursue interesting paths are completely cut-off! It turns out that HPQC treats humans worse than the automated counterparts. In discussion with an automation "expert" at the end of the class I learned that at least you can code some variability into the QTP automation scripts. This is not possible with the same test scenarios executed manually by human beings.

So. Much. Wrong.

So after my first ever QC training session, here are some of my take-aways:

It took me 2 days to "script" 6 test scenarios in this tool and they were rotten test cases to begin with! I suspect that outside of the training environment, it will actually take longer to complete since you won't have a "reviewer" sitting next to you waiting for you to finish your piece.
And they weren't even automated! Who knows how long it would take to tweak the "components" to make them work with a particular automation strategy.
HP QC will never be a useful tool for any agile or rapid development efforts
(HP best practice) Put the tool, data and review processes first, before you think. Maybe always instead of it too.
BPT is a DSL framework for test scripting
BPT component parameters cannot be customised during test execution. They may only be set/defined before you start testing. => No thinking allowed while testing.
The instructor didn't appear to be knowledgeable on anything outside of the tool itself. This includes how we might actually want to use the tool. No, no, no. And I quote: "Testers must change how they work to use the tool in the way it was designed."
As long as there are people pushing these kinds of horrible tools that suck the life and intelligence out of people, inject mountains of wasteful activities that provide no value to the customers or end users, and continue to create barriers between testers and their developer counterparts, I will always have job security in helping organisations recover from these cancers.

At the end of the day, there were several people interested in my idea of randomising tests. The automation expert in the class insisted that it couldn't be done with automation, so I called up my "Unscripted Automation" presentation slides that I gave a few years ago. He said that what I proposed was not a "best practice" and that everyone, the whole industry, was using the tools in the way that he described how they should be used.

My response was to simply say "they are all wrong."

Thoughts on the StarEast 2011 conference

2011-05-14T19:01:00.002-04:00

I first attended StarEast in 1999. I remember the day-long tutorial I attended (by Rick Craig), and two track sessions - one by Cem Kaner on hiring testers, and one by James Whittaker on "Exploiting a Broken Design Process." I know I attended other sessions but I don't have active memories of them any more. I do remember the experience of attending the conference - one of surprise and excitement. Surprise at seeing so many other people in the testing community with similar questions and problems as myself, and excitement at the speakers with lots of great information and advice to give.

Fast forward to 2011 - I returned to StarEast, this time as a speaker. I suppose I didn't need to wait 12 years to return as a speaker. I didn't intentionally ignore the conference. I think I've been busy with other things and it just didn't come up - until last Fall when I received an invitation in my inbox to submit a proposal. I'm really glad I went.

Some things were familiar - the beautiful hotel, the Florida sunshine, the amazingly fresh orange juice, and the basic conference format. One thing that was different for me this time around was the number of people/speakers that I new who were also speaking at the conference. After having attended and spoken at several other conferences over the years, I guess I have gotten to know many of the popular speakers.

I was happy to see many more people speaking that I have never heard about before. That tells me that the community is still growing after all this time and that there are still many more people sharing their knowledge to help enlighten future generations of testing leaders. That's awesome!

I was particularly surprised at the calibre of the Keynote speakers. I was genuinely inspired by every Keynote that I attended (I think I only missed one). (You can find the 2011 program online with details of the speakers and talks but I don't believe they were video recorded.) The Keynote presentations I attended were by: Andy Kaufman, Naomi Karten, and Julie Gardiner. The majority of my #StarEast tweets were from these keynotes. Unfortunately, I missed the Keynote by Gojko Adzic (I heard it was good too), and I'm not including the "lightning" talks here although I did attend those and some were very good.

I was excited to hear that Lisa Crispin and Janet Gregory, authors of Agile Testing, were attending the conference and am happy to have had the opportunity to meet them and speak with them! =) Lisa even attended my track session and wrote up a nice summary about it on the Software Testing Club site.

I won't really say much about my session here. It went okay I guess. I haven't received the session feedback evaluations yet, but I can say that I was really happy when several people came up to me afterwards in the hallway to thank me for my talk. They said that they really enjoyed it. My favourite comment came from Nawwar, my track chair -- i.e. the person who introduces the speaker during the session. He went from asking me "who are you?" just before the session to "Wow. That was the best session I've seen during this whole conference!" Nice. Thanks. =)

I had fun giving the talk - "Real-time Test Design for Exploratory Testing". Test Design is a topic that I am really passionate about and I can talk about for hours. Okay, days. ;-) I think the main thing that I wasn't terribly crazy about my talk was the format. When I talk about Test Design, it is usually when I am teaching it. So I found it hard to just talk about it and not have an interactive session with the attendees to give them a chance to practice it. Don't get me wrong - I had some interaction in my session, but not the sit-in-front-of-a-computer-and-try-things-out kind of way. If I give this talk again, I will try to find some way to make it more interactive. (I may need more glow-in-the-dark straws.)

Back to the conference. I was pleasantly surprised to bump into a former employee - a tester on a team that I managed about a decade ago. Wow! Still in Testing after all this time. AND attending a conference too! Double rainbows! We've hooked another one! ;-)

I've worked with many testers over the last decade and I hear from so few of them. I am always happy to meet them again and know that they are still doing and learning about Testing. I am happy to know when I help testers get better at what they do - I get ecstatic when I find out that they continue to see Testing as a Profession and participate in conferences and networking events!

StarEast had a few more surprises for me. At the Vendor Expo I bumped into Rick Craig - the tutorial speaker from when I first attended StarEast in '99. I introduced myself and told him how I remembered him. He was both pleased and suddenly felt older. Ha ha. We had a good chat.

Continuing my stroll through the Expo, the vendors were vendors. I still think some of the tools are the wrong ones for testers (like, totally and completely without value) but I did see some interesting ones that might have potential in certain circumstances.

As an independent consultant, most vendors weren't interested in me. I don't represent some big company with deep pockets to shell out on their bloatware documentation tracking systems. Of all the vendors, only one stopped to talk to me and really understand what it was that I had difficulty with their tools. He was fascinated by my knowledge of Agile practices, how testers fit on Scrum teams, and why these tools don't help. He asked me for my card and said that he was interested in contacting me to see if I can help provide some feedback for a future generation of tools that might work better. I gave him my card. I'll be curious to see if he contacts me. =) I love to help - just ask!

My final pleasant surprise, and ultimately the best reason for attending the StarEast conference, came between sessions and during the meal breaks. I made an effort to sit with people I didn't know and engage them in conversations. Mostly I was curious to know what they did and what brought them to the conference. I was blown away by the passion that many of these attendees have for Testing. Wow. Some described to me the problems they face at work, the inequalities from other development team members (both in status and in salary), and the barriers preventing them from doing more. And yet, here they were at the conference to "learn from the best", to find ways to provide more value, make life better for them, their teams and their organisations.

They were dripping with hope and enthusiasm and I must say that I was overwhelmed with joy on more than one occasion following a conversation like this. If I had nothing else nice to say about the StarEast conference and received no other benefit from attending, witnessing the passion of the attendees was enough to fill my heart with passion and drive to get out there and do more, teach more!

I have been exposed to a lot of negativity over the last decade in several of the companies I have worked at. It wears you down after a while. You begin to lose faith in yourself and in the [testing/quality/value] mission. Attending StarEast changed that for me and recharged my spirit in a way that I totally wasn't expecting.

I was asked if I planned to speak at StarEast again next year. I don't know. I hadn't planned on it. I might not in 2012. We'll see about the following year or maybe I will speak at one of the other SQE conferences - like Better Software, Agile Development Practices or maybe even StarWest (since I haven't been to the Pacific coast yet).

There is so much to do in this area, in my "local" part of the world. It seems I am gaining recognition everywhere but here and I need to do something about that. StarEast helped recharge me and I hope I enlightened or entertained some attendees of my track session on Test Design and Exploratory Testing. Time will tell if I had a lasting impression on anyone. I hope I did. That wasn't why I went, though, and I got so much more out of it than just the opportunity to share some of what I have learned over the years.

I'm glad I went.

Reflection on my Testing workshop at the KWSQA Targeting Quality Conference

2011-04-16T16:54:00.000-04:00

At this year's KWSQA Targeting Quality conference I gave a half-day workshop titled "Exploratory Testing Basics". I originally proposed that title since I thought it followed nicely from the shorter workshop I gave at the QAI TesTrek conference in Toronto last Fall. I thought to myself - I'd like to redo the exercises again, change up a few things and it should be a piece of cake.

As the Winter months progressed into Spring, I began to worry about my workshop idea more and more. You see, the exercise I gave at the QAI conference, while fun and appropriate, only really covered one aspect of Exploratory Testing - a broader framework. Perhaps that isn't enough? What is enough, then? What makes up the "basics" of ET?

You see, when I teach ET, it's usually one-on-one and I spend 2-3 days just to cover the basics. It takes me a few more days of pair testing and debriefing/coaching to help the new tester put everything into practice. It really is quite complex and a lot of ideas and models may seem abstract until you try them out and adjust with good feedback.

One of the hardest parts, I feel, is trying to teach certain techniques when the tester doesn't see a need for it. For example, Pairwise Analysis. I was introduced to Pairwise Analysis as "Functional Analysis" about 12 years ago and I got it right away. I had done more complex mathematics in university, so it wasn't the math that was the hitch for me - it was knowing when it might be useful and then applying it.

The first project I tried it on was when I was asked to perform Installation Testing for a desktop application. If you have ever done this kind of thing, you will know that there are *many* features and variations that all conspire to convince you that it is a daunting task that may consume every waking moment for weeks on end if you ever want to try and cover all of the possible combinations of systems, hardware, software, feature selections, and so on. Enter Functional/Pairwise Analysis. I did the math; came up with a set number of test scenarios; performed them; and reported my findings in record time -- only a few days instead of the customary 2 weeks that it had taken on previous releases.

That was really cool! I had a new tool in my Tester's Tool belt and I couldn't wait to try it out again.

Years passed and several failed attempts later to teach other testers this cool technique, I finally stumbled upon the idea of Just-In-Time teaching. That is, rather than try to teach a tester all the techniques and models that I have learned over two decades and cram them into a few days, I will wait until they are presented with a problem and introduce the appropriate technique then.

There are two important take-aways for me with this JIT approach. First, it is really effective and the tester gets it - great! Second, it may take a long time before a tester is presented with the situations where certain techniques apply - not so great.

Present day.

So, what do I cover in a 3-hour workshop that I would consider 'good enough' to cover the basics of ET?

The answer, of course, is that proper workshop coverage will be the gap in knowledge from where someone is to where you want them to be. Unfortunately, the knowledge/experience starting point for each individual attending my session will be vastly different, their needs for this information will be different, and unless they have specific concerns some of the ideas very likely won't stick.

There are many unknowns in that equation. So how can I plan an outline to cover this unknown gap for unknown purpose(s) in a short amount of time? Stress.

What did I decide to do? I didn't plan an outline. I took a page out of Jurgen Appelo's book and I had the attendees self-organise and decide.

I handed out index cards and asked each person to create a user story card with their goal for the session. We stuck them to a wall with the heading 'Backlog' on it. On another part of the wall I had a Task board with the headings 'To Do', 'In Progress' and 'Done'.

I asked all the attendees to decide as a group, select the top 4 cards that they wanted to cover this session, and place them in the 'To Do' column. They didn't believe me at first when I asked them to do this and kept asking me to decide. It was really cool to watch the transformation happen and have them take ownership for deciding on the workshop goals.

I read each card, picked one, moved it into the 'In Progress' column and began.

So what did I actually cover? The attendees grouped most of the cards together into one big group and the one they picked said that they wanted to "gain an understanding of ET techniques."

Several years ago, I worked in a Financial Services organisation and was faced with a few audits by Banks where they needed to understand what testing artefacts we produced and why they didn't match their traditional Test documentation expectations.

Knowing that I couldn't just show them test session notes, test guides and other Exploratory Testing artefacts because they wouldn't understand what they were looking at, I created a presentation that had 4 parts:

The Challenges in Testing
What is Agile Development?
An Overview of our Systems Testing Approach
Examples of Testing Artefacts we Create on Software Projects

The auditors wanted to see the last part, but I explained that I needed to cover the first three before they could understand what they saw.

In the "ET Basics" workshop at the KWSQA conference, I covered the important aspects of the first three sections above (I have expanded upon the original presentation over the years), and supplemented that with some additional exercises to cover a critical aspect of Exploratory Testing - Test Design.

Did I meet the objectives? I think so. I covered the underlying principles and ideas that form the basis of good testing, "exploratory" or otherwise. The next step would be to look at some specific models and structures and have them perform exploratory testing.

While performing ET may be an exciting and enlightening activity, it is also a complex and challenging one that requires careful debrief. In other words, we will need more time.

Will I give this same presentation again? Not sure. I might if that's what the attendees want to see. I'll let them decide. If they want to practice something instead, or focus on managing such a testing activity, then we will do exercises for that instead.

I had fun in this workshop and learned some great things from an exercise I tried for the first time. I also thought the training video I showed was an appropriate fit for both a Friday afternoon and the topic covered. ;-) I can't wait to build upon what I've learned from this workshop experience and offer more in the future!

Cheers! Paul.

Radiating Testing Information - Part 1

2011-03-23T23:58:00.000-04:00

This topic is one that I have been asked about many times over the years and I am long overdue for a detailed discussion of it. Back in 2006 I presented an Experience Report at the STiFS workshop in New York titled "Low-Tech Testing Dashboard Revisited." The content of that presentation will be in Part 2. To quote "The Do-Re-Mi Song" from the movie The Sound of Music, "Let's start at the very beginning, a very good place to start."

I attended the StarEast conference in 1999 and there was a talk by James Bach titled "A Low Tech Testing Dashboard." This presentation clicked with me as I was managing several test teams at the time and it addressed a problem that I felt was important. I have used this communication tool many times ever since. If you are not familiar with it, I suggest you read through the PDF slides on the Satisfice web site before you continue. Go ahead. I'll wait.

In this review I will cover some of the who, what, where, when, how and why of the Low Tech Testing Dashboard (LTTD) through examples from past projects I have worked on. I expect your context is different, so my hope is that these examples may help you think about how you might apply this communication tool on your project.

Who are you? Whom is this for?

I have used this tool as a single tester on a development team, and as a team lead/manager managing multiple concurrent teams and projects - using one dashboard for each project.

The first time I used this dashboard, I was one of two testers working on a web application. We were both doing unscripted, risk-based Exploratory Testing and I needed a way to try and understand high-level testing progress since we had no other testing documents to use for reference. That is, it's hard to really understand what a tester means when they say they are "done" testing a feature and you have no way of really knowing if that means the same thing to both of you. Adopting the terminology and scales outlined in the LTTD presentation helped bring us together on the same page in describing progress.

The primary audience for this information was our test team. We stood around the board daily and used it to identify priorities and plans for the day. The secondary audience was the development and project team. Our team was located next to the kitchen so the Testing Dashboard was in a highly-visible location. As soon as we identified our first "BLOCKED" item on the board, the dashboard became a tool for the development team to identify the immediate priorities to unblock our testing. We explained what 'Blocked' meant the first time it happened and then Dev automatically resolved these issues every time after that without additional prompting. Cool!

I have used this board in Waterfall-type projects with a lot of scripted tests and documentation, and on Agile projects too. It is helpful for daily Scrum (stand-up) meetings as it can help you remember what you worked on, what you plan to do, and it identifies any blocking issues or risks that we need help with.

What is the LTTD?

The name has two parts: (1) Low Tech, and (2) Testing Dashboard. They are both important. One describes how you communicate, and the other indicates what it is.

In Agile parlance, I have heard this dashboard referred to as an "information radiator." From the Agile Advice blog, "an information radiator is a large display of critical team information that is continuously updated and located in a spot where the team can see it constantly." One may argue that the entire Testing function/role/job purpose is to radiate (good) information. I agree with that perspective but will save that topic for discussion another day. Here I will focus on the outward presentation of particular information that communicates important project status and quality details to the project team.

The Testing Dashboard communicates different aspects or dimensions of the testing effort in a tabular format for everyone on the project team to see. The Low Tech part reminds you this information can and should be communicated using simple, readily available tools in a highly-visible area - such as a dry-erase board. You could make it high tech but you don't need to. In fact, in my experience, for a single co-located development team (i.e. where the whole team is in the same room), anything more high tech than a dry-erase board is likely a waste of time.

The Testing Dashboard is novel because it communicates (i) Test Effort, (ii) Test Coverage, (iii) Quality Assessment, and (iv) current risks for each Product Area under test, all in one convenient location. It represents a snapshot of these dimensions for a given moment in time or for a particular build. The use of smiley emoticons for the Quality Assessment is particularly effective. I have seen it happen several times where a developer will draw little devil horns on a red unhappy face. This is good because it shows their interest in interacting with the board and it is an expression of their feelings - in a humourous way. Fun interaction like this is always a good thing in my opinion.

Prior to using this dashboard, I was used to being asked for Test Coverage estimates (in terms of percentage tests complete and tests remaining) during Project Status Update meetings. This was likely a measure of the number of Test Plans, Test Cases, or Requirements covered using a Traceability Matrix of some kind. Those percentages always left me somewhat uneasy as I knew that precise measurements and estimates of such things don't reflect the reality of the remaining work to do.

What I like about the LTTD "Test Coverage" scale is that it was like going from Digital to Analog (i.e. a Physics/Electronics analogy). Suddenly I have this dial that I can use to more accurately describe the coverage! This is a good thing. In my opinion, precision in calculating remaining work to do is a waste of time. If humans are involved in the process, especially good testers, then your precise estimates will almost always be wrong. (NB: there is an important difference between accuracy and precision. It's great to have both but if I have to choose I'd rather be more accurate than precise.)

Where do you put the LTTD?

As previously mentioned, an information radiator like this works best when it is in a highly-visible area. In one company, I placed the LTTD on a whiteboard next to the kitchen, which also happened to be on the way to the development area, so everyone in the company saw it sooner or later. In another company, we placed it on a rolling dry-erase board and positioned it at the entrance to the development area (see photo at right). Again, pretty hard to miss.

I've had mixed success with making the Dashboard high tech. At one company, I put a copy of the Testing Dashboard on the wiki. We used Atlassian Confluence at the time and it worked very well. Whenever we updated the dry-erase board during our regular morning stand-up meetings, I would take a few minutes to update the wiki version too. I started receiving comments from team leads in other departments thanking me for the information because they could now get testing updates without having to come up to our floor and see the board directly.

The nice thing about the wiki version was that we could create links to our test strategy pages, bug reports, and other helpful online information. It really does make a handy interactive Testing cover page or executive summary for each project!

This is a good time to mention another tester I know who has implemented a high tech version of the dashboard. Marlena Compton wrote about her experiences with the LTTD and posted it on her blog - see background and CAST 2010 presentation. I like what she's done with it.

Another time at a different company, I tried putting the Testing Dashboard on a wiki to help facilitate the communication for a test team that was distributed geographically. It didn't work very well. Okay, it didn't work at all. Unfortunately, the test team didn't use the wiki for testing purposes. I believe communication in general was a problem on this particular project, so the effort was probably doomed from the start. I haven't ruled out a wiki-based Testing Dashboard as a helpful tool for distributed teams, so I plan to try again when the opportunity presents itself.

When do you update the dashboard?

I create a fresh Testing Dashboard as soon as a project is kicked off. We add testing areas and features to the board as we hear about them and it is a good way to keep track of what is coming and when.

The original presentation suggests updates 2-5 times per week depending on your needs. That sounds about right. One time we were so involved in our testing that we only updated it once a week. This lasted for several weeks. We noticed that the dashboard was failing as a useful, timely communication tool so we returned to more frequent updates. I recommend nothing less than 2 updates per week.

If you are making noticeable testing progress then I would expect to update the board daily or maybe every other day. I find that it is a good place for the test team to gather around each morning and discuss what everyone is working on for the day. We can update it based on what we learned from the previous day, and it helps us to stay on top of the current priorities and risks as they come up.

If you are doing a dedicated Exploratory Testing (ET) effort (i.e. as opposed to ET accidentally happening between scripted test cases), then a tool like this may be useful in helping to communicate testing progress. It's possible that you may already have other tools set up for tracking and communicating progress of your scripted or automated tests. I find that this dashboard communicates more than just progress through a count of tests, so I would still recommend it as a way to supplement your regular testing updates.

How do you build a Testing Dashboard?

Remember that the key here is to build something simple, easy to maintain, and gets the message across clearly. As in Real Estate, the 3 most important things are: location, location, location. Once you have that, then what? Well, it's really up to you, the tools you have available, and your imagination.

In the beginning, I found that drawing straight lines on white/dry-erase boards was a challenge. In the absence of a metre stick, I would grab whatever was handy and worked - e.g. the metal edge strip off a cubicle wall or even a pizza box. I found that the width of a whiteboard eraser is just right for horizontal line spacing. Then it's just a matter of filling in the table using the appropriate marker colours.

One annoyance was that every time we updated the board we would always have to re-draw the lines. By the end of the project the dashboard would look pretty ugly. The answer came at another company when I learnt that a developer had brought in painter's tape to make their agile board.

In particular, he used Painter's Mate Green, 1/4 in x 60 yds or 6 mm x 55 m. It looks like this photo at right:

The 6 mm width is just about the same width as a board marker and the best part is that you can erase over it and the lines are always there! The green is also more visible than standard yellow masking tape on a whiteboard. I have seen blue painter's tape too, but I could never find one thin enough to use. Check your local hardware store to see what you have available.

So the tools I used to make the last few Testing Dashboards are: (1) 6 mm painters tape, (2) a post-it note (for spacing), and (3) colour markers.

I start by marking the width of the whiteboard eraser on a post-it note, and then use the post-it note as a guide for making the lines across the board (see photos below). Follow the general table outline as shown in the presentation slides and you will get the general idea.

A finished board looked like this a few weeks later:

I highly recommend using different-coloured markers when you create and update the dashboard. It makes the different meanings really stand out - i.e. it enhances how you communicate the message. For example, when you write "BLOCKED" in big red letters and have a red unhappy smiley face next to some show-stopper bug report number, people notice.

Why use the LTTD? Why not?

A good reason to use a tool like this is that it helps you and your test team to communicate important information to the project team without them having to ask you every time. In one sense, it does some work for you. For example, project managers can check the dashboard if they want an update and they don't have to interrupt you while you are testing. I don't know many tools that add time to my day. I am happy to say that this is one of them.

If your role as a tester is to provide valuable information, then radiate information! Let everyone know how things are going about things that are important to them before they even ask! Is there other information that people care about? Add it to the board!

The dashboard displays important info at a high level - a level that everyone on the project should understand. I have seen it happen many times when someone asks a tester how things are going and the tester gets lost in the details of the latest problem or issue that he or she encountered. The Testing Dashboard helps pull the tester up out of the details to put such events in a different perspective. Is that latest issue a blocking issue, or just another bug? Is it a show-stopper, or something that you require additional assistance from someone else on the team to help you investigate? What impact does this issue have on the overall quality of this feature or this release? How far along are we in testing this feature or all the features at this point?

The answers to these questions are always immediately visible on the dashboard. While we are fascinated by the puzzles and challenges of the testing problems we face every day, it's nice to have a dashboard around to remind us how to communicate at a level that our customers need.

So why wouldn't you want to use a board like this?

This question is partly rhetorical. I can think of at least one good reason why you might not want to.

For example, if the security policies of your organisation prevent you from leaving information like this up in a visible location. While making information visible is kind of the whole point of this dashboard, if your organisation won't allow it then you may wish to investigate other options - maybe even high-tech solutions. You have a team of smart people working with you. Get together and brainstorm a solution that works for you.

There may be other conditions or times when this may not be the right tool for you. "Because it hasn't been done before" is not a good reason to avoid it. If you are looking to grow, if you are looking for new ways to provide more value in innovative, simple and effective ways, you should give this a try. What will your Testing Dashboard look like? What will it say?

Radiate!

Here ends Part 1, an introduction to the Low-Tech Testing Dashboard. In Part 2, I will delve deeper into some of the aspects of the dashboard - how we modified it in different projects, and how we integrated it with Session-Based Test Management.

Before I post Part 2, I am interested to hear what you think. Have you tried using a Testing Dashboard like this before? Did it work for you? Are you thinking about trying it? Any other concerns that I haven't addressed yet? Please leave a comment and let me know what you think.

Cheers! Paul.

Testing & Programming = Oil & Water

2011-01-29T01:56:00.001-05:00

I was watching a science program just now and it occurred to me that Testing is very much science. And then I wondered about Programming.

I started in IT over 22 years ago doing programming. For me, the process of programming broke down to three parts: figuring out the algorithm to solve the problem, implementing/coding the solution, and cleaning up the code (for whatever reason - e.g. maintainability, usability of UI, etc.). It gets more complicated than that of course, but I think that about sums it up the major activities as I saw them. (SIDE NOTE: I didn't write those to mirror TDD's Red-Green-Refactor, but it does align nicely that way.)

When I think back on my experiences in programming, I don't see a lot of overlap with my experiences in Science (~ 8 years studying, researching and doing Physics & Environmental Science + teaching Science on top of that). Science is about answering questions. The Scientific Method provides a framework for asking and answering questions. Programming isn't about that. Building software isn't about that. I'm having difficulty at the moment trying to see how testing and programming go together.

It occurs to me that schools and universities don't have any courses that teach students how to build software. It also occurs to me that schools and universities provide students with the opportunities to learn and develop the skills required to build software well. The schools just don't know they're doing that and consequently the students don't get that opportunity intentionally.

I'm not talking about learning to program. That's trivial. Building software isn't about programming.

Building software starts with an idea - an idea that someone will pay money for. School courses to watch for here include - Economics, Entrepreneurship.

Building software requires people to work together. School courses that may apply - Business, Math/Finances, Psychology.

Building software requires people to work under constraints. "Project Management" is not really taught in schools, but there are many courses available to the public. I found this really painful to learn by doing. A whole world of insights opened up when I took my first PjM course. Reinventing the wheel really is dumb here. I believe this should be formally taught in schools - in High School actually (the earlier, the better).

Building software requires people to solve difficult problems creatively.

This one is interesting. I think there are many opportunities for people to learn this skill in school. I know that we definitely covered this in Science. I also know that Engineering programs teach students how to do this. There are many more faculties and programs that this would apply to and they all have one thing in common - there's some formula or method for solving problems for some purpose.

The thing is, that purpose is different in each case. See, here's where my mind is doing flip-flops.

In Science, when we solve a problem, the outcome is usually more information. This information feeds back into the original idea to help us check the validity of our initial premise/hypothesis. Sure, there are moments of free-form exploratory investigation, but I don't believe that happens a lot. There are an infinite number of paths that any experiment may go in if you don't have a particular question in mind when you start, so unless you are trying to intentionally waste time and money, you will start with some question in mind before you start your experiment or investigation.

In Engineering, solving a problem is different in that the outcome is usually something real, something tangible, some application or system that fills a need.

The purpose of Science is not to build things but to answer questions. The purpose of Engineering is to build things - and to do so safely, ethically, within desired parameters for intended purpose, and so on.

So where does that leave us in building software? *Building* software is definitely an Engineering task... and then some. I am temporarily over-simplifying the process of building software to focus only on the requirements gathering, design, coding and deployment phases.

"Testing" comes from and is an integral part of Science, so how does it fit in with these software engineering/development phases of requirements gathering, design, coding and deployment? Well, it doesn't. It has nothing to do with them. And yet, it has everything to do with them.

That is, from one perspective, at no time do you ever need to test anything to get through any of those phases.

That last statement, while true, kind of goes against everything I ever learned in school. Whenever I did math problems, I always checked that I got the right answers against the solutions in the back of the textbooks. Why did I do this? Why did I care?

I did it so that I could tell myself that *how* I solved the problem was correct - it got me the same answer that the textbook and my teacher cared about. I discovered there were exceptions, of course. That is, there were times when I got the correct answer but my method was wrong in some way. Dumb luck does play a role in life and that was when I first discovered the evil twins named Type I and Type II errors. (Side Note: I wonder if that's where Dr. Seuss got his idea for Thing 1 and Thing 2? Hmm..)

So, the process of checking answers with the "approved" solutions, and handing in assignments for grading by teachers is a feedback mechanism to tell me that I've learned how to solve certain kinds of problems in ways that provide the desired results. Let's assume for a moment that's a good thing.

Getting back to building software, you can go through requirements gathering, design, coding and deployment without ever once checking that you are producing the desired solution or results. In the end, this is a monumental waste of time and money, and is completely incongruous with the initial premise that you are building a product/service/solution that someone will pay money for. That's bad economics. That's psychotic.

So how do we fix this? What's the problem here?

Well, one problem is that we now have a question, we have doubt at the end of each of these phases. We have a desire to learn if the end result of each stage and for the whole process is meeting [our/someone's] expectations. The answer at the back of the textbook here will be provided by the people who are choosing to pay you and not your competitor for what you produce/release/ship.

Hey! Wait a minute! Science helps us answers question! Testing is a small part of that process. The bigger process starts with a question that stems from some research or exploration of the initial area of interest. This hypothesis is the part we really care about. The "how do we go about gathering enough information to answer this question" part is something different and we should get people who know how to do this to either (a) do it for us, or (b) help us do it for ourselves. Then there is the analysis of that data or observations in context of the initial hypothesis.

But this is a different layer now! We're adding a layer of "science" on top of "engineering". That's weird. That's like trying to mix oil and water together. That is, they don't mix. If you shake them up together you only end up with some cloudy mess that eventually will separate out again.

So what does this mean for building software? We need people who are skilled at engineering solutions, and we need people who are skilled at identifying and answering questions about the solutions being engineered. I believe these are two, very separate skill sets required to be successful.

However, my experience in the Software/IT industry over the last 20 years has been that only one of those skill sets has really been identified as important or relevant -- that of the programmer or engineer in building the solutions.

This is a problem. There's a huge knowledge gap here.

Schools don't teach you how to "test" in the context of software development. Every single Testing "certification" agency I have met to date misses the mark. They don't teach you the correct skills. They "teach" you superficial documentation skills that produce information *like* the kind of information that is required. That would be like me handing out Plumbing certificates to anyone who successfully completes the Mario & Luigi video games because these characters are plumbers in the games. That's just not right. Likewise, there is no actual "science" performed by teaching people to create scientific-like reports. (Although it happens sometimes that testers learn testing skills accidentally if they pay attention to what they're really doing.)

So, where are we here? Building software requires layers of intelligent, creative effort and problem-solving abilities. These layers are complementary and require different skill sets. Just like you wouldn't hire an accountant to deploy your systems, I don't think it's wise to hire a programmer to provide valuable testing insights into the development process. It's the wrong skill set.

Oil and water, or oil and vinegar - the analogy holds for me. Testing is something completely different from Programming and building software. It's a layer on top to help you know that what you are doing is on target for what your paying customers are expecting. Some might call that value.

Software Testing "Popcorn" button

2011-01-10T12:17:00.002-05:00

I made myself some microwave popcorn for a snack just now. Placed the popcorn bag in the microwave, pressed the 'popcorn' button and then 'start'. Someone next to me said: "There's a popcorn button?" Um, yes, there is. In fact, there has been a 'popcorn' button on every microwave oven I've ever seen.

I explained to my colleague that the recommended time on the bag (in this case it was 2 min 30 sec) doesn't work on every oven. Different ovens have different power output and so the actual cook time may vary. If I go with the default time, it might burn or be under-done and leave too many unpopped kernels in the bag. You could figure out the correct time in a few ways.

Method 1: Math
Start by taking a look at the power output of the oven. Full-size ovens deliver 1,000 - 1,600 watts of power, and mid-size ovens yield 800 - 1,000 watts. Higher wattage heats food more quickly. If it's not explicitly written on the bag, assume the default popcorn popping time is for a 1,000 watt oven.

Use a time/power ratio along the lines of: ( t_your_micro_oven / P_your_micro_oven ) = ( 150 sec / 1000 W)

I converted the 2:30 recommended time on the bag to 150 seconds for convenience. The power of your microwave oven should be written on the back somewhere, probably close to the plug. This leaves your popping time as the only unknown variable in the equation, so it should be straightforward to solve.

That might work.

Method 2: Brute Force/Iterative
Put the bag in the oven and follow the instructions exactly. Make no changes. When the time is complete, take the bag out and put the contents in a bowl.

If you are happy with the result, congratulations! You are done. If not, we will need to change the cooking time for the next bag.

On a piece of paper (or in a document on your computer if you prefer), make some observations to capture things like: time settings used, quality of the popcorn output, taste, number of unpopped kernels, and so on. Keep a note next to the microwave oven or on the container where you keep more popcorn bags -- a note to reference these 'test results' so that you can have something for comparison the next time.

Basically, you know where I'm going with this. Pop the first bag with the default recommendations and keep varying the subsequent popping times until you are happy with the result. This will require "n" bags to iterate through until you are happy with the end result.

This should eventually produce high-quality results. It may take you some time and will be costly to do it this way as you may have to go through many bags until you get it just right.

Method 3: Popcorn button

Put the bag in the oven, press the 'popcorn' button, serve and enjoy.

I'm not certain how this works. I shall call it "magic" for now. I can speculate that perhaps the microwave oven manufacturer has a team of engineers dedicated to the "Perfect Popcorn Production" using their equipment and are responsible for programming the correct power and time settings into their ovens.

Maybe they use Method 1 above. Maybe they use a combination of methods 1 and 2 above. Maybe it's something else.

The point here is that someone has already done the thinking for me so that I can focus on the quality of experience of using their oven.

Wow. I have so many comparisons and analogies back to Software Testing running through my head right now.

One that jumps to mind is development & testing terminology/jargon.

When is a test not a test? When it is a check. When it is an inspection. When it is a question. and so on.
Are your [agile] development sprints 2 weeks? No? Why not? That's the default value, so it should work for you, right?

What if software organisations were responsible for their own 'popcorn' buttons? That is, where the 'popcorn button' is a way to communicate the methods and models used within their organisations that produce the desired 'quality' result.

As a consultant, one of the first things I do is watch and listen to the development and project team members. I need to understand their terminology and way of doing things. That helps set a reference frame for me. If I want to make an improvement or change somewhere, I need to have an understanding of how to do that on their terms, not terms according to some industry standard or certification terminology dictionary that may not apply.

What would be the point? Well, I think it would be handy if we could abstract out some of the desired practices from the terminology and implementation details that are specific and custom to every organisation, team or project.

Now that I think about it. Maybe, we as consultants are the "Popcorn Programming Facilitators." That is, the company wants to implement an automated regression testing activity. What does that look like? How can that work for us? What kind of output do we get?

Some days we identify the required popcorn buttons (e.g. Regression Testing, Bug Tracking System, Produce Status/Progress Reports). Some days we help program them.

What do you think?

Fishing for Wisdom

2010-11-12T19:34:00.002-05:00

I just came back from a week at the AYE Conference. My head is full with several new ideas swimming around and stirring up half-baked old ideas - which is a good thing.

One of the thoughts causing my head to spin came from a session one evening where we discussed ideas to improve the conference moving forward. Johanna Rothman led the session and at one point she mentioned that the AYE workshop sessions included pure [Virginia] Satir [ideas, models, etc.] and applied Satir. This got me thinking about some of the subtle differences I had noticed about the sessions and what they meant to me.

In particular, some of the ideas and models I learned from the AYE sessions appear to dwell longer in my mind and apply to a broader spectrum of situations while others seem to be more specific - i.e. an application of a model in a particular context. Don't get me wrong, whether you choose to attend a pure Satir or applied Satir workshop at AYE (and the sessions aren't labelled as such because it doesn't really matter), it's a win-win scenario. :) Sure, different hosts have different styles, but each session is different every time so you sometimes see people attend the same session again to see what new insights they get.

So, what's the big deal here? Why did I get stuck on a small point like this? Well, it reminded me of the time when I was in Teacher's College in the mid-90's, preparing to become a High School Science teacher.

I had 2 main professors in Teacher's College - one for each of my 'teachable' subjects of Physics and Chemistry. Their styles were very different. One professor seemed to keep me busy while the other made me think a lot.

I'll be honest, coming from a university environment to Teacher's College, I expected to be told what I needed to do to be a teacher. You know - I expected a lecture-style learning environment like most of my previous undergraduate courses. What I got didn't match that expectation so I was a bit frustrated at first until I discovered the secret that no one clearly explains to you.

What's the secret? Okay, I'll tell you. The secret is that when you go to Teacher's College, *you* are the teacher, not the student. So, by going there acting like a student expecting to be taught, I had the perspective all wrong.

I don't recall when the paradigm shift happened for me but I'm glad it did. After that point, I didn't consider my professors to be the ones teaching me to be a teacher; I saw them as guides to help me learn the things I needed to be a better teacher.

The real teacher here is experience. And that you can't get unless you are doing what you want to be doing, not sitting in a classroom somewhere talking about what you want to be doing. So what did I get from the Teacher's College experience? I got access to many different 'teachers' to help me deconstruct my experiences so that I could learn from them.

I need to pause for a second and think about that last sentence again. That sounds suspiciously a lot like the AYE conference experience to me. Hmm.

So what was different between my two main professor/guides? I think (now) it was something similar to the difference between the 'pure Satir' and 'applied Satir' AYE sessions. One professor offered tips and ideas that applied in a certain situations - the ones we said mattered to us - while the other discussed models and ideas to help us learn from our own experiences in more broader situations. Looking back, those things weren't clearly stated in that way at the time. I think I understand a bit more now about how they were trying to help us, in different ways, to become better teachers. Of course, both professors provided us with lots of opportunities to practice demos and teaching short topics in an environment where we could safely receive feedback from our peers. (hmm, more AYE conference and PSL familiarity here.)

I recall that someone once mentioned the old Chinese proverb while were at Teacher's College:

Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime.

To me, the quote makes me think of the difference between information and knowledge. I think both professors were trying to teach us to fish (i.e. give us knowledge), just in different ways. They both wanted us to leave the college more confident with knowledge that we could use to help ourselves become better teachers moving forward.

Back to AYE and present day. Reflecting upon the learnings from this past week, I learned new models and ideas that I can apply in many ways - some apply at work and some apply to life beyond the workplace. I think I left with a few fish and a few new fishing techniques.

The old proverb bugs me though. I don't think it completely captures the full experience and feeling of what happened. There's something missing, something meta.

And then it came to me today. It's not the fish. It's not learning to fish either. It's the fishing.

If you pay attention to what you are doing while you are fishing, I think that leads to something other than information or knowledge; it leads to wisdom.

Continuing with this proverb as an analogy, if I learn deep-sea fishing while on vacation, I don't think that will help me very much if I decide to go fly-fishing at a local river. There are many different ways to fish - for the different kinds of fish, the environments in which they live, and the purpose of fishing (e.g. food vs sport). If we pay attention to more than just the types of fish, their environments and techniques to catch them, we can learn something more, something bigger. I find it hard to think in broad ideas like that sometimes. I also find those are the most rewarding moments though.

Attending the AYE conference (and PSL this past Spring) was like that for me. My head doesn't stop thinking about ways to apply the models and ideas we experience at AYE. Meeting wonderful, intelligent, kind practitioners from all over the world helps enrich the shared experiences in ways that bring us closer together. We talk about fish (individual experiences) and techniques to help find solutions to problems we think we see. And then the hosts/speakers go and show us things that help us solve problems we didn't even think about or see!

For me, attending AYE is an opportunity to meet old friends, learn about myself, learn about how to interact better with others (both at work and in personal life), share experiences and knowledge with colleagues, learn some new problem-solving techniques, make new friends, and pause for a moment to reflect upon where I am in life and where I'd like to be. It's a moment to notice that I'm fishing - I'm learning and growing. And that others are fishing too. And while some are fishing for similar things and others for different things, we all recognise that we are fishing so we have that in common.

It's going to take me a bit of time to unpack all of the ideas I was exposed to this past week because learning happened on many different levels. I could post some notes, and I plan to, but I don't believe the notes alone can convey the experience of learning that happened there. Much learning happened between conference sessions too. You meet so many people with similar or related interests and different experiences that once you start talking in the hallways, over dinner or lounging about somewhere, you can't help but continue to learn and think about things in new ways.

If you want, that is. If you're into that sort of thing. If learning, growing, and working better with other human beings isn't your cup of tea, then this conference is definitely *not* for you.

To everyone else, I highly recommend the experience. This conference, and the PSL workshop, are opportunities that shouldn't be passed up. It might even change the way you think about things. It has for me.

To the AYE and PSL hosts, Jerry, Don, Esther, Johanna and Steve, to my Teacher's College professors, Peter and Tom, and to my family, friends, and colleagues who have all provided me with helpful feedback and information to help me grow and be a better person, I thank you.

Using MS Outlook to support SBTM

2010-09-30T23:56:00.004-04:00

Okay, to recap, Session-Based Test Management (SBTM) is a test framework to help you manage and measure your Exploratory Testing (ET) effort. There are 4 basic elements that make this work: (1) Charter or mission (the purpose that drives the current testing effort), (2) Time-boxed periods (the 'sessions'), (3) Reviewable result, and (4) Debrief. There are many different ways that you might implement or apply these elements in your team or testing projects.

Let's take a look at tracking the testing effort from strictly a Project Management perspective. Years ago, when I first became a test manager, I was introduced to the idea of the 60% 'productive' work day as a factor to consider when estimating effort applied to project schedules. That is, in a typical 8-hour workday you don't really get 8 complete, full hours of work from someone. I don't believe it's mentally possible to get that. The brain needs a break, as does the body, and there are many natural distractions in the workplace (meetings, email, breaks, support calls, stability of the code or environments, and so on), so the reality is that the number of productive working hours for each employee is actually something less than the total number of hours they're physically present in the workplace.

That 'productivity' factor changes with each person, their role and responsibilities, the number of projects in the queue, and so on. Applying some statistical averaging to my past experiences, I find that 60% seems about right for a tester dedicated to a single project. I have worked with some teams that have been more productive and some much less.

So what does this look like? If we consider an 8-hour day, 60% is 4.8 hours. I'm going to toss in an extra 15 minute break or distraction and say that it works out to about 4.5 hours of productive work from a focussed employee in a typical 8-hour day. Again, it depends on the person and the tasks that they're performing, so this is just an averaging factor.

4.5 hours is an interesting number. In the SBTM framework, the "Time box" length for the test session revolves around a normalised value. The original SBTM presentation ("How to Measure Ad Hoc Testing") suggests that 1 "Normal" session = 90 minutes (+/- 15 minutes). Your "normal" time box session may be 90 minutes, it may be less. Customise it to suit your team's needs and fit.

I'll use 90 minutes for now because it's the default recommended value to have a focussed test effort to get some solid testing done. 90 minutes. That's 1.5 hours. So 3 * Normal = 4.5 hours.

Therefore, from a project management perspective, 3 Normal sessions is an average tester's target productivity level for completed test sessions in an average 8 hour day.

It doesn't seem like a lot, three 90-minute sessions, but you'd be surprised at the number of distractions that happen throughout the day and development sprints/cycles that can prevent a tester from completing 3 sessions in a day.

Recently, I've been working with a tester on an Agile team who has been facing a high distraction level. As a result of all the interruptions for quick checks, reviews, design meetings, and so on, he has been struggling to complete 3 sessions in a day.

I should mention that I *AM NOT* asking him or anyone else on my team to complete 3 sessions in a day. I am not concerned with the numbers or the math of tracking the sessions completed, especially because the development team environment is very integrated and highly collaborative.

What I am concerned about is trying to find ways to block off his time so that he can get some solid testing done, uninterrupted. One of the powerful aspects of the time-boxed session is to reduce or minimise the distractions so that your brain can maintain the focus on the testing problems at hand. The more the distractions and interruptions, the more likely that something important or interesting will be missed.

Enter MS Outlook. When I look at my Outlook calendar it is riddled with meeting requests throughout the week. I have recurring meetings, impromptu meetings, lunch and learns, scheduled debriefs, sprint planning, retrospectives, demos and so on. I'm the Test Manager/Lead, so that's expected. When I look at my tester's calendar, it is fairly blank/open, save for the daily stand-ups and a few other weekly team meetings. It seems counter-intuitive that someone with an open calendar, so much available 'free' time, should be unable to block off time-boxed periods dedicated to testing specific features and charters.

So I've started to schedule his test sessions in MS Outlook by scheduling Appointments. At the start of the day, we'll sit down and come up with 3 important charters that we'd like to cover in the day. We block off one session/appointment in the morning and two in the afternoon. The "Subject" is a summary of the charter and priority. If the session involves another Dev or team member, they can be included in the appointment, which is now a 'meeting request'.

Outlook is a convenient tool since it (a) is available on everyone's desktop, (b) has built-in reminders, and (c) allows the tester to *see* all the free time available between scheduled test sessions. It's a way for him to try and regain control of the *uninterrupted* test sessions.

SBTM isn't intended to be a time-tracking tool. It's a productivity-enhancing tool. By keeping focussed on the charter, you give your mind time to think about the important things -- to learn about the system and the software, to apply the appropriate tools and techniques, to make the best observations you can in the time you have available. Performing good Exploratory Testing is a delicate and complex thought process.

I see my job as doing whatever I can to help provide the best environment possible for good testing to succeed. I never would have thought that Outlook would factor into that. Who knew?

Test-Driven Development isn't new

2010-09-16T12:10:00.001-04:00

I used TDD as an analogy to a tester today to explain how logging bugs in a bug tracking system drives the development. A bug report represents a failing test (when you verify that it's really a bug that is) according to some stakeholder need/want.

In Test-Driven Development, the programmer writes/automates the test first that represents the user story that the customer/user wants. The test fails. The programmer then writes enough code required to pass the test and then moves on. (refactoring code along the way, etc..)

It's much the same with regular system testing (i.e. in the absence of agile/TDD practices) where a tester identifies and logs a bug in the bug tracking system. One difference is that these bug reports/tests aren't always automated. (Okay, I've never seen anyone automate these bug reports/tests before but I like to believe that some companies/dev teams out there actually do do this.) That doesn't change the fact that a bug report is the failing test. Even if it's a manual test, it drives the development change and then the bug report is checked/retested to see that the fix works as expected.

Bug regression testing, then, is a requirement for good testing and system/software development, not an option.

So, while the agile practices of TDD and others may seem new, I see this one as a retelling of a common tester-programmer practice. If anything, I see TDD as an opportunity to tighten/shorten/quicken the loop between testing feedback and development. With practice, TDD helps programmers develop the skills and habits they need to create code and systems with confidence -- to know that as the system grows, the specific needs of the customers are being met every step along the way. No one gets left behind.

How can we, as testers, help? If your programmers don't practice TDD or automate tests, start investigating ways that you can do this. Investigate Open Source scripting languages. Engage your programmers in discussions of testability of the interfaces. There are many articles and presentations on the internet on the topics of test/check automation, frameworks and Domain Specific Languages (DSL).

Start reading. Participate in discussions (in real life and online). Start developing scripting skills (I recommend Ruby, of course, especially to the tester newbie). If you don't feel confident with your programming skills, help hire someone onto your test team that can help all the testers advance their skills, knowledge, and productivity in that area.

Be the Quality Advocate by putting your words into practice. You want your programmers to start practicing TDD? Show them how you can do it. You are already doing it - scripting/automating the checks that demonstrate a bug failure is just the next step.

Start by automating a single bug failure. Take it from there.

Why New Year's Resolutions Fail

2010-09-03T00:41:00.000-04:00

Someone recently said something to me that made me think. He said that all New Year's resolutions fail because they come at the wrong time.

You know what I mean by New Year's resolutions, right? It's those promises you make to yourself, and maybe to others, right around the end of December that you will change or improve yourself in some way in the new year.

The sentiment may not be wrong, but the timing certainly is. The argument made was that January 1st isn't really the start of the new year - September is. You see, here in North America, whether you are in school or not, most businesses revolve around a "school year" structure of September to June, with July and August being the summer holiday months.

So, if September is the start of the year, we can't make promises to change something in January. That's like starting a 2-week sprint (in Agile Development) and saying half-way through that you are going to have completely new objectives. It doesn't work that way. You already committed to delivering certain goals during the Sprint Planning session at the start.

What's that? What if you didn't set any goals at the beginning of the Sprint/Year in September? Doesn't matter. The Sprint/year started anyway and you are in the middle of it. There's no way you are easily going to shift your life in a totally new direction half way through.

So, the moral of the story is: if you want to make New Year's resolutions, make them in August, not in December. That way you are more likely to follow through with them as the year progresses.

Hm, interesting.

Of course, life changing events can happen any time. You don't need to make a resolution of any kind to change yourself and how you get along in the world. You just need to see yourself how you want to be, and live like you've already reached that goal.

Testing Software is like Making Love

2010-07-10T09:58:00.002-04:00

Work with me for a minute here.

One of the things I dislike about a recent trend in the software testing profession is the lack of analogies and examples that relate to me as an adult. Yes, it's interesting that children are innocent, curious and are natural explorers, but they are also naive, inexperienced and cannot reason or abstract like adults can. I don't find it helpful to me or when training new testers to say you need to be more like children. I'm an adult, so how can you help me now? Do you have an analogy that relates to me as an adult?

Here's an analogy that I think might help.

Testing software is like making love.

What does that mean? For a start, what's the difference between 'making love' and 'having sex'? I think the big difference is caring about the person you are with and wanting to satisfy their needs.

Is there just one way to do it? No. Every person's needs are different, just like every project we help test is unique.

I think people/consultants who specialise in certain things like an automation tool or a process improvement model are kind of like porn stars. They've mastered a technique and go around showing everyone just how good they are. Often, the perceived satisfaction in their results is just superficial and temporary because we are so impressed with their performance and showmanship. Is there real satisfaction or addressing of customer/stakeholder needs in specialists like these? I don't think so. Well, maybe sometimes, but I highly doubt it's the silver bullet/best practice they make it out to be.

Is there a need for specialists? Yes, I think so. We can (usually? often? sometimes?) learn from them. I believe that people (customers, here) fall into high-level patterns and that we (who want to address their needs) should learn from those who have experienced successes in working with certain kinds of customer needs. Does that mean that we should only look to one specialist in our lifetime? That's up to you. That's like saying you only want to learn to use a hammer and will look at all of life's situations with the one tool you know how to use.

Is there value in getting a certificate to show that you've learnt a particular sexual technique? Would you be proud of it? Would you post it up on your wall at work? Would you put it on a t-shirt, business card or advertise that you have this knowledge and/or experience?

There are so many books out there on sex, love and relationships - there are so many different needs and patterns to try and address. If this analogy holds true then the Software Testing industry has a long way to go to catch up on the number of different books published to help people be successful and happy in testing software and systems.

Perhaps we need something like the Kama Sutra of Software Testing. (ooo, I like that. just made it up.)

Does making love involve intercourse? What about tantric sex? What about just cuddling? What if you're alone? Maybe sometimes sex is all you really need at that moment - get in, get the job done, get out. Maybe it's what the customer both needs and wants. Don't think too much about it then. Do your best and move on. Porn stars don't need to build relationships to be famous or successful.

What's the difference between providing someone with what they want compared with what they need? Here's a good follow-up question to ponder: if you provide your customer with what they need instead of what they want, will they still be happy? Will they tell you that? Are you sure that you've really identified what they needed? How do you know? How are you certain about what you think you know?

Talk to your customers. Get to know them. Try to understand both what they are really asking for and what you think they need. Do your best to try and address their needs. Provide everyone on the project with the information they need to be successful in the project constraints. We are only human and therefore make mistakes and miss things sometimes. People can be forgiving too and may provide you with other opportunities to show that you've learned from past experiences and that you care about growing and doing a better job next time.

If the client/organisation isn't forgiving, then take the one-night stand as a learning opportunity and move on. Seek out what you desire. I believe that the best software testers are truly the ones who care about the people we work with and for. They care about doing a good job, about learning, growing and improving their performance and satisfaction level. It's unfortunate that not all of the organisations we work for see that or care. It doesn't stop us though. We're a caring bunch. We're weird that way. ;)

Good software testers are lovers.

Happy Limerick Day! (May 12th)

2010-05-12T17:39:00.001-04:00

The CEO where I currently work sent around the following note by email at the start of today:

Today is Limerick day. A limerick is a five-line poem with a strict form (AABBA rhyming), which intends to be witty or humorous, and is sometimes obscene with humorous intent.

Here is one from me (if you respond in kind let's stay away from the "obscene" part of the definition)!!
[..snip..]

Of course, that challenge went answered and we have had a steady stream of limericks all day! :)

Here were some of the ones I thought up between meetings and test sessions...

(The first one is on hiring testers:)

What is Quality?
How do you check the functionality?
Provide info in a timely fashion
Prefer those with passion
Most important, you need personality!

Doing good work takes time and thought
If you think testing is easy, it is not
We apply test techniques
To see what reeks
And help the stakeholders know what's hot.

In Exploratory Testing we Learn, Design, Test.
We believe no practices are "the best"
Context helps us decide
Which approaches are tried
Until we solve the problem we do not rest.

Jerry Weinberg is an important man
Of his work and ideas I am a big fan
He helped me to learn
People are the concern
So make helping them succeed the plan

Okay, so I'm not a poet. ;-)

Can you do better? Leave some limericks in the comments and have fun! =)

Cheers! Paul.

SBTM ET.XLS spreadsheet with DMY format

2010-04-08T00:33:00.000-04:00

Ha ha! I finally got some time to figure out how to get the ET.XLS spreadsheet to support DMY format instead of the default MDY (US) format. It turned out to be a small change to the macros, but unfortunately required hard-coding the number of columns in the input files to make it work. As long as you aren't changing the number of columns in the TXT files, this should work for you. I also had to remove the forced "m/d" format on one of the worksheet tabs.

You can download a copy of the zipped spreadsheet here: et2_DMY_dates.zip. I also updated my www.staqs/sbtm page to include this file.

Last July I posted the latest tools-ruby scripts and received some feedback that I wasn't the only one with this problem. I fixed the date formatting problem using Excel 2003, so if anyone is using an older version of Excel too bad. ;)

As for Excel 2007, I just started using it this week. I noticed that I had to play with the Macro security settings (once you make the 'Developer' toolbar visible), and then I got it to work. I'll be playing with this a bit more in the coming days so if an update is required I'll post it here to let you know.

If you find this updated file helpful, please drop me a line to let me know. Thanks.

Cheers!

TEDxWaterloo

2010-02-26T16:02:00.008-05:00

I attended the first independent TED event in Waterloo (TEDxWaterloo) yesterday, 25 February 2010. The theme was "Tomorrow StarTED Yesterday." The web site and twitter account page have lots of great info if you're interested to know more about the event and speakers. There's even a nice photo blog of the event at http://www.longexposure.ca/2010/02/tedx-waterloo/

So what can I add that hasn't already been said? Well, I can tell you what the event meant to me.

I wasn't sure what to expect. "TED" stands for "Technology, Entertainment, Design," but that tells me nothing. I've seen and enjoyed a number of the videos that I have come across - i.e. that were recommended to me in one way or another (email, web page links, twitter, passing conversations, and so on). However, separate videos alone hadn't quite made the connection for me.

The opening remarks helped set the context for the event. In those few minutes, it all came together - it clicked. Aha! This event is about more than the sum of the individual words above. Even the motto "Ideas worth spreading" make sense to me now.

The people who spoke at the conference and the selected TED videos that were shown hit me somewhere I didn't expect - in my heart. Most of the conferences that I have attended speak to my mind; help me try to understand or learn something or other. That's nice, but that's not all there is to life.

There's only one conference that I have ever attended that I would compare to the TED event - the AYE Conference. The difference here is that AYE is built from a series of interactive workshops intended to help us understand and work better with each other - human beings, not machines. TED is a perfect complement in that the speakers share stories and ideas which inspire us.

Inspired. Yes, that's a word I would use to describe how I felt during and after the TEDx event.

I spoke with someone at the event whom I hadn't seen in over a year. She described to me how her current project is making her feel dumber every day, dealing with processes and bureaucracy that only serves to confuse and make the project more difficult. At the TEDx event she said she felt like her brain was expanding, opening up again. I think that's another good description of how I felt.

Following the TEDx event, I now have a better understanding of what the words mean when I see them. I have a context for this event and videos - which helps me know what to expect and where/how to process the information.

Here are some notes I took from some of the speakers at the event:

Terry O'Reilly

The Age of Persuasion
idea of friction - sometimes you need to slow down in order to sell and market new ideas. Brilliant examples
He referenced the book "The Checklist Manifesto" by Atul Gawande. (Good example that has come up a few times in testing. I think we need more good, practical examples in our profession.)

Paul Saltzman

(I was really surprised and blown away by this talk. Really left me with a lot to think about afterwards.)
"magic" - that which is real but you do not yet understand. (I've seen magic used a number of times to help demonstrate certain testing concepts. Paul used it in a different sense here and I think there is a lot of magic in what we do every day.)
"humility" - not making yourself feel small, rather, seeing yourself in perspective of the larger universe.
"There is no end to love. Love is infinite. There is no end to creativity. Creativity is infinite."
"Nothing changes until you do."

Caroline Disler

"Western civilisation" - There is no Eastern vs. Western civilisation (unless you happen to be strictly speaking geographically). We are one world.
"Those who are ignorant of the past are prisoners of the present."
she gave a really interesting summary of where knowledge and language (and even numbers!) originated and how every part of the world has helped bring us to where we are today.

Madhur Anand

Heh. I didn't know about Sudbury and the environmental restoration projects happening there. Cool! (My university degree was in Environmental Science, so I had an interest in this talk. As a mining town, Sudbury hasn't had the best rep over the years, so I'm pleased to see that it is leading the way in environmental restoration projects. The world needs more of this.)
she used some interesting quotes. One that was new to me was: "To live in a place, you must first imagine it." by Jay Ruzesky
The environmental crisis is also a crisis of imagination.

Darren Wershler

First one to mention Marshall McLuhan (he said he pulled the short straw backstage before the event. ha ha.)
The roles of different kinds of media (untimely, conceptual and impossible) and their influences on us. Something doesn't have to be real or even possible to have an effect/impact on us, to inspire us to new ideas.
e.g. transporter technology from Star Trek, or the many inventions by Reed Richards of the Fantastic Four. (Nice comic book reference!)

Marty Avery

Be soft to be strong. Namaste.
Kayaking story really touching. What does it mean to be a hero?
There is some of us in everyone we meet. "You am I and I am you."

Amy Krouse Rosenthal

(Google search doesn't do her justice. This was [yet] another amazing talk! Here's a link to reach her: http://www.whoisamy.com/ - watch the videos!)
The 7 [musical] notes on life:

Always Trust Magic (ATM)
Beckon the lovely. Whatever you beckon (attract, look for) will eventually find you.
Connected. We are all connected.
Do
Empty space
Figure it out as you go
Go do it. Howard Thurman quote: "Don't ask what the world needs. Ask what makes you come alive, and go do it. Because what the world needs is people who have come alive."

The [musical reference] 'key' is you
"Make the most of your time here."

Wow. And that doesn't cover all the speakers and videos we watched!

This event was certainly something I needed! I was moved and feel like I want to do something even bigger now! Well done! I can't wait to attend the next one or create my own independent TED event! =)

If an independent TEDx event is coming near you, I highly recommend you take the time to attend. If you do, please tell me what you thought and felt.

Cheers!

Now with minty-fresh visitor counter

2010-02-24T14:39:00.002-05:00

Someone suggested to me this past weekend that I add a visitor counter to this blog. It's one of the most common suggestions made to me over the years and I don't know why.

Back in the mid-90's I had a web site with a counter. It was novel then. I played around with different fonts and features and watched it go up over time. I don't have that site anymore. I set up a new web site about 7 years ago, but adding a counter wasn't one of the important things on my to-do list. Foolish? Dunno. Maybe. Maybe not.

Is Quality measured in numbers?

Anyhoo, I won't go there today. ;-) I'm not going to think philosophically about it right now. I'll reserve thinking and judgment about the visitor counter for a later date.

This is just a placeholder note to indicate that the counter started today.

Cheers! Paul.

SBTM is not ET

2010-02-08T23:48:00.007-05:00

There's a subtle but important distinction that I'd like to talk about. Session-Based Testing is *not* Exploratory Testing. Please stop using those terms interchangeably because they're not.

Exploratory Testing (ET) is a testing approach that puts the emphasis on real-time learning, test design and test execution, as opposed to a more "scripted" approach that puts the emphasis on the separation of these activities - separated in time, space, and usually with copious amounts of documented artifacts.

When I first started in I.T. over 20 years ago, any testing I did as part of my programming contracts were exploratory in nature. I didn't call it 'ET' at the time and I certainly didn't approach it with the same discipline and formality that I do today. Back then, Programming was my main focus and testing was just something I did as required along the way. Ten years later (or about 12 years ago depending on your perspective), I took a workshop class on "Test Case Design" with Ross Collard. That was an amazing class that opened my eyes to a whole new world of analysis and problem solving that I didn't know before. Cool!

After that workshop, I had plenty of opportunities to practice what I learned, try new techniques and tools, and explore additional testing ideas thrown out onto the just-budding software testing mailing lists. One of the things we discussed in Ross' class was the role of "ad hoc" or informal testing. I don't have access to the data, but some study-or-other at the time (90's sometime?) showed that ad hoc testing failed to produce the same amount of testing coverage that formal test design analysis would.

Okay, I buy that. To paraphrase: guessing ideas off the top of your head consistently produced less coverage than having some structured analytical approaches/techniques/heuristics/models at your disposal. Okay. I don't need a formal study to tell me that.

So what's different with Exploratory Testing? Well, when I first learned about ET at the turn of this century, it instantly clicked with me. Rather than the "guessing" attitude normally associated with "ad hoc" testing, ET clearly defined the testing approach in a way that made you think. You learn something; you design something; you test and observe something; repeat. Note that nowhere in there does it say "take a wild-ass guess and call it good, complete or even 'good enough' testing coverage."

Before I was introduced to ET, I had spent several years practising and training other testers on test design techniques. That helped me fill in the "test design" step of ET. That step is the weakest link with most of the testers I have met and spoken with over the years who have tried and given up on (i.e. failed with) ET. You can't really fake your way through test design. That's why I make it an important part of my hiring/interviewing process (you can read the article online).

So, what *don't* I like about ET? There's just one thing really. The ET approach formalised the learning, test design and test execution aspects of testing, but not the interpersonal communication aspect of it.

The 'scripted' (waterfall) approach to testing relies on the documenting (and maintenance) of hundreds or thousands of test cases, each with their own set of pre-conditions, steps, expected results, and so on. While the value of these documented test cases may be questionable, one thing going for it is that you can share these test ideas with other people quite easily. (They're documented; pass it on.)

In ET, not so much. If the important parts of testing takes place in your head as you process all of the inputs and information, and compare them with explicit and implicit requirements and expectations, in order to assess the quality of the A/SUT, then when/how do you share those test ideas with other people (testers, developers, business analysts, etc.)? Well, you don't. Or rather, ET alone doesn't give you any advice for communicating test ideas or testing coverage with others.

Enter "Session-Based Test Management" (SBTM) or just '"Session-Based Testing".

Aha! After a year or two of using ET, I instantly found the merit in SBTM. SBTM provides the framework that you can wrap around an ET approach. It is a way that you can manage the testing effort. It has four main elements: develop specific charters, time-box an uninterrupted work session, create a reviewable result, and review/debrief the session afterwards.

Here's the catch: it is *not* a testing approach! It is a test management framework. Actually, when I teach/describe it to others, I sometimes refer to it as "Session-Based Task Management."

I have taught SBTM to programmers as a way to help them manage their time and reduce the number of interruptions during a work day. I have also successfully implemented SBTM in a waterfall organisation where very little ET was ever performed.

Yes, you read that correctly. I have even wrapped SBTM around a *scripted* testing approach.

Eek! Egad! Gadzooks! Isn't that blasphemy?

Well, actually, no.

You see, I have found that SBTM is an incredibly powerful tool for a test manager. It gives you insights into aspects of testing that you might never have without it.

The four main SBTM elements provide a solid foundation to managing your work, and can be transferred to activities other than just ET. For example: programming, writing, organising/cleaning your basement, any consulting work, and so on.

The original SBTM framework included some Perl scripts that I have long since stopped using. The original archive included a session sheet template, but like any template you can modify it and tailor it to your needs. (If in doubt, just ask James Bach for his thoughts on Test Plan templates! :)) That's the main reason I rewrote the SBTM scripts in Ruby - so that I could customise the session sheets to the needs of the projects I worked on. So, for one project I added a section to the session sheet, and for another project I completely removed the TBS metrics; my Ruby scripts are flexible and can handle the changes easily. (ASIDE: I haven't made this customisable script publicly available on my site yet. Send me a note if you are interested in trying it.)

In fact, if you follow the intent of SBTM, you don't need to use the session sheet template or scripts at all - as long as you have some agreed-upon reviewable result that you can later debrief. In this way, I have heard of some test teams that have implemented SBTM using Wiki's, and others that have integrated old Test Case Management systems into the process. Sounds cool and innovative to me!

So, what's my gripe? In the last several weeks, I have read several times that Exploratory Testing includes time-boxed, chartered sessions with reviewable results. Umm, no, I'm pretty sure you're confusing the framework with the approach, the wrapper with the content, the book format with the story.

If you have implemented SBTM on a project, I can make no assumptions about what testing approach you are following. Likewise, if you include ET in your overall testing strategy, I won't assume you are using SBTM to manage that effort.

If you want to talk about ET or SBTM, please try to describe them in the correct context. It will make it less confusing for beginners and other interested parties. Granted, together you have a very powerful combination. But Superman is still super in a different suit. =)

Time - Bane or Innovation Catalyst?

2010-02-03T12:08:00.006-05:00

Time. What time is it? How much time do we have? When do you want/need it? What's the deadline? I need more time!

If we had all the time in the world for software development, would the delivered results really be of better quality?

A co-worker at a past employer wrote the following when someone sent an email submission for a fun, internal contest the day after the deadline:

The contest ended a long time ago. Trying to submit something now is like submitting your late university assignment.
One of my profs told me:
"I don't care if you have something that's better than all the works of Shakespeare. If you can't get it in before the deadline it's worth nothing to me."

Ha, ha. It was intended as a funny remark at the time but there's some truth in there too.

So, if someone submits an assignment "on time" but of lesser value/quality than they might produce if they had more time, would they still continue to work on their opus or would they give it up to move onto the next project? Do we (as a collective group of intelligent human beings) lose out by putting Time ahead of Quality?

The traditional "Project Management Triangle" puts the emphasis on: functionality/scope, cost and schedule. An experienced consultant can tell the employer: pick/fix any two and we can estimate the third.

I noticed years ago that "quality" isn't in this "triangle". As a novice, I took "scope" and "quality" to be part of the same point. Clearly I was mistaken. When people are focussed on delivering something, on time, at a fixed cost, everyone interprets "quality" in different ways.

I think the Agile manifesto/movement is an interesting response to the "traditional" (a.k.a. Waterfall) approach to software development. It takes the same 3 constraints (of scope, cost and time) and changes up the order of activities to integrate quality into the deliverable products. This is done by embedding customer involvement (via collaboration, user stories, automated acceptance tests) and rapid delivery releases to allow for quicker feedback into the design and implementation. For example, in a traditional/waterfall project, it may take anywhere from 6-18 months to find out your interface/implementation fails to meet the needs of the customers. Or, using agile methods, it might take anywhere from 2-14 days. Your choice.

So what about software testing?

In every waterfall project I have worked on, development always delivered software late into the "test" phase. This meant less time to provide feedback, because the release deadline was fixed. Time is my bane here. I've got less of it and need more of it! ... or do I?

If I stick to a waterfall approach to testing - i.e. develop & document test plans, test strategies, test cases, execute the tests, log the results and communicate the summaries - then, no, time is not my friend here.

But is it a requirement to do testing this way? Whose requirement? How much does their opinion really matter?

I watched my son play a game recently and describe the "glitches" (his lingo, not mine) to his younger brother so that he could try and work around them. I'm pretty sure my boys don't care whether the software team used waterfall or agile methods, or how well their test cases and processes were documented. They found bugs in their game, are annoyed by them, and figured out ways to work around them. Sometimes they just give up on a game altogether.

Personally, I'd say that the customer doesn't really care about how you do your testing - as long as the end result has good enough quality that doesn't interfere with their intended use of the software or system.

Here's a secret: Nobody cares.

Some lawyers may pretend to care when they are paid to do so, but the reality is that I don't know of a single tester who has ever been charged with manslaughter for failure to document critical test cases that may have caught the bugs that resulted in loss of life.

The FDA doesn't care. Their lawyers tell them that they should care about documented tests and results, so they impose regulations. But the FDA doesn't really care about your documented test cases or test processes. What they really care about is that a minimum standard of due diligence has been performed to demonstrate that a particular product will not harm anyone. That's it in a nutshell. You may not even need testers to achieve that level of quality either.

I could go on, but I think I made the point - nobody cares how you do your testing as long as the collective development effort produces a quality product. You remember "quality" - it's that thing that project managers leave off their project management triangle.

So, if we disregard the premise that testing needs to happen in a "waterfall" fashion, what's left? Well, what do we know? We know that (1) we don't have a lot of time, and (2) we have a lot of features to cover. Oh, and it's also very likely that (3) you have a limited number of resources and people - most likely less than what you'd probably like. (Hey, if we're screwed on the 'time' factor, why not get screwed on the 'cost' factor too, right? ;))

So where does that leave us? Time to innovate! Time to become agile! Talk to your customers; collaborate with your developers and business analysts/product managers; learn the software and functionality as you design and execute the tests because there really isn't time to do those things separately.

Risk-based testing (RBT) works on the premise that there might be something bad/undesirable that could happen, so why don't we start by looking in those places first. RBT is also an appropriate response to the statistical impossibility of complete testing coverage for any useful software program with more than 2 lines of code. That is, if it will take an infinite amount of time to test something, how about if we narrow it down to just some of the areas that we think might be risky in some way (i.e. popular, critical, complex, and so on).

What else can you do? You have a lot of features to cover in a short amount of time. Well, start by ditching all the test documentation requirements and focus on: what is necessary to establish a minimum level of understanding of what's going on.

Do you really need all those documented steps for every test case? No, you don't. Unintelligent automated systems and robots need step-by-step instructions, humans don't. And most humans don't follow the steps consistently either, so just let that one go. Instead, describe the scope of the testing you want to do using checklists and decision tables. The important things need to be discussed in person to ensure clarity of requirements and information, but everything else should be fine with using point form.

Worried about how you will capture the test results if you are denied the Pass/Fail test status column? Work it out! Figure out a solution that fits your project's (and organisation's) needs. There are a number of far more useful alternatives out there - e.g. application logging, screen captures, note taking, and so on.

If you don't have enough time to complete a project using the same approach you've used in the past, it's time to try something new. Time to think up of new solutions, new processes, and identify/create new tools to help you reach those goals.

The end goal is a high quality product.. or maybe just "good enough" quality depending on your situation. The end goal is not to produce sparkling, publishable test documentation. (If it is, consider changing your title from "tester" to "test biographer")

Don't lose sight of what's important. What will you do with the time you've been given? How will you choose to react to the situation?

What I learned about Testing from a crazy ex-girlfriend

2010-01-21T22:58:00.004-05:00

I was reminiscing with a tester colleague today about how our mothers used to mess with our stuff when we were younger and how it really got on our nerves.

Picture the scene: you have a desk in your room that's plastered with papers and stuff everywhere. And you know precisely where everything is. It's your mess after all.

Enter the mom. She looks around, maybe she's come in to drop off some laundry or to complain about the state of your room or whatever. You aren't around. She starts to tidy. She tidies the papers on your desk and arranges your action figures/books/pencils/Lego/rubber band collection/whatever into a neat arrangement of some kind.

You return. "Ahhhh! Where's my stuff?!?! You changed the order! I can't find anything now! Don't touch my stuff!!"

Your mom, now hurt because she was "only trying to help," vows to never touch your stuff again unless someone's life depends on it. Maybe. We'll see next week.

We chuckled over the memories, but the connection my colleague made was how that ability to memorise tiny details and the placement of certain pieces of information in a messy desk was perhaps already the mark of a good tester. When you look at a computer screen, you take in all the details and it becomes a new mess of our own design that we track in our heads. We notice when details are moved or changed. If we think there might be something different, if we have a hunch, we can use tools to help us verify it or we can check with an oracle of some kind. It's that ability to make a connection, develop a hunch and act on it, that makes a good tester.

So, you're probably wondering "where does the crazy ex-girlfriend come into the story here?" Good question. No, I haven't forgotten. This is the spot.

In university I had .. er, how would I describe it now.. a short-lived relationship with a girl who was definitely the outgoing/extrovert type. One morning, I met her before classes started and didn't meet up with her again until after lunch when we both had a free spot in our schedules. When I saw her after lunch, I did a double-take but I wasn't sure what I was noticing. Something was different but I couldn't put my finger on it.

Then it hit me! Her earrings were different. She had two piercings in each ear and in the morning she had studs and stars (in that order) and in the afternoon they were reversed - stars and studs.

Being the attentive boyfriend, I asked her if her earrings had changed their places or if I was just losing my mind. She looked at me and didn't say anything for a minute. Then she said 'yes', she was bored in one of her morning classes and decided to switch the order. Then she got mad at me. She was upset that I had noticed because she didn't want anyone to notice that she had changed them.

Umm, really? That includes me? So, I don't get any points for noticing you and paying attention? Okay, I don't get this relationship stuff. I think our relationship lasted perhaps another 48 hours. Oh well. C'est la vie.

In retrospect, that was another example where somewhere in my head I had made the connection - something was different even if I didn't know what. Then I began the process of methodically going through the list of possibilities and checking them off one by one - was it her hair? her eyes? makeup? lipstick? top? something she was carrying? a scent? a mark? necklace? wait, did I check the ears? hey, there are 2 thingies there - could they have changed? Spider sense tingling.. better consult the oracle and check if we have a match. Bingo!

Success in finding the difference! Failure at love. =( You win some, you lose some.

I didn't know it at the time but the best relationship was still to come! =) And on that note, sometimes you notice the little things, and sometimes someone has to hit you over the head with a frying pan to tell you to open your eyes and see what's right in front of you!

Ah, but love makes you blind, doesn't it? ;)

What skill does Exploratory Testing require?

2010-01-11T15:47:00.003-05:00

I've just been challenged with a sobering reality.

I've heard the term "Exploratory Testing" used many times over the last few years by developers and testers at various gatherings. I've practiced it myself for over 6 years in various black-box system testing efforts. When training new testers on my team, I provided them with foundational concepts in context, risk, scientific method, test techniques and communication. Then over the course of several weeks, I reviewed their test sessions and provided feedback during debrief sessions to improve their understanding and application of the various testing skills required to be efficient and effective.

People have told me that I have really high standards, and perhaps I do. To me, testing is a passion and fun, and quality is an ideal achieved through effective communication and interactions with all the stakeholders on a project.

But that's all besides the point. If the question is "what is Exploratory Testing and how do you do it?" then my standards and expectations from team members are irrelevant.

ET is simply an approach to testing software where the tests are not predefined and the focus is on iterative learning, test design and execution (to paraphrase a simplified definition).

How someone learns, how someone designs tests, how someone executes those tests - these things are not defined by any standard; they are applied differently by different people. ET can be performed by anyone. There aren't any requirements for how well or thoroughly someone should perform it.

To quote from the animated movie "Ratatouille": "Anyone can cook. But I realize, only now do I truly understand what he meant. Not everyone can become a great artist, but a great artist can come from anywhere."

So, when I hear the term ET thrown around, I have about as much understanding of how they're testing as I do from a development shop that uses the term "Agile". That is, I don't know anything about what it means to them, how they're applying it, how effective it is, or how it compares to my standards/expectations.

I've been reading articles and research lately comparing ET and Test Case-driven Testing (TCT) approaches, and it never ceases to amaze me how stats and research may be twisted to support everyone's beliefs about which is better than the other.

Developers and Product Managers who have worked with me understand the quality of the information and feedback that my testing style provides. They have said that it is a whole new level of testing feedback they've never seen before. It makes me feel good to hear that - that I'm providing a valuable service.

But when I read one of these comparison articles, I have to assume that the ET applied in the research studies aren't at the same level that I apply it. I have to accept that. I may not like it, but that's the reality. To me, the same research applied would likely show that Agile and Waterfall aren't really all that different in terms of produced output. Sigh.

Am I missing something?

It's My Fault

2009-12-12T08:56:00.002-05:00

Many years ago when I was in university, a friend approached me one day to ask if I'd be interested in going skydiving with her. She said she wanted to go but she wanted the company. I went. It was on my list of things "to try at least once before I die" so why not. =)

There was a full day of training for newbies - which included hands on (wearing the jumpsuits, learning the equipment, how the chutes are packed, jumping off picnic tables, etc.), videos, in-class instruction and discussion, and ended with an exam. The written exam was the last thing before everyone suited up for the plane ride up.

I was a bit surprised when 2 of the instructors pulled me aside after the exam to go over my test results. They wanted to discuss my answer to the final question. I think it was something like: "If something goes wrong, whose fault is it?"

I had run out of room trying to fill in a suitable answer. I had wondered why they didn't give much space to write. My answer was something along the lines of: "Well, if the wind blows me off course and I float into power lines and die it's nobody's fault. Or if I land in some marsh and get eaten by alligators I don't blame the alligators." and so on .. until I ran out of room at the end of the page. (There are no alligators in this part of Canada, by the way.. unless I land in a zoo.)

Looking back at my response now, I was doing what a good tester might do and thought of how many different ways something might "go wrong". But I missed the point. My instructors patiently kept rephrasing the question to see if they could get a different answer from me.

One of them blurted out: "Paul, is someone holding a gun to your head and asking you to jump out of the plane?" To which I replied "no." "Okay, so who is making you do this?" "No one," I replied. Wrong answer again. I still didn't get what they were trying to say.

Finally they explained to me that *I* was the only one making myself do anything here. If something goes wrong, it's simply my fault. Then they told me to cross out what I had written and write in big letters: "IT'S MY FAULT."

They explained that the whole exam didn't matter - none of the answers mattered except for this one question. If I didn't answer this question correctly then I wouldn't be allowed to jump. It was like signing the waiver.

I wrote what I needed to. I went up in a perfectly good plane and then I jumped out. It was an amazing experience and we all had the same silly grins on our faces when we met up with each other on the ground again.

I learned an important lesson that day. I didn't realise that I deflected responsibility for my own actions and decisions. It wasn't intentional, it was just how I thought about things in such abstract ways. When there are billions of different possible events that may occur from any given moment, why would I even consider taking responsibility for one of those outcomes if things go bad?

Well, it's easy. You make the choice yourself. Assuming no one is holding a gun to your head or threatening to harm your loved ones, then the choice is yours. When you make a choice you own the responsibility for the outcome. It's your fault if something goes wrong. It's your fault if something goes right.

Working in the software industry all these years, I have witnessed many times when people don't take responsibility for the decisions they make and how they choose to act - at all levels within the organisation. I have seen employees who whine about not getting the raise or praise that they think they deserve but they don't put in the care/attention/effort required. I have seen managers look for scapegoats when projects/things don't go as planned. I have witnessed the irrational, childish backtracking of senior management who refuse to admit that they ever did anything wrong or made a wrong decision.

Why is it so hard for people to take responsibility for the choices they make?

We make choices every day. Everyone does. Sometimes we have help making them, sometimes we don't. Some are big choices, others not so much.

What about the choices people make when they are at work? How they act? Or rather, how they choose to act towards others?

No one's putting a gun to your head and telling you to "test!" (Well, there was that one scene in the movie Swordfish that was quite entertaining, but that's the movies for you.)

Testing is not easy. Software Development is not easy. If you think it is, it's likely because you don't understand the problem.

As testers, we should aim to provide as much required information about the product/system/service as possible so that the stakeholders can make informed, timely choices. That is, we help others make big/important choices - hopefully good choices. Good information too late doesn't help either. (Note: what the stakeholders choose to do with that information is a different matter. Sometimes all we can do is just provide the information. How they use it is up to them. Good choices are not automatic, even when you do have the information you need in time.)

If the choice you make turns out not to be the right choice (i.e. you get an undesired outcome), then admit it, learn from it, and try not to make the same mistake the next time.

That's the rub - that process right there. If there is no admission of responsibility for the choice you make, do you learn from it? I don't think so. When I see someone deflect responsibility or look for a scapegoat, they aren't learning anything from the situation and will likely make the same mistakes the next time.

I have seen good leaders admit mistakes/poor choices to their departments. Admit that they've learned from it and will try something new or different to try and get a better outcome. That's the kind of leader I like working with/for. Someone who learns. Someone who grows. Someone willing to admit shortcomings and knows how to leverage the strengths others to help make better choices in the future.

No one is omniscient, so why hide your mistakes? There's some risk in every choice you make. That's life. Don't whine about it or blame others if things don't go your way. Take responsibility, learn from it and try not to make the same mistake again.

When was the last time you said "it's my fault"?

Reflections on AYE

2009-11-17T22:29:00.003-05:00

I had the privilege to attend the Amplifying Your Effectiveness (AYE) conference this year. Finally! I've mentioned that conference in several of my presentations and talks over the years, so I was pleased to finally be able to make it out to Phoenix, AZ, this year for the event.

There isn't much I'm going to say about the conference at this time. Browse the conference web site to get an idea of the kinds of sessions and discussions that happen there. Reading about it doesn't do it justice.

Everyone I know who has attended an AYE conference in the past has told me how wonderful it was and how much I would enjoy it. They were right. Even though I was told to expect it, and I hoped it would live up to those expectations, I felt a kind of relief and happiness in knowing that I wasn't disappointed.

In my experience, I've noticed that testers tend to start out only interested in developing their technical skills (e.g. programming/scripting language, automation tools, databases, etc.) - if they show any interest at all in professional development related to their jobs. If you take your career and profession seriously, there will come a time when you realise that the technical skills aren't as important as communication and people skills.

Why does learning happen in that order? Does it make sense? Build People skills upon/after your Technical skills? Shouldn't we start with a good base in communication, understanding and relationship-building, and then work to develop technical skills and expertise afterwards?

Should we focus more effort on teaching teenagers in High School how to understand and communicate effectively with each other to prepare them for developing good working relationships in adulthood? Why is it that the High School/teenage experience tends to do the opposite?

I've seen my children play nicely with other kids, even strange/unknown children in the playground quite nicely. So when do adults forget how to be nice to each other? To play nicely or fairly with others? When do they forget how to show respect and trust, and act with integrity and honesty towards others?

At AYE, those values were apparent. I saw kindness, respect, trust and honesty in abundance. It was overwhelming at times. I wasn't expecting that. I felt a sense of instant community at the conference.

Learning happened. Sharing happened. Discussions and conferring happened. It was fun.

It was everything that I hoped how a group of adults would act. I wish that was a more common occurrence. I wonder what we could accomplish if more people acted that way.

The Price of Clarity

2009-11-03T23:49:00.004-05:00

I went to a "Startup Drinks" event tonight and met some interesting entrepreneurs with lots of great ideas. One thing I discovered (about myself) is that I'm going to need to get on Twitter in the near future. ;-)

One person mentioned a tweet that he posted yesterday from the "Monday Morning Memo" about advertising. The tweet was about the topic "Why Most Ads Don't Work" (by Roy Williams, author of the book "The Wizard of Ads") and is summed up in the nine secret words: "The Risk of Insult is the Price of Clarity."

This was an interesting phrase. I instantly thought about the feedback that we, as testers, give about the products we test. You know what I'm talking about.. the Ugly Baby Syndrome. Sometimes we have to be the bearer of bad news. Hopefully, we can find a nice way to say it, but ultimately, I believe that the risk of insult is the price of clarity.

Thinking about my motto "ubi dubium, ibi opportunitas" (where there is doubt, there is opportunity), we often exploit the vague, unclear areas of products and applications.. because there will usually be lots of bugs there. The bugs we report are about more than just little typos and minor UI issues, though. If we do our jobs well, bug reports can be the catalyst to help bring clarity to features/requirements/implementations and consensus among the stakeholders about the app/system under test.

When I test, I report every bug. It's all information. I can't judge when something will be worthwhile (to report) and when something won't. You won't recognise that your baby is ugly because it has thousands of little things wrong with it if you don't report the thousands of little things. It's not often that you'll test a system where you find a "nose" in the completely wrong place, or an "ear" where an "eye" should be (to continue the 'baby' analogy). Those moments are easy - you *have* to risk the insult or risk completely failing to meet the customers' (and other stakeholders') needs.

If you test, and report bugs, in a way so as to "not offend" (i.e. by watering down the message, not testing certain features too hard, or choosing not to report certain bugs) are you really providing a helpful service?

What are you willing to risk to ensure clarity on your projects? What do you think?

Grey is made up of Black and White

2009-10-06T22:58:00.003-04:00

There's an idea that has been on the back of my mind for a while now.

In certain testing (and development) circles, one hears about "good enough" software. I get it. Nothing's perfect and no software can ever be completely bug free, so the best you can hope for is to have caught and fixed the most important bugs (to your stakeholders) before you ship. Or something like that. I'm paraphrasing.

The thing that irks some people is that "good enough" is not carved in stone. It has a human element to the decision-making process that some bean counters might prefer to do without.

I think I've come to appreciate that definition over time because I've had the benefit of seeing many projects of different types for different companies go out over the last decade. With experience, you develop a sense of when things are in your comfort zone (i.e. acceptable risk) compared with when they are not (i.e. this risk gives me the willies - ship it if you dare but I'm going to duck and cover).

That part I'm okay with. That's not what's troubling me. The thing I'm wondering about is where do you start to teach someone this?

So, when I first started to Test, I started by working on some automation tests, and then moved to developing some written test cases and test plans. Over time, I came to appreciate that the brain tends to move quicker than paper and that there are some testing practices out there that can help you test more and provide you with more good information and feedback in the time given on a project. You can choose to spend that precious quality time with paperwork or do more testing. The choice is yours.

Fast forward a few years. I've chosen to teach Exploratory Testing to people who are new to the Testing field. It's an interesting and challenging approach and I love how it gets you to think bigger than you've thought before.

So, you decide to test something. Did the test "Pass"? Did it "Fail"? How do you know? Are you sure? Can you check? ... and so on...

This leads to discussion of Oracles - i.e. a principle or mechanism by which you can tell something is a bug. Funny thing about Oracles - they're like siblings. Each one is unique, they're similar/related in some ways but different in others, and they all think they're the most important and want to be in front.

As a tester, you need to decide *which* is the most relevant Oracle to help you decide if something is a bug or not.

For example, I often explain to new testers that the app might match the specification when you compare them, so is that a Pass? The simple answer is "yes"... assuming the only thing you care about is comparing the application to a particular claim. So what's the problem?

Well, the problem I have is the assumption that the spec, the claim, is correct in the first place. What if they both match each other but I don't care because it goes against what I *expect* as a user? That is, when you use computers for a while you develop a general understanding of how many apps work in similar ways for similar functions. And then along comes some hot-shot spec-writer who thinks they're going to be awesome and develop something completely new and different for ... oh, I dunno, the 'Print' function.

"Okay," you say to the spec-writer, "you might do that.. but then again, reinventing something that everyone has a certain idea of how it should work might lead to mistakes, frustration and user/operator errors that are not so good."

So, back to the tester, the app matches the spec, but they're both wrong I say because neither matches my expectations. Where's the Pass? Where's the Fail? It's not so easy anymore. In fact, you might say there are shades of grey when it comes to determining Pass/Fail - especially if you find yourself in the situation where there are 3 or more applicable Oracles. (eek!)

That's where I think my line of thinking went wrong a while back. It's not about shades of grey, it's clearly about determining which sibling, which Oracle, is the most applicable here. You *have* to pick one. There has to be a binary, logical Pass/Fail answer to the question "Is there a problem here?" One Oracle needs to edge out more than the others. You may not be able to make that decision on your own, and that's okay. But someone, or some team of people, needs to reach a consensus about whether or not something is a bug according to some oracle that they care about most in the given situation.

At the end of the day, it doesn't matter if your brain is firing on all cylinders or none, a "black and white" decision needs to be made about whether or not something is a bug. Computers are logical things. I think it makes sense that we need to apply logic in this situation too.

But hold on here, where does this fit in with the whole "good enough" software thing I started talking about a few minutes ago? That doesn't sound very logical. In fact, it sounds kind of like the opposite, doesn't it?

Why, yes, it does sound illogical, and yet there is logic in it too. The idea came to me when standing in a Wendy's looking at a poster while waiting to order lunch. The poster had a picture made up of smaller photos.. a photo mosaic it's called. (Aside: Google search turns up lots of cool info on these.)

And then it hit me. Aha!

Determining "good enough" is like finally recognising the picture of the quality of the release when you spend every day looking at all of the smaller 'quality' photos. These dozens, hundreds, or thousands of 'black and white' decisions that are made over the course of a project (i.e. the bugs you find) paint a picture. Some people pay attention to these, while others do not (to their detriment).

Good development teams come up with Release criteria that make sense and often contain human elements in the final go/no-go release decision. Different project team members may have different pieces of the picture. As a Test Lead/Manager, you bring information about the tests that have been run, the risks that were covered, and the kinds of bugs that were found, reported and fixed (and the ones left outstanding). Sometimes you might even have other metrics that you feel are important in helping you recognise "good enough" when you see it.

I like the idea of a photo mosaic as an analogy for good enough software. You may never get to see the complete picture, but you might not need to either. When the picture is too fuzzy, then you should recognise that you might not be collecting all the information you need to help make a good timely decision.

As a tester, you need to be concerned with all the logical decisions in the Pass/Fail criteria world that helps you identify bugs on a daily basis. But don't get attached to any one particular bug, because in the grand scheme it may not be as important as you think it is. (It *might* be, but be aware that it might *not* be too.)

As a Test Lead or Manager, you need to step back from all the separate little details and ask yourself what patterns you recognise in the complete body of [testing] work that has been performed? One might even use the cliché "can you recognise the forest for the trees?" but I like my photo mosaic analogy better. ;)

I like how the colour grey is made up of mixing the colours black and white together. And similarly, good test management decisions, which may seem 'fuzzy' to some, are also based on the understanding of all the little logical decisions made by the testers on the team.

These are different skills (between tester and test lead). I haven't been specifically teaching testers to see the big picture and patterns in work beyond their own. Where does that fit in? When? I don't know that I'm comfortable doing that just yet.

I think I got the teaching testing part down, so I guess the next logical step is looking into the skills required to manage good test teams.

Any thoughts or recommendations? Is it just something that you learn by trial and error, by osmosis, working on different projects for different companies? (Nahh, that's how I learned.. it takes too long.)

This, I think, is going to be the next big need for training in the testing field. Let's say for a moment that you've trained testers in good/different testing practices. So how do you train the managers or leads to manage in new, intelligent ways to complement the productive, agile testing effort? Part of it will be in developing new test management tools, but part of it, clearly, will be in skills development and education too. I just don't see that available right now.

I'll have to keep my eyes open the next time I'm out for lunch. ;)

Think inside the Box

2009-10-03T11:50:00.003-04:00

Last weekend I presented at the Toronto Workshop on Software Testing (TWST). The topic for the workshop this year was "Coaching, Mentoring and Training Software Testers" and I decided to present a process that I had developed with my team to help manage a peer review process. For us, that's a coaching opportunity, so I thought it was relevant to the theme. There was some excitement and hoopla over the presentation that had me baffled for a bit though.

My presentation had some specific content and charts and things which I know were "new" because, well, we developed them. That was the point of sharing it with colleagues. The thing that had me stumped/confused was at a higher level than that though.

You see, I didn't think there was anything particularly new about the whole idea. Our team follows a process that, as someone reminded me, was initially developed 10 years ago for some specific company's needs. That initial presentation is on the web for all to see and I came across it a long time ago.

The high-level process description fits in with our current company's needs and has been working well for us for several years now. In all the articles and descriptions that I've found online, there was just one piece that wasn't described too well.. okay, at all. To clarify the specific example here, I'm talking about Session-Based Test Management (SBTM) - a test management framework to help manage and measure your Exploratory Testing (ET) effort.

Overall, the process has worked for us in our present company/context, but the "Debrief" step/phase is a bit 'vague' or 'light' in description compared to the other important elements. As it happens, I have a teaching (B.Ed) degree, coupled with my 10+ years of experience, so I feel I am able to improvise this step fairly well.

But one thing always bugged me about that step - it didn't fit in with the overall flow of the rest of the framework. That is, according the the framework 'activity hierarchy' you're either testing or your not. Well... I don't think that it's that clear cut. What about the Debrief step? Is that Testing? Or is it clearly "not"? I don't think it's either, but closer to the 'testing' side if I had to pick one.

Okay, so if I look at it as closer to the testing side, then does it fit in with the overall framework? Actually, no. It's an oddball. The catch here is that the basic SBTM framework helps you manage and measure the ET effort, but nothing about the Debrief step. Wait a minute! Hold the phone. Why not?!

So we, as a team, came up with a process for managing and measuring the Debrief step that follows the same basic format as the testing part. When I look at it, it makes sense. It looks more like a complete picture to me now.

For a number of people at the workshop last week, our process seemed like a novel/fresh way of looking at it. I can't see that actually. As far as I can tell, our team is still following the same basic outline and process described over a decade ago. We just filled in some of the blanks in a way that we thought was consistent with the rest of the framework.

When people ask you to "think outside the box" they mean to think in an unconventional way to solve some problem. To me, I looked inside the box and noticed something was incomplete/missing. My solution, as far as I'm concerned, was "thinking inside the box."

Having said that, when I step back from the process to see what the big picture looks like, I feel a bit like Alfred Nobel after inventing dynamite. The reaction from my colleagues at the workshop was kind of like that too - explosive! (It was quite cool actually. I don't recall the last time I've seen such a flurry of excitement and questions in such a short time!)

I don't know what to do with this process right now. I'm changing it to put some "safeties" in place because I recognise the dangers of allowing people to easily generate "metrics" that represent the quality of a tester's work and learning.

Sigh. There is one thing I've gotten from all of this. I have more learning to do. This time, I know it is clearly in the field of Psychology, although I'm not yet sure where to start. I need to understand this Debrief dynamite that we've invented - what it means and how it can be used for good and not for evil in the hands of fools.

I think the new process is working (for me) though, because it allows me to ask new questions that I haven't thought to ask before. For instance, if ET is the simultaneous learning, design and test execution (by one definition), then managing the debrief step helps me to track a tester's learning. Am I really tracking learning though? Am I observing someone's level of interest and attention to detail? How much they care? What are the implications of poor quality work that doesn't improve over time? Should that tester move onto something else? Should I leave them alone if training and reinforcement doesn't help and they do generally "okay" work? ... ?

Aside from the details and all of these interesting new questions, it was just surprising to me that no one has come up with something similar already. I looked inside the box. I filled in the missing piece using a similar-looking piece. I don't see that as being unconventional.

Sometimes you don't have to go out of your way to come up with something fresh. Maybe no one has gotten around to looking at all the corners of the box yet. Maybe someone meant to but got distracted and didn't return. Maybe there's an opportunity waiting. Maybe you are the one to see it.

Have you looked in your box yet? Sometimes a good tester, like an inventor or explorer, is just thorough.

New Ruby SBTM scripts on my web site

2009-07-15T23:15:00.003-04:00

Hi there, for anyone who is interested, I have updated the sbtm-ruby-tools zip file on my web site at: http://www.staqs.com/sbtm/

The current version says it's 1.2, but it's a bit of a mix. I made some updates to some of the scripts last Fall but didn't get around to pushing them onto my site. Just yesterday I ran into an annoying bug when I ran the scripts on a laptop with a newer version of Ruby. It was a one-line change to fix, but this change is worth posting because the bug may cause the 2 most important scripts to *not* run on some systems.

The reason I'm posting this on my blog is to solicit feedback on these scripts. It literally took me 4 hours to create this archive tonight. The reason it took me so long was because the gap between these v1.x files and the v2.x (with the 2.x folder structure that I use on a daily basis) is getting quite large.

You see, last year I completely changed the SBTM 'Sessions' folder structure to help us manage multiple projects simultaneously. To do that, I created new BATCH files and modified ruby scripts to help us work with the different project folders. It's pretty sweet actually. I'm currently managing 3-5 project simultaneously with the SBTM 2.0 framework and it's only a few clicks to switch between any project.

Is this new framework worth sharing? I don't think so. I'm bothered by all the text files and batch scripts (it's so 20th century)... although I have come to really like all the ruby scripts that I have for all my test management needs. From one perspective, it's like a file-based database. On the other hand, it's really a bunch of disjoint text files and command-line scripts (even when you do integrate them with the Windows Explorer).

So, new stuff aside, the help I'm looking for is from someone who is actually using the v1.x SBTM Ruby scripts. Since the gap is so large between my free ones and the ones I use on a daily basis, I'd really like to get some feedback from someone on a completely different system to let me know that the scripts work as advertised.

They should work. They're pretty simple. I'd just like to confirm that.

Any volunteers?

My illumination chamber

2009-07-11T23:20:00.003-04:00

When I was a teenager, I used to bike to work to the downtown of the city. Sometimes a particular coworker/friend would bike home with me most of the way as we both lived in the same general part of the city.

I distinctly remember one night having a philosophical discussion as we pedalled quietly through the dark calm streets. I don't remember the particulars of the discussion anymore, but I recall that that was the first time when I was introduced to the term "Tao." He tried to describe it to me, and said that we were in a state of Tao while we cycled and talked, but that once we began discussing Tao we were no longer in a state of it.

Huh? So, we can be in a state of something until we realise that we're in a state of something and then we're no longer in that state? Is this a Schrödinger's cat kind of problem? Is it like The Game?

It took me a long time, a lot of reading, and personal experiences to begin to understand what my friend tried to explain to me that night all those years ago.

Oddly enough, I had a similar awareness moment just this morning.

First let's rewind about a year or two.. to a presentation I attended on Critical Thinking. Again, I don't remember the particulars of the talk, but I recall that the speaker described the Four Steps of Creativity at one point:

Preparation - Research: Collect information or data
Incubation - Percolation: Milling over collected information
Illumination - Light bulb idea: Aha moment
Implementation - Actual making/creating: Verification

I remember thinking at the time that somehow these steps related to the thinking processes involved when doing Exploratory Testing.

When we "prepare" to test a new feature, we research and discuss that feature. We explore our understanding and ideas and challenge every assumption we have. We *design* tests meant to explore our understanding and observe the results.

There are subtle and simple "aha" moments as we test to help concrete the information we began with. Things change from assumptions to facts, observations, trends and patterns that lead to recommendations.

And yet, things are not always that simple. Sometimes we get stumped when thinking through the problems we face. The answers do not come to us right away. Forcing more information into our heads is not usually the best way to solve a problem, I find. Taking time away from the problem is often what's required... i.e. enter the "Incubation" period.

If you are looking really hard for something and can't find it, sometimes you need to stop looking for it and move onto something else. Often you will find what you are looking for when you don't expect it.

Over the years, I've observed that even though I stop *actively* looking for solutions to a problem, as long as a problem is unsolved, my brain doesn't stop thinking about it. Sometimes, answers come to me in dreams, but I'm not a great sleeper so I don't often remember dreams. More often than not, answers come to me in those moments when I deprive myself of all sensory inputs and let my brain completely relax.

You see, I have a sensory deprivation chamber (of sorts) in my house that I amiably refer to as the Special Hydrogen hydrOxide Wave-particle Emission Room (or SHOWER, for short). I find that this SHOWER helps rejuvenate my energy, and provides a healthy glow to what hair remains on the top of my head.

While I'm in the SHOWER I generally try not to think about any particular problems. It's my only real time to myself all day, so I let the hydrogen hydroxide particles just bounce off my body and lose all sense of everything else.

And then it happens. Many times, under these conditions, ideas just pop into my head! Illumination! Aha moments! I see answers to questions that I had stopped thinking about.

I don't believe that your brain really stops working when you sleep, however, your consciousness needs a break and that's what we get when we sleep. Perhaps a good night's sleep is sufficient incubation period for our minds to mull over the collected information of the previous day(s).

I had noticed previously that I seem to get a lot of interesting ideas when I'm in the SHOWER. However, for some reason, colleagues are not always as happy and eager as I am when I tell them that I was thinking about them in the SHOWER. ;-) I don't know why. It's my illumination chamber.

This morning was slightly different. While in the shower I cheated. I tried to (actively/intentionally) *think* about the answer to a problem I've been working on ... and no new ideas came to me. I think I broke the rules of Tao on this one.

Illumination happens when you let it happen, not when you want it to happen. The other steps in Creativity (and Problem Solving) are a bit more straight-forward, they're done intentionally. This step has a tricky catch to it. Unlike the others, you can't force this one to happen on command.

Darn.

I look forward to my SHOWER session tomorrow. I'll give into the particles and just let them wash all my worries away.. if only for a short time. If illumination happens, that's cool. If not, then that's cool too. I love my showers. Sometimes I'm even moved to sing. =)

Singing is good. At least then I know that I'm using the creative part of my brain and not the analytical side. I've got all day to use my analytical skills. It's good to have time allotted daily to something creative too. There's something very Tao in that balance.

I Found This in My Underwear...

2009-07-06T23:34:00.002-04:00

I purchased a new pair of underwear and small piece of paper fell out of the package. It was a note and it read:

STANFIELD'S
I have personally inspected this garment to be sure it meets the high quality standards that have made Standfields Limited famous. Exclusive fabrics and fine workmanship assure long-wearing comfort and style.
Maureen

Hm. That's interesting. Why is it we don't see notes like this in Software packages?

*Who* would have the courage to put their name as being responsible for assuring the product "meets high quality standards," has "fine workmanship" and that you will have long-term comfort in use?

That's pretty cool. I think the Software Industry still has a long way to go in achieving true customer satisfaction.

The Testing Paradox

2009-05-06T07:54:00.003-04:00

I've seen much debate over the last few years regarding Scripted Testing vs. Exploratory Testing. I think I know the answer as to which approach is the one, correct, true way of doing Testing - it's "Yes."

Before I became a software tester I was a scientist and a High School Science Teacher. I recall a few lessons learned that helped shape how I do testing today.

When teaching (Inorganic) Chemistry, there was an experiment that I performed at the beginning of the year/term. This experiment served a few purposes. The first was to show the students an example of a chemical reaction. You know - the whizz, bang, cool aspect. The second purpose was to stress the importance of reading through the experiment carefully to understand what the important parts were - the warnings, dangers, and critical factors. You see the experiment is designed to fail. That's right, you follow the steps and nothing happens -- the big let-down, and students return to their desks. Denied. *Then* you add some water to dilute the chemicals before discarding the solution and .... POP! WHIZZ! SMOKE! SPARKS! Oooooo, ahhhhhh!

You see, Water is the catalyst required to make the reaction occur. The demonstration experiment is designed to challenge your beliefs that water is fine to dilute any failed experiment. Students need to understand that water is another chemical and that it is not always the best way to deal with a failed experiment. There is no always.

Firefighters understand this when dealing with different kinds of fires. You don't throw water on every type of fire. There are big differences between wood fires, electrical fires and chemical fires. They need to understand the situation before they can effectively deal with it.

When doing experiments in Science, there are times when you can improvise certain variables/steps and times when you clearly can't. So how can you tell the difference? You need to read everything carefully first. You need to understand what you're doing. Only then can you tell the critical steps and components from those that you have some freedom with.

So, what's the tie to testing?

When I first started testing software, many years ago, it was mostly scripted. In fact, I was responsible for an automation suite that tested different kinds of fax modems. The scripts ran through a series of functions in the software apps to test the compatibility of different hardware. Because I knew that, I was able to make variations to the software scripts as long as I knew that the hardware baseline information was still maintained. That is, there were critical functions that we needed to know about, and other, somewhat interesting things that were fine if we knew about them too (and fine if we didn't). I understood the purpose of the tests, so I was able to improvise as long as I didn't negatively affect the bottom line.

Over the last 6 years, I have been doing Exploratory Testing almost exclusively. Does that mean that we do it 100% of the time? No. Why not? Because I can think and tell the difference between when it's good to explore and when it's time to follow a script.

For example, when testing something new, we explore. We don't know anything about it and we don't know how well it meets the customer's needs. Scripting is not the way to go here. When we find problems we log bug reports.

Bug reports are interesting creatures. They are scripts. They indicate the conditions, data, exact steps and expected outcome(s) required to reproduce a very specific problem (or class of problems). Often, if you don't follow these steps as outlined, you will NOT see the unexpected behaviour. It's important that (1) a tester identifies this script as exactly as required and that (2) a programmer follow the steps as exactly as possible so that they don't miss the problem and say "it works on my machine."

When a bug is fixed and returned to our test team for testing, we do a few things. The first is to follow the script and see if the original/exact problem reported is indeed fixed. The second is to now use the bug report as a starting point and explore through the system looking for similar problems. Sometimes we have the time to do that when we first report a bug, sometimes we don't. It depends on what we were thinking/doing/exploring when we first encountered the problem. When a bug comes back to you, though, then that's the centre of your world and there's nothing to keep you from using it to give you additional ideas for finding similar or related problems.

When doing Performance Testing, it is important to understand that it is a controlled experiment.. a scripted test, if you will. You may have done some exploration of the system or risks to identify the particular aspect of the system that you want to observe, but now that you know what you're looking for, you need to come up with a specific plan to control the environment, inputs and steps as best as possible in order to observe and record the desired metrics. This is just good science. Understand your controls and variables. If you don't know what I'm talking about, DON'T DO PERFORMANCE TESTING. Leave it to the professionals.

I have a few stories about incompetent testers looking for glory who took my Performance Test Plans and improvised them in unintended ways or didn't even read the whole thing because they were lazy or thought they knew better .. just to have meaningless results that couldn't be used to infer anything about the system under test. My plans weren't the problem, the testers were.

So how do you do good testing? It starts with your brain. You have to think. You have to read. You have to understand the purpose of the activity you are being asked to perform and the kind of information your stakeholders need to make a good, timely decision.

Sometimes Exploratory Testing is the way to go, sometimes it's not. Note: I recognise that at this point there are still many, many testers out there who don't know how to do ET properly or well. Sigh. That's unfortunate. Those of us who do understand ET have a long way to go to help educate the rest so that we can see a real improvement in the state of the craft of testing.

Ditto for Scripted Testing. If you're going to follow the exact steps (because it is important to do so), then follow the steps and instructions exactly. Can't follow the steps exactly because they are incomplete or no longer relevant? Well, what do you think you should do then?

The point of this note is just to say that no one side is correct. There is no one true, correct, testing approach/method. They both are and they both aren't. It's a paradox. An important one. Practice good science and understand what you're doing before you do it. Improvise only when you know you can. Understand the strengths, weaknesses, and risks of any approach in the given situation and you should do fine.

Exploration in Literature

2009-02-09T00:09:00.002-05:00

While reading a book recently, I came across a quote by T.S. Elliot. I looked up "Little Gidding" and found the whole poem on the internet. Here's a paragraph from the last quartet:

We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.

I tend to avoid deep poetry because I sometimes find it depressing more often than uplifting. This one made me reflective. In life, we explore so that we may gain wisdom and appreciate where we came from - where we started. As children and teenagers we may not appreciate things quite the same until we strike out on our own. "Home" always feels different when you've been away for a while.

Is there a tie here to software testing? Dunno. Haven't thought that far. If in life we explore to gain wisdom, how does that compare to exploring software? Do we appreciate the requirements more if we have done an effective job of test exploration? Do we gain wisdom? If so, about what? The people we're working with? The processes or technologies? The development practices or industry?

Just thought I'd share the find.

Great Example of Exploratory Testing Session notes

2008-10-30T10:23:00.004-04:00

Exploratory Testing (ET) can be done well or it can be done poorly. How do you know how well you're doing ET? I believe you need 2 things: (1) you need to keep notes as you perform your testing, and (2) you need good feedback from a competent, experienced tester.

Taking good notes is not easy. It takes practice. A lot of it. In the process, you will learn something about how you organise your thoughts. You will also learn about biases, assumptions, techniques, critical thinking and communication of important facts.

Let's return to organisation of thoughts for a minute. One analogy that I often use is for a tester to think of your test notes like a Science Report. You know.. one of those reports that you likely had to do in elementary or high school for a science project, assignment or fair. There are basic and important sections/elements in a Science Report, and I believe those elements are also key for good test session notes.

I'm not going go into much detail about the comparisons here (for that, see my thoughts on my web site at http://www.staqs.com/sbtm/), rather I thought I'd share a link to a news story that I just came across on the Time.com web site. One journalist decided to perform a test of the new Google "Mail Goggles" feature.

Read the article here: http://www.time.com/time/business/article/0%2C8599%2C1849897%2C00.html?imw=Y

What do you think?

There are a number of things I like about that article. One of the things that struck me the most was how much I liked the "test notes" in that article. I think it's a great example of test notes that you should keep during an ET session. I've reviewed more session notes over the last 5 years than I can remember. The test notes in this Time article are very good.

The "science report" structure is both amusing and helps to efficiently communicate what the author did. I doubt the journalist knows anything about ET or session-based test management. The flow and clarity of the report is very good because the journalist has a skill that I often find lacking in the "poor" ET session reports I've reviewed over the years. What's that skill? A journalist knows how to focus on and communicate the facts. The science report structure helped the author organise her thoughts and communicate the facts efficiently. I liked it. I think it worked. And it was funny too! :)

Which brings me to another important point. Just because you add structure to a report doesn't mean you lose your personality. You can be both clear and funny. Emotion and impressions are important in good notes too because they help raise awareness of the "qualitative" aspects of testing. Too many people put too much value on the "quantitative" aspects of testing.

I like Albert Einstein's quote in regards to this point:

"Not everything that can be counted counts and not everything that counts can be counted."

A good ET session report/test notes should reflect both qualitative and quantitative elements. The science report structure might provide you with a good framework for organising your thoughts. Oh, and if you're looking to improve your technical writing in the Software Testing profession, perhaps you might consider a journalism course.

Testers Create Bugs, Not Bug Reports!

2008-04-25T23:25:00.003-04:00

I've been thinking a lot about this phrase lately: "Your thoughts shape your reality." I think it's from a Buddhist teaching but I'm not sure because I read it as a second-hand quote somewhere that I can't recall right now.

I'm also familiar with the tester's lament: "hey, I didn't put the bug in the code, I just report it."

But what about the phrase: "if a tree falls in the forest and no one is around to hear it, does it make a sound?" That one has always annoyed me. The way I see it, the answer to the question depends mostly on how you define what a sound is. If a sound requires a listener to be real, then, no, no sound was made.

What about a bug? If there is a bug in the software but no one encounters it, is there really a bug in the software?

Now, I report bugs. A lot of bugs. I employ many creative and imaginative tricks to find them. They love me. They come to me even when I don't expect it. One of the tricks I use is to change my perspective on how I look at software. I also exploit vague areas, or areas of doubt within the development team, development processes or methods of communication. For example, if it looks like there might be some vague statement in a specification that could be confusing or interpreted in different ways, I'm pretty confident there will be lots of bugs there.

But hold on. What if I don't go looking for the bugs? What if I don't report them and no customer ever sees the problem or complains about it? Does the bug really exist?

Or does the mere act of identifying this discrepancy between intention and execution create the entity that I call a bug? Did I create it or was it already there?

Scenario: A spec-writer wrote a document that was interpreted in some way by one or more programmers. According to their interpretation, the implementation (the software) is working as intended. They check that their code measures up to their understanding (unit testing?) and then build the software into a deliverable product.

Then along comes a tester. A tester looks at it in a completely different way and suddenly the software is full of bugs. But wait a minute! The bugs weren't there until someone looked and said they were there.

This reminds me of Schrödinger's cat. Is the cat alive? Is it dead? Is it both alive and dead, or neither? You have to look to find out, but once you've looked you've changed everything. What a psych trip!

So, what if finding bugs falls into the same category? In that case, the mere act of observing has changed the reality of what's in the box. By looking inside and asking the question, we have shaped the reality by our thoughts.

Doesn't that mean that we have therefore created the bugs that we report? The programmers didn't put the bugs there. They interpreted the specs and design documents according to their thoughts and shaped the software according to their reality. They built no bugs.

The bugs were called into existence when we, the testers, said they were there with our mind tricks and clever tools.

The bug report, therefore, is just a record of how we have reshaped reality with our thoughts in a way that is different from how reality looked before we started testing. We created those bugs in the bug reports.

One might argue, then, that if you make the bug reports go away you make the bugs go away. If no one (customer) sees the bug, does it really exist?

That's a change in perspective from what I was taught about testing. I don't report bugs. I create them!

I Am The Arrow

2008-03-17T20:10:00.003-04:00

One of my favourite television series was called Babylon 5. Good stuff. From time to time I recall particularly well-written scenes and moments in that series that seem to apply well to some situation happening in my life. That happened again today.

I had a meeting today that changed everything. I've spent the last decade searching for answers that apparently weren't there. Today I found a well, no a sea, of good answers. It's going to take me a while to assimilate everything and find a way to communicate it back to people. It's not the right piece of information, it's the right way of looking at things.

I said to someone this afternoon that this is the first time in 9 years that I've really learned something NEW. Something that makes me think. Something that makes me think in a new way.

For the first time in almost a decade, the phrase that comes to mind is "I am the arrow." I know where I'm going and what I need to do.

The context for that phrase is from an episode in Babylon 5 - Season 3, episode 61: "War Without End (Part Two)" There was this scene where Sinclair is absolutely certain he knows what is going to happen (in his future which is really everyone else's past - Time paradox stuff) and what he needs to do. That scene has always haunted me. It's not every day that you know with such certainty exactly what you need to do.

In all fairness, that has happened to me once before. It was the day that I started the relationship with my wife -- that was almost 20 years ago. I had no doubt. I was absolutely certain.

Unfortunately, moments like that don't come around very often. I needed to put a little note here to mark the occasion that it happened to me again today. It feels good for a change - to know the solution to a decade-long problem.

Happy St. Patrick's Day! =)

Ross Collard's Tea-Test (Technique)

2008-03-03T21:54:00.003-05:00

I was reviewing the test notes for one of my testers today when I came across an interesting note. I decided to look up the problem report in the bug tracking system to read the details. The bug report said that if you go to particular new page in a web app (currently in dev't) and enter some information, wait 35 minutes, and then press a button to continue, you get an error. If you wait less than 30 minutes there is no error, and if you wait over 60 minutes then the application will timeout (as expected), so you have to wait just the right amount of time.

Suddenly I started laughing out loud saying: "Hey, it's the Tea Test!"

I called over the tester whose notes I was reviewing to thank him for his good work in finding, isolating and reporting the bug and to tell him this story. You see, there are many kinds of test techniques out there. At the very least, most programmers and testers have heard about BVA and Equivalence Classes, and some testers who take an interest in their profession learn about other techniques as well. In our test team, we can rattle off at least a dozen techniques at any given time and we are usually selecting from among a pool of 30 or more techniques on any given project, but that's not important right now.

What was important at this moment was that the technique that I could apply to the bug found wasn't on any list I had ever seen, but it was a technique that I knew about.

Back in the summer of 2003, I drove down to Virginia to attend a special 5-day "Black-Box Software Testing" workshop offered by both Cem Kaner and James Bach. It was a great opportunity and I didn't want to miss it. Much to my surprise and delight, Ross Collard had also come to attend the course. Ross had taught me my first courses in Test Case Design and Test Management some 5 years prior, and it is information I still use to this day.

One day during the BBST workshop Cem and James asked the participants to name some test techniques. Ross offered two that I hadn't heard before. The first was the "shoe test". That's where you take off your shoe and put it on the keyboard. Then you wait to see how the app handles the non-stop input.

I've seen this technique happen in real life. I've seen someone lean back against their desk while talking to others not realising that they were leaning on the keyboard. Another time, I saw someone put a magazine on their desk which accidentally landed partly on the keyboard and proceeded to cause a beeping noise from the computer as the keyboard input cache filled up.

This can be an interesting technique as you ponder which key on the keyboard to place your 'shoe' for maximum effect in the App Under Test. For example, how well do you think your web app can handle you pressing [F5] to refresh the page non-stop? Can you find a web page that makes several database calls and then try [F5] repeatedly again? Think you can bring down a database server by doing this? [evil grin]

The second technique Ross mentioned was the "tea test". I hadn't heard of that one before, so I asked if the "T-test" was related to the Statistics t-test. He said "no, it wasn't." He said that what he would do here is enter some input in an app, get up from the desk, walk over to make a cup of tea, have the tea, walk back and enter the next input. And he punctuated this by saying that since he doesn't walk very fast these days, this process could take anywhere from 30-40 minutes. Ha! That was funny. I hadn't heard of any test technique like that before.

Fast forward to today. That was exactly the amount of time (30-40 mins) that my tester had to wait for the bug to appear! The tester told me how he had spent over an hour trying to reproduce that bug in a background VMWare session, so that he could continue with other testing while waiting for the right amount of time to pass. We both laughed at how the "Tea test" applied here.

The developer assigned to fix the bug (who sits on the other side of the desk partition from me) must have overheard me telling this story. He piped up and said: "I hate that bug! I have to wait a long time to try and reproduce it!" We laughed harder. =D

This was the first time I've seen Ross' "Tea test" actually work -- i.e. actually find a bug. I thought it was just a joke at the time. I now know there's truth in that technique. It's not that I really doubted Ross, it's just that he's a funny guy and sometimes you can't tell when he's pulling your leg. =)

Ross, you really are the Test Master. Next time I see you, the tea is on me. Cheers!

Something Interesting about Reporting Bugs

2007-11-19T01:20:00.000-05:00

I just happened by this link just now: http://cwe.mitre.org/ for the "Common Weakness Enumeration" (CWE) project.

Here's the blurb:

"International in scope and free for public use, CWE™ provides a unified, measurable set of software weaknesses that will enable more effective discussion, description, selection, and use of software security tools and services that can find these weaknesses in source code."

I'll have to look into this later. Don't know anything about it yet.

To be a Good Tester you must think like a Scientist

2007-08-30T22:10:00.001-04:00

It's funny how many times this particular analogy keeps coming up. The comparison between Testers and Scientists, and the similarity between testing and the Scientific Method.

Most recently it occurred to me while reading one of my son's chapter books. FYI: "chapter" books are short books broken into chapters for kids just beginning to read on their own. Usually for kids ~ 7-8 years old. These aren't Harry Potter books.. most of them barely reach 80 pages.

The book that caught my attention is called "Jigsaw Jones #9: The Case of the Stinky Science Project" by James Preller. The main characters are in Grade 2 and in this particular story their teacher was giving a Science lesson:

"The world is full of mystery. Scientists try to discover the truth. They ask questions. They investigate. They try to learn facts. Scientists do this by using the scientific method."

The teacher then handed out sheets of paper which read:

THE SCIENTIFIC METHOD

1. Identify the problem. What do you want to know?
2. Gather information. What do you already know?
3. Make a prediction. What do you think will happen?
4. Test the prediction. Experiment!
5. Draw a conclusion based on what you learned. Why did the experiment work out the way it did?

Back when I used to teach High School Physics, I recall giving a set of steps very much like this one. I might have used the word "inferences" instead of "conclusion" but otherwise it's a pretty good list.

When you think about testing software, generally you run through the same process and set of questions. If you don't think about each of these questions, then you're probably not doing something right.

For example, here are some questions that come to mind when I think of the Scientific Method applied to testing software:

1. Identify the problem.

What are the risks?
What is the particular feature of interest?
What is it you want/need to 'test' and 'why'?

2. Gather information.

What references are around to tell you how something should work? (e.g. Online Help, manuals, specifications, requirements, standards, etc.)
What inferences can you deduce (or guess) about how something should work? (i.e. based on your experiences testing similar apps, or other parts of the same system, etc.)
What can you determine by asking other people? (e.g. customers, programmers, subject-matter experts, etc.)

3. Make a prediction.

Design your tests.
What is your hypothesis?
What are the expected results?
Think about any assumptions or biases that might influence what you observe. How can you compensate for these?

4. Test the prediction.

Set up the environment
Execute the tests
Be creative! Make as many observations as you can.
Collect data

5. Draw a conclusion based on what you learned.

Did you observe the expected result? Does this mean the test passed? Are you sure?
If the test didn't turn up the predicted result, does this mean the test failed? Are you sure?
Revise the test design and any assumptions based on what you observe.
Do you have a better understanding of the risks that drove the test in the first place?
Do you have any new questions or ideas of risks as a result of this test?
If you collect a lot of data, summarise it in a chart that can help demonstrate the trend or pattern of interest.
Write a few words to describe what these results mean to you. (You might not have all the information, but don't worry about that. Just say what you think it means.)

In general, I find the Scientific Method to be a very good guideline for both beginners and experienced testers alike. Wikipedia has some entries on the Scientific Method as well as a Portal. I think it's a good read. I'd recommend those pages to anyone serious about becoming a good tester.

If there are things on those pages that you aren't sure about, look them up! You might just learn something new about how to think about things that will help you do your job better.

Happy Learning!

Ubi Dubium, Ibi Occasio (Opportunitas).

2007-08-01T22:03:00.001-04:00

Where there is doubt, there is opportunity.

That's my new motto as a Software Tester. =)

It came to me when I read a comic that I borrowed from a friend recently. You see, I'm a fan of the writer J.M. Straczynski, so my friend told me about a comic that JMS had written a few years ago called "Supreme Power" (Max Comics). If you've ever read a comic, you'll know that each issue or story usually has a separate title. Issue #8 of Supreme Power has the episode title "Ubi Dubium, Ibi Libertas" which he translates for you on the last page as: "Where there is doubt, there is freedom."

That title made sense in the context of the story, however I couldn't stop thinking about that phrase for several days afterwards. There was something about it that I liked, and yet it didn't completely fit for what I feel I do as a tester.

When there is doubt, I go to work. I have fun. I explore. I discuss and test. Most of software testing is about working in the space between the vagueness of specs or requirements and their ever-changing interpretation into working software code. So really, doubt is everywhere. Doubt is the whole thing! Where there's doubt, there's opportunity.

Over the years, I have often contemplated different analogies and ways of describing what I do as a software tester so that I could explain it to others who don't really understand the role. (For some reason, if you're not a programmer and if you're not doing Support or Sales, then most people don't really understand what else is there.)

So now I feel that I'm really close to a good analogy. Doubt is the space where I work and play in. I'm a Doubt Management Specialist or Facilitator, if you will. Someone writes up some specifications based upon what they think the customer wants to the best of their knowledge and understanding. (There's doubt.) Someone else interprets those requirements and transforms them into mathematical algorithms that perform some function on a computer. (There's more doubt. Is that like Doubt-squared? ;-) )

Enter the Software Tester, the go-between. We see the doubt in the specs and come up with some ideas (i.e. tests) to explore the meanings and possible interpretations. We see the doubt in the software as features are incomplete, don't perform as expected, are insecure in some way, unusable or not robust enough according to our interpretations and experiences as users of the technology.

If there was no doubt in the whole process, I don't think we'd have anything to do. We'd totally be out of jobs. Maybe we could be Project Managers or Programmers I suppose. ;-)

So you see, where there is doubt, there is opportunity for us. Opportunity to explore, to test, to ask questions, to find bugs, to strengthen understanding, to clarify, to add value.

If you don't see the doubt, I don't believe you are adding any value. At the end of the day, I believe the best testers are the ones who add value by reducing the doubt in the development project.

It's another way of looking at the problem. I kind of like it. What do you think?

Bonitatem et disciplinam et scientiam doce me (et gaudium ostende me)

2007-07-09T20:05:00.000-04:00

There are a few things that I recall from my high school days -- the school motto was one of them: "teach me goodness, discipline and knowledge." It served me well as a guide over the years but I always felt like there was something missing in that motto alone.

Years later, when I was working as a high school teacher, the three main things that I focussed on getting across to my students were: Attitudes, Skills and Knowledge. That was pretty similar to my old high school motto, so it was easy for me to remember and apply. I thought those three things were a good guide for my classes, but again I felt like there was something a little too dry about it that I couldn't quite put my finger on.

One day, during a contract placement teaching one of my favourite topics - Physics - I got some interesting advice from a seasoned teacher on my teaching approach. He said that I did well in the class but that he lived by a different motto: "any class that goes by without some humour and laughter is an hour wasted of the students' lives." And of course he said it with a smile. =)

It's interesting because I have always had this joie de vivre that everyone who gets to know me notices. I like to smile. I love to make jokes. I love to stop and smell the roses and listen to the wind and the trees. I have a penchant for puns and when the going gets tough, I get silly. =)

Yes, I suppose there are times when seriousness is called for. However, for the most part, life is too wonderous and entertaining not to be silly and enjoy and to make other people smile and lighten up too.

Up until that moment when I got that feedback from that teacher, I had been working on a different model: one where you are serious in your professional life and save the humour for your personal life.

After that moment, I ditched the old "professional = serious" model and decided for myself that every hour worth living was worth enjoying, regardless of whether I was at work or at play.

Needless to say, after that advice I began to share my passion for the subjects I taught in the funnest ways I could think of while still adhering to my teaching goals and objectives. A decade later, if you have ever attended one of my QA or Software Testing workshops, you would also know that it is always with a certain amount of levity that I share my experiences and knowledge. After all, if you can't laugh at yourself, who can you laugh at, right?

At work I'm the same. Always the professional, and always with the smile on my face or joke to add. Don't get me wrong.. I'm not a clown every hour, but not a day goes by that I don't try to do or say something to add a little levity or have some fun.

Here's where it gets interesting. 8-)

Not everyone sees it the same way. Some people are simply "no fun at all." In fact, I even worked with someone once with the nickname "the Director of No Fun."

(Aside: Hmm, I wonder if it is a particular trait of middle-management to have the least amount of a sense of humour? Maybe it's because they realise that their jobs are the most expendible, so they tend to try to look and act important all the time?)

Oh, but it's not just people at work.. there are many people out there in the world who have simply forgotten what it is like to have fun when interacting with other human beings. To these people, adding a smile to a thought or reply suddenly takes on a different meaning. Now you are being facetious or patronizing or sarcastic. Your words are not only not meant to be taken seriously, but you are insulting as well. Nice.

Interesting. Personally, I find that these words tell me more about the listener than they do about the speaker. In fact, it tells me that the listener is not really trying to listen at all because they are too busy imposing their own negativity on the world around them thereby tainting everything that they hear.

I feel very sad for these people. It really is a shame that negativity appears to be more predominant in modern society than positivity. Acting and speaking in ways that emphasize goodness, joy and happiness generally makes you the odd one out.

(Aside: It's fun to meet other 'positive' people at parties and elsewhere. It really causes an exponential increase in positive and silly energy that just invigorates and excites me. I don't think that most people can handle having more than one of us around at any one time. =D )

Which brings me back to my old alma mater's motto: "Bonitatem et disciplinam et scientiam doce me." That's a good start, however, I'm also adding: "et gaudium ostende me." My Latin might be a bit rusty, but I'm pretty sure that translates as "and show me joy/happiness/fun."

It's not enough to teach and be taught the right attitude, the right information or the right skills. I want to see the fun that other people have in their work and I want others to see the fun that I have in mine.

If you happen to see me with a smile when you meet me, please feel free to smile back. =)

The Three Physical Requirements of a Good Software Tester

2007-05-28T23:11:00.000-04:00

There are three physical elements that I find a good software tester must have:

Good working senses
Brain - ability to think
Heart - someone who cares

Your senses (sight, sound, touch, etc.) give you the information that you need to process.

Your head helps you process the information and form them into ideas and models to work with.

Your heart gives the information meaning. Someone with heart is someone who cares about others and about the quality of their work. Without this, you are little more than a computer.

Interestingly enough, there are some people with reasonably good-working senses who are still unable to see. I think that it is likely an impediment from their head or their heart that prevents them from seeing. Can this be fixed? Perhaps -- if the person genuinely wants to see. Not everyone wants to see.

The problem is no longer a mechanical one but rather a psychological one. That's tricky.

All written communication is fundamentally flawed. It tries to capture some of the above 3 elements, but usually fails to really grasp the element of 'heart'. (And Poetry is likely the exact opposite - more heart than anything else.) I don't believe that any useful communication can take place without being physically present with the other person. There are many things said and understood between people that may be poorly or incorrectly inferred through written communication alone.

A software tester is an Information specialist. Do you have what it takes to be really good at it? Do you really care?

Decompression

2007-05-27T22:33:00.000-04:00

Sometimes after a long day at work, I need some time to settle my thoughts from the day before I can resume normal life. I call this my 'decompression' time - similar to the decompression period required for deep-sea ocean divers who have to sit in a decompression chamber before they can return to our normal Nitrogen-Oxygen atmosphere at sea level.

There are days when I'm just so wound up about things at work - someone or something that preoccupied a lot of active think-time - that even by the time I return home I'm still not able to absorb new information until my brain can settle. Sometimes when someone tells me something during this period, it just never makes it into my long-term memory.

I read a phrase in a book recently that I think captures this perfectly. The book is Dragonquest by Anne McCaffrey, and this phrase is used to describe what one of the leaders is thinking when he just returns from a battle after a particularly intense day:

"A man needed a few minutes to digest chaos and restore order to his thinking before he plunged into more confusions."

Sometimes, I find that this decompression time can take up to an hour after I leave work. And that's from a job that I like working at! (I've worked at some places where my thoughts never settle and the stresses prevent me from sleeping. You know you need to move onto someplace else when that happens.)

I call this a short-term decompression period -- which is based on the events of a particular day.

Sometimes when the stress and activity level is really high and prolonged at work for several weeks, I notice a different kind of conditioned response happen. My head and body become accustomed to a high-level of thinking and action so that when the stress is removed, my body feels almost lost and weightless for up to several days afterwards. I think of this as a long-term decompression period.

Here's an example. Last summer, we had a particularly difficult software release because the deadline was tight, there was a lot of complex functionality to cover and there was clearly insufficient people to help us reach the target. As a result, my colleague and I put in over 200 overtime hours in the course of a few months in order to help make our targets. When the release finally shipped, I found myself wandering around my office space for a few days looking for a fire to put out, a problem to solve, a meeting to go to, or some late-night that I needed to come in for. But there wasn't anything. So I just had this doe-eyed, deer-in-the-headlights look about me for a few days until my brain and body could become re-accustomed to a normal workload and workday schedule again.

Sometimes you may not even be aware that your brain is not ready to process new information until the other chaos had been digested. When I come home from work after a particularly hard day, I try to avoid doing anything that will require long-term memory for a short while. My kids are a great distraction for me.. I can get lost in their world for a short time to help me clear my head so that I can process new information and new chaos from my second career - home life. =)

Why do we Test?

2007-05-18T10:01:00.000-04:00

As with other good questions, this one can be answered with another question:

Why does someone want you to test?

Do you know the answer to that question? If you aren't sure, start by asking the person you are working for.

Can Practitioners write Academic Quality papers?

2007-02-21T13:20:00.000-05:00

I've thought about the idea of the AST Journal for some time now. In principle, I really like the idea. One thing that I've worried/wondered about though is the idea of writing a paper that stands up to academic scrutiny (or pretty close to it anyway).

Today, I happened to notice the following Quote Of The Day in the weekly StickyLetter (from StickyMinds.com):

"Science is supposedly the method by which we stand on the shoulders of those who came before us. In computer science, we all are standing on each other's feet."
~ Dr. Gerald J. Popek

At first it made me laugh, then it made me wonder about what it would take to write a good paper. I've been reading some good articles and papers on Software Testing lately and if there's one thing I know it's that I don't have the time to do all the required research to produce a really good paper.

I, like perhaps many experienced testers, learned my craft by doing. I picked up ideas here and there over the last 15 years (a conference presentation here, an email thread there, a passing conversation with a colleague or manager, and so on), that when applied I started to notice the patterns of what works and what doesn't. I then began to build up some notion of the importance of contextualisation in my successes and failures.

I've written small snippets of perhaps good ideas and thoughts in the past, and I've communicated some other good ideas during workshops and presentations, and I like the idea of sharing knowledge. One thing I don't think I have the time to do, though, is go through all the previously published material to see who thought of what first. It may be important for research historians, but I don't really believe that I have the time or resources available to me to really do a proper job of it. It almost seems weird or absurd to me from one perspective too.. who thought of it first? "Well, I thought of idea 'foo' all on my own. I can list you all of the experiences, conditions and factors that led to these inferences and the outcomes of the applications of these thoughts. I didn't read the idea anywhere, so how can I attribute the idea to someone else?"

So where would I begin to look in the published literature to reference who actually came up with the same idea or portions of it before me? Or worse. What if to do a really good job of it, you need to reference articles and papers from across various disciplines ( i.e. computer science, psychology, education, engineering, philosophy, sociology, etc.)? That is, what if the profession of Software Testing is really just the centre of a whirlwind of various professions and disciplines all combining into patterns that we each interpret in different ways to successfully complete the tasks before us? How would you know that you've referenced enough people or ideas to do a proper job in your paper?

I just feel so overwhelmed at the prospect sometimes. It's not writer's block.. it's the thought that spending a day or two articulating a few good ideas and the contexts in which they seemed to be successful for me might require weeks of research to support in good academic fashion. And even then, I know I would likely miss some other good referenceable point or idea or person.

Is it possible to do a good job writing a good paper and still have a day job? Perhaps. Is it possible to do a good job writing a good paper and still have a life? I don't think I could. Maybe we all are standing around on each others feet sometimes. So how do we get past this? How do we turn all this information into knowledge so that we can have some progress? How do we help the next generation so that they don't have to reinvent all of the same ideas that we've had to discover on our own over the last three decades?

Learning not to be the best to win friends

2007-02-06T19:45:00.000-05:00

When I was young, I developed a habit that I don't really know how I got started on. I'm sure some Shrink could probably extract it from my memories through some quality sofa time, but I'm not really interested in that. The problem is that the habit stuck and it seriously affected how I interacted with others.. but I didn't notice right away. I would have had to have been paying attention to notice. Unfortunately, I only developed that observation skill much later in life.

So here's the thing: I was a perfectionist. I know what you're thinking - every tester says that. Well, I was pretty methodical in my approach and fairly obsessive about it too, right from an early age. If I didn't get perfect on a Math test, I practiced the questions I got wrong until I always got them correct. If there was skill that I didn't excel at, I practiced until I got them down - from video games to languages, from sports to mechanics, from music to cooking ... and so on and on. I was only limited by the resources available to me and as a result I became quite good at a lot of things and a real Information Investigator too. By the time I was twelve I could navigate any and all of the libraries in the city of Toronto (it's a big city). By the time I was in my teens (in the mid-1980's), I learned to navigate the newly developing electronic information systems using a modem hooked up to my Atari 8-bit computer. The advent of the Internet in the 90's was an absolute dream for me! I was an information junkie and I loved to learn. It became a habit: learn something, learn as much about it as possible, get really good at it.

One day I discovered that there's a more efficient way of describing such a person: a know-it-all. Ouch. Kind of harsh. What was worse was discovering that my girlfriends were intimidated by how much I knew and how well I did at school. That was stupid, right? I mean learning was like a video game for me. I did it to see how much I could cram into my brain before I got a brain cramp or ran out of information sources. I didn't do it to feel superior or make anyone else feel inferior. It was just who I was and what I did.

Well that sucked. I wanted people to like me. So I did what came naturally - I pretended to be dumb and intentionally did worse in school. That became my new game - to try and intentionally not do my best, not be perfect. The problem was that my habit was still there, so I had to keep finding ways of hiding what I knew. My teachers didn't like this new turn of events, of course, but I had some good friends and good times so I didn't really care what the teachers thought.

That worked okay for a while I guess. I only thought about dumbing down when I cared about making a good impression. When you're a teenager, that's not really all the time. ;-)

Fast-forward a decade. After graduating from University (finally! they had to force me out because I didn't want to leave :-) ), I discovered a similar but more distressing dynamic in people interactions in the workplace. So here's a question - don't you want to work with people who are good at what they do? Don't you want to be the best at what you do? Apparently, I discovered, a lot of people kind of don't really care all that much about work. It's just a day job that pays the bills for them. And that whole perfectionist thing I've got going on not only intimidates some people, but it also (apparently) makes them look and feel bad.

Well that sucks. Again.

Oh ya, and it gets worse. If you don't kiss the butts of the people in upper management, lie to them and praise them and make them look better than you, you're not likely going to advance within your organisation. It was at this point that I discovered another trait that I didn't know I had - integrity. Basically, anyone who wanted me to kiss their butts could just go ahead and kiss mine. :-b

I wasn't about to dumb myself down in the workplace. Certainly not for some management position amongst all the other self-gratifying, ego-centric, self-praising half-wits. No way. Beware the people you surround yourself with indeed!

Of course, a new lesson I learned was how to balance Integrity and Tact (diplomacy) so as to maintain working relationships. After all, I still wanted a paycheck!

Over the course of several years and several companies, I explored and observed the various nuances of interpersonal dynamics in the workplace with regards to the impact of a product and/or technical expert. Basically, I keep getting better at what I do so I've had to learn how to be good at what I do without making anyone else feel bad about themselves and not sacrificing my integrity when dealing with self-absorbed managerial staff. It's not easy, but I love a challenge! =)

So, why do I bring this up now? Today I had a conversation with my boss - just a regular one-on-one chat that we have every other month at work - and he said something that reminded me of that old habit. He said that other people at work have noticed how much more relaxed and approachable I've been lately.

I know that when I first started at this company, I was a little gung-ho on the whole perfectionist thing (again) but I thought I had turned down the volume on that habit. Some of it is left-over from my Software Quality Assurance days as a "Quality Crusader". I try to remain focussed on Software Testing these days, but it's hard to keep my mouth shut sometimes when people are doing blatantly improvable tasks, and since I have a stake in the success of this company I want to see everyone doing a good job - err, well, the best job they can be doing at the time, that is. Alright? =)

I know I had turned down the volume on that aspect of myself, so I had to think for a minute about why my boss was suddenly remarking on the observation that other people had noticed I was more relaxed around the office.

There was something, an event, that happened last October (2006) that has changed my life forever. It's still a bit too personal to mention here right now, but needless to say it got me thinking about where I was spending most of my time. Last summer I had spent too much overtime at work, and perhaps that got me a bit more stressed than usual -- and was likely the comparison benchmark for my boss' observation.

Since November 2006 my attitude towards work (in general) has shifted again. I kind of don't care about it anymore. I think something inside me snapped. Don't get me wrong, I still love Software Testing and still want to keep getting better at it, it's just that now I think I've finally broken free from the expectation of perfection in others. The transference of my own preferences onto others is a dangerous thing in sneaky and subtle ways that you don't usually see coming.

The reality is that I've got more important things to worry about than what other people should care about. If someone cares about what I think, then that's nice, I'll offer my opinion. Otherwise, do whatever the heck you want so long as it doesn't affect my ability to get my job done.

I've read some good stories and articles over the years, and one day I think I would still like to work in an environment where I could have a mentor that I could learn from. Someone that would expect me to be better than who I am and help me to reach my potential. Somewhere where I could demand the best from my team members and actually get that quality because they care too.

Right now I'm content to work with people I like and who like and support me. That means a lot too.

Never Test Before 4

2007-01-25T12:02:00.000-05:00

Kind of a silly thought, I know, but it keeps coming back.

I work in a small agile development environment. Development works according to 2-week cycles to complete chunks of code. I keep noticing that anything prior to Cycle 4 or 5 is usually incomplete and unstable for testing. The first several cycles are when all the foundational architectural changes are usually happening.

So we can never really test before (cycle) 4. That's fine. I've got these Ruby scripts to keep me busy in the meanwhile. =)

Observation on the Proofreader Effect

2007-01-25T10:24:00.000-05:00

I've been working on some performance test scripts using Ruby (Watir actually) over the last few weeks, and have been happily rewriting the scripts I first wrote a year ago. (Programmers call this activity refactoring.) I've learnt a lot about Ruby and scripting web apps over the last year. One of the biggest helps came when I read Brian Marick's new book "Everyday Scripting with Ruby". Thanks to that book, my performance test scripts are really slick now and look more like a programmer wrote them. But I digress..

The thing that I've been thinking about over the last few days is the problem of testing the scripts that I've written. Any good tester would never trust a programmer to write error-free code, so why should I trust myself to? But then who should test my scripts? Well, there really isn't anyone else around who can right now so I have to do it myself. Is that a problem? I don't think so.

I'm the biggest stakeholder who cares about these scripts working correctly, while my boss is mostly interested in the numbers and my analysis. So I ran the scripts and worked out the kinks one section at a time until I was able to run them straight through several times without error.

Is that good enough testing? Well, I got the coverage and measurements I wanted, so I guess so. The scripts don't have to be perfect, they just need to give me the data I need. So, it's all good.

Right. I completed the analysis for this run and then started to compare the numbers against the benchmark numbers from last year. It wasn't until several hours later that I noticed a typo in the output. Eek!

I'll just sneak back into the code and fix that. No one saw that. I'll just re-run the scripts and make sure the output looks "clean" this time. Great! Looks fine now.

So how did I miss that typo? I thought about this for a while. I think the proof-reader effect is like a FIFO buffer. That is, I don't think I could have seen this bug until I got the other bigger bugs out of the way.. you know, like the ones that prevented the script from completing or collecting the data I needed in the first place.

First in, First out. Get the big ones out of the way and then I can see the smaller ones that were hiding underneath. The typo was always there but I was just temporarily blinded to it because my attention was so focussed on the bigger fish.

So was I unqualified to test my own code? I don't think so. I caught all the bugs I cared about. It just took me a few days to find them. Would a separate tester have found the typo before me? Maybe, maybe not. The FIFO effect only affected *my* ability to see the little things until the bigger ones were out of the way because I was the one who wrote the scripts. A separate tester would have a different perspective and shouldn't be affected by this FIFO/Proofreader Effect in the same way.

We do Exploratory Testing almost exclusively on our products. When I test, I don't see the same effect happening to me. It's just a matter of time until I get to a feature or page and then I hit it like a whirlwind and move on. It's quite cool and effective. Defect finding rate starting to slow down? Switch to another Test Technique - voilà! More bugs. All the Risks addressed? Move on.

I've seen a number of conversations happening on some of the message boards questioning whether or not a programmer is able to test his or her own code. After this recent experience, I think if the desire is there and there is enough time, then yes, she should be able to find all the bugs that matter.

Once again, a separate pair of eyes not constrained by the FIFO effect would likely speed up the process. Nothing we didn't already know. A Tester helps you to find the bugs that matter sooner rather than later. Well, a good one will anyway.

Sometimes "Good Enough" isn't good enough

2007-01-01T22:39:00.000-05:00

I've been a big fan of the idea of "Good Enough" software testing over the last decade. Rather than thinking that the problem of doing good Software Testing is akin to "Digital" technology with it's complete, precise values, I've thought of it more like "Analog" technology with the big dials and reasonable, approximate (and cheaper) signals.

This past week, I've watched my seven year old son play with a new LEGO set that he got for Christmas. It's a neat mechanical lego set that lets him build a motorised helicopter, cool car, or attack crab thingy. (ASIDE: I can't begin to imagine what the Marketing team's conversation was like when they thought up that last one!) I noticed when he completed the helicopter and turned on the motor, that it didn't sound right to me. So I went over and took a close look at his creation. It looked correct. There didn't seem to be any missing pieces, but when he turned it on again, I noticed that not all of the gears turned together consistently. I picked it up and took a really good look at it. Not knowing much about how it was built, I just randomly squeezed together lego pieces that weren't tightly packed together whenever I came across them.

There was one set of lego pieces that had a gap of about a millimetre. When I squeezed them together, it made a (good) snap sound. I asked my son to turn on the motor again and this time it not only sounded correct, but the gears all worked together in perfect synch also. Voila!

I thought about this for a few moments afterwards. Up until then, my son had worked on the premise that if the lego pieces were reasonably attached, that it was "good enough". He didn't need to have a tight fit between every single piece to see the finished product. I mean, it looked like the complete picture of the helicopter in the instruction manual, so what difference would a small gap between a few pieces make?

In this case it made a big difference. If it needs to work like clockwork, then "good enough" is probably not enough.

So what's the tie in to Software Testing? Well, just how scalable is the "Good Enough" approach? For me, it's always been about testing to the most important Risks and using whatever tools and techniques seem appropriate to the situation at hand. It's always seemed kind of foolproof to me.

Maybe my Digital/Analog analogy is a flawed one. I mean, Analog technology has its limits and is not very scalable. Digital technology is more precise and can handle more information. Is there a point when a Digital solution gets so large that it requires an Analog approach again? (I think the answer here is 'yes.')

Is there a time when "good enough" needs to be replaced with a more complete, structured or methodical approach to software testing? I can't think of any situations like that right now, but that doesn't mean there aren't any. That is, I can't think of a time when I wouldn't want to say that good software testing has to strike a balance between the economics, quality and time to market for a product or system. Shipping with bugs is okay if you know that they aren't critical or life-threatening.

So perhaps "good enough" doesn't always apply when we're dealing with real-world objects like lego creations, automobiles, watches, et cetera. I think that it still holds pretty well to the virtual world of software testing. Until someone can give me a good example or two of when "good enough" wouldn't be good enough for testing software, I think I'll chalk this up to another distinction between testing software and testing hardware.

Ruby at Work - Internet access through a Proxy Server

2006-08-29T14:42:00.000-04:00

While reviewing Brian Marick's book "Scripting for Testers" (due out later this year or early next), I discovered that some of the internet access RUBY commands and scripts always failed when I tried them from work. Here's the weird thing - the same commands and scripts worked from my home computer. So what gives?

I found the answer when I googled comp.lang.ruby and tinkered around in IRB from my work computer. It turned out to be the Proxy Server at work. For lack of a better place to store this nugget of knowledge, this seems like a good place for now.

THE PROBLEM:

Open a command prompt on a Windows machine, and launch IRB. Issue the following commands:
----
irb(main):001:0> require 'open-uri'
=> true
irb(main):002:0> url = 'http://www.amazon.com/gp/product/0974514055'
=> "http://www.amazon.com/gp/product/0974514055"
irb(main):003:0> page = open(url)
Errno::EBADF: Bad file descriptor - connect(2)
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `initialize'
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
from c:/ruby/lib/ruby/1.8/timeout.rb:48:in `timeout'
from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
from c:/ruby/lib/ruby/1.8/net/http.rb:555:in `do_start'
from c:/ruby/lib/ruby/1.8/net/http.rb:544:in `start'
from c:/ruby/lib/ruby/1.8/open-uri.rb:245:in `open_http'
from c:/ruby/lib/ruby/1.8/open-uri.rb:629:in `buffer_open'
from c:/ruby/lib/ruby/1.8/open-uri.rb:167:in `open_loop'
from c:/ruby/lib/ruby/1.8/open-uri.rb:165:in `open_loop'
from c:/ruby/lib/ruby/1.8/open-uri.rb:135:in `open_uri'
from c:/ruby/lib/ruby/1.8/open-uri.rb:531:in `open'
from c:/ruby/lib/ruby/1.8/open-uri.rb:86:in `open'
from (irb):3
----

THE SOLUTION:

You need to add the Proxy Server address (and user & password, if required) to the URL you are trying to access. Here's an example of something that worked for me from my work machine (replacing the relevant proxy address as required, of course):
----
require 'net/http'
host = 'www.amazon.com'
path = '/gp/product/0974514055'

proxy_addr = 'your.proxy.host'
proxy_port = 8080

response = Net::HTTP::Proxy(proxy_addr, proxy_port).get_response(host,
path)
----

=> response.body now returns the page source, just like it does from my home computer. (Response '200')

The second problem I had accessing a remote page was fixed with the following:
----
require 'open-uri'
url = 'http://www.amazon.com/gp/product/0974514055'

proxy_addr = 'http://your.proxy.host:'
proxy_port = 8080

page = open(url, :proxy => (proxy_addr + proxy_port.to_s))

text = page.read; nil
----

Note that the proxy_addr is slightly different here than with the 'net/http' commands above. (You could probably put it all in one variable here.)

If your Proxy server requires Username and Password, then the open command will be longer. (Luckily I don't need to enter that here at my work.)

I don't have any RUBY scripts that do this sort of thing yet, but I was happy to figure out the solution anyway. Nice change of pace. Ruby totally rocks. I have yet to find something that I can't do with it for testing automation.

"The Effects"

2005-11-17T17:19:00.000-05:00

Patterns are funny things. Sometimes it is so easy to jump to silly conclusions because we notice the obvious even if we know there might be valid explanations for them.

Take for example patterns of behaviour. Here are a few things that I see on a regular basis:

The Developer Effect - The software application always works as expected on a Developer's computer.
The IT Effect - Any weird or unexpected computer behaviour (e.g. unable to print) will suddenly start to work correctly when you call an IT Helpdesk person over and they are looking over your shoulder as you retry the steps.
The QA Effect - Any normally working system will suddenly stop working as expected when a Tester starts to look at something.

The "QA Effect" was my most recent addition to this list. A programmer sent me a link to a web site today and when I followed the link from the email message a new browser window appeared with the message: "We're sorry - an error has occurred. It has been logged and will be attended to shortly."

Ha ha. Is that Murphy's Law, or what?

I wonder how many other "Effects" there are in this industry? Write back a Comment to this Blog entry to let me know if you've come across any other immediate patterns. I'd be very interested to hear them.

Cheers!

I Know What I'm Doing!

2005-11-11T22:27:00.000-05:00

This will sound weird but I had a strange realisation the other day. I thought to myself: "I know what I'm doing!"

Okay, this didn't just come up out of the blue and I wasn't practicing some sort of confidence-building exercise. It all started when I happened to notice a bug in the software I was testing and I decided not to report it.

What?! But you're a tester, man! Report the bugs you find!

Ah, well, yes, I plan to. But you see, I was busy at that particular moment thinking about something else, and I didn't want to distract myself by following up on a different thread that might make me lose my train of thought. To justify not reporting it I had to reflect for a moment as to how likely it would be that I would find the bug again. I had absolute confidence that I would find the bug again, so I simply forgot about it and went back to what I was working on.

I think it was later that night that the event started to settle in my mind and it started to make synapse connections with other memories in my brain. One particular memory came to me from my teenage years, some 20 years ago.

I recall one time when I was at a party and was talking to some of my older sister's friends. Coincidentally enough I think the topic of conversation was (creepy, crawly) bugs. There was this one guy who said that he was playing a video game in his home when he saw a millipede scurry across the floor. He said that his reaction was to pause for a moment, smile, and then keep playing his game. Someone asked him why he didn't just drop what he was doing and catch the icky bug right away?

His reply? He said that he knew those bugs and that he would easily find it later and take care of it then. He wasn't worried, so he didn't let it interrupt his game since there was no pressing need to take care of it right away.

I remembered that story for a long time and I always thought that if I ever saw an undesired bug I would simply take care of it then and there. Too many variables in life to let opportunities pass you by.

And so I have. For many years, I have always taken care of the bugs that I've found right away - both the creepy crawly kind, and the software kind.

So then, what happened this time? Why did I let a software bug go and not report it right away?

I realised that I was absolutely confident that I would find that bug again, and since there was no immediate need to take care of it, I let it go for the time being.

Sure enough, I got around to testing that particular area of the application and I had the opportunity to properly focus on the bug and report it with sufficient detail (it was a tricky one). I felt really great about it too.

When you do Exploratory Testing you are faced with the reality that none of your tests are pre-scripted. Exploratory Testing is kind of like what improvisation is to acting: there's no script. It certainly doesn't mean that the Improv people don't know how to act. In fact, it's quite the opposite. They are very talented actors. They just happen to know how to react to the circumstances that arise and can adapt to it with style.

I must confess that I've still been a bit nervous about Exploratory Testing. It doesn't help that I am bombarded with doubt by every other tester in the profession still doing it the traditional way: What if you miss something? How can you be sure of coverage? How do you know when to stop testing? and so on.

Well, I now *know* that I know what I'm doing. If I can totally forget about a bug and then stumble upon it again at a later date, then there must be something to this "improvisation" stuff after all.

Do you feel like the Stanford Bear?

2005-10-26T14:27:00.000-04:00

I was asked just now if I felt like the Stanford Bear. I had to laugh, because sometimes I do.

What's the Stanford Bear, you ask? That's a good question. I'll tell you what I know..

I heard a story recently of an introductory computer class at Stanford University where the Teaching Assistants were so tired of answering the same questions over and over again that they programmed a machine to answer all the commonly asked questions. They added voice-recognition software to it and packaged it up nicely in a large teddy bear that sat on a shelf at one end of the class. Now, when a student had a problem or question, the TA's wouldn't help the student unless they knew that they had gone to the bear first. This relieved much of the workload from the TA's so that they could focus on the problems that really required their attention.

I'll admit that I don't know how much truth there is in this story. I spent some time scanning the internet and I couldn't find any pages to corroborate the tale. I'd like to think that it's true though.

True or not, it got me to wondering as to whether or not you could program something to give out canned answers on Software Testing to commonly asked questions.

I don't think that you could do it so that it would recognise every possible context and provide you with an appropriate response for every situation. (While a cool idea, let's be realistic here!) However, within a particular context - say revolving around the particular testing procedures for a particular class or industry, I'd think this might be do-able.

Better yet, I think it might be cool to program a Testing Bear that's hooked up to a central database somewhere. That way, people could program the Bear to answer all the common questions within their context, but then people all over the world would have access to the same questions and answers.

Now that might be something! Hmm.

Software Testing is more like Fencing

2005-09-03T11:55:00.000-04:00

I got into a discussion with my colleague at work the other day about Martial Arts and Fencing. He asked me if they were similar and I said that based on my experiences with both that I didn't think so.

Martial Arts is about the harmony between mind and body. Different Martial Arts stress different skills and techniques but there is always the spiritual side, and with your mind focussed and at peace you can do almost anything. I remember some of my first Kung Fu lessons as a teenager when I was taught that my first reaction to a fight situation should always be to find a quick way out of it. Fighting should only be the last resort.

Fencing, on the other hand, is like both a Science and an Art. I remember learning the blocks (the "parries"), the lunges, and all the other technical maneuvers, but I also remember that when I was in an actual duel my mind almost forgot everything and I went with my instincts. You see, an actual duel is *much* faster than you can imagine. You almost don't have time to think or you're toast. With lots and lots of practice the technical moves become part of your instincts and then you become better at fencing.

So what about Software Testing? Despite the fact that some people amicably refer to it as a Martial Art, I don't see the spiritual side of it. When I see a bad application my first reaction isn't to run away. ;-) I dive in and go with my instincts. And project schedules are almost always faster than you'd like them to be.

With lots and lots of practice, my testing techniques have become part of my gut reaction now, so I usually make my own luck when it comes to dealing with bad programs.

Perhaps Quality Assurance is more like Martial Arts. You need a belief system to keep you motivated when you do *real* QA. You need to believe that you can eliminate the defects, the inefficiencies and the variances from the development process by applying certain methods and models.

I don't believe that. Not for a moment. We could stand here and talk about how we're going to do something or we could just do it and tweak things as we go along. Me? Let's dive in and do the best we can because time is running out. =)

En garde!

A bit about Software Testing Certifications

2005-09-01T18:25:00.000-04:00

Recently I was asked about my "certification". Here, more or less, is the response that I gave. I thought I'd note it here in case I was asked again.

Back in June 2002 I received my certification through the American Society for Quality (ASQ) as a "Software Quality Engineer". While I may use the term "Certified Software Quality Engineer" in the US, due to Canadian laws, I may only refer to myself as an ASQ-CSQE here in Canada.

Here is a link where you can find out more information about this certification: http://www.asq.org/certification/software-quality-engineer/

Of particular interest to you may be the Requirements for this certification and the Body of Knowledge covered by it. (FYI: Note that "Software Testing" is merely a subset of section "VI - Software Verification and Validation" to the "QA" folk. Many people use "QA" and "Testing" interchangeably, but to me they are quite different.)

This ASQ certification covers many different aspects of Software Quality throughout a Software organisation (i.e. from defining Quality in the Business Goals, through to the Engineering methods, Requirements, Development, Testing, and Configuration Management). The exam itself was interesting in that many of the questions were designed in such a way as to demonstrate application of knowledge. (So people without experience who can memorise well may not pass the exam.) I spent about 4 years preparing for the ASQ exam by applying the principles, techniques and advice from the reading list materials into my regular work duties.

Why did I get this certification? It was applicable to what I was doing at the time.

Over the last few years I have changed my focus from Quality Assurance to Software Testing.

Most of my energy is currently on the Software Testing profession and I am an active member of the Association for Software Testing (AST). You can find out more about the AST at their web site: http://www.associationforsoftwaretesting.com/

I find it interesting that my ASQ involvement has provided me with a good knowledge of Quality-related issues that are, in general, quite difficult and complex to apply and manage on a day-to-day basis in a Software organisation. By contrast, the AST aims to provide us with the knowledge and support we need to be effective Software Testing practitioners. (i.e. the former is laden with bureaucracy and paperwork while the latter aims to make us useful employees.)

There are several Software Testing certifications available to the public, but none of them are particularly meaningful right now. The AST does not yet have any certification program, but we have discussed the development of a certification that would be based on demonstration of skill and not just one's ability to memorise the terms and definitions from an assigned reading list (which is what all of the current Testing certifications are based upon).

In the past, when I have interviewed candidates for Tester positions, I have used "Testing certifications" as an indicator that someone has more than a passing interest in this profession. Unfortunately, I can confirm to you that the certification itself has absolutely *no correlation* to testing ability whatsoever.

Too bad.

Team Leads need to be Potion Masters

2005-08-29T08:44:00.000-04:00

Okay, maybe it's because I just read the latest Harry Potter book, but in discussing Test Team leadership recently with someone, I used this as an analogy.

I'm always surprised to meet people who think that the stages of team development (i.e. "Forming, Storming, Norming, and Performing") are some kind of kiddie joke or else they may not have heard of them at all. That's too bad. I believe that anyone who's serious about team leadership/management should take these stages very seriously.

If you are not prepared to recognise the moments when your team members start to pick each other apart or are challenging and questioning everything you do (in a bad way), then your team will never grow into a constructive, cohesive whole. As a team lead, you need to detect the poisons and provide the antidotes.

Most importantly of all, if you at least notice the poisons, but if you don't know what to do then ASK FOR HELP!! I'm tired of meeting people with EGOs so large that it prevents them from asking for help from others. ("Pride" is one of the Seven Deadly Sins, isn't it?) If you had seen someone fall but didn't know what to do you would call an emergency number (like "911"), wouldn't you? After all, that would be the right thing to do.

So why is it different when it comes to team management? If you notice that your team is falling apart and you don't know what to do to make it better, ask for help.

The biggest problem that I've noticed in this industry is not the managers who don't ask for help though - it's the managers who don't even notice that there's a problem. I've seen many teams just completely fall apart because the team members picked each other apart by getting on each others' nerves right in front of an oblivious team lead/manager.

Perhaps there's only a certain type of person who makes a good manager. Perhaps there's something in the MBTI that correlates with the traits of a good leader or manager. Perhaps I'll have a chance to look into this some more in the near future. Something to think about.

Shego: The first thing that every villain needs is...

2005-05-18T21:42:00.000-04:00

"The basic tools of the trade."

There's a kids show that I watch with my sons every now and then called Kim Possible. It's a smart, unoffending show that empowers youth to face their fears and foes in a creative and humourous way.

There was one episode where a villain (Sr. Senior, Jr.) was getting trained by another more-experienced villain (Shego, pronounced "She Go"). Shego started her lesson by asking Junior what's the first thing that every villain needs. The response got me to thinking about how I might answer that question if I were in a similar situation.

What are the "basic tools of the trade" for a Software Tester?

I think that in our case, we need to:

Arm/prepare ourselves with Testing Techniques that uncover bugs
Be able to effectively communicate our findings and testing information
Be able to identify Risks.

What else? I think those 3 are the most basic, and they don't cover things like attitude, motivation, or the ability to deal with stressful situations. You'd think that a Villain wouldn't ask for training in those, and I don't think a Tester would either.

I'd be curious to know if there are any additional "basic tools of the trade" that I've left out from my short list above. I tend to lean towards the "softer skills" during interviews and when assessing potential testers, but I'm not sure which one(s) are worth identifying as "Basic Tools". Many of the ones that I've been exploring are not "basic" -- they are more "intermediate" or "advanced."

Shego: "The second thing that every villain needs is a Plan."

Good advice. =)

Welcome to the Knowledge Age

2005-05-17T21:03:00.000-04:00

Okay, maybe I'm splitting some hairs between "Information" and "Knowledge", but I'm choosing to define "information" as simply the summary or statement of the raw data itself (i.e. observed facts without opinion), and "knowledge" as providing you with the ability to make good use of the available data.

You can see some of this beginning to take shape right now. Blogs, like the one you are reading right now, are a key factor in helping us to interpret fact through opinion and personal experiences.

How do you know which car to purchase? Pick up the Lemonade Guide. Make an informed decision before you buy!

How do you know which company to work for? Research them first. There are many places to look.

How do you know which TV or Stereo to buy? DON'T start by asking the sales staff at your local electronics store. Start by asking your friends or family their preferences and experiences. If you can, check out some of web forums where they talk about their purchases.

If we're going to survive in this world, we're going to have to know how to make the right decisions. To do that, we're not only going to need access to timely information, but we're also going to need access to the risks involved in certain situations.

Want to travel abroad? Don't just look at the pretty brochures! Take into account the weather conditions, health risks and the political conditions. If you don't, you might get lucky and have a good time. If you're not so lucky, you might end up in a foreign hospital, jail or even dead.

Do you still want to let a coin flip make that decision for you?

We need more than just Information. We need to know how to process that information to help us make good, informed decisions. When we can do that, we will truly have "Progress".

I do believe that we are on the cusp of a new "Knowledge Age". The signs are appearing around us every day. Unfortunately, the capitalists and war mongers are also out there helping to drag us back into the Dark Ages. We can get through these challenging times if we're strong and help each other out. Not everything needs to be about who's right, who's wrong, and how you can make a fast buck. (Note that I don't think that making money to provide for yourself, your family, and/or to enjoy life is a bad thing. Making money solely to achieve power, prestige, or at the expense of other life is wrong in everybody's books.)

I've read enough Science Fiction stories to know that there are probably an infinite number of ways that we can make the Earth a really bad place to live. It's my planet too, so I'd like to think that we can make the best of it if we just work together.

Okay, so what's the connection to Software Testing? If we apply this idea to what we do, a good Software Tester provides information. Good, credible information is unbiased and reproducible (whenever possible). That doesn't excuse the Software Tester from their responsibility to provide additional opinions, alternatives, and risks that may not be apparent simply by looking at the testing information that we provide.

Testers are a part of the development team, and therefore a part of the overall decision-making process. If you don't care enough to play an active role in the decisions affecting product releases and the future of the company you work for, then you waive your right to complain if things don't turn out the way you think they should.

I've no patience for whiners. They shouldn't work in the Software Industry at all. Neither should scapegoat-hunters. Next time a problem arises at work, how will you react? What will your contribution be?

Welcome to the End of the Information Age

2005-05-16T21:33:00.000-04:00

I remember having some discussions about 15 years ago about the Internet and what it was useful for. At the time I was still in University, and the Internet was primarily used to link together educational institutions and government agencies.

We didn't have the Windows operating systems that are so common (rampant?) today, so most of the applications we had were text/character based. Applications like "gopher" and "telnet" were used to navigate the various libraries and information sites on the 'net, and programs like "Pine" and "Elm" were popular for checking email from our Unix accounts.

For the most part, if you were in college or university and you needed to do research on some topic, the Internet was likely to provide you some limited access to information databases online. (Real libraries provided the really useful information though.. no shortcuts.) Other than that, email was very useful for trying to get a hold of authors and researchers anywhere in the world. (Cool!) That was it though. If you weren't into Newsgroups, then you probably had a life and never gave the Internet a second thought.

By the early 1990's the Mozilla web browser appeared on the scene (running on the X-Windows interface on our Unix accounts) and changed the way we viewed information forever. The gopher sites were slowly replaced by more flexible and visually-appealing web sites. By the mid-90's, commercial businesses were jumping on the HTML-bandwagon and starting to make use of the Internet to appeal to a slightly different market - the technologically-savvy! Email accounts were gaining in popularity, as graduating students started to expect these accounts as a standard part of their professional lives too.

Sometime before anyone really started to worry about the "Year 2000" (Y2K) problem, the Internet was already becoming the "Information Superhighway" and the World Wide Web was becoming the standard interface to that information.

With any advancement in Communication we usually have Progress. When information travels faster, we can work with it sooner. Just think about what the Telegraph, Telephone, Radio and Television did for the world. Now we have the Internet, and the world is a smaller place once more. We can watch News as it happens from anywhere in the world. We can research the latest papers, articles, journals, and theories. We can shop and start a business within a global market. We can be entertained, post our thoughts, send messages instantly to others, and more! The Internet is truly a rich, multi-media opportunity to express our ideas and share information with everyone on the planet.

There's just one problem. There's a lot of crap out there too. I don't just mean the obscene material that some people of little intelligence or low moral standards think we should know about... I'm talking about the poorly-researched, incomplete, often-misleading, assumption-driven, over-hyped, or just plain incorrect information that is passed off as "fact". If you don't know how to tell the good information from bad, then just remember this simple rule:

ALWAYS IDENTIFY THE REFERENCES AND CHECK THE SOURCES!

"Houston, we have a problem!"

So, now we find ourselves floundering in a sea of information, and we can't always tell the useful from the useless. Welcome to the end of the Information Age.

Just knowing the facts (true or otherwise) isn't going to help you. You have to know what to do with that information. You have to know what the assumptions were that the information was based upon. You have to know when the information you are reading is incomplete. You have to know how to separate the useful from the useless. You have to know how to act upon receiving the correct information. In essence, you have to know how to process the information into useful action.

Welcome to the beginning of the Knowledge Age.

Some incomplete thoughts...

2005-05-15T23:14:00.000-04:00

There are a series of related ideas that I want to discuss, but I don't think I'll have the time to properly describe them here. I'll make a quick attempt to put some of my unpolished ideas down in upcoming blogs, but I'll have to save the clarity and completeness factors for a later date and/or forum.

As with all ideas, discussion helps me bring out the best arguments. If you have any ideas or comments to share, please do so with the functionality available through this blog mechanism.

Thanks. Cheers!

Use your BRAIN!

2005-04-27T22:27:00.000-04:00

I bumped into an old colleague last week and found out that he's expecting his first child in a few months. He asked me if I had any advice for the delivery "experience".

I had to think about it for a minute, but then I recalled a lesson I learned before my first child was born.. useful advice that worked for me at the time, and advice that I continue to apply in other areas as well today.

Since you don't really know what will happen, you kind of need advice that will prepare you for just about any situation that might arise. In that case, use your BRAIN.

BRAIN is an acronym - a mnemonic for some heuristic advice. Here's what it stands for:

B = Be aware - of everything going on around you.
R = Know the Risks (Don't know them? Ask!)
A = What are the Alternatives?
I = What do your instincts tell you?
N = What if you say "No"?

There are times when you will find yourself alone in a situation where an important decision needs to be made. Don't forget to use your BRAIN.

Sometimes your heart tells you one thing and your head tells you something else. This handy little mnemonic should help you find a reasonable compromise between the two. It's worked for me.

New Tester's Hardware Research Tool -- eBay

2005-04-25T21:57:00.000-04:00

Ever come across a piece of hardware in a lab (like a modem or network hub) and you don't know which power supply goes with it? If you can't find the spec sheet for it, you might be out of luck. Or are you?

I picked up an inkjet printer out of a junk pile last week. It looked like it was in good shape but it had no power supply. I checked the manufacturer's web site for some technical spec sheets and I found them, but they were useless when it came to telling me the exact specs of the power supply.

So I googled the web and I found a few hits on eBay. Why not? It's worth the look. Turns out it was my lucky day! Not only did I find the exact power supply for sale on eBay, but some really thorough sellers include close-up photographs of the power supplies too. Spankin! =)

I printed up a copy of the photographs and rummaged through a bunch of boxes of miscellaneous mice, power supplies, cables and other small parts. Bingo! I found the matching power supply. I brought it home, plugged it in and, presto, my kids now have a new inkjet printer all their own. (Of course, there's the added cost of the overly expensive inkjet cartridges, but considering the printer was free, I'll overlook that for now.)

Marketing Research - How to make your meetings work better

2005-04-20T22:13:00.000-04:00

I got a phone call yesterday. I was contacted by a Marketing Research group that periodically holds Focus Groups on particular subjects in my town. I signed up with them a few years ago and I get a call from them maybe a few times a year, so it's not really a bother. (I like to contribute to research and data whenever I can.)

There are usually the pre-screen questions though to make sure that you are a part of their target group. Turns out that I didn't fit the profile they were looking for this time around (I'm a Generation-X, and they're looking for a Gen-Y), but I asked them the details of the Focus Group anyway. It was to be a 2-hour discussion forum on some topic and it pays $50.00.

Wow. That's cool. $50.00 for sitting around in a meeting room and talking for a few hours? I can do that. (Too bad I didn't qualify this time.. better luck next time.)

Hey, wait a minute!

How come when I sit in a Project status meeting for 2 hours at work I don't get $50 spending cash?!

How much more efficient do you think these status meetings would run if the Meeting Chairperson had to dole out cash for each attendee for each hour? I'm pretty sure that these meetings would run a lot quicker and be far more productive! ;-)

Maybe there's a lesson in there somewhere.

Interlude

2005-04-08T00:03:00.000-04:00

I haven't forgotten about this blog.. I've just been crazy busy at work for the last month. We had a major software release at the end of March, followed shortly afterwards by a planned "hotfix" release to patch the last remaining things that we couldn't finish in time for the deadline. So between long work hours and spending my remaining waking hours with family, I've let this chill for a while.

That reminds me.. I think some of my friends called about 4 weeks ago.. I should probably get back to them.. maybe get out, go see a movie. I need some air.

I've got lots to write about here, so I'll probably be adding several entries over the next few weeks. One thing that I should mention is that one of my hard drives has also failed on my home computer. Just happens to be the one that has all my email on it. (That kinda sucks.) I'll be [carefully] taking the computer apart this weekend to see if I can recover the data before I replace the drive.

I actually unplugged the drive several weeks ago, so if I can go almost a month without retrieving my email, I think I can wait a few more days before I start writing entries. ;-)

Cheers! Write soon.

The Paradoxes of Software Development

2005-02-17T21:13:00.000-05:00

When you've been working in Software Development for a while, you eventually wise up to the three important factors that drive any project: Time, Money, and Scope. (Sometimes people replace that third option with "Quality", but I think that the three of them together make up the three sides of a "Quality" triangle.)

They are certainly the three things that drive any Project Manager, although you sometimes hear them talk about it using different words:

Schedules, Milestones and Deadlines
Budgets, Resources, People, Equipment, Tools, Training, etc.
Features, Requirements, Work Breakdown Structure

When you start any project, you always desire lots of each of these. In reality, you are constrained in each. You can't have all the time in the world. You can't have all the people, tools, training, or equipment that you'd like. And someone is always trying to sneak in more features, but there are limits to what you can put into any software release if you ever want to get it out the door and make some money from it.

Consultants use these three factors (Time, Cost, and Scope) during contract negotiations. The experienced consultant will be able to take any two fixed factors and give you a practical estimate for the third.

So what are the paradoxes? These three factors are, but not for the reasons listed above so far.

Time

You never have enough time. Never. Live with it.

It's more than that though. When you are at work, your time is always interrupted. This is unfortunately unavoidable. It is the nature of Software Development that we miraculously create a complex masterpiece through a process of continuous interruption.

Interruptions come in different forms. Everybody expects to take bathroom and mealtime breaks. But what about the interruptions by email? What about when someone calls you on the telephone or shows up at your desk? Bam! There goes your train of thought!

Of course, it's not polite to yell at people to go away and leave you alone. (Although it may be very tempting with some people.) The reality is that, as a complex human activity, software development requires that a lot of people get the information they need to do their part of the project. Everybody has a part in creating information, so everyone has a responsibility to make sure that the information they know is passed onto the people who need it.

You could be on the verge of solving a complex problem that has eluded you for days, when all of a sudden a Project Manager shows up at your desk to ask you your opinion on the risks involved in making a change somewhere. Or it could be an impromptu meeting. Or a technical writer asking you to review some documentation that you have some familiarity with.

I knew a programmer once who said that he was far more productive when he worked at home for two reasons: (1) he didn't have to get up and leave the building whenever he needed to smoke (he could just smoke at his desk at home), and (2) no interruptions.

The problem of course is that when he does get back into the office, he will eventually be inundated with questions about the work that he did -- in order to pass that information onto the next person who needs it. He also eventually needed to collaborate with other programmers to understand the integration of his work with theirs.

Try as you might, you cannot make software in isolation forever. You may need to get away from others for a while to help you focus on a problem or goal, but you cannot stay away for long. You just can't.

Money

This is a funny one. Not everyone is immediately aware of the importance of the money available for a project until they try to get something and realise they can't. For example, it would be really cool if we could get this "super analyzer metrics-reporting database-query problem-solving automatic-tester all-in-one XML-friendly scriptable-API user-friendly reliable coffee-making appointment-remembering time-saving dry-cleaning fully-customizable easy-to-use tool" ... until they see the price tag.

Or worse, you realise that you can get some of the tools you need cheaply, but you don't have the time to spend figuring out how it can be integrated into your project or environment. Is there any way that we can hire a contract person (e.g. a high school or university student) to figure it out for us so that we can reap the benefits of the tool without actually spending any of that time figuring it out for ourselves?

What? No money in the budget for extra hires? But how can we meet the deadlines if we don't have the help/tools/equipment/training/whatever that we (think we) need?!

Start-up companies generally have a harder time with budgets and available finances than larger, more established, global organisations, but big companies have their share of money problems too.

The funny thing about needing money is that often you get it - but at a price! You will end up "owing" someone, in one way or another. Nothing comes free, and money is at the top of the list.

Many years ago, I had a terrible experience working for one company when we had a person leave our test team. I approached the Director of Development about when I could start hiring her replacement, but then he started giving me the run-around, talking about the "return on the value" of our test team, and that perhaps the funds allocated to the test team salary budget could be better reallocated to hire another programmer instead. After all, we would certainly be more productive with another programmer and one less tester, wouldn't we?

I quit a few months later. I have been extremely happy with my decision and never regretted it. That company no longer exists since it lost all of its customers a short time after I left.

Money is a tricky thing. It should never be placed in the hands of fools, because they will always do foolish things with it.

It's about more than just the Economics of the business though. "Time to Market" can have a BIG impact on the price of the Shares in your company (if it is a Publicly traded organisation). You don't meet the release deadline for a particular Quarter >> share prices drop >> No bonuses this year! In fact.. there might even be "restructuring" in the near future.

To do a great job, you may need more people, tools, training or equipment, but those things all cost money. You may or may not have money "in the budget" for these things. Getting those things may come at the expense of a lower salary or absence of financial bonuses. Not having those things may mean that you will miss the important deadline and cause your business to miss its Sales goals... which in turn mean lower salaries, no financial bonuses, or even layoffs.

Money. How do you want it?

Scope

It all starts innocently enough with the customer. This "customer" may take different forms depending on the industry you are working in.

You start with a list of features. These features are deemed to be the critical success indicators for the software if it is to be bought and used.

Software Development is a complex and iterative process whereby some really smart people take that feature list and break them down into hardware and software components from a particular technology. Suddenly that list of ten or so items, is now broken down into hundreds of smaller pieces that all need to work together in harmony to make up the features that the customers want.

Think of it this way: In Ancient Egypt, a Pharaoh once said: "Build me a Pyramid! I want a tomb of magnificent proportions that will leave a legacy for eons to come!" The Architects designed it, and then the Engineers built it.

Have you ever looked closely at a Pyramid? It's made up of thousands of stone cubes! Where did these stones come from? How did they get there? How did they put them together to make the finished product? What about the workers? How did they feed and clean up after thousands of slaves?

Thinking about some of these questions will begin to give you an idea of the orchestra at work in a software company to produce a product that the customer needs and will pay for.

But then a funny thing happens. Someone looks at the software. It might be a Product Manager, a Software Tester, or even the customer. And they discover something that isn't quite right. Sure, perhaps the big, important features are there, but the little details between them might be clunky, incomplete, or too complicated to make sense.

Uh oh. We need to make changes to the software.

And we keep "testing". And we keep discovering more "little things". And before you know it, that "critical list of ten features" that was broken down into hundreds of components is now tied together with thousands of little details!

It takes a keen mind to be able to keep all of these details in perspective of "the big picture". That is, does it make sense to make a particular change now, or should we put it off until the next release or version of the software?

I've met many testers who believe that every bug they find should be fixed right away. That's a foolish and narrow-minded perspective, in my opinion. I believe that if the problem is worth reporting, it should be worth fixing (whenever feasible), but each problem needs to be assessed for the risks involved and the priority or relevance for that particular release.

For example, I once tested a client-server application and found a bug that crashed both the client and server at the same time. (This is unusual because usually you only find bugs that affect one or the other.) In the Project Status Update meeting that week, I argued that we should NOT bother fixing the bug for that release. That is, even though it was a serious crash, I knew that no customer had ever reported the problem in the five years that the software was on the market. This led me to conclude that no one was in fact actually using that feature. So, I could only conclude that it would be a waste of time to fix a feature that no one was actually using.

Being able to keep things in perspective like this is what helps you to avoid one of the strongest Failure-factors in the business: "Scope Creep." Changing Requirements.

It's just a game sometimes. I've seen it many times. It goes something like this..

You ask the customer what they need to do their jobs. They tell you and you make the list. You come up with the requirements and double-check them with the customer to make sure it looks right. You make the software, work out the bugs, and release it to the customer for approval.

Now that the customer has the working software in front of them, the bar has been raised. "Oh well, it almost does what we need. It just needs these many more features, and then it will be perfect."

The smile falls from your face, but you go back and work to get these new features into the product. When you bring the software back to the customer, the bar has been raised again! "Well, it's almost perfect, but it still needs these features for it to actually work for us with the way we do things."

Just for the record, these are the kinds of customers that cause software companies to go out of business. More importantly, the Project Managers who don't recognise this situation and allow these scenarios to play out late in the development cycle are really the ones who cause the software companies to fail.

"Scope Creep" or "Feature Creep" simply refers to the changing of the requirements while the product is still in development. What may seem like a straightforward list of requirements at the outset may suddenly become conflicting requirements by the time you expect to release the software.

Changes may be subtle and sneak up on you in inconspicuous ways, or they may just hit you in the face like a closing door when you are looking the wrong way.

Getting a handle on Scope requires you to understand: (1) the Development Models (Life Cycles) used to provide the guidance in the way you build the software, (2) the customer and their appropriateness to help you achieve your desired results, and (3) the Requirements gathering process and the iterative nature required to truly flesh out the useful features from all the unnecessary "Bells and Whistles."

Scope. Feature list. You think you know what you need to do. You will be wrong every time. Software Development is a process whereby you usually engage on a journey with nothing more than a general sense of direction but no clear understanding of the final destination. That's the way it is.

Go to the ocean and learn to surf. Things will become clearer.

New Year's Revolution

2005-01-05T23:17:00.000-05:00

I'm at an interesting crossroad in my career. Allow me to think out loud and recap a few things..

On the one hand, I have a formal Science education. I love Science. I got into it because of Physics. I loved the problem solving. However, I didn't see myself as a Lab rat for the rest of my life, despite the coolness factor.

Enter my second love: Teaching. I went to Teacher's College and became a High School Science teacher. Interesting thing about teaching is that you actually learn more about a subject than you ever did as a student. There's something about looking at a particular concept from many different perspectives to try and explain it to a variety of students with a range of skills and interests. I learned even more about Science and I loved it even more. However, the reality was that I needed a break from school so the timing was a bit off. I can always return to it later if I choose.

Somewhere in between all this, a hobby of mine (fooling around with computers, and software in particular) turned into something of a recurring job. I did the programmer thing for a few years. Did my time in Tech Support for almost a year, and then I fell into QA almost by accident. I liked Testing. It was somewhere in between the programmers and the customers, but there was something I couldn't quite put my finger on that kept me interested in it.

When I chose to not teach anymore, I decided to go back to QA & Software Testing. I didn't know what the difference was (between "QA" and "Testing") but I knew that: I enjoyed doing it, that I could do it, and that not a lot of people really wanted to do it. I spent about 6 years trying to figure it all out. Eventually I came to understand what Quality Assurance (QA) really was about and how different it was from Software Testing. At that moment, I chose to continue to pursue learning and doing Software Testing.

The years have passed, and I have become quite Senior in this field now. That means that I can be dropped in pretty much any situation now and have a reasonable chance of success (but more on that later). One of the problems when you reach this point though is that there aren't many people you can turn to when you have questions. As luck would have it, I stumbled across the right set of email discussion forums where all of the industry experts hang out and discuss ideas. These are some of the smartest and nicest people I have met in my life -- outside of the circle of Physicists I once knew who still ponder the mysteries of "Life, the Universe, and Everything in it" today.

I gotta say that being able to talk with some of these people is really cool! I don't think I've ever been this close to the cutting edge of creating a new profession and affecting an entire industry as I am today.

Here's my dilemma: I'm enjoying my work as a Tester too much, but the "teacher" side of me is aching to get out and spread the word! BUT.... to become a teacher here, it would mean either becoming a trainer/consultant of some kind or going into an educational institution of some kind (a university or college).

Going the independent consultant route involves risks that I'm not prepared to handle at this particular moment. I could do the consulting and training part, but I don't have the business skills I would need to keep myself busy. There's a recipe for disaster! So, until I find a partner who can help me with the marketing and finances side of things, I'll keep this one on the back-burner for a bit.

Going the educational institution route has it's own interesting problems too. If I wanted to teach at the local college, I could probably do that. There's not a lot of money in that though, so I don't know how long I would be able to keep it up. Teaching at the university has a great many attractive qualities. Just one thing: I don't have a Ph.D. Oh ya, I'd need one of those.

The funny thing is that I know exactly where I would go to get one (can you say the Florida Institute of Technology?), but there's a few things keeping me from packing up and doing that right now.

Okay, so fine. I have a few options, but I have relegated them for the time being in favour of staying where I am to see what else I can learn and help contribute to the Testing Profession as a "practitioner".

So, what now? I wrote an email to the Software-Testing mailing list a few days ago asking for a sense of direction. Maybe it was an unfair question to ask. Maybe it wasn't the right place to ask it. But I felt that I wanted to open myself up and see what kind of response I would get.

I got some good replies, but one of the best by far came from James Bach when he wrote:

We are headed toward a world that recognizes and respects testing as a skilled investigational activity, and is able to systematically describe and foster those skills.

Can you list your testing skills?

Can you demonstrate how to develop testing skills?

Can you describe a model of the testing activity as you see it?

Can you explain and defend your work without appeals to authority?

Can you relate testing and testing skills to other established fields of intellectual work?

Let these questions, or others along the same vein, guide you. Anyway, they work for me. Please suggest others, if you want.

I like James. He makes me think. =)

I'm pretty sure that's why I love Testing so much. It really is the closest thing (for me) to Physics when it comes to the think-work required to do a good job of Testing. I've met so many drones in my career and only a few who ever gave testing a passing thought. But here, now, is the paradigm we are striving to communicate: to do a good job of testing, you have to use your head.

Imagine that!

Well, of course it's actually a bit more complex than that, and smarter people than me can explain it really well, but I'm content to just know what I'm doing right now.

This is it. The beginning of a movement that will change the way Software is developed forever. (Welcome to the Revolution!) Historians will look back at the early 2000's one day and try to piece together the events that led to the paradigms that affected future thinking on the subject of Software Development and Software Testing in particular.

That history - our future - isn't written yet. Someone's gotta make it happen. In the words of William Shakespeare:

All the world's a stage, And all the men and women merely players: They have their exits and their entrances; And one man in his time plays many parts..

This is my part and the curtain hasn't fallen yet. So I have some improvisation left to do.

At least I can use James' questions as a guide in the short-term.

Let's see where it takes me, shall we?

Don't ask me to look.. I'm tired.

2004-12-29T21:48:00.000-05:00

This year it was our turn to host the family Christmas lunch/dinner party. My wife planned to cook a turkey with all the trimmings and I was in charge of baking some cakes for dessert. (I love to bake.)

About a week ago, I found myself sitting in the kitchen making a shopping list of the things that I would need for the baking extravaganza on Christmas Eve:
- I need milk. Don't we have milk? Better get some more.. Add it to the list. Check.
- Need sugar. Check the cupboard. Got plenty. What's next?
- Flour. Hmm.. we don't seem to have enough in the flour container. Didn't we have another bag of flour in the cold cellar(*)? (* Our "cold cellar" is an uninsulated storage room in the basement that we use as a pantry.) I've already been in there three times today and I didn't see it. In all fairness, each time I was in there I was looking for something else, so I'm not really sure if it was there or not.

My wife happened to be nearby in the living room sitting with our two boys watching a Christmas special on TV. "Honey? Do you know if we have any flour in the cold cellar?" I ask. She hummed and hawed for a moment and then said that she thought she remembered that there was a bag in there somewhere.

"I thought so too, but I've been in there a few times already and I didn't see it. Do you know where it might be in there?" I ask. She wasn't really sure but restated that she was pretty sure there was a bag in there.

At this point, I should add that this was near the end of the day and I was pretty tired. I don't think that I had slept very well over the previous few days, so in all I was feeling pretty run-down. Oh, and it was absolutely freezing in the cold cellar and I didn't really want to go in there again not knowing exactly where I was going to find what I needed. "Honey? Since you seem to think you know where it is, do you think you could go in and check for me please?" I plead.

She tells me that she didn't really want to get up and check at that particular moment. (In all fairness, when I'm spending time with the boys I usually defer such requests too.) That was too bad. I had started my shopping list. Once I started, I wanted to finish it. I can't just stop the list now and get back to it later. I need to know if we have some so that I can move onto the next item on the list!

Arrg! Fine. I'll go check myself. So I go in there, I check where I think it should be - nothing. I check the shelves - nope. I look in various plastic bags and underneath some bags with Christmas presents that the boys don't know about - nada. No luck. I'm freezing. Okay, back up to the warmth of the kitchen.

I march on up and mention to my wife in passing that I didn't see any flour so I'll just add it to the list. She asked me if I had checked the shelf above the wine bottles. I said that I looked there in passing, but I'm pretty sure it wasn't there. In fact, I was pretty sure that the bags on that top shelf contained leftover Halloween candy. She seemed surprised at me when I said this and told me that we shouldn't have any of that stuff left.

Damn. I have a doubt. Arrg. Back down into the cold cellar I go.

I go in and check the top shelf. Sure enough... no candy. One (plastic shopping) bag contained a bag of whole wheat flour (I can't use that for the cakes I want to make), and another bag contained some rice. (Wow, I didn't know we had extra rice. This is a good thing to know too.) No Halloween candy. And definitely no white flour. I can't feel my toes. Gotta get back into the warmth.

Up I go again. Again, when I pass my wife I mention that I checked and that she was right about the bags not containing candy, but that I still didn't find the flour I needed.

I sit back down in the kitchen and scribble "flour" onto the shopping list. I had checked. I double-checked. Didn't find any. We will definitely need some. Ahh, I can now happily move onto the next item.

I finish the list and go onto doing something else. About an hour later I found myself sitting pretty much where my wife was in the living room doing something with the boys. I heard a rustling noise coming from downstairs followed by some laughing. (Hmm, do I want to know what she's laughing about?)

My wife returned to the living room to let me know that there was indeed a bag of white flour sitting on the shelf in the cold cellar -- exactly where I had looked. (Sigh. It can't be true. Can it?)

"Really?" I ask. "Yes, really." She replied. "You're not just pulling a fast one on me?" I counter. "No, I'm not. It is sitting there in the middle of that shelf and you just didn't see it."

Doubt returns. Of course I have to go and see for myself.

I walk into the cold cellar and sitting there staring straight at me is this big bag of flour - right in the middle of the shelf. Some fleeting silly thoughts entered my mind: maybe a flour fairy came by and put it there after I had left; maybe my wife found the bag buried somewhere else and she just put it there so that I would think that I had missed it. No. I shake my head. I know exactly what had happened.

The Moral of the Story (a.k.a. "Lesson Learned"):

I was tired and my observation skills were clearly not up to the task at hand.

Also, I did not want to go in there and look for it. This feeling was so strong that I pretty much believed that there was NO bag in there to begin with.

As a result I didn't see the bag even when I was looking directly at it. I knew precisely what I was looking for, but I had the wrong disposition or frame of mind for looking (at that particular moment). In fact, upon reflection, I think I had even moved the bag of flour out of the way at one point in my search so that I could check to see what was behind it.

What changed my frame of mind? I had to believe my wife. Unlike me, she believed that there was a bag in there. She had an idea of where to look. She just opened the door and saw it sitting there plain as the nose on your face.

Sigh. I shut the door and went back up and scratched "flour" off the shopping list. We clearly have enough.

The Software-Testing Tie-In:

Over the years, I've always been very vocal about making sure that testers aren't forced to work overtime to make up for really stupid last-minute schedule crunches. "Just because development was late doesn't mean that the testers have to work overtime every night for the next four weeks to make up the lost schedule time."

I know that when you're tired you make mistakes. To compensate at work, I have always employed a risk-based testing strategy to help me clearly prioritise and communicate what we will (and won't) test under the new schedule timeline. No overtime - that is not an option.

Until the flour-in-the-pantry incident, I cannot recall such a blatant example of this phenomenon ever happening to me. I'm sure it has. This time it was easy to check though.

I can only imagine some poor tired (tester) soul trying to look for a particular bug (or class of bugs) but being too tired to see what is directly in front of them!

I didn't need this "flour" event to happen to me to drive the point home, but it certainly did. In the future, not only will I continue to force the "no overtime" issue, but I will also try to ensure some "recovery" time between testing projects to make sure that the testers have enough time to recover from the intense think-work involved on big testing projects.

Tired brains are just no good to anyone.

To Blog or Not To Blog...

2004-12-01T23:44:00.000-05:00

Don't mind if I do, thank you. =)

After much ado from many of my friends, family, and coworkers over the last year or so, I have finally gotten around to starting a journal again. It's all about the writing. Gotta keep up the writing.

So, dear reader, what can you expect from this journal of thoughts?

To begin with, I am a student of life. Always with the learning am I. As a result, I'm big on analogies. It's like everything in life can somehow be compared to something else.. to some degree or other, that is. Analogies are my coping mechanisms to help me see if I understand something well enough, as well as provide quick and dirty mental models to help me describe something to someone else.

So what is it that I'm mostly learning about now? Software Testing is taking up most of my actual thinking brain cells these days. For most of the rest of things in life I'm kind of winging it as I go along. ;)

This spot will be reserved for my thoughts on what I'm learning, what I'm thinking, and what is generally going on with me in this grand adventure. You see, Software Testing is an emerging discipline.. uncharted territory.. undiscovered country. I have the privilege of partaking in discussions with many of the Pioneers in this field and sometimes I can't help but feel a bit overwhelmed at the prospect of helping to shape the future of Software Engineering in some small way. Cool. =)

There's so much to say.. so much to do..

And so it begins..