So, I made some supper for my daughter. Fruits and a sandwich. Sometimes she might be a bit tired at this hour and act up. Slightly crummy apples, those browns sports on bananas or wrong amount of drink just won’t do then. The routine that follows usually entails diplomacy, arguments, bribes, threats and pretty much anything that would improve the odds for her eating something before she goes to bed. This time the problem was the sandwich, more precisely the butter coverage of it.
I started with a slice of rye bread.
Then I applied some butter over the bread, which was not enough for my daughter.
Some more butter. Still not enough.
More butter, which was now too much. I ate the bread and gave her only some fruits.
Even though I have iterated this scenario for countless of times now, I still fail to understand what would be the right butter coverage for this bread. When it is covered?
When you’re trying to gather information from a system, process, product, service, whatnot, when can you safely say that it is covered? Can you ever? 9 out of 10 times when I’ve been asked to help teams to shape up I’ve seen test suites that cover only few key functionalities. Login, some read/write operations, links and buttons, payment, logout. Then they spent hefty amount of time and money to automate that suite, and now a full row of green in Jenkins is the pinnacle of success.
Did you know that the color green in Hudson/Jenkins was originally blue and that the balls turned green because of a plugin? This originates from Jenkins’ creator Kohsuke Kawaguchi’s Japanese heritage: Japanese people consider green to be a shade of blue and for a long time they didn’t even have a word for green. I’m not sure how Japanese people feel about using green or blue for that matter as a sign of approval or coverage, but I think the original Jenkins color blue has more use for testing purposes than the color green.
Consider a requirement: “This electronic equipment has to work within the nominal range of 100-250 VAC.” If you’re a formally trained tester or otherwise inclined to think things through computer sciences, you might start dividing this challenge into equivalence classes. Something under 100 VAC, something in between 100 and 250 VAC, and something over 250 VAC. Boundary value analysis to the rescue! 99 VAC, 100 VAC, 101 VAC, 249 VAC, 250 VAC, 251 VAC. Six test cases. That should cover it, right? If those go green in Jenkins, we are ready to go live, right?
But wait. Some smart-ass asked something about ranges 110-120 VAC and 220-240 VAC. “WTF?! Those were not in the requirement!” shouts product owner. “But, but… those are the most used voltages in the world”, pleads the trainee who just joined the project. I buy this trainee a beer and exhale the following:
- “In what environment does this equipment has to operate?”
- “To what kind of market are releasing this equipment? Domestic? International? What kind of voltages they use there?”
- “To what kind of usage is it subjected?”
- “What do you mean with “has to work”? Can it work partly? How?”
- “Is hazardous operation allowed? If yes, in what terms?”
- “What does nominal mean?”
- “What does VAC mean?”
- “Does it work with DC?”
- “What about amperes?”
- “By whom will it be used?”
- “How long does it have to work, and in what kind of stress/load?”
- “Are there competing products? How they fare against this equipment?”
- “Are other systems dependent on this equipment?”
- “Is it possible to maintain this equipment? Or test a bit further than just six voltage tests?”
- “Is there any user manual, dev notes or something that would help us understand this equipment and reflect reality to the claims done about it? Have we made other claims, perhaps to the public?”
- “Does this equipment bring value to the company? Should it?”
- “Do we have anyone who knows anything about electronics?”
Now, imagine something a bit more complex than one requirement about electronic equipment. Something about your own professional context. If you use even a shred of imagination, you can come up with an endless flow of questions similar to what I exhaled above. The more you do this questioning the more you understand that even the most simple product or service cannot be covered exhaustively, let alone a complex one. The chase for that darker shade of green (or a thicker layer of butter) becomes a futile effort.
So, blue. Have you considered a heatmap? Collection of focus areas you either burn your fingers on or not? Areas from which you either find something or not? Blue works in that sense beautifully. It doesn’t give any empty promises nor does it build ignorant confidence, which ultimately grows into hubris. It just indicates that nothing has been found with the effort that has been made very clear to everyone.
I’ve used four colours to indicated all of my testing and feedback efforts for over tens years now: red to indicate problems, yellow to indicate possible problems, purple to indicate (mainly to myself) that I need to return to that focus area and blue to indicate that I have not found anything despite my efforts, which is bad, because there is always something. Even the simplest focus area has countless variables that can be subjected to endless amount of product, quality, consistency, behavioral, environmental, etc. heuristics, and those pesky questions, which means that my blue pen is almost as unused as my green pen.
While automation suites should use blue, because their capability to actually find anything is quite limited, instead of colors human beings should focus on bringing visibility to their actual efforts and what they have found. That’s it. That generates fruitful discussion between stakeholders and the dangers of empty promises, ignorance and hubris can be avoided. So involve everyone to the effort and be open about the work you do. Be interested about the work others do. Do it together. My daughter has eaten most of the sandwiches I did for her, but she has eaten every sandwich we made together.