Got Test?
In August I had a discussion with a VP of Technology about testing techniques, strategies and the quality of the software being developed in the group. The explanation of his driving philosophy for development and quality assurance was simplistic at best. Further discussion, proves a superficial understanding of the problem and as a result this philosophy is the direct cause of the high concentration of defects in the software systems, under his lead.
The goal, as explained to me is very simple; it is to undergo very little testing and iterate releases every two weeks, whilst reacting to defects in production. Testing quality in production, identifying defects aside your customer!!? I thought he was speaking in jest.
Continuing, factually, the current goal, as he explains it, is to achieve a testing strategy that can meet less than 1 hour of testing for every hour of development. Clearly misleading the executives of the round-table to seek favor in these words toward an idea that faster is better.
"Go On!!" I thought, "You're pulling my leg!! No one really believes that!!! Do they???" Well, when one researches the software the defects render this philosophy to a "T".
Sadly, No, my leg was not being pulled; the statement was stone cold serious. Though, you could cut the cloud of confusion, innocence and fear with a knife as it lingered in the air from the lips of the naïve to my disbelieving ears. This strategy is “text book”, exactly what not to do.
What is so much more comical is the fact the current software's quality at the time suffered from about 65% rollbacks on all attempted implementations. That number maybe high and can be fairly disputed. However, let's split the difference and reduce it to less than half. Shall we say, 25%, think that's fair? Certainly, the rollbacks are undisputable. Either way, cause and effect is illustrated here. This philosophy, in terms of software development and the important roll of testing proves its own problems.
Scratching my head on these statements I thought through the quality of software that existed in this group. What future direction did this philosophy drive the company toward??? Something imminent, something insecure... All I could see was the current strategy causing the infliction of a financial laceration to the company's bottom line. This laceration causes a financial hemorrhage that will prove to become more serious as time goes on. This strategy, philosophy and traction cause a financial hemophilia which offers a grim prognosis. Of course, if the revenues come in at a faster pace then the bleeding, mediocrity prevails, alas, the status quo is tolerated. If executive management only knew the ill-effects I’m sure there would be a proper shellacking in store. Unfortunately, I’m not sure anyone with real expertise in these areas will ever draw focus.
As wasteful as this seems, many companies behave in this manner. So, the saying goes, "Despite themselves, they make money, albeit more canbe mad issue, the money still comes in." Se la vie...
It was explained further that having 80% of functionality (yes, functionality) was what the IT group tries to achieve. Seems like I have heard of something like this before (80/20)... The justification for this strategy was due to the market place and the drive of the business. Furthermore, testing in production (Yikes) was the goal and testing up front does not fit their model. The current mantra is "The business drives the development." Well of course!! But, when the business tries to control applications like tuning a radio's frequency or equalizer without taking real study, well then, rollbacks, financial hemorrhaging and defects, ensue.
This was a strategy for failure at any level, in any company, in any effort. Test, test, test, until your sick of it!! You can never test enough, testing in production COSTS YOU MONEY! Releasing defects can cause more damage the one can count. Remember, your eyes and your senses don't always tell you the truth, for example (example here). Profiling, instrumentation and facts prove the system results. Increase to production makes the bottom line profits swell. Decrease to production costs the company money, as you might infer.
Testing, testing, testing eliminates defects. Instrumentation proves reveals facts and sets the truth free.
My favorite analogy is when an automobile manufacturer decides to engineer a safety air bag system for the automobiles. First off, let's say it took 5 years to design, develop and implement the air bag system. Let's further say the aforementioned testing strategy was employed on the system. Would you take your children, spouse, parents or anyone else for that matter in for a drive for an air bag system with 1 hour of test for 1 hour of development; essentially, the testing plan described above? Or would you favor a proper testing strategy? Of Course not!! Well, then would you, if you were a CEO employ such a testing strategy on applications?? Knowing defects cost you money?? Knowing that testing in production, again, costs you money??
So, then why would you expose your company to applications that are tested with the same recklessness?
I'm not suggesting these two software development efforts are equivalent Far from it, building a car and building an app are both mission critical, however, at different levels of responsibility and severity. Make no mistake about it, the purpose, effort and results in either case are serious. Of course, different levels of severity and serious cause/effect results in their specific domains, nonetheless.
I'm sure you can understand the difference here and can practically apply the analogy to the discussion. Not having a good testing plan, strategy and testing, testing, testing to your development process can increase defects that cost the effort or company money.
Which brings me to the XBOX360 and Ubisoft's game of KING KONG for the XBOX360. Though, I am sure they did testing, in fact, "World Class". I am also certain they did not do enough. This week in the news, it was reported by BBC; Ubisoft's King Kong's X360 version is to dim to play on non HD televisions.
Ouch!
Where did the testing come in? Well it did come in... It must have been completed in many areas, just deficient and not in the form of testing on disparate platforms. This reflects in the company not knowing their client...Ehe!?! They tested on systems with HDTV screens, thus the problem was not even a fleeting thought for their customers with non-HDTV platforms.
The defect is within the rendering of the graphics on a regular television. The images are just to dark and the user is unable to see and keep up with the game play.
Ubisoft's comments:
"We have a problem on the 360," said Mr Guillemot. "The screen is dark on some TVs and it totally changes the experience. When it's dark, you don't see where you have to go."
Guillemot further commented: "I'm a bit disappointed that we didn't see it when we were developing the game." Guillemot says Ubisoft will try to fix the game code, but for now, suggests gamers try the PlayStation 2 or Xbox versions if the X360 game is too dark.
Obviously, Ubisoft does not subscribe to the testing strategy above! They make some really awesome games and are top-notch developers whom understand the value of proper testing. This should not be a slam to Ubisoft, not at all!! But it should be a wake up call to corporate IT departments.
When developing software inclusion of testing as a primary effort, during the development process is responsible and correct. Testing in production is wrong, irresponsible and costs more to the bottom line.
Testing 3 to 1 at the minimum will save your bottom line, it matters in production and production quality directly adds to the bottom line.
When the testing is less than 3 to 1, the software defects simply increase. Defects in software cause financial loss, definitively, realizing it is the confusion. If you do not have a responsible testing plan, strategy and direction the defects may make U-B-soft, too.
cheers