I should thank Santhosh Tuppad who prompted me to write this post. This is quite related to his post Virus or Trojan or Malware or Adware – Follow up testing.
Many of us (TESTERS) know that Microsoft releases one or the other security patch, hot fixes and services packs on Tuesdays. In this post, I will share information about how Microsoft Updates can impact on testing and how we have been able to manage this problem. I do not have intention to use this post to describe about compatibility testing but as it is related will have a note in the end.
What is the Impact of MS updates on Product/Application?Application starts behaving weirdly. User complaints, affecting the credibility of the product. Testers and Developers cycling to simulate and debug. In short the impact User Complaints –> Customer Escalations – > Pressure from Management (Follow-up) –> Frustrating Night outs/Weekends for Testers and Developers
How does MS Updates get into User Machine?
Most of the product users computers will have windows auto update option enabled and hence MS can get into the machine (it is good practice to avoid security implications). The other way is to manually install some of the hot-fixes to solve a specific problem. It is worth saying that most of these hot-fixes are not completely tested and Microsoft does warn that “Do not install the hot-fix if you are unsure of it and installation of incorrect hot-fix can cause problem to your machine, etc.,”.
How does it impact on Testing?
A security patch on IE has all the potential to block the application from working. Here, any patch or hot-fix can restrict or change the behavior of normal application. Usually we do not anticipate problems from these Patches as we(at least most of us, if not all) trust MS.
We would have spent our energy and time when we prove that the bug is not from the application but something else? To find out that something else you would spend some more time. Once you get to know it could be because of the patches and finally find out which patch is causing the problem among so many(MS does not release one patch at a time) patches. This process of bug isolation becomes critical and challenging. Here, the challenge is not only the effort put on to simulating the bug but also other factors such as pressure from the management, pressure from the customer, time constraints etc.,
How to handle this situation, or compatibility of the product?
It is easier to use the Virtual PCs for sorting out these problems.
Have 2 separate Virtual Machines (VM), one with patches updated(with the latest one) and other without the latest one but until the previous patch.
Conduct all your tests on the VM with patches updated while have your Smoke test on the VM without the latest patch. This way you can be ahead of the problem and giving information/bug to the customer rather than getting a complaint/escalation.
We have implemented this for a leading Scandinavian content management and Web publishing product that had to support 46 different combination of Windows OS, IE, SQL Server, Office etc., we had set up Virtual Machines with automation and continuous integration using QTP, QualityCenter and Team Foundation Server. The Continuous Integration takes care of running the automated tests on most of the environments while the Manual exploration of product and bug hunting is done on the latest patch VM.
You cannot practically test all the combination of OS, Web Server,Browser and other components with the common project constraints such as time, budget etc., It is good to consider Pairwise Independent Combinatorial Testing (PICT) Tool or ALLPAIRS to get effective minimal number of combinations.
--LN
