Thursday, May 13, 2010

Impact of Microsoft Updates on Testing – Compatibility Testing

I should thank Santhosh Tuppad who prompted me to write this post. This is quite related to his post Virus or Trojan or Malware or Adware – Follow up testing.

Many of us (TESTERS) know that Microsoft releases one or the other security patch, hot fixes and services packs on Tuesdays. In this post, I will share information about how Microsoft Updates can impact on testing and how we have been able to manage this problem. I do not have intention to use this post to describe about compatibility testing but as it is related will have a note in the end.

What is the Impact of MS updates on Product/Application?
Application starts behaving weirdly. User complaints, affecting the credibility of the product. Testers and Developers cycling to simulate and debug.  In short the impact User Complaints –> Customer Escalations – > Pressure from Management (Follow-up) –> Frustrating Night outs/Weekends for Testers and Developers

How does MS Updates get into User Machine?
Most of the product users computers will have windows auto update option enabled and hence MS can get into the machine (it is good practice to avoid security implications). The other way is to manually install some of the hot-fixes to solve a specific problem. It is worth saying that most of these hot-fixes are not completely tested and Microsoft does warn that “Do not install the hot-fix if you are unsure of it and installation of incorrect hot-fix can cause problem to your machine, etc.,”.  

How does it impact on Testing?
A security patch on IE has all the potential to block the application from working. Here, any patch or hot-fix can restrict or change the behavior of normal application. Usually we do not anticipate problems from these Patches as we(at least most of us, if not all) trust MS.
We would have spent our energy and time when we prove that the bug is not from the application but something else? To find out that something else you would spend some more time. Once you get to know it could be because of the patches and finally find out which patch is causing the problem among so many(MS does not release one patch at a time) patches. This process of bug isolation becomes critical and challenging. Here, the challenge is not only the effort put on to simulating the bug but also other factors such as pressure from the management, pressure from the customer, time constraints etc.,

How to handle this situation, or compatibility of the product?
It is easier to use the Virtual PCs for sorting out these problems.
Have 2 separate Virtual Machines (VM), one with patches updated(with the latest one) and other without the latest one but until the previous patch.
Conduct all your tests on the VM with patches updated while have your Smoke test on the VM without the latest patch. This way you can be ahead of the problem and giving information/bug to the customer rather than getting a complaint/escalation.

We have implemented this for a leading Scandinavian content management and Web publishing product that had to support 46 different combination of Windows OS, IE, SQL Server, Office etc., we had set up Virtual Machines with automation and continuous integration using QTP, QualityCenter and Team Foundation Server. The Continuous Integration takes care of running the automated tests on most of the environments while the Manual exploration of product and bug hunting is done on the latest patch VM.

You cannot practically test all the combination of OS, Web Server,Browser and other components with the common project constraints such as time, budget etc., It is good to consider Pairwise Independent Combinatorial Testing (PICT) Tool or ALLPAIRS to get effective minimal number of combinations.



Santhosh Tuppad said...

Cool post. Thanks for getting it done so soon and not ONE WEEK :)

Points about handling such kind MS updates which you have mentioned is a good learning.

Looking forward for more cool posts like this.

I have updated my post by adding "Related Post" by linking to your blog post of MS Updates as my blog readers would find it helpful.

Santhosh Shivanand Tuppad

Lakshminarasimha Manjunatha Mohan said...

Hi Santhosh,
Thanks for the comments.:-) Will plan to write more posts.


Mahendra said...

LN it's great post and valuable info... this would certainly help testers to isolate the defects in such situations with minimum effort... but, how do we get to the root cause, which patch has impacted our AUT.... I know we can write a script which would fetch info on all patches that has been installed on machine as said by you, MS would release # of patches at one shot.. how can we identify the exact culprit patch :)


Lakshminarasimha Manjunatha Mohan said...

Hi Mahendra,
Thanks for reading the post and feedback.

I will try to answer your question.
As written in the post when you are testing on 2 VMs one with latest patches and other VM with pathes but not the latest. If you find a bug and know that it is from the MS Update. You have already isolated it to the latest patches level. Microsoft does not relase one patch at a time. But, based on the nature of the defect you can get to know which specific component realted patch is causing the bug. Further, exporation and testing should tell you the right patch.
We should remember that every patch is relased to fix a specific bug/s. Also, the security patches will be released to fix specific vulnerabilities. MS provides information such as what could be the impact of installing a specific patch etc., based on these information you can filter out many possibilities. Finally, I should say that there is no substitute for exploration. You have to explore to find the right reason for the bug.


Mahendra said...

Thanks LN for the quick response and I do agree with you for exploration further.

Will be waiting for your next posts :)


seema said...

nice post