Test Automation ROI with COFFEe (at) Coffee Day

Most of the customers that I work with and management that I work with are interested to get automation testing implemented for their application or product and sometimes a portfolio of applications. The reason for implementing automation that they would give for implementing automation testing and plan for budget is they believe or in some cases they are made to believe one or more of the following.

1. Automation is the way of saving testing costs
2. Automation is the only way for improving the quality of testing
3. Automation is the way to improvise the testing process
4. Automation is for Regression testing
5. Once automation is implemented it is possible to cut billing and reduce some head count for the project
6. Automation can do magic by testing the application to pile the bugs
7. Automation can reduce the testing effort

Then does it mean all the statements mentioned above is TURE in all cases or does it mean it is FALSE in all the cases. There is no immediate answer to this question rather it needs to be decided based on the context. We all know and most of the time customer as well (at least in my case as of now) understands that automation is an investment and unless done with care it can lead to failure and loss of money. So now the question is how to make such a critical decision of implementing or not implementing automation for any application. We as TESTERs are responsible for providing information to the stakeholders so that they can make informed decision about it.

The Management and Customer are more interested to know about how much cost savings can be expected and achieved by implementing automation. However, it is not always true that cost savings is achieved. The value of automation is to be presented with much more benefits that we can gain out of it. Doug Hoffman, Michael D Kelly and many others have already written quite a lot about ROI for Test Automation. I thank them for such wonderful information. In this blog post I make an effort to explain how I have presented test automation benefits to customers with a heuristic COFFE CD (easily remembered as COFFEe at Coffee Day). (Earlier I have used the mnemonic FCCOFED for the same)   

The mind map below answers some of the basic questions that one may have about Test Automation ROI.

Below is the heuristic/mnemonic that I use while analyzing the automation ROI and further for presenting the same to customer and the management. Thanks to Pradeep Soundarrajan for correcting the method by helping me with the “Coverage” (second C) in COFFE CD.

Let me know if this helps you in some manner and also share your experiences with presenting ROI of Automation.

Feedbacks – A Lesson Learnt

Recently, I had an opportunity to sit with the customer and take the customer satisfaction feedback. During this meeting, I had a good lesson learnt which is the reason for this post.

Usually Customer Satisfaction Review (CSR) is conducted quarterly. In a collaborative project where customer is also part of the project CSR means feedback by customer about the Project Manager, Project Team and the Organization on different aspects.

We both (I and my customer contact) sat together in a silent room and started the discussion. He explained about all the good achievements that we had made and all other concerns that he had. These are all usual stuff. The interesting part was, he said it is easier for me to say that you have not done this and need to improve on this and so on. But, for me as a manager to you, all these feedbacks raise question “How did you help to the team to achieve this?”.

Instantly, my thoughts went back thinking about “How I have helped my team members for whom I have commented saying you have not done this, you need to improve on that and so on..” 

This looks very basic but an impressive question that many of us as managers never think about. This kind of retrospective is very important for  both of us ‘managers’ and 'receiver of the feedback’.

Further, I thought about reaction of a manager to a question by a reportie saying, I agree to you that I could not do this (some task or goal …) but “How did you help me to achieve this?”  Most of the managers would feel uncomfortable to these questions.

This is good lesson learnt for me. After some thinking around this, I have decided to implement this every time I need to give a feedback.

Let me know if you have different thoughts or comments.

Regards,

LN

Low Cost Performance Testing with free tools - Front-End Performance Testing

In this article I will share information on conducting Front-End Performance Testing (FPT) which is low cost yet very effective and does not need any commercial tools to be used. Before starting with the core topic I should thank Steve Souders for writing great books High Performance Web Sites and Even Faster Websites and Scott Barber for making it easier by bringing in the heuristic approach.

So, what is Front-End Performance Testing?

The Front-End Performance Testing is Monitoring and measuring the user experience from  the performance point of view and validating the front-end design with respect to the practices for high performance websites without any focus on the server side load. FPT is conducted to find any client side performance problems in the application.

You may ask why do we need to conduct FPT?

FPT is important for the reasons listed below (few of the points I have taken it from Scott Barber’s article on FPT)

• 50-90% of a web page's response time results from front-end design and implementation  (Taken from Scott Barber’s article)
• The front-end design and development of websites is conducted with little to no thought on performance (Other than, possibly reducing the size of the graphics)
• Usually neglected by development considering the client system not in control. Also, neglected by test team during testing
• Checking/Testing for potentially largest and cheapest performance improvements
• Relatively easy to conduct tests and investigate the Front-End performance issues (in comparison with backend performance tests)
• Minimal learning curve, can be tested by developers or testers [Capturing the data may not need any experience but effective analysis of the captured data needs reasonable performance testing skills and understanding]
• The client side processing and activities done at the client side by Java Script and Silverlight or not captured by LoadRunner like performance test tools that work on the HTTP/HTTPs layer (E.G. You would have seen in Google that suggests the search criteria on typing every single letter in the search box. This is a JSON request that should be quick enough not to let the user know that the response is from the server. Measuring the response time and user experience of this type of scenarios could be real hard without FPT)
• It needs less effort and can be done with Functional Testing, Can be automated as well. [For testing and reporting a website it would take about 4 hours based on my experience (here it also matters how big is the website) :-)]. I usually combine it with manual exploration during initial navigation through the application for learning it. In fact the information captured by HTTP sniffers is worth taking a look as you can understand about the application much better and device your tests.(not only performance but also other tests such as security and even functional or validation etc.,)

The objective of FPT is to

• Find client side performance problems without any focus on server loading or resource utilization by the server
• To understand the end user experience

Few considerations to be made while conducting FPT include the following

• It is important to consider all the web pages in the application including any alternative flows
• Use Free HTTP Sniffer tools and Browser plug-ins to gather information about the end user experience
• Document any functional defects that you may find during FPT

Now, let us see how to do the real test. The strategy that I follow for FPT is based on the heuristic created by Scott Barber.

Test Strategy

Use the heuristic SCORN for comprehensive Front-End Performance Testing. The heuristic can be defined as below (I have changed a few points based on my understanding but you can see the original heuristic as defined by Scott at Use "SCORN" to test the front end of a website for performance):

• Size – Focus on testing for
  • Uncompressed graphics and media
  • Object or code duplication
  • Script and styles living outside of the base HTML
  • Code "minification."
• Caching – Focus on testing for
  • Expires settings,
  • Etags
• Order – Focus on testing for the order sequence of components
  • Styles/style sheets
  • Critical content (i.e. what the user came to see the page)
  • Relevant media (i.e. graphics related to the critical content)
  • Incidental content (i.e. non-critical graphics)
  • Scripts
• Response Codes – Test for
  • Requests for objects that don't actually exist (Any exception is costly - MSDN)
  • Superfluous redirects
  • Errors that are not obvious from the browser
  • Redundant Requests
  • Invalid URL references (E.g. http://:)
  • Unused Request
• Number – Ask questions related to
  • Number of Requests
  • 1 Heavy Graphic Vs many small graphics
  • Inline Scripts Vs 1 External Script
  • Inline Styles Vs 1 External Style Sheet

Testing Procedure

• Use browser plug-ins or online tools to capture page load times.
• YSLOW – FireFox FireBug Add-on
• Episodes – YSLOW Add-on
• HammerHead – YSLOW Add-on
• HTTPWatch Basic
• Page Speed – FireBug Add-on
• Microsoft Visual Round Trip Analyzer
• Fiddler with nXpert – Performance Analysis Plug-in
• Conduct FPT with functional tests and/or user acceptance tests
• Devise the tests based on the project context and criteria – Pick the tool based on the need, best for the context
• Additionally, for monitoring the resource utilization during the test you can leverage on Perfmon (for windows based systems ) and NMon (Unix based systems). PAL is a good tool for analysis of Perfmon Logs.

My favorite tools for FPT are Net tab of FireBug, YSLOW, Episodes, HTTPWatch and VRTA. 

Reporting

The performance errors and slow pages need to be analyzed and reported for both executive and technical audience.

Although, I call it as Low cost performance test in the title of this article. FTP is not a substitute for comprehensive performance test. In fact the FPT is part of the complete performance testing strategy. When customer is not ready to invest for performance testing and in situations wherein you have customer complaints on the performance of the applications, FPT can be the first step of performance testing.

Hope this information helps you to device your performance tests. In the next post I will plan to write about automation of FPT with different tools such as QTP, TestPartner, TestComplete  and also without using any tools but VBScript.

--LN

Impact of Microsoft Updates on Testing – Compatibility Testing

I should thank Santhosh Tuppad who prompted me to write this post. This is quite related to his post Virus or Trojan or Malware or Adware – Follow up testing.

Many of us (TESTERS) know that Microsoft releases one or the other security patch, hot fixes and services packs on Tuesdays. In this post, I will share information about how Microsoft Updates can impact on testing and how we have been able to manage this problem. I do not have intention to use this post to describe about compatibility testing but as it is related will have a note in the end.

What is the Impact of MS updates on Product/Application?
Application starts behaving weirdly. User complaints, affecting the credibility of the product. Testers and Developers cycling to simulate and debug.  In short the impact User Complaints –> Customer Escalations – > Pressure from Management (Follow-up) –> Frustrating Night outs/Weekends for Testers and Developers

How does MS Updates get into User Machine?
Most of the product users computers will have windows auto update option enabled and hence MS can get into the machine (it is good practice to avoid security implications). The other way is to manually install some of the hot-fixes to solve a specific problem. It is worth saying that most of these hot-fixes are not completely tested and Microsoft does warn that “Do not install the hot-fix if you are unsure of it and installation of incorrect hot-fix can cause problem to your machine, etc.,”.  

How does it impact on Testing?
A security patch on IE has all the potential to block the application from working. Here, any patch or hot-fix can restrict or change the behavior of normal application. Usually we do not anticipate problems from these Patches as we(at least most of us, if not all) trust MS.
We would have spent our energy and time when we prove that the bug is not from the application but something else? To find out that something else you would spend some more time. Once you get to know it could be because of the patches and finally find out which patch is causing the problem among so many(MS does not release one patch at a time) patches. This process of bug isolation becomes critical and challenging. Here, the challenge is not only the effort put on to simulating the bug but also other factors such as pressure from the management, pressure from the customer, time constraints etc.,

How to handle this situation, or compatibility of the product?
It is easier to use the Virtual PCs for sorting out these problems.
Have 2 separate Virtual Machines (VM), one with patches updated(with the latest one) and other without the latest one but until the previous patch.
Conduct all your tests on the VM with patches updated while have your Smoke test on the VM without the latest patch. This way you can be ahead of the problem and giving information/bug to the customer rather than getting a complaint/escalation.

We have implemented this for a leading Scandinavian content management and Web publishing product that had to support 46 different combination of Windows OS, IE, SQL Server, Office etc., we had set up Virtual Machines with automation and continuous integration using QTP, QualityCenter and Team Foundation Server. The Continuous Integration takes care of running the automated tests on most of the environments while the Manual exploration of product and bug hunting is done on the latest patch VM.

You cannot practically test all the combination of OS, Web Server,Browser and other components with the common project constraints such as time, budget etc., It is good to consider Pairwise Independent Combinatorial Testing (PICT) Tool or ALLPAIRS to get effective minimal number of combinations.

--LN

OWASP Top Ten 2010 Released

 

On April 19, 2010, final version of the OWASP Top 10 for 2010 has been released. You can find more information about it at OWASP Top 10 2010 Press Release and OWASP Top Ten Project.

The OWASP Top 10 Web Application Security Risks for 2010 are:

• A1: Injection
• A2: Cross-Site Scripting (XSS)
• A3: Broken Authentication and Session Management
• A4: Insecure Direct Object References
• A5: Cross-Site Request Forgery (CSRF)
• A6: Security Misconfiguration
• A7: Insecure Cryptographic Storage
• A8: Failure to Restrict URL Access
• A9: Insufficient Transport Layer Protection
• A10: Unvalidated Redirects and Forwards

The Web Application Security Consortium provides Threat Classification Taxonomy Cross Reference View which gives a clear mapping between WASC Threat Classification, MITRE's Common Weakness Enumeration, SANS Top 25 and OWASP Top Ten.

Regards,

LN

Evaluating Test Tools for Automation – Criteria to be considered

Before starting with any automation it is very common to evaluate various tools commercial and free tools available in the market and choose the best suited one for the application under test. This article describes the criteria to be considered while evaluating tools for automation and a methodology for evaluating the tools. This article does not aim to provide a comparison for all these tools. However, it clearly indicates the criteria to be considered for evaluation of testing tools.

Following are the most common criteria to be considered while evaluating tools for automation and any application. In addition to these there may some specific criterions to be considered based on the application under test.

Evaluation Criteria:

1. Technology Support - Support for various controls and technologies used in the application such as iFrames, AJAX controls, PDF forms, tree view, cold fusion controls etc.,
2. Ease of Script development/enhancement
3. Reporting Results - The tool under evaluation should have a feature of producing result log which should be easy to analyze and pinpoint the defect.
4. Test Independence - The failure of one test script should not have any impact on the entire test suite.
5. Maintenance of Script - As there is very high maintenance overhead for the automated test scripts, the tool should provide ease of script maintenance.
6. Multi browser support - The tool under evaluation should support different flavors of Windows OS and multiple browsers (at least IE6, IE7 and IE 8.0)
7. Data Driven Capability - The tool should provide a means to have an external data store to store all the data and read/ write into the data store.
8. Ease of Object Store Maintenance - There should be a means to have easy maintenance of Object Store. Object store is the repository of all the objects captured by the tool.
9. Ease of Continuous Integration for nightly regression tests – The tool under evaluation should provide a easy means to integrate the tests to the build environment to have nightly regression tests conducted in a continuous integration environment
10. Limitations – Limitations of the tool with respect application under test
11. Advantages – Advantages of the tool with respect application under test
12. Cost of Licensing - Tool should not be expensive and should have a flexible licensing option.

Evaluation Procedure:

1. Select the tools that you want to consider for evaluation.The common tools considered for evaluation for automation include QTP, TestPartner, TestComplete, Visual Studio Team Edition, Selenium, Microsoft Lightweight Test Automation Framework, ArtOfTest WebAii etc..,
2. For each of the tool considered for evaluation identify the pros and cons with respect to each of the criterion listed above
3. Give a score for each of the tools for each of the criterion. Scale of scoring is
   • 1 - Below Average
   • 2 - Average
   • 3 - Good
   • 4 - Very Good
   • 5 - Out Standing
4. Prepare a Score card of all the tools for each of the criterion considered. A sample score card is shown below. Please do not consider the data provided below as the actual comparison data. This is provided only to show an example.

   Evaluation Criterion\Tool	QuickTest Pro	Test Partner	Test Complete	VSTS
   Technology Support	4	3	3	2
   Ease of Script development	4	3.5	3	3
   Reporting	4	3.5	3	3
   Test Independence	4	3	4	4
   Script Maintenance	4	3.5	3	4
   Cross-Browser Support	4	3	2	2
   Data Driven Capability	4	4	4	4
   Ease of Object Store Maintenance	4	3	3	NA
   License Cost	2	3	4	3
   Final Score	3.78	3.28	3.22	3.13

5. Provide a Rank for each of the tools considered based on the score provided earlier. Below is a sample example of tool ranking. This ranking does not represent the actual comparison of the tool rather it represents the suitability of the tools for a specific application it was evaluated.

   Tool	Final Score	Tool Rank
   QuickTest Pro	3.78	1
   TestPartner	3.33	2
   TestComplete	3.22	3
   Visual Studio 2008 Team Edition for Software Testers	3.13	4

6. Recommend the best suited tool based on the Rank

Although, you can make a recommendation for a specific tool based on technical analysis with specific criteria. It is always not true that your recommendation will win the race. Most of the time it is a business decision based on the cost and budget and hence be open to work any tool and try to find workarounds for the issues.

--LN

A Checklist for Performance Testing – Requirement Questionnaire

A Checklist for elicitation of the Performance Testing Requirements.

Performance Testing of an application involves the various phases outlined below. This article is an attempt to   provide a list of questionnaire that can help the test leaders/managers and testers to elicitate the performance testing requirements. This information is most important to know before starting with the performance test. Note that the questionnaire provided here is generic list of questions that is common to most of the applications. However, it need to be tailored based on the nature of the application under test. Usually I use this questionnaire by sending it to the stake holders of the application for answering. I have been using this questionnaire successfully for conducting performance tests.

---

---

Performance Testing Requirement Questionnaire

1. Please provide the URI's and credentials of the application for testing
2. Please provide the test environment (Hardware and Software) configuration.
3. Where is the test environment setup? – Inside the Firewall in a isolated LAN environment or outside the firewall.
4. What technologies have been used to develop the application?
5. What are the interfaces of the application? e.g., Payment gateways, web services etc.,
6. Briefly describe the business/domain of the application.
7. Is the application already in production? - Is this performance testing being conducted pre-production or post-production?
8. Are the web server logs for the application available? Applicable only if the application is already in production.
9. What are the critical workflows of the application to be considered as candidates for performance testing?
10. What is the expected workload (number of simultaneous virtual users) to be tested?
11. What is the average session duration of a user? - Average time a user would be logged into the application.
12. How many hours in a day the application would be available/accessed by the users?
13. Do you have any specific performance objective(SLAs) for this test? E.g. 1000 Invoices to be processed in a Day
14. Is test data required for performance testing available in adequate quantity in the required format.
15. Does the test team members have the  necessary privileges for the test server machines.
16. Do you aware of any performance problems in the application that is experienced by the users or observed by other stake holders?

--LN

Estimating Effort for Automation Test in Agile/Scrum

Effort Estimation has been an art since a long time and it will continue to be an art. Especially estimating time for testing effort is usually hard because of various uncertainties. That too when in an Agile/SCRUM environment where change is quite natural it is very hard to estimate. Here is an estimation methodology that I have used and successful to a good extent.

This methodology is specifically suited for distributed agile teams where development and manual testing of the application is done by one team while test automation is done by another team.(it can be used anywhere else as well) Here, in this scenario the Product Owner of the automation team is a member of the development sprint team who is a subject matter expert.

Although in SCRUM usually the estimations are done during the sprint planning, it is insisted to estimate before the sprint planning meeting during the Product Backlog Items (PBIs) Grooming sessions as the team would be developing automation scripts independent of the development sprints and have reasonably good understanding of the application and does not need the Product Owner’s(PO) presence in estimating or understanding the PBIs. However, in case if there is a need for POs involvement in understanding the PBI it is recommended to estimate in the grooming meeting or after it but necessarily after understanding the PBIs.

Note that the story point estimation does not provide the effort estimation as such rather it represents the complexity and size of the task in terms of relative complexity. Based on the knowledge of Sprint Velocity (Rate of productivity) one can identify the effort in terms of time and cost.

For example: If you estimate 100 story points and have the velocity as 4 story points per hour then your team would need about 25 hours to complete the 100 story points which can be further planned based on the capacity (Available time) of the team for the sprint.

Story Point Estimation

• PBIs are estimated based on the Agile/SCRUM principles of estimation in terms of story points
• PBI’s are estimated during the Grooming sessions
• PBI’s are estimated by all the team members and arrived at an amicable number that is considered as the actual estimate
• For story points Fibonacci series of numbers 1, 2, 3, 5, 8, 13, 21 are considered
• PBI with 1 story point is considered the least complex while 21 is considered to be relatively most complex
• Story points are assigned to reflect the complexity and size of the PBI based on the relative complexities
• For estimating the story point for a PBI, below criteria needs to be considered
  1. Available (Already developed) Business Utility Libraries that can be reused – More the BULs available for reuse less complex is the PBI
  2. BULs to be developed – Number of BULs to be developed and their complexities are considered
  3. General Utility Libraries to be developed/changed -- Number of GULs to be developed/changed and their complexities are considered
  4. Data Preparation – Amount of Test Data to be prepared and its complexity is considered
  5. Unit Testing -- Complexity of unit testing the script is considered
  6. Checkpoints – The number of checkpoints to be implemented based on the COAs and their complexity is considered
  7. Reviews – Consider the effort for review of the automation script
• Following Rules are to be followed while estimating story points for automation.
  1. All the team members estimate individually for a given PBI
  2. To avoid influences of one team member to other, Planning Poker is played. Planning Poker is an estimation game where all the team members estimate separately and disclose their estimates together at the same time as per the Scrum Master’s (SM) instruction
  3. Most of the time highest Estimate wins the race - only if all the team members and SM are in agreement
  4. SM is responsible for handling discrepancies in estimates between team members
  5. Discrepancies in estimates between the team is handled by Poker Votes, Reasoning and discussion as applicable and decided by the SM
  6. Sprint Backlog Items(SBI) are to be created for the PBI’s -- one SBI for each script, BUL and GUL to be developed or maintained. This is basically decomposing the PBI into multiple components called as SBIs for automation.
  7. Time estimates are provided for the SBIs – based on the velocity
  8. “Estimate” should be set once and not adjusted
  9. SBI’s not to exceed 8 hours – In case if a SBI cannot be broken any more and on breaking it loses the logic then the estimate can exceed 8hours for a SBI.
  10. Previous sprint Velocity (Rate of Productivity) and the capacity of the team is to considered for projecting the velocity of the current sprint

 

-- LN

TESTER by INSTINCT, not by CHANCE.

Testing for Exceptions in Web Application – An Approach

Last week when I had been to Norway, one of my colleague test manager casually asked question on testing a requirement for a banking application. I think it is interesting challenge to conduct such a test and this article is an attempt to provide testing approach for such a situation.

Requirement: Application to log every exception it encounters and fail gracefully (of course securely without any sensitive information disclosure)

Question: How to test and plan for testing the exceptions in a Banking and Finance application?

Solution:

This is not that easy testing that can be done with ease as simulating exceptions is not always possible.

Before thinking about the solution it is pivotal to understand the requirement clearly. In an effort to understand the requirements we need answer the below questions.

• How are the exception stack traces logged by the application? Usually it logged to a secure database or logged with audit trail logs.
• What exceptions are caught as specific exceptions in code and what exceptions are caught as generic exceptions? This is important to understand what need to be done to simulate an exception.

Now, having understood the requirement reasonably well we can think about testing.

It is hard to anticipate different types of exceptions caught on a specific event. However, 2 approaches to test this situation are described below.

Black box approach: This is best suited for the manual functional testers with limited competency in programming. As seen by the black box tester any exception is thrown only on an event such as page submission or request either GET or POST of a request.

• List all the scenarios in the application where there is GET or POST, predominantly form submissions.
• For each and every request list down the possible exceptions based on the functionality being processed and data input. For example on page where integer inputs are made it is necessary to consider IntergerOverflow, IndexOutOfBound and NullPointerException
• Design test cases to simulate each of the exceptions by relevant test input data and expected results. It is important to consider the Post Conditions or Follow-ups after the exception.
• Execute the test cases and record the results with defects

The advantage of this approach it does not need programming skills for a tester to conduct the tests although it requires basic programming knowledge for designing the tests. The cons of this approach are it does not ensure coverage as many exceptions may not be practically possible to simulate.

Grey box approach: This is the approach that ensures greater coverage and best suited for testers with programming knowledge. I recommend this approach. It is good to plan and estimate for this testing during test design and planning phase.

• Review the code with focus on the catch clause
• Verify if all the specific exceptions are handled in the catch clause depending on the scenario under context
• Verify that there is a generic catch clause that takes care of all the exceptions not handled by specific exceptions
• Here, it is important to validate the exception reports for accuracy and consistency. Usually the exception are report to a secure database or to a log file stored securely.
• It is also important to look that exception report for sensitive information that can be juicy for a hacker.
• The stack trace of the exception should not be transferred as clear text in the HTTP requests and responses. It should be encrypted with a strong cipher.

The testing approach is described based on my experience in testing such scenarios with an intention that it could be of some help to the testers community. Comments and feedback on this are appreciated.

-- LN

TESTER by INSTINCT, not by CHANCE.

Blogging after long time

It’s been long time I have written any article on the net. Now I think it is time again to be actively blog on the net. You will be able to see more frequent articles sharing my experience and knowledge from next year. I plan to dedicate some of my time to blogging about Testing especially Automation, Performance and Security Testing.

Regards,

LN  

How is CreateObject different from GetObject?

One of my junior colleague had asked the question "What is the difference between CreateObject and GetObject and in what situation GetObject is to be used?". This post is the answer that I have given him.

1. CreateObject creates an instance of a class
2. GetObject gets the reference of an running object to another object
3. Always use CreateObject except for Windows Management Instrumentation (WMI) and Active Directory Service Interfaces (ADSI)

The below script demonstrates the use of these two VBScript functions.

Set objExcel = CreateObject("Excel.Application") 
objExcel.Visible = True 
Set objWorkbook = objExcel.Workbooks.Add 

Set objExcel2 = GetObject(, "Excel.Application") 
Set objWorkbook = objExcel2.Workbooks(1) 
Set objWorksheet = objWorkbook.Worksheets(1) 
objExcel2.Cells(1, 1).Value = "In god I TRUST everything else I TEST."

 

-- Lakshminarasimha Manjunatha Mohan

In God I TRUST everything else I TEST.

Windows Hotfixes and Patches Enumerator

Below is a simple VBScript developed using Windows WMI that can be executed from Windows Script Host to enumerate the installed hotfixes, patches and security updates on any given machine.

You might be wondering what for this script is useful. While product testing that supports different environments it happens so that suddenly one fine day you might notice some mis-behavior or unusual defects in the application. Actually these mis-behaviors or defects are not always related to the application. However, it may be a compatibility issue with one of the patches or security updates made by the windows automatic updates.

As a solution to this problem I am this script to find out all the patches, hot fixes and security updates on a server where the product is working fine and compare the that with another server where the mis-behavior or defects are seen and thus classify a defect from compatibility issue. This is really a very handy script for compatibility testing. Let me know your comments on this. I intend to extend this script for comparing two different lists of hotfixes and patches so as to make it easier to find the differences.

For executing this script it is necessary have an EXCEL file named test.xls at C:\

'========================================================
'Description: This script enumerates the Windows hotfixes 
' and other security updates
'Author: Lakshminarasimha M.
'=========================================================
strComputer = "."

Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set colQuickFixes = objWMIService.ExecQuery ("Select * from Win32_QuickFixEngineering")

sDataTable = "C:\test.xls"
sDataSheet = "Sheet1"

iReqRow = 2
Dim oXLApp 'As Excel.Application
Dim oWorkBook 'As Excel.Workbook
Dim oWorkSheet 'As Excel.Worksheet
Dim iRowsCount 'As Integer
Dim iColsCount 'As Integer
Dim sColHeader 'As String
Dim sReqColumn 'As String

Set oXLApp = CreateObject("Excel.Application")
'oXLApp.Visible = True
Set oWorkBook = oXLApp.Workbooks.Open(sDataTable)
Set oWorkSheet = oWorkBook.Sheets(sDataSheet)
oWorkSheet.Activate
iRowsCount = oWorkSheet.UsedRange.Rows.Count
iColsCount = oWorkSheet.UsedRange.Columns.Count
iReqRow=2
' Set Column Headers
     oWorkSheet.Range("A1").Value = "Caption"
     oWorkSheet.Range("B1").Value = "CSName"
     oWorkSheet.Range("C1").Value = "Description"
     oWorkSheet.Range("D1").Value = "FixComments"
     oWorkSheet.Range("E1").Value = "HotFixID"
     oWorkSheet.Range("F1").Value = "InstallDate"
     oWorkSheet.Range("G1").Value = "InstalledBy"
     oWorkSheet.Range("H1").Value = "InstalledOn"
     oWorkSheet.Range("I1").Value = "Name"
     oWorkSheet.Range("J1").Value = "ServicePackInEffect"
     oWorkSheet.Range("K1").Value = "Status"

For Each objQuickFix in colQuickFixes
     oWorkSheet.Range("A" & iReqRow).Value = objQuickFix.Caption
     oWorkSheet.Range("B" & iReqRow).Value = objQuickFix.CSName
     oWorkSheet.Range("C" & iReqRow).Value = objQuickFix.Description
     oWorkSheet.Range("D" & iReqRow).Value = objQuickFix.FixComments
     oWorkSheet.Range("E" & iReqRow).Value = objQuickFix.HotFixID
     oWorkSheet.Range("F" & iReqRow).Value = objQuickFix.InstallDate
     oWorkSheet.Range("G" & iReqRow).Value = objQuickFix.InstalledBy
     oWorkSheet.Range("H" & iReqRow).Value = objQuickFix.InstalledOn
     oWorkSheet.Range("I" & iReqRow).Value = objQuickFix.Name
     oWorkSheet.Range("J" & iReqRow).Value = objQuickFix.ServicePackInEffect     
     oWorkSheet.Range("K" & iReqRow).Value = objQuickFix.Status
    iReqRow = iReqRow + 1    
Next
    oWorkBook.Save    
    oWorkBook.Close
    oXLApp.Quit
    Set oWorkSheet = Nothing
    Set oWorkBook = Nothing
    Set oXLApp = Nothing
    MsgBox("---Hotfixes Enumerated Sucessfully.---")

-- Lakshminarasimha Manjunatha Mohan

In God I TRUST everything else I TEST.

How to automatically delete unused folders?

Below is a Visual Basic Script that can delete all the sub folders older than 30days from the current date. The folder from which the sub folders need to be deleted can set by  changing the value for strMainFolder variable. This script can also be used to manage files and folder in remote computers just by setting the required computer name/IP to strComputer variable.

The script can be used very effectively by scheduling a job from Windows Task Scheduler.

strComputer = "."
strMainFolder = "C:\"
Set objWMIService = GetObject("winmgmts:" _
& "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")

Set colSubfolders = objWMIService.ExecQuery("Associators of" _
& {Win32_Directory.Name='" & strMainFolder _
& "'} Where AssocClass = Win32_Subdirectory ResultRole = PartComponent")

For Each objFolder in colSubfolders
   dtmFolderCreationDate= CDate(Mid(objFolder.CreationDate,5,2) _
    & "/" & Mid(objFolder.CreationDate, 7, 2) & "/" _
    & Left(objFolder.CreationDate, 4) & " " _
    & Mid (objFolder.CreationDate, 9, 2) & ":" _
    & Mid(objFolder.CreationDate, 11, 2) & ":" _
    & Mid(objFolder.CreationDate,13, 2))
    If (DateDiff("d",dtmFolderCreationDate,Now)>30) Then
    'Wscript.Echo objFolder.Name
    'Wscript.Echo objFolder.CreationDate 
    errResults = objFolder.Delete
    End If
Next

-- Lakshminarasimha Mohan

In God I TRUST everything else I TEST.

IE popup window disappears while QTP Script Execution!!

A strange problem of Mercury QuickTest Pro 9.2 with Internet Explorer 7.0 is discussed with solution.

Problem: While executing scripts in IE 7.0 if another browser is opened by the script as a result of an operation or test step, the pop up window automatically disappears. The issues looks strange but the window disappears without any clue.  

Due to this issue there are many other problems including problem in executing Navigate method of IE object etc., I have not tested it on IE6. However, the problem might also exist in IE6 as well.

Example scenario: A QTP 9.2 automation script for the following scenario in the admin mode of EPiServerCMS web site fails to execute on IE 7.0.

Steps for Simulation:

1. Login to EPiServerCMS Admin Mode (Ensure that the Role and membership providers are set to Multiplexing providers with SQL Server providers as default provider)
2. Click on Administer Groups
3. Create a new Group
4. Click on the Delete button corresponding to the newly created Group the pop up window disappears only when a QTP script for this scenario is executed. However, the scenario works perfectly fine on manual execution of the same steps.

 

Solution: The solution for this problem is just enable the "BHOManager Class" Add-on in the IE. (This can be done by going to IE-> Tools -> Manage Add-ons -> Enable/Disable Add-ons...)

What is "BHOManager Class" add-on or BHOManager.DLL?

A Browser Helper Object, or BHO, is just a small program that runs automatically every time the Internet explorer browser is started.

BHOManager.DLL is installed on the system when QTP is installed. This is a helper class that assists QTP for execution of QTP scripts on the browser. BHOManager.DLL is located in the folder C:\Windows\System32. The unique ID or CLSID of this BHO is {474264BC-9571-47C1-85B9-780F756DC9CE}.

Additional Information about BHOs:

BHOs are not stopped by personal firewalls, because they are identified by the firewall as browser itself. They are also known to conflict with other running programs, causing a variety of page faults, run time errors, and the like, and generally impede browsing performance.

BHODemon is small tool that can be used to view, enable or disable all the BHO's on a computer. It is a free tool that can be downloaded from BHODemon.

Other known BHOs:

The other known BHOs include the following:

1. Microsoft.VisualStudio.QualityTools.RecorderBarBHO.dll - This BHO is used by Visual Studio Team Suite for web tests

For more information BHOManager.dll and other BHOs refer to

http://www.file.net/process/bhomanager.dll.html

http://www.sysinfo.org/bhoinfo.html

Performance Testing: Resource Monitoring

What is Load Testing?

The behavior of the application and the utilization of the resources by the application under test is monitored under different load patterns and it is validated against the expected measurements to identify any potential performance errors or bottlenecks.

What is necessity for Monitoring Resources?

Any normal ASP.NET web application will be developed using three tier architecture of Presentation, Business and Data Layers. When a request is made by the client to the server the request is processed by the web server which in turn would be forwarded to the application server and then to the database server. During this processing the System, web server application server and the database server resources are utilized. It is necessary to monitor these resources utilization under load before the application is moved to production in order to identify the performance errors and bottlenecks before it breaks the application and impacts the business of the application.

What Resources need to be monitored?

It is necessary to monitor every logical or physical layer of the web application. In a simple scenario where the application is hosted on a web server and application interacts with the database the sources to be considered for monitoring are System resources, the web server resources and the database resources.It is necessary to identify the specific resources and counters (with their threshold levels) to be monitored at each layer. This identification needs expertise and experience in performance testing. Below is the list of counters that can be used for first level of performance analysis. This table applies for the general ASP.NET applications. However, it needs to be tailored for the specific requirements.

Note: The counters column in the below table to be read as shown in the example: % Processor Time (Processor _Total). Here, the Processor is the Object, % Processor Time is the Counter and _Total is the instance.

How to Monitor the Resources? What tools to use?

On a Windows based machine it is possible to monitor the resources using a utility called perfmon. The same way as perfmon Linux/Unix based machines facilitate the resource monitoring by means of utility known as rstdmon.
Almost every Load Testing tools has integration with Windows Perfmon or has some other has means to provide the facility to monitor the resources. E.g. LoadRunner, Visual Studio 2005 Team Edition for Testers, WebLoad, etc., 

Conclusion

The table of counters provided here provides a possible list of counters that can be monitored for while Load Testing an ASP.NET application. However, it needs to be tailored for specific requirements. For example, while testing a application that uses ADAM process one would be interested to know the processor and memory utilization by the ADAM worker process (dsamain) in specific. It is advised to use the minimum number of high level counters for the first round analysis. Based on the first level analysis one can added more detailed and specific counters for the areas that warrant performance and remove few counters related to areas where the performance seems be good.

-- Lakshminarasimha Mohan

"In god I TRUST, everything else I TEST."

Object Identification Mechanism

QTP identifies the objects by mandatory properties set as default. If the mandatory property values are not sufficient to uniquely identify an object in the Application Under Test (AUT), QTP adds some assistive properties and/or an ordinal identifier in order to create a unique description.

The default Mandatory, Assistive Properties and Ordinal Identifiers can be changed for a specific Class of objects in QTP at Tools -> Object Identification.

The Smart Identification can also be enabled to ensure more comprehensive object identification. The User defined Class of Objects can be mapped to the standard class of Objects so that QTP identifies the user defined objects as standard objects.

What is Smart Identification?

QTP makes use of the special feature Smart Identification when it  fails to identify objects using the defined properties finds an ambiguity in identifying objects. 

The Smart Identification mechanism is more complex and more flexible. If configured logically, a Smart Identification definition can probably help QTP identify an object, if it is present, even when the learned description fails.

The Smart Identification mechanism uses two types of properties, Base Filter Properties and Optional Filter Properties.

Base Filter Properties: The most fundamental properties of a particular test object class; those whose values will not be changed. E.g. html tag = <A> for a object of Link Class

Optional Filter Properties: Other properties that can help identify objects of a particular class.

Smart Identification Process

The QTP follows the below process to identify the objects using Smart Identification Mechanism:

1. QTP creates a new object candidate list within the Objects Parent Object containing all the objects that match all of the properties defined in the Base Filter Properties list.
2. New Object Candidate List (OCL) is created by filtering out the objects in the OCL that does not match the first property listed in the Optional Filter Properties list.
3. The OCL is evaluated in the following manner:
   1. If the new object candidate list still has more than one object, QTP uses the new (smaller) OCL to repeat elimination process for the next optional filter property in the list.
   2. If the new object candidate list is empty, QTP ignores this optional filter property, returns to the previous object candidate list, and repeats the elimination process for the next optional filter property in the list.
   3. If the object candidate list contains exactly one object, then QTP concludes that it has identified the object and performs the statement containing the object.
   4. QTP continues the process described above until it either identifies one object, or runs out of optional filter properties to use.

Summary

• QTP Identifies the Object using the learnt description.
• If unable to identify the objects using learnt description Smart Identification is invoked.
• If after completing the Smart Identification elimination process, QTP still cannot identify the object, then QTP uses the learned description plus the ordinal identifier to identify the object.
• If the combined learned description and ordinal identifier are not sufficient to identify the object, then QTP stops the run session and displays a Run Error message.