Evaluating Test Tools for Automation – Criteria to be considered

Before starting with any automation it is very common to evaluate various tools commercial and free tools available in the market and choose the best suited one for the application under test. This article describes the criteria to be considered while evaluating tools for automation and a methodology for evaluating the tools. This article does not aim to provide a comparison for all these tools. However, it clearly indicates the criteria to be considered for evaluation of testing tools.

Following are the most common criteria to be considered while evaluating tools for automation and any application. In addition to these there may some specific criterions to be considered based on the application under test.

Evaluation Criteria:

1. Technology Support - Support for various controls and technologies used in the application such as iFrames, AJAX controls, PDF forms, tree view, cold fusion controls etc.,
2. Ease of Script development/enhancement
3. Reporting Results - The tool under evaluation should have a feature of producing result log which should be easy to analyze and pinpoint the defect.
4. Test Independence - The failure of one test script should not have any impact on the entire test suite.
5. Maintenance of Script - As there is very high maintenance overhead for the automated test scripts, the tool should provide ease of script maintenance.
6. Multi browser support - The tool under evaluation should support different flavors of Windows OS and multiple browsers (at least IE6, IE7 and IE 8.0)
7. Data Driven Capability - The tool should provide a means to have an external data store to store all the data and read/ write into the data store.
8. Ease of Object Store Maintenance - There should be a means to have easy maintenance of Object Store. Object store is the repository of all the objects captured by the tool.
9. Ease of Continuous Integration for nightly regression tests – The tool under evaluation should provide a easy means to integrate the tests to the build environment to have nightly regression tests conducted in a continuous integration environment
10. Limitations – Limitations of the tool with respect application under test
11. Advantages – Advantages of the tool with respect application under test
12. Cost of Licensing - Tool should not be expensive and should have a flexible licensing option.

Evaluation Procedure:

1. Select the tools that you want to consider for evaluation.The common tools considered for evaluation for automation include QTP, TestPartner, TestComplete, Visual Studio Team Edition, Selenium, Microsoft Lightweight Test Automation Framework, ArtOfTest WebAii etc..,
2. For each of the tool considered for evaluation identify the pros and cons with respect to each of the criterion listed above
3. Give a score for each of the tools for each of the criterion. Scale of scoring is
   • 1 - Below Average
   • 2 - Average
   • 3 - Good
   • 4 - Very Good
   • 5 - Out Standing
4. Prepare a Score card of all the tools for each of the criterion considered. A sample score card is shown below. Please do not consider the data provided below as the actual comparison data. This is provided only to show an example.

   Evaluation Criterion\Tool	QuickTest Pro	Test Partner	Test Complete	VSTS
   Technology Support	4	3	3	2
   Ease of Script development	4	3.5	3	3
   Reporting	4	3.5	3	3
   Test Independence	4	3	4	4
   Script Maintenance	4	3.5	3	4
   Cross-Browser Support	4	3	2	2
   Data Driven Capability	4	4	4	4
   Ease of Object Store Maintenance	4	3	3	NA
   License Cost	2	3	4	3
   Final Score	3.78	3.28	3.22	3.13

5. Provide a Rank for each of the tools considered based on the score provided earlier. Below is a sample example of tool ranking. This ranking does not represent the actual comparison of the tool rather it represents the suitability of the tools for a specific application it was evaluated.

   Tool	Final Score	Tool Rank
   QuickTest Pro	3.78	1
   TestPartner	3.33	2
   TestComplete	3.22	3
   Visual Studio 2008 Team Edition for Software Testers	3.13	4

6. Recommend the best suited tool based on the Rank

Although, you can make a recommendation for a specific tool based on technical analysis with specific criteria. It is always not true that your recommendation will win the race. Most of the time it is a business decision based on the cost and budget and hence be open to work any tool and try to find workarounds for the issues.

--LN

IE popup window disappears while QTP Script Execution!!

A strange problem of Mercury QuickTest Pro 9.2 with Internet Explorer 7.0 is discussed with solution.

Problem: While executing scripts in IE 7.0 if another browser is opened by the script as a result of an operation or test step, the pop up window automatically disappears. The issues looks strange but the window disappears without any clue.  

Due to this issue there are many other problems including problem in executing Navigate method of IE object etc., I have not tested it on IE6. However, the problem might also exist in IE6 as well.

Example scenario: A QTP 9.2 automation script for the following scenario in the admin mode of EPiServerCMS web site fails to execute on IE 7.0.

Steps for Simulation:

1. Login to EPiServerCMS Admin Mode (Ensure that the Role and membership providers are set to Multiplexing providers with SQL Server providers as default provider)
2. Click on Administer Groups
3. Create a new Group
4. Click on the Delete button corresponding to the newly created Group the pop up window disappears only when a QTP script for this scenario is executed. However, the scenario works perfectly fine on manual execution of the same steps.

 

Solution: The solution for this problem is just enable the "BHOManager Class" Add-on in the IE. (This can be done by going to IE-> Tools -> Manage Add-ons -> Enable/Disable Add-ons...)

What is "BHOManager Class" add-on or BHOManager.DLL?

A Browser Helper Object, or BHO, is just a small program that runs automatically every time the Internet explorer browser is started.

BHOManager.DLL is installed on the system when QTP is installed. This is a helper class that assists QTP for execution of QTP scripts on the browser. BHOManager.DLL is located in the folder C:\Windows\System32. The unique ID or CLSID of this BHO is {474264BC-9571-47C1-85B9-780F756DC9CE}.

Additional Information about BHOs:

BHOs are not stopped by personal firewalls, because they are identified by the firewall as browser itself. They are also known to conflict with other running programs, causing a variety of page faults, run time errors, and the like, and generally impede browsing performance.

BHODemon is small tool that can be used to view, enable or disable all the BHO's on a computer. It is a free tool that can be downloaded from BHODemon.

Other known BHOs:

The other known BHOs include the following:

1. Microsoft.VisualStudio.QualityTools.RecorderBarBHO.dll - This BHO is used by Visual Studio Team Suite for web tests

For more information BHOManager.dll and other BHOs refer to

http://www.file.net/process/bhomanager.dll.html

http://www.sysinfo.org/bhoinfo.html