Saturday, January 16, 2010

Evaluating Test Tools for Automation – Criteria to be considered

Before starting with any automation it is very common to evaluate various tools commercial and free tools available in the market and choose the best suited one for the application under test. This article describes the criteria to be considered while evaluating tools for automation and a methodology for evaluating the tools. This article does not aim to provide a comparison for all these tools. However, it clearly indicates the criteria to be considered for evaluation of testing tools.

Following are the most common criteria to be considered while evaluating tools for automation and any application. In addition to these there may some specific criterions to be considered based on the application under test.

Evaluation Criteria:

      1. Technology Support - Support for various controls and technologies used in the application such as iFrames, AJAX controls, PDF forms, tree view, cold fusion controls etc.,
      2. Ease of Script development/enhancement
      3. Reporting Results - The tool under evaluation should have a feature of producing result log which should be easy to analyze and pinpoint the defect.
      4. Test Independence - The failure of one test script should not have any impact on the entire test suite.
      5. Maintenance of Script - As there is very high maintenance overhead for the automated test scripts, the tool should provide ease of script maintenance.
      6. Multi browser support - The tool under evaluation should support different flavors of Windows OS and multiple browsers (at least IE6, IE7 and IE 8.0)
      7. Data Driven Capability - The tool should provide a means to have an external data store to store all the data and read/ write into the data store.
      8. Ease of Object Store Maintenance - There should be a means to have easy maintenance of Object Store. Object store is the repository of all the objects captured by the tool.
      9. Ease of Continuous Integration for nightly regression tests – The tool under evaluation should provide a easy means to integrate the tests to the build environment to have nightly regression tests conducted in a continuous integration environment
      10. Limitations – Limitations of the tool with respect application under test
      11. Advantages – Advantages of the tool with respect application under test
      12. Cost of Licensing - Tool should not be expensive and should have a flexible licensing option.

Evaluation Procedure:

  1. Select the tools that you want to consider for evaluation.The common tools considered for evaluation for automation include QTP, TestPartner, TestComplete, Visual Studio Team Edition, Selenium, Microsoft Lightweight Test Automation Framework, ArtOfTest WebAii etc..,
  2. For each of the tool considered for evaluation identify the pros and cons with respect to each of the criterion listed above
  3. Give a score for each of the tools for each of the criterion. Scale of scoring is
    • 1 - Below Average
    • 2 - Average
    • 3 - Good
    • 4 - Very Good
    • 5 - Out Standing
  4. Prepare a Score card of all the tools for each of the criterion considered. A sample score card is shown below. Please do not consider the data provided below as the actual comparison data. This is provided only to show an example.

    Evaluation Criterion\Tool

    QuickTest Pro

    Test Partner

    Test Complete

    VSTS

    Technology Support

    4

    3

    3

    2

    Ease of Script development

    4

    3.5

    3

    3

    Reporting

    4

    3.5

    3

    3

    Test Independence

    4

    3

    4

    4

    Script Maintenance

    4

    3.5

    3

    4

    Cross-Browser Support

    4

    3

    2

    2

    Data Driven Capability

    4

    4

    4

    4

    Ease of Object Store Maintenance

    4

    3

    3

    NA

    License Cost

    2

    3

    4

    3

    Final Score

    3.78

    3.28

    3.22

    3.13

  5. Provide a Rank for each of the tools considered based on the score provided earlier. Below is a sample example of tool ranking. This ranking does not represent the actual comparison of the tool rather it represents the suitability of the tools for a specific application it was evaluated.

    Tool

    Final Score

    Tool Rank

    QuickTest Pro

    3.78

    1

    TestPartner

    3.33

    2

    TestComplete

    3.22

    3

    Visual Studio 2008 Team Edition for Software Testers

    3.13

    4

  6. Recommend the best suited tool based on the Rank

Although, you can make a recommendation for a specific tool based on technical analysis with specific criteria. It is always not true that your recommendation will win the race. Most of the time it is a business decision based on the cost and budget and hence be open to work any tool and try to find workarounds for the issues.

--LN

5 comments:

hs said...

Thanks for specifying criteria for automated software testing. could you please help e to get a free automated testing tool?

hs
http://www.softwaretestingnet.com/

seema said...

Hi. i need some help.
Can you please list out the limitations of Testpartner?
Please send them to vinay.nayak@boci.co.in

Lakshminarasimha Manjunatha Mohan said...

YES. Will try to list out the generic limitations of TestPartner. However, you will have to consider it with respect to your application.

Santhosh Tuppad said...

@LN,
Good one. Most of the organizations just jump on tools and do not do much research on them before using those tools with their product(s).

The points you have provided are really very cool :)

Thanks,
Santhosh Shivanand Tuppad

Lakshminarasimha Manjunatha Mohan said...

Hi Santhosh,
Thanks for reading the post and feedback.

Regards,
LN