Technology Evaluation Test Automation

From wiki.gpii
Jump to: navigation, search

Testing Tool Evaluation - Kavya Komuru


Why image based tool?

  • Image based tools validate the layout and appearance of each and every (complete) screen of the application under test (AUT), running on different browsers, screen resolutions, devices, and operating systems. It is expected to validate the correctness of the UI when it is zoomed in and out, when hovering over various UI elements, and when the application’s window is resized.It is also expected to facilitate fast and easy test maintenance and failure analysis. In other words,it is expected the tool to eliminate the need for subsequent Manual Visual Testing.

Why not object base test automation?

  • Object-based Test Automation is preferred tools of choice when Functional Test Automation focuses on testing the functionality of applications rather than their visual appearance. These tools are tightly coupled with the UI framework of the AUT. If the UI elements of the AUT are properly identifiable it is easy to develop robust, cross-platform tests using these tools.

What is user experience?

  • User Experience testing lets user to see application from the perspective of a real user.It’s about observing what people actually do, and hearing what they think as they interact with website or application. As GPII is an application where users with tokens key in to interact with variety of devices and platforms with multiple preferences it is more suitable to test GPII application user experience with image based test tool.

Criteria For Test Tool:

Below are the requirements identified for automation tool from high to low priority.

  • Tool must automate GUI testing exactly the way user does via image and text processing.
    • Features
    • Test Script Reliability
    • Scalability of scripts
    • Test suites and image library
    • Consistency of scripts on Vm’s
  • Ease of use
  • Ease of setup and installation
  • Cross platform/Cross browser
    • Windows 7, 8, 10 32 bit/64 bit
    • Linux, Mac
    • Safari/Firefox/Chrome
  • Integration into Jenkins, JIRA and other tools
  • Reporting
    • Documentation
    • Support
  • Price
  • Vendor Qualification

Tools which are evaluated against the criteria are:

  • SikuliX
  • Eggplant Functional
  • Ranorex
  • TestComplete
  • Routinebot
  • Autoit


From the above tools, SikuliX, Eggplant Functional and Squish are considered as best criteria match, and they were evaluated against each other. Results are pasted below. The link to Google Docs is:

From the evaluation, it is observed that SikuliX and Eggplant Functional are the most suitable to satisfy APCP testing requirements.

Recommended Decision Approach for Eggplant Functional: Two week evaluation against our requirements with vendor.

EggPlant Functional




1. EggPlant Functional uses a two-system model, consisting of a controller machine, where scripts are authored and executed, and a system under test (SUT) which runs running a VNC server or connected via RDP

2. Eggplant Functional helps test applications better, faster, and with less effort by automating the execution of functional testing.

3. EggPlant Functional uses advanced image analysis technology to conduct UI testing. This approach allows it to drive and validate the system under test (SUT).

4. Eggplant can find images and even read text to help drive or validate tests.

1. Sikuli is open-source test automation tool that locates anything from the screen, based on image recognition techniques.

2. It uses image recognition powered by OpenCV to identify and control GUI components.

1. GUI test automation tool for functional GUI regression tests.

2. Squish runs a small server (squish server) that handles the communication between the Application Under Test (AUT) and the test script.

3. The “squish server” starts the AUT and injects the Squish hook into it.

Best used

1. If the aplication is designed to give a human being an interface to interact with… on a Website, Tablet, Smartphone, Desktop Application, or any combination of these

2. Eggplant Functional gives you a way to build automated tests that will ensure the quality of your product.

3. Eggplant Functional allows to test the functionality and performance of a developed app on any device or operating system regardless of coding language.

1. Sikuli automates anything user see on the screen. It uses image recognition to identify and control GUI components.

2. Automate some repetitive tasks in daily usage of applications or web pages

3. Playing games

4. Administration of IT systems and networks


1. Test the user experience.

2. The eggplant UX (user experience) helps you understand how your application will behave in the real world in the hands of real users.

3. EggPlant can interact with any device (including mobile, tablet, desktop, server, and Internet of Things devices) in the same way a user does, by looking at the screen.

4. EggPlant uses sophisticated image and text search algorithms to locate objects on the screen in a completely technology agnostic manner and then drive the device.

5. Eggplant can find images and even read text to help drive or validate tests.

6. Expansive library of scripts that interactively call to each other for different combinations of functionality to be tested. The modular nature of this method provides flexibility in testing and functional test expansion.

1. Tests the user experience

2. cannot read text from images

3. Application screen can be transferred to a desktop (Windows/Mac/Linux) where Sikuli is being run.

4. User input to application can be transferred from desktop to application

1. Squish uses property-based object identification (independent of screen position.

2. It is a two-component system, consisting of a runner, which interprets and executes scripts, and a server, which hooks in and controls the application under test (AUT) by injecting a module into it that provides a TCP/IP connection between the AUT and the program running the test

3. Provides script less test creation interface to tester similar to Test Studio.

4. It supports separation of test data from test cases, which allows reusability of test cases.

Cross platform/Cross browser

Test any device Cross platform/cross browser technology

Has some limitations. Supports windows XP and later including windows 8 and 10(32 bit or 64 bit) Mac OSX 10. 6 10. 11 10. 10 (64 bit only) Linux Depends on Prerequisites

Cross platform/cross technology

1. Test application from any host operating system. Eggplant can be installed on Windows, Mac OSX, or Linux.

Not supported

Not supported

Test Script Creation

1. Create automated tests quickly with uniquely robust record-and-playback, a flexible keyword driven testing framework, the intuitive test definition language Sense Talk, or any language (e. g. Java, C#, Ruby) via the egg Drive interface.

1. No Record and play back. SikuliX supports two approaches:

a. Scripting - SikulixIDE provides support for basic workflows based on searching images and acting on GUI elements and basic editing and running scripts.

b. Programming - Needs mature IDE like NetBeans or Eclipse and testing environment which integrates SikuliX as a feature library using API's. Supports scripting languages jython, jruby, scala, clojure.

1. Record and playback is available on high level interactions and playback

2. Very Good tool to use to ensure 100% code coverage Squish's support for BDD is unique because it tightly combines and integrates the BDD approach and GUI test automation.

3. Supports scripting languages Python, JavaScript, Ruby, Perl and TCL


1. Tests created on platform are easily scalable by adding new images to image library

1. Tests and images must be created for multiple platforms.


1. Tests once created runs consistently without failures

1. Test fail often.

2. SikuliX doesn’t provide any built in error handling or debugging.

Ease of use

1. EggPlant Functional user-centric test automation approach also makes it easy to write tests,testers can be productive with eggplant within a few hours.

2. Easy Drag and drop image creation

3. Efficient Image library management

4. Using one or more scripts, testers can establish test scenarios that test functionality across multiple devices or operating systems.

1. Sikuli uses a visually driven approach (i. e. , drag/drop images to test against), a test script writer can write test cases without any knowledge of the target application code.

2. Sikuli provides an IDE that allows for rapid development of test scripts with application screen shots.

3. The Sikuli image dictionary (key feature of Sikuli) allows to manage images to drive tests.

Easy to setup

1. Eggplant license on one machine can be easily connected to any VMS through VNC or RDP and tests can run on AUT on these machines.

2. EggPlant has open interfaces and an open philosophy. It can be easily integrated into any test environment and work with any ALM, continuous integration, JIRA or other tools too.

1. SikuliX need Java 7 (JRE (runtime only) or JDK (runtime + development kit)). Up to SikuliX version 1. 1. 0 java 6 vaersion can used.

2. Highly recommended, to always have the latest Oracle/Sun Java version on system (currently Java 8, does not work with Java 9). On 64-Bit capable systems a Java 64-Bit version is strongly recommended.

3. SikuliX will detect Java's bitness at runtime and select the fitting native library set automatically.

4. The system running SikuliX scripts must have a real screen connected.

Ease of Maintenance Test Scripts

Write one test, and run it on multiple platforms and/or browsers hence easy to maintain testscripts

Need different scripts on different platforms and image creation

overhead on testers

Continuous integration

1. Integrate eggplant Functional with popular ALM tools such as Jenkins. The Jenkins plug-in will report full detailed status information back from eggplant Functional to Jenkins so that it can report on exactly what went wrong when errors have occurred.

Can be integrated with Jenkins http://netjunki. org/jenkins-and-sikuli-integration. html


1. EggPlant Functional out-of-the-box results

2. Accessing report information from within a script

3. Using Test Cases to gather information for reporting

4. HTML reporting

5. Using Assert commands for reporting

6. Using Capture Screen for reporting

7. Email reporting using Send Mail 8. Results reporting in eggplant Manager

No Explicit test reporting

1. If using unit test method then implement HTMLTest Runner to generate reports(Test results)

2. Implement robot framework Which provides better reporting.


Yearly Renewal of license Priced at $7600


Squish has separate licenses prices for different platforms(Mac, windows, Linux) priced at 6700Euros One time License

Vendor Qualifications

1. Extensive Documentation

2. User Forums

3. Training by Vendor

4. Full Support

1.Limited Documentation

2. No support

3. No Training

1. Good Documentation

2. Email based support

3. Training by Vendor


1. High accuracy finding images and text on multi platforms Eggplant uses a sense Talk as the scripting language and this language is open source and similar to English level language

2. Eggplant comes with all the features for constructing test suites, creating reports etc

3. Test scripts are scalable

4. Test Scripts are reliable

5. Scripts can run on any platform guest OS via RDP or VNC

6. Scripting is not platform dependent

7. Easy and explicit error handling

8. Easy to setup

9. Ease of Test script creation and maintenance

10. Efficient Error handling

1. Open-source

2. Sikuli work as a stand-alone tool meaning that it will record as well as execute the scripts on the same machine.


1. Need to have VNC or RDP SUT connection

2. License is pricey

1. sikuli doesn't come with features for constructing test suites, creating reports etc.

2. Its not easy to match images if their dimensions/locations change(Resolution dependent) – Sikuli offers various primitives but depending on nature of change, matching logic may or may not work.

3. Not suitable for testing on virtual machines. Desktop using VNC/Screencast is not very smooth. Connection trough RDP has issues

4. Sikuli scripting is platform dependent.

5. No Explicit error handling - Error handling during run time has to be taken care properly in scripting if we run the scripts for long time.

6. Scripts are not scalable

7. Test Scripts are not reliable for cross platform testing

8. sikuli is resolution dependent