Never too early to start thinking about your tests

This post is spurned by taking a reflection on my journey as a tester and it’s amazing to see how much i have moved from designing large UI test suites to very small UI test suites. For emphasis, the largest test suite had taken over 8 hrs and the smallest has been 3mins of execution time.
There are a lot of posts on the internet about how fragile a UI test framework could be and i would expect the next question to be how have you been able to achieve this. I would try to address this in this post.

1. Thinking about tests while designing the system: There are some decisions made in the early stages of the projects that has a direct impact on how easy or how difficult It would be to test a system. E.g. a web application that has been built with RESTful api and with minimal business logic embedded within the UI lend itself to proper layering of test automation when compared to a system with loads of logic embedded in the UI logic.

2. Testing is done at the cheapest level:
There is always a cost associated with test automation, cost in terms of test creation, test maintenance and test execution. As testers and developers we ought to think about the cost we incur and determine at what level the tests should be written.

Test Creation: The project test suite should be split between unit test, integration tests and end-to-end tests. The end to end tests can be further split into two categories, api tests and user interface tests. The cost associated with test creations would generally increase as we move up from lower levels of test automation to more abstract levels such as User interface tests. However, in some instances e.g. rewrite of legacy applications without any form of test automation, it might be easier to create a quick UI test suite using record and play tools. There isn’t any rule of the thumb on this but I would advise the team to have this conversation at all times discussing how to best automate tests for new features and updates to existing features.

Test Maintenance: This is the cost associated with updating tests as the system continues to evolve, and in my experience its generally cheaper to update lower levels of test automation and this can be done in parallel with development of features.

Test Execution: This can best be expressed in amount of time it takes for a test suite to run from start to finish, and I would like to include the time elapsed when the tester and/or developer is waiting for the test result. It also makes sense to include efforts involved in interpreting the any test results including test failures. Bearing in mind that the automated test suite is expected to be executed several times in a day, over the life time of the project this is one aspect that could potential be the most expensive. The teams test strategy should ensure that the lower level tests which are generally faster needs to contain the majority of the tests I.e. unit, integration, api tests.

3. Collaboration with developers: No matter how solid a test automation strategy is, it would never be complete if there hasn’t been any thoughts about how to collaborate with developers. It is very common to have the unit tests and integration test be designed and maintained by developers while testers would design and maintain the end-to-end test suite. The consequences of such practices include test duplication and test omission. At the end of the day we should all be working as a team, testers should take interest in Unit tests and integration tests; ensuring that these tests are relevant at the least and we should be able to suggest scenarios to be added to the test suite even if we do not have the skills to write such on our own.
On the other end, developers need to be involved with design and maintenance of end to end tests, developers should be able to advice testers if a proposed end to end test (usually expensive) can be written cheaply in a unit test and/or integration test.

These are some of the ideas that have helped me with achieving a lean UI test automation suite and i do hope you find this useful as well.

Posted in Agile Delivery, Test Strategy | Tagged , , , , , | 3 Comments

Copying objects between two different amazon s3 accounts

I have recently taken on ownership of a website and as part of the migration task i’ve had to copy over a few artefacts;I will be posting another blog about what i have learnt from this process.

The website uses Amazon s3 for storing users’ uploaded photos and document. I was faced with the task of copying these files over to my S3 account.

Quite a number of way to achieve this; but i stumbled on a link which is a python script that copies object between two buckets in the same account.

I have then updated this script to handle copying objects between two different accounts.

The edited script is as below:

from boto.s3.connection import S3Connection
from boto.s3.key import Key
from Queue import LifoQueue
import threading

source_aws_key = '*******************'
source_aws_secret_key = '*******************'
dest_aws_key = '*******************'
dest_aws_secret_key = '*******************'
srcBucketName = '*******************'
dstBucketName = '*******************'

class Worker(threading.Thread):
    def __init__(self, queue):
        self.source_conn = S3Connection(source_aws_key, source_aws_secret_key)
        self.dest_conn = S3Connection(dest_aws_key, dest_aws_secret_key)
        self.srcBucket = self.source_conn.get_bucket(srcBucketName)
        self.dstBucket = self.dest_conn.get_bucket(dstBucketName)
        self.queue = queue

    def run(self):
        while True:
            key_name = self.queue.get()
            k = Key(self.srcBucket, key_name)
            dist_key = Key(self.dstBucket, k.key)
            if not dist_key.exists() or k.etag != dist_key.etag:
                print 'copy: ' + k.key
                self.dstBucket.copy_key(k.key, srcBucketName, k.key, storage_class=k.storage_class)
                print 'exists and etag matches: ' + k.key


def copyBucket(maxKeys = 1000):
    print 'start'
    s_conn = S3Connection(source_aws_key, source_aws_secret_key)
    srcBucket = s_conn.get_bucket(srcBucketName)

    resultMarker = ''
    q = LifoQueue(maxsize=5000)

    for i in range(10):
        print 'adding worker'
        t = Worker(q)
        t.daemon = True

    while True:
        print 'fetch next 1000, backlog currently at %i' % q.qsize()
        keys = srcBucket.get_all_keys(max_keys = maxKeys, marker = resultMarker)
        for k in keys:
        if len(keys) < maxKeys:
            print 'Done'
        resultMarker = keys[maxKeys - 1].key

    print 'done'

if __name__ == "__main__":

Pre-requisites are you have to install python and install the boto s3 library.

and obviously you have to put your credentials in there

Posted in Programming | Tagged , , | 2 Comments

Assign AutomationIds to DevExpress ComboxBoxEdit Control

For some time i have been trying to automate a desktop application which uses a lot of DevExpress controls. For someone coming from a web application environment; this was  a very difficult task for me as i had no experience of WPF or any of the other custom controls framework out there.

So that i dont have to repeat myself i would post a link to the post i put on the devexpress website but i hope to flesh out a detail blog post sometime in the future so that other people can learn from me.


Posted in Software Testing | Tagged , , | Leave a comment

type into a textfield with keypress event disabled

A colleague asked me how to automate a disabled text-box which has a calendar attached to it. The customer has specified that they want keyboard entries disabled for this text-field, hence every user of the website has to use the calendar drop down.

testfield calendar

Previously, this has been automated by using webdriver to type the text into the text box but as it was disabled, this was impossible.

imagine, this is the html for the textbox:

<input type="text" id="text_box_identifier" value="" disabled >

Using a javascript library such as jQuery, i was able to set this field using


Hope that saves you some time today ….

Posted in Software Testing | Tagged , , , , , , , | 2 Comments

Cucumber unicode support

I ran into a problem today where some french and spanish characters were being displayed strangely.

And this made my test to fail,

After some poking around in google i can across a post that suggested adding

require ‘cucumber/formatter/unicode’

to the env.rb and that seem to have sorted the problems i was having.

Hopes this saves you some time.

Posted in Software Testing | Tagged , , , , , , | Leave a comment

Performance Testing experience using Ant and Jmeter – Part 2

In my last blog post, i described how i have used jmeter-plugins at my current client site, now i am faced with another problem , i need to be able to run these performance test from command line, in other for these performance tests to be executed in a CI environment such as teamcity.

A quick search brings to light the ant jmeter task which is an ant task for running jmeter test. This fulfils a number of the objectives i have set in this first part of this blog post as teamcity has got an inbuilt runner for ant. To install this, you would need to include ant-jmeter-1.1.1.jar in your Ant classpath. See the link for more instructions.

And the author of the ant jmeter tasks has alsop produced an xsl file which can be fed into the xslt ant task to produce a nicely formatted html result page.

I have put all these into an ant script that looks like the below:

<project name="Performance Testing for my project" basedir="." xmlns:ac="antlib:net.sf.antcontrib">
  <taskdef uri="antlib:net.sf.antcontrib" resource="net/sf/antcontrib/antlib.xml">
    <pathelement location="${basedir}/lib/ant-contrib-1.0b3.jar"/>

  <target name="clean">
    <delete includeemptydirs="true">
      <fileset dir="Results" excludes="**/PerformanceCharts.html"/>

  <target name="run-jmeter" depends="clean">
    <taskdef name="jmeter" classpath="${basedir}/jakarta-jmeter-2.5.1/extras/ant-jmeter-1.1.0.jar"

      <testplans dir="${basedir}/TestPlan" includes="*.jmx"/>
      <property name="request.threads" value="300"/>
      <property name="request.loop" value="10"/>

  <target name="create-reports" depends="run-jmeter">

  <target name="create-graphs" depends="create-reports">
    <ac:for param="reportType" list="TimesVsThreads,ResponseTimesOverTime,ThreadsStateOverTime">
        <java jar="${basedir}/jakarta-jmeter-2.5.1/lib/ext/CMDRunner.jar" fork="true">
          <arg value="--tool"/>
          <arg value="Reporter"/>
          <arg value="--generate-png"/>
          <arg value="${basedir}/Results/@{reportType}.png"/>
          <arg value="--input-jtl"/>
          <arg value="${basedir}/Results/JMeterResults.xml"/>
          <arg value="--plugin-type"/>
          <arg value="@{reportType}"/>
          <arg value="--width"/>
          <arg value="800"/>
          <arg value="--height"/>
          <arg value="600"/>

  <target name="main" depends="clean,run-jmeter,create-reports,create-graphs"/>

I have added an additional target for deleting the test result files as jmeter would append subsequent test results to the previous test results in the file, if file is not deleted.

You would also notice that there is a target named “create-graphs”, this uses the CMDRunner.jar which is a command line utility shipped with jmeter-plugins. This tool has helped me to re-create the graphs which i referred to in part 1 of the blog. It parses the xml result created by the ant jmeter task and creates png images for each of the graphs specified.

This graph images can then be embedded into an html page to be displayed in teamcity.

I hope this helps someone.

Posted in Software Testing | Tagged , , , | 7 Comments

Performance Testing experience using Ant and Jmeter – Part 1

Jmeter is a well known open source performance/load testing tool and to be fair it does a lot of stuff really well.
if you wants to do some quick performance testing without a whole lots of infrastructure around it then it is great.

I started out my task with jmeter with some objectives:

  • A tool that i could integrate into a CI tool such as teamcity
  • Meaningful graphs that could be easily interpreted by any one in the team
  • Able to integrate the graphs into teamcity
  • Able to monitor the performance of the website under test.
  • After doing an initial round of tests, i didnt like the graphs which were being produced by jmeter and i search for plug-ins to enable me plot better graphs.
    I came across jmeter-plugins which is an awesome collection of jmeter plugins. Instructions to install can be found on this page.

    Whilst, there are lots of usefuil graphs bundled within this plugin, i found the following graphs to be most useful

  • Active Thread Over Time
  • Response Times Over Time
  • Response Times versus Thread.
  • I could easily put “Active Thread Over Time” and “Response Times Over Time” to get a complete picture of my performance results solely based on response times.
    And that was good enough for me.

    Also jmeter only allow you to specify a ramp up period and the maximum number of thread you want to execute your tests with. I desperately wanted to be able to specify a ramp down time as well and fortunately, the Ultimate Thread Group which ships with the jmeter-plugins solves that problem for me as well.

    Having jmeter-plugins installed along side with my installation of jmeter helped to get the best out of jmeter, and i was able to specify my load more realistically as well as having clear and better graphs.

    See below.

    Response Time Over Time

    Thread State Over Time

    Time Vs Thread

    In the next blog post i would discuss how i have integrated jmeter with ant, enabling me to run jmeter in command line

    Posted in Software Testing | Tagged , , , , | Leave a comment