Copying objects between two different amazon s3 accounts

I have recently taken on ownership of a website and as part of the migration task i’ve had to copy over a few artefacts;I will be posting another blog about what i have learnt from this process.

The website uses Amazon s3 for storing users’ uploaded photos and document. I was faced with the task of copying these files over to my S3 account.

Quite a number of way to achieve this; but i stumbled on a link which is a python script that copies object between two buckets in the same account.

I have then updated this script to handle copying objects between two different accounts.

The edited script is as below:

from boto.s3.connection import S3Connection
from boto.s3.key import Key
from Queue import LifoQueue
import threading

source_aws_key = '*******************'
source_aws_secret_key = '*******************'
dest_aws_key = '*******************'
dest_aws_secret_key = '*******************'
srcBucketName = '*******************'
dstBucketName = '*******************'

class Worker(threading.Thread):
    def __init__(self, queue):
        threading.Thread.__init__(self)
        self.source_conn = S3Connection(source_aws_key, source_aws_secret_key)
        self.dest_conn = S3Connection(dest_aws_key, dest_aws_secret_key)
        self.srcBucket = self.source_conn.get_bucket(srcBucketName)
        self.dstBucket = self.dest_conn.get_bucket(dstBucketName)
        self.queue = queue

    def run(self):
        while True:
            key_name = self.queue.get()
            k = Key(self.srcBucket, key_name)
            dist_key = Key(self.dstBucket, k.key)
            if not dist_key.exists() or k.etag != dist_key.etag:
                print 'copy: ' + k.key
                self.dstBucket.copy_key(k.key, srcBucketName, k.key, storage_class=k.storage_class)
            else:
                print 'exists and etag matches: ' + k.key

            self.queue.task_done()

def copyBucket(maxKeys = 1000):
    print 'start'
	
    s_conn = S3Connection(source_aws_key, source_aws_secret_key)
    srcBucket = s_conn.get_bucket(srcBucketName)

    resultMarker = ''
    q = LifoQueue(maxsize=5000)

    for i in range(10):
        print 'adding worker'
        t = Worker(q)
        t.daemon = True
        t.start()

    while True:
        print 'fetch next 1000, backlog currently at %i' % q.qsize()
        keys = srcBucket.get_all_keys(max_keys = maxKeys, marker = resultMarker)
        for k in keys:
            q.put(k.key)
        if len(keys) < maxKeys:
            print 'Done'
            break
        resultMarker = keys[maxKeys - 1].key

    q.join()
    print 'done'

if __name__ == "__main__":
    copyBucket()

Pre-requisites are you have to install python and install the boto s3 library.

and obviously you have to put your credentials in there

Advertisements
Posted in Programming | Tagged , , | 2 Comments

Assign AutomationIds to DevExpress ComboxBoxEdit Control

For some time i have been trying to automate a desktop application which uses a lot of DevExpress controls. For someone coming from a web application environment; this was  a very difficult task for me as i had no experience of WPF or any of the other custom controls framework out there.

So that i dont have to repeat myself i would post a link to the post i put on the devexpress website but i hope to flesh out a detail blog post sometime in the future so that other people can learn from me.

http://www.devexpress.com/Support/Center/Question/Details/Q452914/focus-answer-ee71c273-5f00-11e2-bf12-f0bf97cd2df9

 

Posted in Software Testing | Tagged , , | Leave a comment

type into a textfield with keypress event disabled

A colleague asked me how to automate a disabled text-box which has a calendar attached to it. The customer has specified that they want keyboard entries disabled for this text-field, hence every user of the website has to use the calendar drop down.

testfield calendar

Previously, this has been automated by using webdriver to type the text into the text box but as it was disabled, this was impossible.

imagine, this is the html for the textbox:


<input type="text" id="text_box_identifier" value="" disabled >

Using a javascript library such as jQuery, i was able to set this field using


$('#text_box_identifier').attr("value","27/05/2012");

Hope that saves you some time today ….

Posted in Software Testing | Tagged , , , , , , , | 2 Comments

Cucumber unicode support

I ran into a problem today where some french and spanish characters were being displayed strangely.

And this made my test to fail,

After some poking around in google i can across a post that suggested adding

require ‘cucumber/formatter/unicode’

to the env.rb and that seem to have sorted the problems i was having.

Hopes this saves you some time.

Posted in Software Testing | Tagged , , , , , , | Leave a comment

Performance Testing experience using Ant and Jmeter – Part 2

In my last blog post, i described how i have used jmeter-plugins at my current client site, now i am faced with another problem , i need to be able to run these performance test from command line, in other for these performance tests to be executed in a CI environment such as teamcity.

A quick search brings to light the ant jmeter task which is an ant task for running jmeter test. This fulfils a number of the objectives i have set in this first part of this blog post as teamcity has got an inbuilt runner for ant. To install this, you would need to include ant-jmeter-1.1.1.jar in your Ant classpath. See the link for more instructions.

And the author of the ant jmeter tasks has alsop produced an xsl file which can be fed into the xslt ant task to produce a nicely formatted html result page.

I have put all these into an ant script that looks like the below:


<project name="Performance Testing for my project" basedir="." xmlns:ac="antlib:net.sf.antcontrib">
  <taskdef uri="antlib:net.sf.antcontrib" resource="net/sf/antcontrib/antlib.xml">
    <classpath>
    <pathelement location="${basedir}/lib/ant-contrib-1.0b3.jar"/>
  </classpath>
  </taskdef>

  <target name="clean">
    <delete includeemptydirs="true">
      <fileset dir="Results" excludes="**/PerformanceCharts.html"/>
    </delete>
  </target>

  <target name="run-jmeter" depends="clean">
    <taskdef name="jmeter" classpath="${basedir}/jakarta-jmeter-2.5.1/extras/ant-jmeter-1.1.0.jar"
             classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>

    <jmeter
            jmeterhome="${basedir}/jakarta-jmeter-2.5.1"
            resultlog="${basedir}/Results/JMeterResults.xml">
      <testplans dir="${basedir}/TestPlan" includes="*.jmx"/>
      <property name="request.threads" value="300"/>
      <property name="request.loop" value="10"/>
    </jmeter>
  </target>

  <target name="create-reports" depends="run-jmeter">
    <xslt
            in="${basedir}/Results/JMeterResults.xml"
            out="${basedir}/Results/JMeterResults.html"
            style="${basedir}/jakarta-jmeter-2.5.1/extras/jmeter-results-detail-report_21.xsl"/>
  </target>

  <target name="create-graphs" depends="create-reports">
    <ac:for param="reportType" list="TimesVsThreads,ResponseTimesOverTime,ThreadsStateOverTime">
      <sequential>
        <java jar="${basedir}/jakarta-jmeter-2.5.1/lib/ext/CMDRunner.jar" fork="true">
          <arg value="--tool"/>
          <arg value="Reporter"/>
          <arg value="--generate-png"/>
          <arg value="${basedir}/Results/@{reportType}.png"/>
          <arg value="--input-jtl"/>
          <arg value="${basedir}/Results/JMeterResults.xml"/>
          <arg value="--plugin-type"/>
          <arg value="@{reportType}"/>
          <arg value="--width"/>
          <arg value="800"/>
          <arg value="--height"/>
          <arg value="600"/>
        </java>
      </sequential>
    </ac:for>
  </target>

  <target name="main" depends="clean,run-jmeter,create-reports,create-graphs"/>
</project>


I have added an additional target for deleting the test result files as jmeter would append subsequent test results to the previous test results in the file, if file is not deleted.

You would also notice that there is a target named “create-graphs”, this uses the CMDRunner.jar which is a command line utility shipped with jmeter-plugins. This tool has helped me to re-create the graphs which i referred to in part 1 of the blog. It parses the xml result created by the ant jmeter task and creates png images for each of the graphs specified.

This graph images can then be embedded into an html page to be displayed in teamcity.

I hope this helps someone.

Posted in Software Testing | Tagged , , , | 7 Comments

Performance Testing experience using Ant and Jmeter – Part 1

Jmeter is a well known open source performance/load testing tool and to be fair it does a lot of stuff really well.
if you wants to do some quick performance testing without a whole lots of infrastructure around it then it is great.

I started out my task with jmeter with some objectives:

  • A tool that i could integrate into a CI tool such as teamcity
  • Meaningful graphs that could be easily interpreted by any one in the team
  • Able to integrate the graphs into teamcity
  • Able to monitor the performance of the website under test.
  • After doing an initial round of tests, i didnt like the graphs which were being produced by jmeter and i search for plug-ins to enable me plot better graphs.
    I came across jmeter-plugins which is an awesome collection of jmeter plugins. Instructions to install can be found on this page.

    Whilst, there are lots of usefuil graphs bundled within this plugin, i found the following graphs to be most useful

  • Active Thread Over Time
  • Response Times Over Time
  • Response Times versus Thread.
  • I could easily put “Active Thread Over Time” and “Response Times Over Time” to get a complete picture of my performance results solely based on response times.
    And that was good enough for me.

    Also jmeter only allow you to specify a ramp up period and the maximum number of thread you want to execute your tests with. I desperately wanted to be able to specify a ramp down time as well and fortunately, the Ultimate Thread Group which ships with the jmeter-plugins solves that problem for me as well.

    Having jmeter-plugins installed along side with my installation of jmeter helped to get the best out of jmeter, and i was able to specify my load more realistically as well as having clear and better graphs.

    See below.

    Response Time Over Time

    Thread State Over Time

    Time Vs Thread

    In the next blog post i would discuss how i have integrated jmeter with ant, enabling me to run jmeter in command line

    Posted in Software Testing | Tagged , , , , | Leave a comment

    Screenshare tool for my trainings

    Occasionally i invite people round to my house for testing or automation training which sometimes i find inconvenient.
    In other situation, someone on other side of town need my help and i cant.

    I have recently come across the tool – join.me and it is awesome, it allow mwe to share my screen with people and you can give them control ove your machine as well.

    I think this is great !!!!!!

    And guess what it is a free tool, there is a pro-version which is a paid version, but i think the free one satisfy my need. One more tool for my testing and automation coaching.

    Enjoy!!!!!

    Posted in Software Testing | Leave a comment