Monday, April 28, 2014

Stress Testing with Siege and Bombard, know your limits

I couldn't find a clear quick intro on getting siege and bombard in action, so I'm writing one here.

Siege is a load testing and benchmarking utility that has been available for quite a while, It allows you to hit your web application on a specific url (or a set of urls in a file) with specific concurrency and size settings.  siege summarizes the measures of the test outcome including:

  • Transaction rate (requests/sec)
  • Actual concurrency (even if you hit with 200 concurrent connections, your server might be responding with just 80)
  • Average, longest and shortest response time
example:

triggering 200 concurrent users, each hitting the url twice

siege -c200 -r2 http://www.modsaid.com/

summary:

Transactions:         400 hits
Availability:      100.00 %
Elapsed time:       38.44 secs
Data transferred:        0.26 MB
Response time:       10.09 secs
Transaction rate:       10.41 trans/sec
Throughput:        0.01 MB/sec
Concurrency:      104.96
Successful transactions:         400
Failed transactions:           0
Longest transaction:       34.19
Shortest transaction:        0.52


Siege is available in most repos and can be installed directly on ubuntu/debian via

apt-get install siege

However, in order for POST requests to work normally, I'd recommend installing the latest version (currently 3.0.5) from source

wget http://www.joedog.org/pub/siege/siege-3.0.5.tar.gz
tar zxvf siege-3.0.5.tar.gz
cd siege-3.0.5/
./configure && make && sudo make install

Similar tools exist, mainly ab of apache,  but ab is very basic compared to the flexibility of siege

Using siege is great for a single test, but you'll need to run it several times with different parameters in order to be able to understand your limits. and there comes the handy wrapper, bombard.

Bombard is a wrapper for siege that allows you to start your load testing with certain parameters and increase the load (concurrency) incrementally. and it draws charts that let's you see how your server reacts.  plotting makes things more clear and gives you a better sense of your server current limits.


Installing bombard requires dependencies:
  • GD2 perl module
    sudo apt-get install libgd-graph-perl
  • Chart-2.x
    wget http://www.cpan.org/authors/id/C/CH/CHARTGRP/Chart-2.4.6.tar.gz
    tar zxf  Chart-2.4.6.tar.gz && cd Chart-2.4.6 && \
    perl Makefile.PL && make && make test  && sudo make install
Then you can install bombard from the source

git clone git@github.com:allardhoeve/bombard.git
cd bombard
./configure
make && sudo make install

Now you can check all options through bombard -h

let's try experimenting out site:
Let's start with:
  • starting concurrency 50
  • increment 10
  • runs 10 times  (so it will try 50, 60, 70, 80, ..., 140)
  • each run will take 1 minute
Note: Bombard expects the absolute path of the file, it will not work with relative paths

bombard  -f /home/me/bombard-test-urls.txt -i10 -r10 -s50 -t1







From both graphs we can see that the transactions rate increases as we increase test concurrency. but it nearly saturates at concurrency 80.  The response time starts at nearly half a second, and remains so as we increase the load. until we reach 80 concurrency.  then it increases. the increase here is not from our web application itself, It is actually from having the requests buffer at the web server and application.

The two graphs indicate that our setup is able to handle up to 80 concurrent requests decently. then the performance will degrade as the load increases.


If you find this useful, please share your experience via comments, or get it touch with @modsaid



6 comments:

Aaron Fay said...

Thanks for this post, I found it a little late, but it still helped.

It's worth noting that siege will take a relative path to the URL file, while siege appears to need an absolute path.

Thanks again.

Tony Andriano said...

i managed to work my way through the dependencies on os/x, but the charts don't work. here's what happens after a successful run:

Generating graphs...
No min_val or max_val is defined at /usr/local/bin/siegechart line 51.
No min_val or max_val is defined at /usr/local/bin/siegechart line 51.
No min_val or max_val is defined at /usr/local/bin/siegechart line 51.
No min_val or max_val is defined at /usr/local/bin/siegechart line 51.

any ideas?

Tony Andriano said...

oddly enough, the problem was the siege logfile path. it was entered as $(HOME)/var/siege.log, which didn't work in bombard. bombard also doesn't check that the log file can be opened, which is very bad obviously. if you put "or die(...)" after "OPEN $logf" in bombard, it will show the error.

i figured this out by working backward from siegechart. i did "print Dumper \@data" in siegechart to look at what it was trying to graph, but the array was empty. i eventually got back to bombard to see that it wasn't making a copy of siege.log because it couldn't open it.

i hope this helps someone. what a pain it is when people fail to write extremely simple error handling. i wasted 4 hours because someone was too lazy to write "or die $!;". ugh.

Mohamed Omer said...

i figured this out by working backward from siegechart. i did "print Dumper \@data" in siegechart to look at what it was trying to graph, but the array was empty. i eventually got back to bombard to see that it wasn't making a copy of siege.log because it couldn't open it.

Seo Job Training

mahmoud said...

@Mohammed Omar

the data being plotted is also generated as csv (or tsv) next to the charts themselves.

surath wije said...

[root@server bombard-upstream]# bombard -f /etc/siege/urls.txt -i1 -r1 -s10 -t1
The run directory must already exist

My urls.txt is as follws:

[root@server siege]# ls
siegerc urls.txt

siege is installed as follows:
[root@server etc]# ls siege*
siege:
siegerc siegerc_good urls.txt

siege-3.1.3:

What am I missing ?