Stress Testing with Siege and Bombard, know your limits

I couldn't find a clear quick intro on getting siege and bombard in action, so I'm writing one here.

Siege is a load testing and benchmarking utility that has been available for quite a while, It allows you to hit your web application on a specific url (or a set of urls in a file) with specific concurrency and size settings.  siege summarizes the measures of the test outcome including:

  • Transaction rate (requests/sec)
  • Actual concurrency (even if you hit with 200 concurrent connections, your server might be responding with just 80)
  • Average, longest and shortest response time
example:

triggering 200 concurrent users, each hitting the url twice

siege -c200 -r2 http://www.modsaid.com/

summary:

Transactions:         400 hits
Availability:      100.00 %
Elapsed time:       38.44 secs
Data transferred:        0.26 MB
Response time:       10.09 secs
Transaction rate:       10.41 trans/sec
Throughput:        0.01 MB/sec
Concurrency:      104.96
Successful transactions:         400
Failed transactions:           0
Longest transaction:       34.19
Shortest transaction:        0.52


Siege is available in most repos and can be installed directly on ubuntu/debian via

apt-get install siege

However, in order for POST requests to work normally, I'd recommend installing the latest version (currently 3.0.5) from source

wget http://www.joedog.org/pub/siege/siege-3.0.5.tar.gz
tar zxvf siege-3.0.5.tar.gz
cd siege-3.0.5/
./configure && make && sudo make install

Similar tools exist, mainly ab of apache,  but ab is very basic compared to the flexibility of siege

Using siege is great for a single test, but you'll need to run it several times with different parameters in order to be able to understand your limits. and there comes the handy wrapper, bombard.

Bombard is a wrapper for siege that allows you to start your load testing with certain parameters and increase the load (concurrency) incrementally. and it draws charts that let's you see how your server reacts.  plotting makes things more clear and gives you a better sense of your server current limits.


Installing bombard requires dependencies:
  • GD2 perl module
    sudo apt-get install libgd-graph-perl
  • Chart-2.x
    wget http://www.cpan.org/authors/id/C/CH/CHARTGRP/Chart-2.4.6.tar.gz
    tar zxf  Chart-2.4.6.tar.gz && cd Chart-2.4.6 && \
    perl Makefile.PL && make && make test  && sudo make install
Then you can install bombard from the source

git clone git@github.com:allardhoeve/bombard.git
cd bombard
./configure
make && sudo make install

Now you can check all options through bombard -h

let's try experimenting out site:
Let's start with:
  • starting concurrency 50
  • increment 10
  • runs 10 times  (so it will try 50, 60, 70, 80, ..., 140)
  • each run will take 1 minute
Note: Bombard expects the absolute path of the file, it will not work with relative paths

bombard  -f /home/me/bombard-test-urls.txt -i10 -r10 -s50 -t1






From both graphs we can see that the transactions rate increases as we increase test concurrency. but it nearly saturates at concurrency 80.  The response time starts at nearly half a second, and remains so as we increase the load. until we reach 80 concurrency.  then it increases. the increase here is not from our web application itself, It is actually from having the requests buffer at the web server and application.

The two graphs indicate that our setup is able to handle up to 80 concurrent requests decently. then the performance will degrade as the load increases.


If you find this useful, please share your experience via comments, or get it touch with @modsaid



Comments

Unknown said…
Thanks for this post, I found it a little late, but it still helped.

It's worth noting that siege will take a relative path to the URL file, while siege appears to need an absolute path.

Thanks again.
12:49 PM
Anonymous said…
i managed to work my way through the dependencies on os/x, but the charts don't work. here's what happens after a successful run:

Generating graphs...
No min_val or max_val is defined at /usr/local/bin/siegechart line 51.
No min_val or max_val is defined at /usr/local/bin/siegechart line 51.
No min_val or max_val is defined at /usr/local/bin/siegechart line 51.
No min_val or max_val is defined at /usr/local/bin/siegechart line 51.

any ideas?
3:43 AM
Anonymous said…
oddly enough, the problem was the siege logfile path. it was entered as $(HOME)/var/siege.log, which didn't work in bombard. bombard also doesn't check that the log file can be opened, which is very bad obviously. if you put "or die(...)" after "OPEN $logf" in bombard, it will show the error.

i figured this out by working backward from siegechart. i did "print Dumper \@data" in siegechart to look at what it was trying to graph, but the array was empty. i eventually got back to bombard to see that it wasn't making a copy of siege.log because it couldn't open it.

i hope this helps someone. what a pain it is when people fail to write extremely simple error handling. i wasted 4 hours because someone was too lazy to write "or die $!;". ugh.
9:56 AM
Unknown said…
i figured this out by working backward from siegechart. i did "print Dumper \@data" in siegechart to look at what it was trying to graph, but the array was empty. i eventually got back to bombard to see that it wasn't making a copy of siege.log because it couldn't open it.

Seo Job Training
2:48 AM
mahmoud said…
@Mohammed Omar

the data being plotted is also generated as csv (or tsv) next to the charts themselves.
11:11 PM
Unknown said…
[root@server bombard-upstream]# bombard -f /etc/siege/urls.txt -i1 -r1 -s10 -t1
The run directory must already exist

My urls.txt is as follws:

[root@server siege]# ls
siegerc urls.txt

siege is installed as follows:
[root@server etc]# ls siege*
siege:
siegerc siegerc_good urls.txt

siege-3.1.3:

What am I missing ?

9:05 AM
Post a Comment

Popular posts from this blog

Success of Startups

Entrepreneurship has become one of the one of the most interesting topic in the business side in any country in the world. While it is digging its way in the middle east, there is some sort of confusion and a lot of aspects related to that topic. A Startup can act and advance to tackle business opportunities in a way that competes with large enterprises. and most of the times, they need to be supported by Investors to back them up. One of the things that started to really bother me in the culture is the definition of success of a startup. No one denies that a decent revenue generating startup is a successful one. However, some people believe that just raising an investment is a success.... umm, debatable. I believe we can make some kind of analogy with some entities that showed up in the past decade like biznas and skybiz (I'm sure some of you will remember those names).   they used an invalid scheme summarized as: People can register and join the network for a fee (as an i
Read more

Prime Numbers Generator

Image
did u know that 34136029 is a prime number?!! well.. that's what i recently realized :) I thought of writing a piece of code for that purpose.. wondering how far i can get. The idea of the code is simple We know that 2 and 3 are prime numbers the code checks every following odd numbers to be prime or not. To verify a prime number, the number divisibility by the prime numbers generated so far is checked. (29 is a prime number.. because it id not divisible by 2, 3, 5, 7, 11, 13, 17, 19 nor 23) We only need to verify divisibility against the numbers smaller than or equal to the root of the number in hand. (We needn't check the divisibility of 33 with 11, as we already must have found out it is divisible by 3).. I did not want to go with the mathematical root evaluation as i am not sure about its load. I used bits handling instead. the root of a number uses at most half of the number of bits that the original number uses. i only made a naive java implementation so far. downloa
Read more

Converting a batch of files of csv format to xls or xlsx

While working with data, we tend to deal a lot with CSV files. It is more efficient and trivial to manipulate csv data either through command line tools or through programming languages like ruby. However, occasionally we need to convert files to MS Excel format before sending them to 3rd party. Gnumeric is a wonderful opensource spreadsheet program.  It contains a command line convertion tool called ssconvert that can be used to perform direct conversion among lots of supported formats easily from the command line. For example, conversion from csv to xlsx can be done via ssconvert  file.csv file.xlsx OR ssconvert  file.csv --export-type=Gnumeric_Excel:xlsx It can be installed in ubuntu via apt-get install gnumeric In order to convert lots of files, I wrote a simple bash script that will process all csv files in the current directory and generate the corresponding xlsx files.  The script came pretty handy in data preparation. I believe i can be helpful to many people
Read more