Benchmarking Web Socket servers with wsbench

September 24 2010

Web Sockets are gaining traction as a realtime full-duplex communication channel, with several leading browsers (Chrome 5, Safari 5, Firefox 4, etc) having implemented support for some flavor of the protocol. Server support exists, but is not widespread, and is (entirely?) limited to specialized servers, with Socket.IO (based on NodeJS) being perhaps the most well-known. Finally, there appear to be no tools to do load or other testing on Web Socket services.

Enter wsbench, a benchmarking tool for draft76 Web Socket servers featuring

• Ability to generate a high degree of load using a single client process.
• Easily change the number and rate of connections and number and size of messages to send/receive with each connection.
• Ability to target a specific port, path, and Web Socket protocol in the target server.
• The core request/session engine is easily scriptable using JavaScript.

You'll find that wsbench is quite easy to use.

Here, we open and close 1000 connections to a Web Socket server running on localhost, port 8000. We generate 50 connection requests per second.

    % time ./wsbench -c 1000 -r 50 ws://localhost:8000
    Success rate: 100% from 1000 connections

    real    0m20.379s
    user    0m1.340s
    sys     0m0.517s

We can see a few interesting things in the output above.

• The error rate is tracked by wsbench and reported at the end. Errors include failure to open a connection, failure to send a message, failure to close the connection cleanly, etc.
• The wsbench process running on a late-model Mac Book is able to generate this load using less than 10% of the CPU.

The above benchmarking run only tested establishing connections. We didn't send (or receive) any messages. By passing the -m 5 and -s 128 options to wsbench, we can send 5 128 byte messages per connection. Invoke wsbench with the -h option to see full usage:

usage: wsbench [options] <url>

Kick off a benchmarking run against the given ws:// URL.

We can execute our workload in one of two ways: serially, wherein each
connection is closed before the next is initiated; or in parallel, wherein
a desired rate is specified and connections initiated to meet this rate,
independent of the state of other connections. Serial execution is the
default, and parallel execution can be specified using the -r <rate>
option. Parallel execution is bounded by the total number of connections
to be made, specified by the -c option.

Available options:
  -c, --num-conns NUMBER   number of connections to open (default: 100)
  -h, --help               display this help
  -m, --num-msgs NUMBER    number of messages per connection (default: 0)
  -p, --protocol PROTO     set the Web Socket protocol to use (default: empty)
  -r, --rate NUMBER        number of connections per second (default: 0)
  -s, --msg-size NUMBER    size of messages to send, in bytes (default: 32)
  -S, --session FILE       file to use for session logic (default: None)

Beyond performance testing, it can be useful to run load against a server continually to discover any resource leaks. This can be done by passing the -c 0 option -- 0 connections is interpreted as a special "infinite" value. For example -c 0 -r 100 will open/close 100 connections per second indefinitely (or until wsbench is terminated with a ^C).

For information on how to script the core of wsbench, take a look at the Session Scripting section in the project page on GitHub. Because this tool is written entirely in JavaScript (using NodeJS), you'll find that its easily extensible using a familiar language.