coderrr

January 17, 2009

MemcacheQueue: a pure memcached queue

Filed under: ruby — Tags: — coderrr @ 1:25 am

Shameless Plug: Gain Uncensored Access to the Internet with a VPN Service.

MemcacheQueue is a very simple message queue which relies solely on memcached and requires no extra servers. I got the idea when watching the Patterns in Distributed Computing talk by Mike Perham at RubyConf 2008. He also has a queue based on memcached but it requires some extras as well.

I wanted to come up with something even simpler and with less dependencies. MemcacheQueue clocks in at 90 lines of ruby, requires only memcached to function, but still ensures that there are no duplicate or lost messages. It is made possible by the atomic incr command of memcached. Thanks to Costan for pointing this out to me and for reviewing my code.

Here is an example of a process which adds messages to the queue based on user input:

mcq = MemcacheQueue.new('writer', 'localhost:1234')
mcq.create_queue

trap("INT") { mcq.shutdown }

while msg = gets
  begin
    mcq.add_msg(msg)
  rescue MemcacheQueue::ShutdownError
    puts "shutdown"
    break
  end
end

And here’s an example of a worker reading messages from the queue:

mcq = MemcacheQueue.new('reader', 'localhost:1234')

trap("INT") { mcq.shutdown }

loop do
  begin
    msg = mcq.get_msg
    puts "message received: #{msg}"
  rescue MemcacheQueue::ShutdownError
    puts "shutdown!"
    break
  end
end

You can have any number of workers writing and reading from the queue without having to worry about lost or duplicate messages (except in extreme circumstances like server failure or hard process kill, see the source for comments about this). There are a few caveats though. Firstly, you must make sure you have enough memory in your memcached servers to store all your unread messages. Secondly, if you are going to stop the readers, you will need to make sure you start up the same number (or more) of readers with the same names (name is first argument to initialize) to prevent losing messages.

Performance

Reads are about 5% slower than Starling while writes are usually around 2-4x faster. Things could probably be sped up more by switching to the memcached gem instead of memcache-client.

More examples…
In the examples directory.

All comments and bug reports welcome.

3 Comments »

  1. [...] is pretty clever… Might have to code this up in PHP… memcachequeue-a-pure-memcached-queue Subscribe to the comments for this post Posted on : Jan 16 2009 Posted under Random Thoughts [...]

    Pingback by CodeWord: Apokalyptik » Blog Archive » A Pure Memcached Queue — January 17, 2009 @ 5:05 am

  2. Seems to break on creating a queue. The first line of the first example fails with:

    Error on get state, #, retrying

    I’ve got memcached running on the port in question… Any ideas what might be going on?

    Comment by Joseph Weissman — March 24, 2010 @ 3:29 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 28 other followers

%d bloggers like this: