MemcacheQueue: a pure memcached queue

MemcacheQueue is a very simple message queue which relies solely on memcached and requires no extra servers. I got the idea when watching the Patterns in Distributed Computing talk by Mike Perham at RubyConf 2008. He also has a queue based on memcached but it requires some extras as well.

I wanted to come up with something even simpler and with less dependencies. MemcacheQueue clocks in at 90 lines of ruby, requires only memcached to function, but still ensures that there are no duplicate or lost messages. It is made possible by the atomic incr command of memcached. Thanks to Costan for pointing this out to me and for reviewing my code.

Here is an example of a process which adds messages to the queue based on user input:

mcq = MemcacheQueue.new('writer', 'localhost:1234')
mcq.create_queue

trap("INT") { mcq.shutdown }

while msg = gets
  begin
    mcq.add_msg(msg)
  rescue MemcacheQueue::ShutdownError
    puts "shutdown"
    break
  end
end

And here’s an example of a worker reading messages from the queue:

mcq = MemcacheQueue.new('reader', 'localhost:1234')

trap("INT") { mcq.shutdown }

loop do
  begin
    msg = mcq.get_msg
    puts "message received: #{msg}"
  rescue MemcacheQueue::ShutdownError
    puts "shutdown!"
    break
  end
end

You can have any number of workers writing and reading from the queue without having to worry about lost or duplicate messages (except in extreme circumstances like server failure or hard process kill, see the source for comments about this). There are a few caveats though. Firstly, you must make sure you have enough memory in your memcached servers to store all your unread messages. Secondly, if you are going to stop the readers, you will need to make sure you start up the same number (or more) of readers with the same names (name is first argument to initialize) to prevent losing messages.

Performance

Reads are about 5% slower than Starling while writes are usually around 2-4x faster. Things could probably be sped up more by switching to the memcached gem instead of memcache-client.

More examples…
In the examples directory.

All comments and bug reports welcome.

---

Possibly related posts: (automatically generated)

• memcached as simple message queue
• Microsoft Message Queue Example and Overview
• Working with Worker Role in Windows Azure