coderrr

October 21, 2008

When to use readpartial instead of read in Ruby

Filed under: network, ruby — Tags: , — coderrr @ 6:34 pm

In all my network related ruby projects I end up using this simple loop:

while buf = (socket.readpartial(1024) rescue nil)
  (data||="") << buf
  # process data here
end

First let me explain why in general you want to use readpartial instead of read for network IO in ruby. I’m pretty sure people get confused over read/readpartial a lot because ruby’s read doesn’t work the same as BSD-sockets’ read which you find everywhere in C code and other languages. readpartial is actually much closer to the BSD-sockets’ read. When you call socket.read(length) it will block until either the socket is closed from the other side or it has read length bytes from the socket. In some cases this is ok, but a lot of the time it’s not what you want.

Take for example a protocol in which you receive instructions from a server. Each instruction can be from 10 to 100 bytes long and the instructions come approximately once a minute. In this scenario if you did socket.read(100) you could potentially have to wait up to 10 minutes before that method returned and you could process the instruction(s).

readpartial to the rescue. It will return immediately with whatever data is available on the socket. If there is no data available it will wait until any amount of data becomes available and return it immediately. This means if a 10 bytes instruction came from the server and you did socket.readpartial(100) you would get the 10 byte instruction back right away.

The one thing that kinda sucks about readpartial is that instead of returning nil on an EOF it raises an EOFError. This doesn’t really help us create beautiful code. Thus the: (sock.readpartial(1024) rescue nil) to catch the error and convert it into a nil. The problem with this is it will rescue any error not just EOFError, which is not optimal. I’ve often misspelled readpartial and the rescue nil will swallow the NameError, making it hard to figure out the problem. So I decided to do some refactoring and come up with a better solution for this loop I have to use all the time.

Here’s what I came up with:

class IO
  def while_reading(data = nil)
    while buf = readpartial_rescued(1024)
      data << buf  if data
      yield buf  if block_given?
    end
    data
  end

  private

  def readpartial_rescued(size)
    readpartial(size)
  rescue EOFError
    nil
  end
end

Now I can write:

socket.while_reading(data = "") do
  # process data here
end
# the data var will still exist after we exit the loop's scope

This saves me a line, only rescues from EOFError, and still looks decent, probably even nicer than what I had before. Alternatively, if I don’t want it to automatically append the incoming data for me I can do:

socket.while_reading do |buf|
  # process each buf here
end

Anyone have an idea for a better name than while_reading?

Also, just a note on when you should use read instead of readpartial. If you simply want to read all the data from a socket until it is closed then socket.read will suit you fine. There are probably some other scenarios but I’ll leave those up to you to figure out.

6 Comments »

  1. Nice use of block semantics. Good write up on the general problem as well.

    Comment by Julio Capote — October 21, 2008 @ 11:45 pm

  2. socket.drain ?

    Comment by Leif Eriksen — October 22, 2008 @ 2:39 am

  3. haha i like socket.drain

    Comment by coderrr — October 22, 2008 @ 7:11 am

  4. [...] When to use readpartial instead of read in Ruby In all my network related ruby projects I end up using this simple loop: [...]

    Pingback by Top Posts « WordPress.com — October 23, 2008 @ 12:17 am

  5. r, _, _ = IO.select([socket], nil, nil, 0)
    data = r && r.first.read_nonblock(1024)

    will not raise an exception (which is expensive if you’re trying to do this a couple of thousand times per second)

    Comment by Theo (@iconara) — November 6, 2013 @ 11:07 am

    • sorry, should have been r && r.first && r.first.read_nonblock(1024)

      you don’t have to write it on one line, of course…

      Comment by Theo (@iconara) — November 6, 2013 @ 11:08 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 28 other followers

%d bloggers like this: