coderrr

December 20, 2008

Automatically flushing redirected or piped stdout

Filed under: c, linux — Tags: , — coderrr @ 12:12 am

Whenever you pipe or redirect a program’s stdout it will automatically be set to buffered mode. Meaning the output won’t be written until a certain amount of bytes have been buffered. I think the default on most systems is 4096.

For example:

ruby -e 'loop{puts 1; sleep 1}' | cat

You wouldn’t see any output for 4096 seconds. This can be annoying if you have a long running program whose output you want to tee to a file. (tee writes the output to a file while at the same time printing it to the console). If you just pipe the output to tee you’ll see the output come in chunks. Some programs come with options to automatically flush stdout after every write. Some scripting languages allow you to do this as well

# ruby
STDOUT.sync = true
# perl
$| = 1

And in C you can set the buffering of a stream with:

# unbuffered
setvbuf(stdout, NULL, _IONBF, 0)
# line buffered
setlinebuf(stdout)

But if you can’t modify the program or script a generic solution is required.

expect_unbuffer, or just unbuffer depending on where you get it, does just that. I’m not 100% clear on exactly how it works but it’s something like this. It sets up a pseudo terminal (pty) which it redirects your program’s output to. Since the program thinks it’s writing to a console it remains in line buffered mode. This way even after you redirect or pipe the programs output, it will be flushed as it is being written.

How to get expect_unbuffer varies on the system you use. On ubuntu 8.10 apt-get install expect-dev is enough. On others which don’t include it in the expect package you may need to install the expect-dev package and then download/compile unbuffer.c with:

sudo apt-get install expect-dev
wget http://expect.nist.gov/example/unbuffer.c
sudo gcc -o /usr/bin/expect_unbuffer unbuffer.c -I$(dirname $(find /usr/include -name expect.h | head -n 1)) -lexpect

Here’s the man page.

If anyone knows of a better or simpler way to do this generically I’d like to hear it.

4 Comments »

  1. Any idea how to do this in bash shell

    Comment by Anonymous — August 16, 2011 @ 1:53 pm

  2. A better way is to use the stdbuf command to force a command into line buffer mode, e.g:

    stdbuf -oL ruby -e ‘loop{puts 1; sleep 1}’ | cat

    For more info see ‘man stdbuf’.

    Props to this fourum post which I found in the same google search as you post: http://compgroups.net/comp.unix.shell/bash-how-to-flush-the-output-from-command-tee

    Comment by robconnolly — January 24, 2012 @ 1:15 am

    • awesome, thanks for that

      Comment by coderrr — January 24, 2012 @ 2:08 am

      • really helped!

        Comment by Anonymous — October 25, 2013 @ 3:25 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 27 other followers

%d bloggers like this: