May 6, 2008

Facebook Chat API

Filed under: facebook, network, ruby — Tags: , , — coderrr @ 7:37 pm

Shameless Plug: Browse anonymously to protect your browsing habits from data tracking sites with a VPN Service.

Update: It’s been quite a while since I wrote this post and I don’t think the code below even works anymore because I haven’t kept up with Facebook’s API changes. So I’ve put the code up on github so anyone can fork it and make fixes if they like.

Update: Simon just mentioned that when using facebook’s new profile layouts this code will break. You can find the fix in his comment. Also note, as he says, you only need this fix if you’ve changed to facebook’s new profile layouts.

Update: Ryan just notified me that facebook has made a change and you must now pass force_render=1 to presence/update.php to retrieve the buddy list. I’ve updated the example client below.

Update: Turns out Adium utilized my post in adding Facebook chat to their client, cool!

Update: Facebook recently made a breaking change to their API to require post_form_id to be passed to presence/update.php. I’m assuming they did this to prevent CSRF attacks against it. I’ve updated the client code below to reflect these changes.

Facebook Chat was released recently and I was interested in looking under the hood at how its protocol works. It turned out to be simpler than I expected…

Receiving Messages

Facebook uses a Comet and JSON approach to receive incoming messages. Comet is a technique where you open a long running HTTP connection to a server (usually AJAX). The connection stays open until the server has something to tell you at which point it sends the data through the connection and the connection can be closed (or in some cases remain open). JSON is just a way of representing data in Javascript. Let’s look at how it works.

A connection is opened to: First let’s look at the url and what the different placeholders represent.

ANYTHING can be anything, Facebook has setup *.channel[NUM] as a catch-all subdomain. So no matter what you put there it will resolve to the same IP. The ANYTHING in the path can also be anything. I assume both the ANYTHING values are used to prevent the browser from caching these requests.

NUM represents the channel number you are using. It appears there are 20 channels (01-20), as that range of subdomains resolve and the rest don’t. I am assuming that a “channel” is just one of Facebook’s comet interface servers. Every time you log in you are assigned to a specific channel and it doesn’t change throughout your session. It may also be fixed per user, I’m not sure about this though.

UID is just your user id. SEQ is the sequence number you are requesting from the server. If you give an out of range seq number the server will tell you what the latest seq number is so that you can request it. If you give an old seq number the server will show you the (old) messages that correspond with that number. Requesting will immediately respond telling me I should re-request with a seq number of 0 (or whatever my current seq number is) since the seq number -1 is never valid.

// request
for (;;);{"t":"refresh", "seq":0}

After we build the correct URL and make the connection, it will just “sit there”. This is how Comet connections work. The connection will remain inactive in a “requesting” state until someone sends us a message. If a message is not received in less than a minute (approx 55 seconds) Facebook will close the connection and instruct you to reopen it. I assume they do this to get around browser timeout issues. If a message is received you will get the message data through the connection which is then closed. After a message is received the seq number should be incremented for the next request. After either a message is received or the connection times out a new connection will immediately be established.

// request times out
for (;;);{"t":"continue"}
// request receives a message!
for (;;);{"t":"msg","c":"p_MYID","ms":[{"type":"msg","msg":{"text":"yo","time":1209557234412,"clientTime":1209557232415,"msgID":"4177168544"},"from":FRIENDSID,"to":MYID,"from_name":"Myfriends Name","to_name":"My name","from_first_name":"Myfriends","to_first_name":"My"}]}
// request for the next message

Sending Messages

Sending messages is a lot simpler. All you need to do is make a POST request to with a few simple values: msg_text with the text of the message, to with the uid of the recipient, msg_id with a random unique number id for the message, client_time with the current time in microseconds, and post_form_id which can be found as a hidden form value on any facebook page. Facebook will respond with some JSON indicating there was no error and the message will be sent!

// POST request to:
for (;;);{"error":0,"errorSummary":"","errorDescription":"No error.","payload":[]}

Buddy List
The list of your friends who are online is polled about every 3 minutes with a POST request to The ‘userInfos’ hash contains the actual buddy list. Some extra info is provided to let you know how the buddy list has changed since the last request.

// POST request to
for (;;);{"error":0,"errorSummary":"","errorDescription":"No error.","payload":{"buddy_list":{"listChanged":true,"availableCount":1,"nowAvailableList":{"UID1":{"i":false}},"wasAvailableIDs":[],"userInfos":{"UID1":{"name":"Buddy 1","firstName":"Buddy","thumbSrc":"http:\/\/\/pics\/q_default.gif","status":null,"statusTime":0,"statusTimeRel":""},"UID2":{"name":"Buddi 2","firstName":"Buddi","thumbSrc":"http:\/\/\/pics\/q_default.gif","status":null,"statusTime":0,"statusTimeRel":""}},"forcedRender":true},"time":1209560380000}}


Notifications are polled through the same mechanism as the buddy list. You need to pass &notifications=1 to update.php to receive info about them. Other than that I won’t be covering them here.

The Fun Part
So here’s a very simple implementation of a Facebook Chat client written in Ruby. It will require you have the json and mechanize gems installed.

require 'rubygems'
require 'mechanize'
require 'json'
require 'ostruct'
require 'pp'

class FacebookChat
  def initialize(email, pass); @email, @pass = email, pass; end

  def login
    @agent =
    @agent.user_agent_alias = 'Windows IE 7'
    f = @agent.get("").forms.first"email").value = @email"pass").value = @pass
    body = @agent.get("").body

    # parse info out of facebook home page
    @uid = %r{<a href=".+?/profile.php\?id=(\d+)" class="profile_nav_link">Profile</a>}.match(body)[1].to_i
    @channel = %r{"channel(\d+)"}.match(body)[1]
    @post_form_id = %r{<input type="hidden" id="post_form_id" name="post_form_id" value="([^"]+)}.match(body)[1]

  def wait_for_messages
    determine_initial_seq_number  unless @seq
      json = parse_json @agent.get(get_message_url(@seq)).body
    end  while json["t"] == "continue"   # no messages yet, keep waiting
    @seq += 1

    json["ms"].select{|m| m['type'] == 'msg'}.map do |msg|
      info = msg.delete 'msg'
      msg['text'] = info['text']
      msg['time'] =['time']/1000) msg
    end.reject {|msg| msg.from == @uid }  # get rid of messages from us

  def send_message(uid, text)
    r = "", 
      'msg_text' => text, 
      'msg_id' => rand(999999999),
      'client_time' => (*1000).to_i,
      'to' => uid,
      'post_form_id' => @post_form_id

  def buddy_list
    json = parse_json("", 
                        'buddy_list' => 1, 'force_render' => 1, 'post_form_id' => @post_form_id, 'user' => @uid).body)
    json['payload']['buddy_list']['userInfos'].inject({}) do |hash, (uid, info)|
      hash.merge uid => info['name']


  def determine_initial_seq_number
    # -1 will always be a bad seq number so fb will tell us what the correct one is
    json = parse_json @agent.get(get_message_url(-1)).body
    @seq = json["seq"].to_i

  def get_message_url(seq)
  # get rid of initial js junk, like 'for(;;);'
  def parse_json(s)
    JSON.parse s.sub(/^[^{]+/, '')

if __FILE__ == $0
  fb =, ARGV.shift)

  puts "Buddy List:"
  pp fb.buddy_list

  Thread.abort_on_exception = true do
    puts usage = "Enter message as <facebook_id> <message> (eg: 124423 hey man wassup?) or type 'buddy' for buddy list"
    loop do
      case gets.strip
      when 'buddy' then pp fb.buddy_list
      when /^(\d+) (.+)$/
        uid, text = $1.to_i, $2
        fb.send_message(uid, text)
        puts usage

  # message receiving loop
  loop do
    fb.wait_for_messages.each do |msg|
      puts "[#{msg.time.strftime('%H:%M')}] #{msg.from_name} (#{msg.from}): #{msg.text}"

Run with: ruby chat.rb [email] [password]

April 30, 2008

Facebook Chat Protocol

Filed under: facebook, network, ruby — Tags: , , — coderrr @ 2:39 pm

Shameless Plug: Browse anonymously to protect your browsing habits from data tracking sites with a VPN Service.

moved here

March 12, 2008

Quickly creating lots of Facebook accounts with FireWatir

Filed under: facebook, ruby — Tags: , — coderrr @ 8:17 am

What you’ll need:
1) A catch all email address or as many email addresses as users you want to create (,, …) which should all forward to a single email inbox. I used Postfix to setup virtual aliases which forward to my gmail account.
2) Ruby
3) FireWatir

A few months ago I wrote this script to quickly register a lot of accounts on Facebook. I registered 100 users in probably 10 minutes. I haven’t really made use of too many of them yet, but they can be useful for a lot of things, especially testing Facebook Apps. The script uses FireWatir, a browser automation tool written in Ruby which allows you to control a Firefox browser.

After you launch the script, it will navigate to the register page and fill in all necessary values for you except for the CAPTCHA. It will put the focus on the CAPTCHA field so all you have to do is type out the captcha and then press 0 (zero) to let the script know you’ve finished. At this point it will submit the registration and move onto the next user.

The script takes a single parameter, a starting number to append to the base email address. For each successive user it will increment this number. You’ll also want to change the BASE_EMAIL constant to match whatever your email addresses look like.

After you’ve run the script you’ll have a bunch of validation emails from Facebook sitting in your inbox. You need to turn off cookies in your browser, then go through and click every link. After that, all your accounts should be new, valid facebook accounts.

Here’s the script:


BASE_EMAIL = "fbook.<NUM>"

require 'firewatir'

email_start = (ARGV.shift || '1').to_i

# use => 'facebook')  if you want to use a specific profile
f =
(email_start..1000).each do |email_num|
  f.text_field(:id, :name).value = "#{FIRST_NAMES.rand.capitalize} #{LAST_NAMES.rand.capitalize}"
  f.select_list(:id, :lifestage).value = '3'
  sleep 0.1
  f.text_field(:id, :reg_email__).value = BASE_EMAIL.sub(/<NUM>/, email_num.to_s)
  f.text_field(:id, :reg_passwd__).value = "aaaaaa"
    f.text_field(:id, :reg_passwd2__).value = "aaaaaa"
  rescue Exception
  f.select_list(:id, :birthday_month).value = (1..12).rand
  f.select_list(:id, :birthday_day).value = (1..28).rand
  f.select_list(:id, :birthday_year).value = (1945..1987).rand
  captcha = f.text_field(:id, :captcha_response)
  sleep 0.1  while captcha.value !~ /0$/
  captcha.value = captcha.value.sub(/0$/,'')
  f.checkbox(:id, :terms).click

  f.button(:id, :submit_button).click


class Range
  def rand
    Kernel.rand(self.end+1 - self.begin) + self.begin

class Array
  def rand

# name lists shamelessly ripped from some most-common-names website


February 29, 2008

Fixing GET/POST HTTP methods for Facebook applications in Rails

Filed under: bug, facebook, patch, rails — Tags: , , , — coderrr @ 4:58 am

If you’ve written a Facebook app you probably know that all requests get proxied to you as POSTs, regardless of what type of request the user made to Facebook. The reason they do this is most likely that many web servers have size limits on GET requests (I think ~1k). And since Facebook passes a user’s entire friends list on every request along with alot of other data it would be over the maximum size most of the time.

This can cause problem with routing especially with resources. Luckily Facebook passes the true request method to you in the POST data. So here’s a hack to convert the method of the request based on the original one passed as a parameter.

class << ActionController::AbstractRequest
  def new_with_facebook_request_method_hack(*a)
    returning new_without_facebook_request_method_hack(*a) do |request| 
      if request_method = request.request_parameters['fb_sig_request_method']
        request.instance_variable_set :@request_method, request_method.downcase.to_sym

  alias_method_chain :new, :facebook_request_method_hack

Throw this in wherever you want: environment.rb, a plugin, lib dir, etc.

Now you can do things like, request.get?, or use ActiveRecord resources which rely on the request method.

February 20, 2008

Problems with RFacebook and named routes and Rails 2.0

Filed under: bug, facebook, patch, rails — Tags: , , , — coderrr @ 2:06 pm

Synopsis: If you are using named routes with RFacebook and Rails 2.0 you should turn off route optimization by adding “config.action_controller.optimise_named_routes = false” to your Rails::Initializer block.

The problem?
RFacebook defines its own url_for method so that your links to Facebook canvas pages have the correct host and/or path prefixed to them ( The problem is, when you use named routes and positional arguments instead of a hash ( user_path(user) versus user_path(:id => user) ) Rails 2.0 does some optimization so that url_for never gets called. This bypasses RFacebook’s url_for so all those paths are screwed up.

The solution?
Turn off route optimization. Here was my first solution to that:

class ActionController::Routing::Optimisation::PositionalArguments
  def applicable?

I later found out a better way to do it from the Facebooker gem, apparently they were already aware of this problem and had fixed it. Taken from their init.rb:

# We turn off route optimization to make named routes use our code for figuring out if they should go to the session
# If this fails, it means we're on rails 1.2, we can ignore it
  ActionController::Base::optimise_named_routes = false 
rescue NoMethodError=>e

Alternately, if you don’t use positional arguments anywhere then you don’t need to turn off route optimization. EXCEPT if you are using named routes with NO arguments, then you run into the same problem. There are two fixes for that. First, pass in {} to any routes w/ no arguments: user_path -> user_path({}). Of course that is a sucky solution cuz you have to change a ton of code, so the second solution is this ugly hack:

ActionController::Routing::Routes.named_routes.routes.keys.each do |route_name|
  [ActionController::Base, ActionView::Base].each do |klass|
    klass.class_eval do
      alias_method "old_#{route_name}_path", "#{route_name}_path"
      define_method("#{route_name}_path") do |*a|
        old_method = method("old_#{route_name}_path")

        if a.empty?

Of course I don’t really see any reason not to turn off route optimization unless you have a bajillion users in which case you probably have much bigger problems than this.

RTunnel 0.3.6 is out

Filed under: facebook, network, ruby — Tags: , , — coderrr @ 1:50 pm

I just released a new version of RTunnel. It seems that most of the bugs are gone, but I’m sure there are still a few lurking.

The biggest change is that a new connection is no longer made for every tunneled connection. It originally worked that way for simplicity’s sake. If every tunnel had its own connection it made the program a lot simpler. This caused a lot of extra lag though because for every new tunneled connection, you had to wait for a new TCP connection to be established.

Now all connections are tunneled through a single TCP connection. This required me to make a little command based protocol, but it was worth it cuz now everything’s faster!

gem install rtunnel

December 6, 2007

Backgrounding tasks in Rails with threads and using resolv-replace to make them faster

Filed under: facebook, rails, ruby — Tags: , , — coderrr @ 10:56 am

require ‘resolv-replace’ if you are doing any DNS resolution inside of {}

I ran into an issue with threads recently while working on some Facebook applications (TrustBox, TooCute, and Interpret The Bible). It’s important in general for your actions to respond quickly in Rails, but it’s even more important in a Facebook environment because if your app doesn’t respond to facebook w/in some fixed timeout period it will display an error to the user. Since the Facebook API calls generally take between 1-5 seconds to respond I obviously couldn’t do these synchronously inside the action. Thus I needed to background them somehow.

I decided on using Threads. A few quick notes: ActiveRecord IS thread-safe but if you want to use AR models inside a thread block you need to set ActionController::Base.cache_classes = true (and if you do this, you need to restart your dev server anytime you change code). Personally I didn’t need to use any AR models inside the thread, only call the facebook API.

The issue I ran into was that { … } was taking somewhere from 0.3 up to 10 seconds sometimes to finish (I’m on a shitty connection). Not the thread block, but the actual spawning of the thread. Spawning a thread should never take anywhere close to that long, it should immediately return and launch the thread into the background.

Long story short, by default Ruby’s DNS resolution blocks the entire interpreter (since it’s done in C) and the thread was hanging the entire process while trying to resolve The solution was to require ‘resolv-replace’ which transparently changes the DNS resolver to use pure Ruby which doesn’t block.

In general DNS resolution is pretty damn quick but if you are running a high traffic website you don’t want your mongrel processes locking up for any amount of time. So if you are doing any name resolution inside of threads I recommend requiring ‘resolv-replace’.

update: This doesn’t really changing anything but… It wasn’t the actual spawning of the Thread that was hanging, but the entire interpreter that was stopped, while ruby did the DNS lookup, which it must have scheduled to run before returning the new Thread object.

update: This problem still exists in ruby 1.9 even though Threads are native. This is because the Threads are still controlled by the VM and they do not run simultaneously (there is a global mutex lock around each thread).

The Silver is the New Black Theme. Create a free website or blog at


Get every new post delivered to your Inbox.

Join 31 other followers