SOCKS 4 server with ProxyMachine

I proxy almost all my web traffic through a SOCKS proxy. After going through a bunch of different servers, each with its own issues, I decided to just write my own. I ended up using ProxyMachine and it only took seven lines!

# socks4.rb
proxy do |data|
  next  if data.size < 9
  v, c, port, o1, o2, o3, o4, user = data.unpack("CCnC4a*")
  return { :close => "\x0\x5b\x0\x0\x0\x0\x0\x0" }  if v != 4 or c != 1
  next  if ! idx = user.index("\x0")
  { :remote => "#{[o1,o2,o3,o4]*'.'}:#{port}", :reply => "\x0\x5a\x0\x0\x0\x0\x0\x0", :data => data[idx+9..-1] }
end

run it with:

$ proxymachine -h 127.0.0.1 -p 1080 -c socks4.rb

* note this only actually covers half of the SOCKS4 protocol, but it’s the only half that pretty much anyone uses

How to force Flash or any program to use a SOCKS proxy using Transocks and iptables in Linux

The Flash plugin for Linux does not respect any browser’s SOCKS proxy settings. This means sites which stream video through a protocol other than HTTP will go direct to the host rather than through your SOCKS proxy.

One way to force Flash or any program through a SOCKS proxy is to use iptables in combination with transocks.

First download my transocks+dante tarball here: http://github.com/coderrr/transocks/tarball/master

Unpack that, and then simply run make. This will compile dante (which contains a SOCKS client library that transocks depends on), and then compile transocks with that library.

Now that transocks is ready, we need to setup rules for iptables which will redirect our traffic to be handled by transocks. You can put the following rules in a sh script.

iptables_transocks.sh:

#!/bin/sh

LOCAL_NET=192.168.1.0/24

# Flush all previous nat rules, you might not want to include this line if you already have other rules setup
iptables -t nat --flush

iptables -t nat -X SOCKSIFY
iptables -t nat -N SOCKSIFY

# Exceptions for local traffic
iptables -t nat -A SOCKSIFY -o lo -j RETURN
iptables -t nat -A SOCKSIFY --dst 127.0.0.1 -j RETURN
iptables -t nat -A SOCKSIFY --dst $LOCAL_NET -j RETURN
# Add extra local nets here as necessary

# Only proxy traffic for programs run with group 'transocks'
iptables -t nat -A SOCKSIFY -m owner ! --gid-owner transocks -j RETURN

# Send to transocks
iptables -t nat -A SOCKSIFY -p tcp -j REDIRECT --to-port 1211

# Socksify traffic leaving this host:
iptables -t nat -A OUTPUT -p tcp --syn -j SOCKSIFY

Once you’ve created the script, run it:

chmod +x iptables_transocks.sh
sudo ./iptables_transocks.sh

Note, if you need to, you can clear out all these rules with:

sudo iptables -t nat --flush

The setup I have chosen here is to only proxy traffic for programs run with the group-id of group ‘transocks’. This makes it easy to socksify any program by just running it as a specific group. So the first thing we’ll want to do is create this group:

sudo addgroup transocks
sudo gpasswd transocks
# set an empty password

Next, start up transocks

# -f means run in foreground
./transocks -f

Now that we have created the group with an empty password and started transocks we are ready to socksify whatever program we want:

sg transocks 'firefox'
sg transocks 'opera'
sg transocks 'lynx http://whatismyip.com'

sg (set group) will run the program with your current user but with the group you specify. This is a semi-non-invasive way of notifying iptables you want it to proxy the traffic from this program. Note that any files this program writes out will have the group of transocks. In most cases this won’t matter but you should be aware of this.

Although sg will prompt you for a password (even though you set a blank password), if you create an application launcher through your windowing system it should launch without having to respond to or seeing a prompt.

Note, if your kernel supports it, you can tell iptables to only proxy traffic for programs with certain names by using the -m owner --cmd-owner [cmd name] option. The other option is to use UIDs instead of GIDs (-m owner --uid-owner) to notify iptables which traffic to socksify. This of course means you’ll have to run programs as a different user which will probably cause you more pain.

More Ordered Hashes for Ruby 1.8

Here’s one more ParseTree based technique for ordered hashes, it’ll be the last one, I promise. This one doesn’t require any Ruby2Ruby string building/evaling craziness. The syntax is the same as the previous ParseTree example:

hash = H{{ :key1 => value1, 'key2' => value2 }}

But with this implementation you can only use literals as keys, meaning strings, symbols, and numbers.

First we call the block with yield to get the original unordered hash. Next we use ParseTree just to find the keys of the hash in order. Since we only support literals we can use the value straight out of the s-expression, no conversion is needed. And we don’t need to look at the keys’ values in the s-expression since we already have them in the unordered hash. Finally, we create a new ordered hash and set each key in order, grabbing the value from the original (unordered) hash. This allows us to simplify the use of the s-expressions from ParseTree and not have to deal with code generation at all (Ruby2Ruby). It also gives about a 4x speedup over the previous technique.

Here’s the code:

class K;end
def H(&b)
  K.send :define_method, :x, &b
  sexp = ParseTree.translate(K, :x)
  hash_sexp = sexp[2][2]  # skip method definition stuff

  raise "block doesn't contain hash!"  if hash_sexp.first != :hash
  hash_sexp.slice! 0

  unordered_hash = yield

  oh = OrderedHash.new
  until hash_sexp.empty?
    (type, k), _ = hash_sexp.slice! 0, 2
    raise "bad key type: #{type}"  if ! [:lit, :str].include?(type)

    oh[k] = unordered_hash[k]
  end

  oh
end

This and alternative implementations are on my github.

Ordered Hashes for Ruby 1.8 using ParseTree

Update: Added another ParseTree based ordered hash technique here

Here’s another, less invasive (depending how you look at), way to get ordered hashes in ruby 1.8:

hash = H{{ :key => val, 'key' => val, 55 => val }}
method_needs_ordered_pairs {{ :key1 => :val1, :key2 => :val2 }}
method_needs_args_and_ordered_pairs arg1, arg2, H{{ :key1 => :val1, :key2 => :val2 }}

This technique uses ParseTree and Ruby2Ruby. The reason for the double curly braces ( {{}} ) is that we need a block for ParseTree to parse (ParseTree can’t just parse any argument unless we run it on the entire file). The first set of braces is the block, the second is the actual hash.

After ParseTree breaks down the hash into an s-expression we use a slightly modified Ruby2Ruby to rebuild the hash representation using an OrderedHash class instead of Hash. Once we have the new representation as a string we eval it in the binding of block so that any method calls or local variables inside the hash definition will work as expected.

hash = H{{ :key => val, 'key' => val, 55 => val }}
# turns into
hash = OrderedHash.new.merge!(:key => val).merge!('key' => val).merge!(55 => val)
# note: this could be optimized to something like this
hash = (__tmp1 = OrderedHash.new;__tmp1[:key]=val;__tmp1['key']=val;__tmp1[55]=val;__tmp1)

The benefits of this approach over the previous are that it doesn’t require modifying any core classes and it will work with any type of object as a key. Of course using ParseTree and Ruby2Ruby makes it much slower, about 300x.

Code is on github at ordered_hash_syntax repo: implementation and test with examples of usage.

Ordered Hashes for Ruby 1.8

For those of you who want a more declarative way to define ordered hashes in ruby 1.8 (perhaps for a DSL or something) here’s something I came up with:

class Object
  def >>(v)
    [self, v]
  end
end

def H(*pairs)
  pairs.inject(OrderedHash.new) {|oh, (k, v)| oh[k] = v; oh }
end

my_hash = H :some_key >> 'some value', 'nother key' >> another_value

Of course this won’t work for Fixnums as keys.

Firefox 3 internals, blocking alerts and XMLHttpRequests

In my quest to find something which acts similar to an alert() box in Firefox 3 (for anti-anti-frame-busting), but without the annoying user-experience, I discovered a few things that I thought I should share. Note this is all FF3 specific.

Why do alert() and other javascript dialogs block javascript execution and in particular javascript timers?
It actually turns out that they do this implicitly, as a side effect of being modal dialogs. Modal dialogs halt the execution of the thread they have been created from until the user closes the dialog. Since all the javascript from a single page is run in a single thread the dialog effectively stops all of it.

embedding/components/windowwatcher/src/nsWindowWatcher.cpp:949

// alert() eventually calls this function
nsWindowWatcher::OpenWindowJSInternal(...)
/* ... */
    printf("About to show the dialog...\n");
    newChrome->ShowAsModal();
    printf("You won't see this until the user closes the dialog\n");

Why don’t synchronous XMLHttpRequests block javascript timers?
Synchronous XHRs do block the execution of the javascript they were called from but not the execution of other javascript timers. That’s a little unexpected given the nature of other synchronous things like the above modal dialog. You would expect some code which blocks your javascript function to be blocking the actual thread its executing on. But Firefox does something interesting while making the requesting.

content/base/src/nsXMLHTTPRequest.cpp:1986

 // If we're synchronous, spin an event loop here and wait
  if (!(mState & XML_HTTP_REQUEST_ASYNC)) {
    nsIThread *thread = NS_GetCurrentThread();
    while (mState & XML_HTTP_REQUEST_SYNCLOOPING) {
      if (!NS_ProcessNextEvent(thread)) {
        rv = NS_ERROR_UNEXPECTED;
        break;
      }
    }
  }

Internally Firefox processes the synchronous request asynchronously, in an event loop. And one of the things that this event loop does is schedule timers for execution.

Even though this event loop is asynchronously processing the XHR request. It is still synchronous in the sense that it will block normal execution of the thread until it returns. This can produce some interesting effects. Take this example which performs two overlapping synchronous XHRs:

setInterval('document.body.innerHTML += "."', 1000)

xhr = new XMLHttpRequest()
xhr.open("GET", "http://request-which-takes-1-second.com/", false)

setTimeout(function() {
  xhr2 = new XMLHttpRequest()
  xhr2.open("GET", "http://request-which-takes-10-seconds.com/", false)

  document.body.innerHTML += 'start2'
  xhr2.send(null)
  document.body.innerHTML += 'done2'
},500)

document.body.innerHTML += 'start1'
xhr.send(null)
document.body.innerHTML += 'done1'

In this example we queue the 2nd XHR to start about half a second after the first one starts. Since the first one only takes a second we might expect to see the text done1 before we see done2. But since it was the event loop of the first XHR which actually initiated the 2nd XHR, the first XHR won’t return until the function which initiated the 2nd XHR returns. We end up getting output like this:

start1start2...............done2done1

anti anti Frame Busting

In this post I presented a way to prevent a site you were (i)framing from frame busting out. Jeff Atwood recently contacted me to see if I knew a way to get around the prevention (to prevent sites from framing stackoverflow.com), which of course I didn’t, but I told him I’d think about it and see if I could come up with something. A week later I had come up with a few ideas but none actually worked.

See updates below for latest/best solution

But, due to some extra motivation from his post today (which links to my original post), I may have just come up with something.

  if (top != self) {
    top.location.replace(document.location)
    alert('busting you out, please wait...')
  }

It’s so stupid simple, but it seems to actually work. If you present an alert() box immediately after changing top.location you prevent the browser from running any more JS, either from your framed site or from the framing site. But you don’t prevent the browser from loading the new page. So as long as the alert box stays up for a few seconds until the browser has loaded enough of the new page to stop running scripts on the old page, then the anti frame busting code never runs and you successfully are busted out once the user clicks OK on the alert.

I’ve just done a preliminary test of this in FF3 and IE7 and it worked in both. So I’m calling it, anti-anti-frame-busting is here!

Update: Jason brought up in a comment the issue of a user clicking OK before the page finished loading in which case the anti-frame-bust will still prevent you from busting. One thing you can do to make sure that the page loads extremely quickly so that no user will be able to click OK before that is to (pre-)cache it. Here’s an example:

<!-- this is the page being framed -->
  <script>
    function bust() {
      if (top != self) {
        // this page is now cached and will load immediately
        top.location.replace("http://page-with-expires-headers.com/")
        alert('busting you out, please wait...');
      }
    }
  </script>
  <!-- cache it! -->
  <iframe onload="bust()" src="http://page-with-expires-headers.com/"></iframe>

You should have these headers on the page to bust out with.

      Expires: Sun, 19 Aug 2012 14:10:44 GMT    <-- far enough in the future
      Last-modified: Fri, 19 Jun 2009 04:24:20 GMT    <-- now

First the framed page will do the initial load of the cached page in an iframe (which you can make hidden). Now that page will be cached and the next time you visit it the browser should make no network requests, loading the page completely from its local cache.

For this technique to work you’ll probly want to use it with a blank page which contains only a javascript/meta redirect to the actual page that was being framed. For example:

<html><body>
  redirecting...
  <script> if (self == top) top.location='http://site-to-redirect-to.com/'; </script>
</body></html>

Update: On IE7 this caching technique alone is good enough to prevent anti-frame-busting. Meaning no alert() is required after the top.location change. At least this is true for a simple page which only consists of a javascript redirect:

<html>
  <body>
    we've busted out!  redirecting...
    <script>
      // only redirect if we're busted out
      if (top == self)  top.location = "http://original-page.com";
    </script>
  </body>
</html>

Update: The holy grail of anti-anti-frame-busting: This code, along with the caching technique described above, works in both FF3 and IE7 and has no negative user-experience (ie. alert boxes or frozen browsers):

  <script>
    function bust() {
      if (top != self)
        setInterval("top.location.replace('http://cached-bust-out-page.com/with/redirect')",1);
     }
  </script>
  <!-- cache it! -->
  <iframe onload="bust()" src="http://cached-bust-out-page.com/with/redirect"></iframe>

Fixing constant lookup in DSLs in Ruby 1.9

Ruby 1.9 puts an extra hurdle in DSL development. Consider this simple DSL example which runs fine in 1.8:

class ProxyObject
  def initialize
    @o = Array.new
  end

  def method_missing(m, *a, &b)
    @o.send m, *a, &b
  end
end

def dsl(&b)
  ProxyObject.new.instance_eval &b
end

module A
  class B; end

  dsl do
    p Module.nesting # => [A]
    push B
    p pop # => A::B
  end
end

This DSL simply proxies to an Array object. In ruby 1.8 everything will work, in specific, resolving the constant B, which resolves to the class A::B. This constant resolution works because we are in the lexical scope of class A, as we can see by the module nesting. (For more info on module nesting see this post)

Now run the code in ruby 1.9:

module A
  class B; end

  dsl do
    p Module.nesting # => [#<Class:#<ProxyObject:0x97beab8>>]
    push B  # => uninitialized constant ProxyObject::B
    p pop
  end
end

Now things break. Our module nesting no longer contains A because instance_eval has changed the nesting to the metaclass of our ProxyObject instance, which doesn’t help us at all. But wait… my last blog post to the rescue! In ruby 1.9 we can dynamically add modules to our lexical scope. So all we need to do is find the original module nesting of the block, then add those modules to our nesting, and then eval the block as we normally would:

def dsl(&b)
  modules = b.binding.eval "Module.nesting"
  Kernel.with_module(*modules) do
    ProxyObject.new.instance_eval &b
  end
end

module A
  class B; end

  dsl do
    p Module.nesting  # => [#<Class:#<ProxyObject:0x8d6d190>>, #<Class:A>, A, Kernel]
    push B
    p pop  # => A::B
  end
end

Please let me know if you see any issues with this approach or have other ideas.

Credit goes to Lourens for mentioning this problem with 1.9 to me.

Dynamically adding a constant nesting in Ruby 1.9

Wanting to see if it would be possible to somehow dynamically modify Module.nesting I hacked around looking for ways to do this for a long time in 1.8 to no avail. But it seems that it’s somewhat trivial in 1.9.

module Kernel
  def with_module(*consts, &blk)
    slf = blk.binding.eval('self')

    l = lambda { slf.instance_eval(&blk) }
    consts.reverse.inject(l) {|l, k| lambda { k.class_eval(&l) } }.call
  end
end

Allows you to do:

module X
  module Y
    module Z
    end
  end

  module Y2
    class Z2
    end
  end
end

x, @x = 5, 6
with_module(X) do
  p Y, Y2 # => X::Y, X::Y2
  with_module(Y, Y2) do
    p Z, Z2  # => X::Y::Z, X::Y2::Z2
    p x, @x  # => 5, 6
  end
end

Without losing the scope of your blocks.

Now should you ever do this? Probably not. It’s usually better to actually nest inside the modules you want to be in the lexical scope of. Or another technique, used sometimes testing situations, is to shorten a long constant name with an abbreviation constant like:

XYZ10 = X::Y::Z::Ten
...
XYZ10.new

Please let me know if you think of a use case which one might consider “valid” or useful for with_module.

One interesting aspect of figuring out how to make this method work was discovering some special properties of the *eval() methods.

The following will not work:

class X
  class Y
  end
end
l = lambda { p Y }
X.class_eval { l.call }  #  uninitialized constant Y (NameError)

But this will

l = lambda { p Y }
X.class_eval(&l)  # => X::Y

And so will this

l = lambda { p Y }
s = self
X.class_eval { s.instance_eval(&l) }  # => X::Y

When you eval a block directly rather than calling call on it, it will use the lexical scope of the calling scope rather than its own lexical scope at the time the block was created.

module G
  class N; end
  $l = lambda { p N }
end
# using the lexical scope of the lambda when it was created
$l.call  # => G::N
# using the current lexical scope + X
X.class_eval &$l  # uninitialized constant X::N

ActiveRecord’s with_connection is now useful!

A few days ago my with_connection (originally cleanup_connection) patch finally went in to rails trunk. It changes the way with_connection works a little bit, but should be mostly backwards compatible.

Note, you’re probably only going to care about this if you use ActiveRecord in long running threads.

Before this patch, with_connection was useless for most Rails devs’ situations unless we wanted to execute raw SQL:

ActiveRecord::Base.connection_pool.with_connection do |conn|
  User.find(1)  # this will NOT use the connection passed to the block (up to Rails 2.3.x), it
                    # will create a new one, or use the one that existed before we called with_connection
  conn.execute("select raw sql here")  # this is the only way to use the connection
end

So that kinda sucked. Because the idea of with_connection is exactly what you want when you have lots of long running threads. You want to check out a connection, use ActiveRecord as you normally would (and have it use that new connection), and then check it back in whenever you’re done instead of leaving it lying around. This way you could have hundreds of threads using ActiveRecord sporadically with a small sized connection pool.

That’s what this patch allows you to do. Now with_connection will checkout a connection and set it as the main connection for the current thread (if it doesn’t already exist). If a connection already exists it will do nothing. This has the nice effect of allowing you to wrap your ActiveRecord code as closely as possible with with_connection. If with_connection always checked out a new connection no matter what, then you’d want to wrap your code at as high a level as possible which in turn would mean you would be keeping connections checked out of the pool when you didn’t really need them.

Here’s a (very contrived) example:

# shorthand
def db(&b); ActiveRecord::Base.connection_pool.with_connection(&b); end

class User
  def update_score
    db do
      self.score = calculate_score
      save!
    end
  end

  def calcuate_score
    db { self.score_cards.sum(&:points) }
  end
end

100.times { User.all.each {|u| Thread.new { u.update_score } } }

In this example we could call calculate_score directly, or indirectly through update_score, and we would never checkout more than one connection from the pool. We could also feel free to call these methods from any number of threads and know that when the calls finish their connections will be checked back into the pool without having to deal with any special ActiveRecord cleanup methods.

If this is the type of thing you are doing or want to be doing but you don’t want to actually to have wrap all your code in those annoying blocks, you can check out this monkeypatch (blog post here) which essentially wraps all DB touching ActiveRecord methods with with_connection for you.