Firefox 3 internals, blocking alerts and XMLHttpRequests

In my quest to find something which acts similar to an alert() box in Firefox 3 (for anti-anti-frame-busting), but without the annoying user-experience, I discovered a few things that I thought I should share. Note this is all FF3 specific.

Why do alert() and other javascript dialogs block javascript execution and in particular javascript timers?
It actually turns out that they do this implicitly, as a side effect of being modal dialogs. Modal dialogs halt the execution of the thread they have been created from until the user closes the dialog. Since all the javascript from a single page is run in a single thread the dialog effectively stops all of it.

embedding/components/windowwatcher/src/nsWindowWatcher.cpp:949

// alert() eventually calls this function
nsWindowWatcher::OpenWindowJSInternal(...)
/* ... */
    printf("About to show the dialog...\n");
    newChrome->ShowAsModal();
    printf("You won't see this until the user closes the dialog\n");

Why don’t synchronous XMLHttpRequests block javascript timers?
Synchronous XHRs do block the execution of the javascript they were called from but not the execution of other javascript timers. That’s a little unexpected given the nature of other synchronous things like the above modal dialog. You would expect some code which blocks your javascript function to be blocking the actual thread its executing on. But Firefox does something interesting while making the requesting.

content/base/src/nsXMLHTTPRequest.cpp:1986

 // If we're synchronous, spin an event loop here and wait
  if (!(mState & XML_HTTP_REQUEST_ASYNC)) {
    nsIThread *thread = NS_GetCurrentThread();
    while (mState & XML_HTTP_REQUEST_SYNCLOOPING) {
      if (!NS_ProcessNextEvent(thread)) {
        rv = NS_ERROR_UNEXPECTED;
        break;
      }
    }
  }

Internally Firefox processes the synchronous request asynchronously, in an event loop. And one of the things that this event loop does is schedule timers for execution.

Even though this event loop is asynchronously processing the XHR request. It is still synchronous in the sense that it will block normal execution of the thread until it returns. This can produce some interesting effects. Take this example which performs two overlapping synchronous XHRs:

setInterval('document.body.innerHTML += "."', 1000)

xhr = new XMLHttpRequest()
xhr.open("GET", "http://request-which-takes-1-second.com/", false)

setTimeout(function() {
  xhr2 = new XMLHttpRequest()
  xhr2.open("GET", "http://request-which-takes-10-seconds.com/", false)

  document.body.innerHTML += 'start2'
  xhr2.send(null)
  document.body.innerHTML += 'done2'
},500)

document.body.innerHTML += 'start1'
xhr.send(null)
document.body.innerHTML += 'done1'

In this example we queue the 2nd XHR to start about half a second after the first one starts. Since the first one only takes a second we might expect to see the text done1 before we see done2. But since it was the event loop of the first XHR which actually initiated the 2nd XHR, the first XHR won’t return until the function which initiated the 2nd XHR returns. We end up getting output like this:

start1start2...............done2done1

anti anti Frame Busting

In this post I presented a way to prevent a site you were (i)framing from frame busting out. Jeff Atwood recently contacted me to see if I knew a way to get around the prevention (to prevent sites from framing stackoverflow.com), which of course I didn’t, but I told him I’d think about it and see if I could come up with something. A week later I had come up with a few ideas but none actually worked.

See updates below for latest/best solution

But, due to some extra motivation from his post today (which links to my original post), I may have just come up with something.

  if (top != self) {
    top.location.replace(document.location)
    alert('busting you out, please wait...')
  }

It’s so stupid simple, but it seems to actually work. If you present an alert() box immediately after changing top.location you prevent the browser from running any more JS, either from your framed site or from the framing site. But you don’t prevent the browser from loading the new page. So as long as the alert box stays up for a few seconds until the browser has loaded enough of the new page to stop running scripts on the old page, then the anti frame busting code never runs and you successfully are busted out once the user clicks OK on the alert.

I’ve just done a preliminary test of this in FF3 and IE7 and it worked in both. So I’m calling it, anti-anti-frame-busting is here!

Update: Jason brought up in a comment the issue of a user clicking OK before the page finished loading in which case the anti-frame-bust will still prevent you from busting. One thing you can do to make sure that the page loads extremely quickly so that no user will be able to click OK before that is to (pre-)cache it. Here’s an example:

<!-- this is the page being framed -->
  <script>
    function bust() {
      if (top != self) {
        // this page is now cached and will load immediately
        top.location.replace("http://page-with-expires-headers.com/")
        alert('busting you out, please wait...');
      }
    }
  </script>
  <!-- cache it! -->
  <iframe onload="bust()" src="http://page-with-expires-headers.com/"></iframe>

You should have these headers on the page to bust out with.

      Expires: Sun, 19 Aug 2012 14:10:44 GMT    <-- far enough in the future
      Last-modified: Fri, 19 Jun 2009 04:24:20 GMT    <-- now

First the framed page will do the initial load of the cached page in an iframe (which you can make hidden). Now that page will be cached and the next time you visit it the browser should make no network requests, loading the page completely from its local cache.

For this technique to work you’ll probly want to use it with a blank page which contains only a javascript/meta redirect to the actual page that was being framed. For example:

<html><body>
  redirecting...
  <script> if (self == top) top.location='http://site-to-redirect-to.com/'; </script>
</body></html>

Update: On IE7 this caching technique alone is good enough to prevent anti-frame-busting. Meaning no alert() is required after the top.location change. At least this is true for a simple page which only consists of a javascript redirect:

<html>
  <body>
    we've busted out!  redirecting...
    <script>
      // only redirect if we're busted out
      if (top == self)  top.location = "http://original-page.com";
    </script>
  </body>
</html>

Update: The holy grail of anti-anti-frame-busting: This code, along with the caching technique described above, works in both FF3 and IE7 and has no negative user-experience (ie. alert boxes or frozen browsers):

  <script>
    function bust() {
      if (top != self)
        setInterval("top.location.replace('http://cached-bust-out-page.com/with/redirect')",1);
     }
  </script>
  <!-- cache it! -->
  <iframe onload="bust()" src="http://cached-bust-out-page.com/with/redirect"></iframe>

Fixing constant lookup in DSLs in Ruby 1.9

Ruby 1.9 puts an extra hurdle in DSL development. Consider this simple DSL example which runs fine in 1.8:

class ProxyObject
  def initialize
    @o = Array.new
  end

  def method_missing(m, *a, &b)
    @o.send m, *a, &b
  end
end

def dsl(&b)
  ProxyObject.new.instance_eval &b
end

module A
  class B; end

  dsl do
    p Module.nesting # => [A]
    push B
    p pop # => A::B
  end
end

This DSL simply proxies to an Array object. In ruby 1.8 everything will work, in specific, resolving the constant B, which resolves to the class A::B. This constant resolution works because we are in the lexical scope of class A, as we can see by the module nesting. (For more info on module nesting see this post)

Now run the code in ruby 1.9:

module A
  class B; end

  dsl do
    p Module.nesting # => [#<Class:#<ProxyObject:0x97beab8>>]
    push B  # => uninitialized constant ProxyObject::B
    p pop
  end
end

Now things break. Our module nesting no longer contains A because instance_eval has changed the nesting to the metaclass of our ProxyObject instance, which doesn’t help us at all. But wait… my last blog post to the rescue! In ruby 1.9 we can dynamically add modules to our lexical scope. So all we need to do is find the original module nesting of the block, then add those modules to our nesting, and then eval the block as we normally would:

def dsl(&b)
  modules = b.binding.eval "Module.nesting"
  Kernel.with_module(*modules) do
    ProxyObject.new.instance_eval &b
  end
end

module A
  class B; end

  dsl do
    p Module.nesting  # => [#<Class:#<ProxyObject:0x8d6d190>>, #<Class:A>, A, Kernel]
    push B
    p pop  # => A::B
  end
end

Please let me know if you see any issues with this approach or have other ideas.

Credit goes to Lourens for mentioning this problem with 1.9 to me.

Dynamically adding a constant nesting in Ruby 1.9

Wanting to see if it would be possible to somehow dynamically modify Module.nesting I hacked around looking for ways to do this for a long time in 1.8 to no avail. But it seems that it’s somewhat trivial in 1.9.

module Kernel
  def with_module(*consts, &blk)
    slf = blk.binding.eval('self')

    l = lambda { slf.instance_eval(&blk) }
    consts.reverse.inject(l) {|l, k| lambda { k.class_eval(&l) } }.call
  end
end

Allows you to do:

module X
  module Y
    module Z
    end
  end

  module Y2
    class Z2
    end
  end
end

x, @x = 5, 6
with_module(X) do
  p Y, Y2 # => X::Y, X::Y2
  with_module(Y, Y2) do
    p Z, Z2  # => X::Y::Z, X::Y2::Z2
    p x, @x  # => 5, 6
  end
end

Without losing the scope of your blocks.

Now should you ever do this? Probably not. It’s usually better to actually nest inside the modules you want to be in the lexical scope of. Or another technique, used sometimes testing situations, is to shorten a long constant name with an abbreviation constant like:

XYZ10 = X::Y::Z::Ten
...
XYZ10.new

Please let me know if you think of a use case which one might consider “valid” or useful for with_module.

One interesting aspect of figuring out how to make this method work was discovering some special properties of the *eval() methods.

The following will not work:

class X
  class Y
  end
end
l = lambda { p Y }
X.class_eval { l.call }  #  uninitialized constant Y (NameError)

But this will

l = lambda { p Y }
X.class_eval(&l)  # => X::Y

And so will this

l = lambda { p Y }
s = self
X.class_eval { s.instance_eval(&l) }  # => X::Y

When you eval a block directly rather than calling call on it, it will use the lexical scope of the calling scope rather than its own lexical scope at the time the block was created.

module G
  class N; end
  $l = lambda { p N }
end
# using the lexical scope of the lambda when it was created
$l.call  # => G::N
# using the current lexical scope + X
X.class_eval &$l  # uninitialized constant X::N 

ActiveRecord’s with_connection is now useful!

A few days ago my with_connection (originally cleanup_connection) patch finally went in to rails trunk. It changes the way with_connection works a little bit, but should be mostly backwards compatible.

Note, you’re probably only going to care about this if you use ActiveRecord in long running threads.

Before this patch, with_connection was useless for most Rails devs’ situations unless we wanted to execute raw SQL:

ActiveRecord::Base.connection_pool.with_connection do |conn|
  User.find(1)  # this will NOT use the connection passed to the block (up to Rails 2.3.x), it
                    # will create a new one, or use the one that existed before we called with_connection
  conn.execute("select raw sql here")  # this is the only way to use the connection
end

So that kinda sucked. Because the idea of with_connection is exactly what you want when you have lots of long running threads. You want to check out a connection, use ActiveRecord as you normally would (and have it use that new connection), and then check it back in whenever you’re done instead of leaving it lying around. This way you could have hundreds of threads using ActiveRecord sporadically with a small sized connection pool.

That’s what this patch allows you to do. Now with_connection will checkout a connection and set it as the main connection for the current thread (if it doesn’t already exist). If a connection already exists it will do nothing. This has the nice effect of allowing you to wrap your ActiveRecord code as closely as possible with with_connection. If with_connection always checked out a new connection no matter what, then you’d want to wrap your code at as high a level as possible which in turn would mean you would be keeping connections checked out of the pool when you didn’t really need them.

Here’s a (very contrived) example:

# shorthand
def db(&b); ActiveRecord::Base.connection_pool.with_connection(&b); end

class User
  def update_score
    db do
      self.score = calculate_score
      save!
    end
  end

  def calcuate_score
    db { self.score_cards.sum(&:points) }
  end
end

100.times { User.all.each {|u| Thread.new { u.update_score } } }

In this example we could call calculate_score directly, or indirectly through update_score, and we would never checkout more than one connection from the pool. We could also feel free to call these methods from any number of threads and know that when the calls finish their connections will be checked back into the pool without having to deal with any special ActiveRecord cleanup methods.

If this is the type of thing you are doing or want to be doing but you don’t want to actually to have wrap all your code in those annoying blocks, you can check out this monkeypatch (blog post here) which essentially wraps all DB touching ActiveRecord methods with with_connection for you.

C function thread safety in Ruby 1.8

In ruby 1.8 (I’m speaking only of MRI here) you don’t have to worry about threads being context switched in the middle of (most) C functions. 1.8 switches threads in really only 1 way: a call to rb_thread_schedule(). This function is called from various other ruby internal functions and macros (CHECK_INTS) but as long you aren’t hitting any of those in the C function in question you won’t be switched.

You may know about the infamous SIGVTALRM that ruby sends to itself to schedule threads every 10ms. But all that signal handler does is set a global flag saying that it’s time to switch. CHECK_INTS uses the flag to determine whether or not to call rb_thread_schedule(). So that signal by itself does not actually switch threads.

signal(SIGVTALRM, catch_timer);
...
void catch_timer(int sig)
{
    if (!rb_thread_critical) {
        rb_thread_pending = 1;
    }
}
...
#define CHECK_INTS ...  if (rb_thread_pending) rb_thread_schedule(); ...

Now you DO have to worry about your C function being context switched if you are calling back to methods on ruby objects from your C function. rb_call() calls CHECK_INTS once every 256 times. Another way to get switched is dealing with ruby’s IO functions or rb_thread_select(). But if your C function is primarily self contained then you can rest safe that it will block the whole interpreter until it returns :).

This is one of if not the only reason that ruby’s primitive data structure operations (Array, Hash, String, etc) ARE thread-safe. Because they are implemented in C. If you were to reimplement all of Array, Hash, and Strings methods in ruby by directly translating the C code into ruby you’d probably end up with non thread safe methods.

Even though 1.9 uses native threads I believe it acts in a similar matter, but that’s for another post.

And if someone knows more than me and I misrepresented something, please let me know!

||= is NOT thread safe, neither is Hash.new {|h,k| … }

||= is an extremely useful ruby operator that lots of people learn to love and use often. The problem is it doesn’t mix well with threading (shared writable variables in specific) because it’s not atomic. I’ve been noticing a lot of people making these mistakes recently, even in code which touts being thread safe (even in my own x_x).

It’s an easy mistake to make so I figured I should explain what’s wrong with it and how to avoid it. Let’s take a simple example:

class X
  def lock
    @mutex ||= Mutex.new
    @mutex.synchronize { yield }
  end
end

lock uses ||= to lazily initialize the @mutex instance var, which is normally fine, except in this case. First you need to remember/realize what ||= actually expands to:

@mutex or @mutex = Mutex.new

Ok so let’s assume two threads call the lock method at the “same time”. Here’s the scenario which demonstrates how it’s not thread safe. {{}} represents what the interpreter is currently evaluating:

# >> thread 1
{{@mutex}} or @mutex = Mutex.new
# >> thread 2 (context switch)
{{@mutex}} or @mutex = Mutex.new
# At this point, both threads evaluated @mutex to nil, so they will both go ahead with the assignment
# >> thread 1 (context switch)
@mutex or {{@mutex = Mutex.new
@mutex.synchronize}} { yield }
# thread 1 is already referencing the object stored in @mutex, ready to call synchronize on it, so if another thread changes it, it won't make a difference to thread 1
# >> thread 2 (context switch)
@mutex or {{@mutex = Mutex.new}}
# thread 2 has now "won" the race and set @mutex to the Mutex object it created.  BUT both threads
# will have be acting on different Mutexes, rendering their synchronization useless.

So usually in cases where you want to use the ||= operator with a shared variable you want to actually synchronize the ||= operation with another “more global” mutex. Or in this situation just setting @mutex = Mutex.new in the initialize method would be best. Here’s an example with the global mutex.

class X
  @@class_mutex = Mutex.new
  def lock
    @@class_mutex.synchronize { @mutex ||= Mutex.new }
    @mutex.synchronize { ... }
    # or
    @@class_mutex.synchronize { @mutex ||= Mutex.new }.synchronize { ... }
  end
end

Along the same lines a Hash with a default proc can run you into thread safety problems. Here’s an example:

h = Hash.new{|h,k| sleep Thread.current[:sleep].to_i; h[k] = Mutex.new }
t1 = Thread.new { Thread.current[:sleep] = 1; h[1] }
t2 = Thread.new { h[1] }
p t2.value # => #<Mutex:0xb7c67840>
p t1.value # => #<Mutex:0xb7c63614>
p h[1] # => #<Mutex:0xb7c63614>

The default proc will be called twice, two Mutexes will be created and you could end up in the same situation as the ||= example. The bottom line here is that when you use a default proc in a Hash or the ||= operator, you expect that the variable or hash key in question will only be set a single time. When dealing with threads this isn’t always the case.

How to get a list of un-pushed commits in git

Assuming your remote repository is named origin and you’re dealing with the master branch:

$ git log master ^origin/master

This shows what commits are in master but not in origin/master (which is the remote branch).

This same syntax can be used to see the difference between two local branches, but will show cherry-picked commits as differences which can be confusing:

$ git log master ^production

Determining how much memory a require uses

Here’s a quick little monkeypatch of the require method which will print out a tree showing how much memory each required file used:

# mem.rb
$require_level = 0
module MemTrackingRequire
  def require(name)
    $require_level += 1
    mem = lambda { GC.start; `ps -o rss= -p #$$`.to_i }
    before = mem.call

    required = super
  ensure
    after = mem.call
    $require_level -= 1
    if required
      puts "#{" "*$require_level}#{name} #{after - before}"
      puts "-------"  if $require_level.zero?
    end
  end
end

Object.class_eval { include MemTrackingRequire }

Here are some examples of how to use it (assuming it’s saved as mem.rb)

$ ruby -rmem -e 'require "rubygems"'
  rubygems/rubygems_version 0
  rubygems/defaults 0
   thread.so 24
  thread 28
  rbconfig 36

...

  rubygems/config_file 928
   rubygems 0
  rubygems/custom_require 0
 rubygems 1788  # <-- this is the number you are looking for, it represents the total amount of memory used by requiring 'rubygems'

$ ruby -rubygems -rmem -e 'gem "activerecord", "=2.2.2"; require "activerecord"'
...
   active_record/i18n_interpolation_deprecation 0
  active_record 5896
 activerecord 5896

$ ruby -rmem -e 'require "net/http"'
...
   uri/mailto 0
  uri 60
 net/http 800

Ruby 1.8 define_method scope bug

Update: Patch accepted, bug fixed!

While writing some tests for my tunnel_splitter project I ran into this really weird bug. It’s hard to believe no one has run into this before. I’ll give you the isolated version (which took considerable time to narrow down).

a = 1
Object.send :define_method, :x do
  lambda do  # lambda is necessary to reproduce bug
    p a # => 1
    a = 2
    p a # => 2
  end.call
end
x(nil) # passing at least one arg is necessary to reproduce bug
p a # => 1

This is wrong. Instead of 1, 2, 1, the output should be 1, 2, 2.

Here’s another reproduction, closer to the actual way I encountered the bug. This one is sensitive to the timing of when threads are scheduled. By commenting out the p :blah line you can actually prevent the bug from happening. That makes this bug way more evil and caused me considerable pain.

def callblock; yield; end

a = 1
Object.send :define_method, :x do
  p :blah # comment this line out and things will work as expected
  begin
    callblock do
      raise 'blah'
    end
  rescue
    p a
    a = 2
    p a
  end
end
Thread.new{ x(nil) }.join
p a # => 1

Offender #1
eval.c:8626 proc_invoke()

    if (_block.frame.argc && DMETHOD_P()) {
        NEWOBJ(scope, struct SCOPE);
        OBJSETUP(scope, tmp, T_SCOPE);
        scope->local_tbl = _block.scope->local_tbl;
        scope->local_vars = _block.scope->local_vars;
        scope->flags |= SCOPE_CLONE;
        _block.scope = scope;
    }

If the block being invoked was a block passed to define_method (DMETHOD_P()) and it is being invoked with some number of args other than 0 (_block.frame.argc) then Ruby will “clone” the scope. Effectively, this doesn’t really do much. It creates a new scope struct, but then it points the local vars tables to the exact same place as the original, meaning if you modify a local in this scope, it will be reflected in the original scope. The issue is that it loses the flags of the original scope. We will see why this is important in a second.

Offender #2
eval.c:8214

static void
scope_dup(scope)
    struct SCOPE *scope;
{
    ID *tbl;
    VALUE *vars;

    scope->flags |= SCOPE_DONT_RECYCLE;
    if (scope->flags & SCOPE_MALLOC) return;

    if (scope->local_tbl) {
        tbl = scope->local_tbl;
        vars = ALLOC_N(VALUE, tbl[0]+1);
        *vars++ = scope->local_vars[-1];
        MEMCPY(vars, scope->local_vars, VALUE, tbl[0]);
        scope->local_vars = vars;
        scope->flags |= SCOPE_MALLOC;
    }
}

scope_dup will copy the local_vars (local variables’ values) table into a new memory address. Whenever this is done, changes to the variables in this scope will NOT affect the original scope’s variables (unless they are explicitly copied/pointed back). Note that scope_dup will return right away and do nothing if the scope has the SCOPE_MALLOC flag set, which scope_dup sets after duping a scope, effectively making it perform only one dup per scope, even if called multiple times.

Offender #1 + Offender #2 = fail

Each of these pieces of code by themselves will not cause the bug to occur. But put them together, specifically the scope cloning conditional and then the scope_dup function, and voila, your weekend has become an MRI debug session.

The issue is that the proc_invoke() scope cloning conditional will not copy the parent scope’s SCOPE_MALLOC flag. Which means if you call scope_dup on a cloned scope you will end up duping the local variables even if they had already been duped once before. Essentially you will end up duping a scope’s local variables twice, which means the parent scope’s variables will not be affected at all by changes to the current scope’s local variables.

Solution

If we just copy the SCOPE_MALLOC flag when cloning a scope all our problems are solved:

      ...
        scope->local_vars = _block.scope->local_vars;
        scope->flags |= SCOPE_CLONE;
        scope->flags |= (_block.scope->flags & SCOPE_MALLOC);  // add this line
        _block.scope = scope;
    }

This way if a duped scope is cloned then duped, the local var tables will still be pointing to the same place.

By itself this actually causes ruby to segfault due to multiple free()s on the same memory.

gc.c:1271

            if (!(RANY(obj)->as.scope.flags & SCOPE_CLONE) && vars[0] == 0)
                RUBY_CRITICAL(free(RANY(obj)->as.scope.local_tbl));
            if (RANY(obj)->as.scope.flags & SCOPE_MALLOC))
                RUBY_CRITICAL(free(vars)); // double free segfault here!

We can modify the second conditional to make sure the scope doesn’t have the SCOPE_CLONE flag before free()ing.

            if (!(RANY(obj)->as.scope.flags & SCOPE_CLONE) && vars[0] == 0)
                RUBY_CRITICAL(free(RANY(obj)->as.scope.local_tbl));
            if (RANY(obj)->as.scope.flags & SCOPE_MALLOC)) && !(RANY(obj)->as.scope.flags & SCOPE_CLONE))
                RUBY_CRITICAL(free(vars));

Conclusion

I’m not sure if this is the best solution. I’m not even sure what the scope cloning conditional is needed for. I can’t think of a reason why you would need to make some modification to the scope of a block passed to define_method called with 1 or more arguments, which you wouldn’t need to make if it were called with 0 arguments. There probably is some reason though, and it’d be nice if someone could point out what it is. Also I’m not sure if it might be better to just copy all the scope flags instead of just the SCOPE_MALLOC one.

Also, I’m definitely not an expert on the internals of MRI so it’s possible this solution actually causes some other bugs.

Here’s the patch

1.8.7 and 1.9

This bug also occurs in 1.8.7 but not in 1.9.

bug filed