coderrr

January 10, 2010

Tunnel Splitter: Accelerating a single TCP connection over multiple ISPs

Filed under: network, ruby, tunnel_splitter — Tags: , , — coderrr @ 12:15 pm
        LOCAL HOST                                          REMOTE HOST 

                                +--> (ISP1) ---+
                               /                \
(SOCKS client) -> (ts_client)  ---> (ISP2) ------ > (ts_server) -> (SOCKS server)
                               \                /
                                +--> (ISP3) ---+

About

At its essence Tunnel Splitter is a ruby and EventMachine based client/server pair which allows you to split a single TCP connection into multiple parts and send them over multiple TCP connections. The client does the splitting, the server does the reconstruction. At first glance this doesn’t seem to be anything special, but it actually allows you to do something very cool: use the aggregate the bandwidth of multiple ISPs for a single TCP connection.

There are already plenty of load balancing routers out there. Although they do load balance connections between multiple ISPs, any single connection will only be going out over a single ISP. Not to their discredit, it’s the best you can do with only a router. Tunnel splitter requires a little more. It requires that you have access to a server sitting on a connection twice as fast (up+down) as the sum of all your ISP connections. This is easily available nowadays. I currently run my tunnel splitter server on an $8/month VPS.

If it’s not already obvious, tunnel splitter can save you a lot of money if you need a fast connection. Getting a single 50Mbps connection to your house will cost a lot more than getting 6 15Mbps connections (or whatever speed your ISP provides). The increase in price as you need more and more bandwidth from a single ISP connection is almost never linear.

Tunnel splitter also provides transparent redundancy if ISPs go down. It uses a custom protocol on top of TCP to make sure all packets get delivered. If 10 packets have already been sent out through ISP1 before it is determined that it went down, they will be resent out through another link, so your tunneled TCP connection will not drop.

By itself tunnel splitter only tunnels one local port to one remote host:port. This isn’t that useful by itself. But when you put a SOCKS server at that remote host:port now you can tunnel any application over it which supports SOCKS proxying. Add a little routing magic (which I will discuss later) and you can proxy _anything_, even if it has no SOCKS support.

Usage

Now I’ll show how to download and run tunnel splitter. Either download or clone it from github. To run it you will need to install ruby and the eventmachine gem: gem install eventmachine.

Let’s assume you are connected to two ISPs, one on eth0 and one on eth1. And let’s say your server on a fast connection is running with the address tsplitterserver.mayhem.org. And let’s say you have a SOCKS server running on the server listening on localhost port 1080. On the server you would simply run:

ruby ts_server.rb 0.0.0.0:6000 localhost:1080

This tells it to listen for tunnel splitter client connections on port 6000 and forward the tunneled connections to the SOCKS server on port 1080.

Client side we would run this:

ruby ts_client.rb localhost:1080 tsplitterserver.mayhem.org:6000:eth0~10 tsplitterserver.mayhem.org:6000:eth1~10
# note that the localhost:1080 on the client and the server are unrelated and do not actually need to match

This tells the tunnel splitter client to create 10 tunnels to the server over the eth0 interface and 10 tunnels to the server over the eth1 interface, and then wait for incoming connections on port 1080. When the tunnel splitter client receives connections on port 1080 it will split them up and load balance them across all 20 connections. The reason we create 10 tunnels over each ISP here is to make sure we are maxing out each ISPs bandwidth. Some ISPs impose per connection caps so you can’t get your full bandwidth without using multiple connections. If you’re sure that your ISP doesn’t do this then you are fine with just a single connection and can leave off the ~10.

Now if you point any SOCKS capable app to localhost:1080 it will be using the aggregate bandwidth of both your ISPs, we’re done! Not quite. If you’re using Linux there is another issue we need to address first.

If you’re using OSX or another BSD (I’m assuming they’re the same, but I’ve only tested on OSX) then you can skip this next section.

Linux Routing

The way tunnel splitter makes connections over a specific ISP is to bind the source address to the address of that ISP. The problem is that Linux by default does not route packets based on source address. So with our current setup all the packets will go out over the same ISP and the other one will be totally unused. The simplest way to fix this is to install the shorewall firewall. You don’t need to use it as a firewall but it provides the least involved solution (that I know of) to get Linux to route based on source addresses. The instructions can be found in README.shorewall.

There _is_ a way to set up the needed routing without shorewall but I haven’t looked into it too much yet and it seems to be a little more complicated.

BSD Routing

It works out of the box (at least on OSX).

Applications which don’t support SOCKS proxying

For these applications you can use transocks_em. It is a server which receives all connections from specific apps (via some routing magic) and then forwards them to a SOCKS proxy of your choice. I’ll make a post about it soon, so stay tuned.

Other Uses

Tunnel splitter can also be useful in single ISP situations. As I mentioned above some ISPs impose per TCP connection bandwidth limits. If you are just downloading a file you can often overcome these with download acceleration software which makes multiple HTTP requests. This doesn’t work however for things like streaming video (youtube, hulu, etc). Tunnel splitter solves this problem because it allows you to have a single TCP connection use your ISPs maximum total bandwidth.

The fact that connections will never be dropped by tunnel splitter can be useful as well. You can sleep or hibernate your computer, or all your internet connections can go down for any amount of time, and when you restore things your remote connection will still be active. This works because of a few things. First, whatever program you are proxying is actually connected to localhost (to the tunnel splitter client), so your ISPs going down doesn’t affect that conneciton. Second, the remote connection from the tunnel splitter server to the final destination is of course not affected by your computer’s local status. Thirdly, it is only the tunnel splitter client and server which will detect that the tunnel connections have gone down. But as a side effect of its redundancy protocol, it will keep retrying forever to connect the tunnel connections. Once they do reconnect, any pending data on either side will be sent across as if nothing ever happened. As far as the client you are proxying and the remote server you are proxying to are concerned they have both had active TCP connections the entire time. Of course if you are proxying a protocol which does it own protocol level connection health checking or pinging (IRC) it will detect that there are no responses and most likely drop the connection. I have experimented a little with adding support for both the tunnel splitter client and server to be protocol aware and respond to these pings so that the even a protocol with connection health checks could stay up indefinitely. Not sure if this is something people will really care much about though.

Reliability

Tunnel splitter is still very beta although I have been using it locally for all my web traffic and movie streaming for close to a year now and it’s been quite a while since I’ve run into any bugs.

Feedback

All comments, criticisms, and suggestions welcome. Also feel free to contact me at coderrr.contact@gmail.com

Check out the README for more info.

17 Comments »

  1. Great solution!
    A little question: How can you have multiple ISPs? Here in Germany, you are limited to one because you only have 1 phone line over which the DSL stuff is sent. Is this different in the US (or where you are)?

    Comment by Jonas Schneider — January 10, 2010 @ 5:34 pm

  2. @jonas:
    Even if there is only one ISP available where you live, you should be able to have them install multiple lines to your house.

    Comment by coderrr — January 11, 2010 @ 6:29 am

  3. Who offers an $8/months vps?

    Comment by Brian McManus — January 11, 2010 @ 4:32 pm

  4. i guess these guys raised their prices, i’m currently paying $8 for their $10 vps:

    http://elitedatahosting.com/store/?page_id=51

    Comment by coderrr — January 11, 2010 @ 4:41 pm

  5. Source based routing on linux is not too tricky, though definitely messier than “out of the box” working.

    Here’s a good start:
    http://lartc.org/howto/lartc.rpdb.multiple-links.html

    Comment by Fenn — January 12, 2010 @ 1:25 am

    • thanks for that link i’d been meaning to try this for a while

      Comment by coderrr — January 12, 2010 @ 1:31 am

  6. [...] Tunnel Splitter: Accelerating a single TCP connection over multiple ISPs [...]

    Pingback by Daily Links #140 | CloudKnow — January 12, 2010 @ 8:17 pm

  7. This program looks so great.

    Does it work on Windows ?

    Is it possible to share the same (VPS) server between a few users or the software will work correctly only if one user is connected to the server component at a time ?

    Comment by Bob — March 4, 2010 @ 7:28 pm

  8. @bob,

    currently doesn’t support windows, you could use the same VPS for multiple instances of ts_server, but each ts_server instance will only support a single client.

    Comment by coderrr — March 7, 2010 @ 9:29 am

  9. Tunnel Splitter: Accelerating a single TCP connection over multiple ISPs ? coderrrroy@gigemail.net

    Comment by Esmeralda — May 28, 2010 @ 4:51 am

  10. Here is a project that I would like to do. (Server Side) I have a TiVo and Slingplayer on a LAN in the USA. (Client Side) I am in China with Home network, PC Running Win7, but have Virtualbox and VMWare. Is there a way of running this software on a Router to bridge PC or LAN to the LAN in the USA, possibly using the Router itself to run the server software?

    Comment by Mark — June 27, 2010 @ 6:46 am

  11. What happened to the github repo für tunnel_splitter? Has it vanished?

    Comment by Daniel — August 11, 2010 @ 3:29 pm

  12. Is this project still available?

    Comment by Josh — December 29, 2010 @ 6:35 pm

  13. Great project.

    What happened to the source repository? Is it still available?

    Comment by Maxim Veksler — January 10, 2011 @ 6:01 pm

  14. hello, the git repository is not available. Could you please to share it in another site?

    Comment by gotamorg — August 17, 2011 @ 9:10 am

    • yes please make it available again.

      Comment by Anonymous — September 18, 2011 @ 5:30 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Customized Silver is the New Black Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 28 other followers

%d bloggers like this: