Bonding/teaming internet connections

So, perhaps you are too far from your local exchange to get a decent DSL speed, or you have an urgent large download and want to use a secondary internet connection to boost your speed. This article will allow you to combine 2 connections together. No ISP support is required. Connections could be all DSL, 2.4Ghz wireless, or HSDPA or any combination of, indeed any type of internet connection (latency permitting).

There are a number of expensive products on the market which do this. I want to bring this into the realms of the sys admin/computer hobbyist, after all this is the information age.

In this example I am using 1 DSL connection, with 1 HSDPA connection, but assuming you know enough to be able to implement this you should be able to change interface names or implement for more than 2 connections if you want.

You will need:-

  1. A computer of some sort with Ubuntu loaded on it (only for the purposes of this article, could easily be any other distro if you are prepared to adapt it).
  2. A dedicated server of some sort on the internet with Ubuntu (again another distro if you want to adapt, I can recommend a Kimsufi 2G for €14.99 per month ex VAT, can’t get better value than that)
  3. 2 internet connections, obviously.
We will be using a VPN and multi-link PPP to bundle the 2 connections together. The traffic will go through the dedicated server onto the net as follows:-
LAN <-> Masquerading/SNAT <-> VTUN <-> multi-link PPP <-> VTUN <-> SNAT <-> Internet.
First of all we have to setup policy routing so that the 2 VPN tunnels go through the correct interfaces, otherwise this is all a waste of time. We are going to setup 2 routing tables 1 for each connection.

/etc/iproute2/rt_tables

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep

# Routing table for DSL connection
101     dsl-connection
# Routing table for HSDPA connection
102     hsdpa-connection

We then have to remove any default route/gateway from the network setup and add the policy routing rule which will tell the computer to route traffic through the correct interface.

/etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 192.168.1.254
        netmask 255.255.255.0

# Rule to route everything originating from the DSL interface through the DSL gateway
post-up ip rule add from 192.168.1.254 lookup dsl-connection
# Gateway for DSL connection
post-up ip route add default via 192.168.1.252 table dsl-connection

The following file is the setup for the HSDPA connection (in my case to Three), adjust accordingly for your own provider. Refer to http://johnlewis.ie/mobile-broadband-from-the-command-line-in-ubuntu/ for the chat script and pap-secrets.

/etc/ppp/peers/provider

user "user"

connect "/usr/sbin/chat -v -f /etc/chatscripts/pap -T *99#"

# Serial device to which the modem is connected.
/dev/ttyUSB0

# Do not use this connection as the default route.
nodefaultroute

# Makes pppd "dial again" when the connection is lost.
persist

# no compression - ppp is used only until the modem
novj
novjccomp
nopcomp
nodeflate
noccp

The following file sets up the correct routing for the HSDPA connection once it is up.

/etc/ppp/ip-up.d/bond-route

#!/bin/sh

# Rule to route everything originating from the HSDPA interface through the HSDPA gateway
# Notice we use a network here as opposed to a singular IP address or interface name
# This is because the IP address is dynamic but always in the 10.0.0.0 range
# And also because routing rules by interface name don't appear to work at the moment
ip rule add from 10.0.0.0/8 lookup hsdpa-connection
# Gateway for HSDPA connection
# Similarly using the fake gateway assigned by pppd in the absence of provider supplied
# Succesfully routes out of the correct interface, again as interface name doesn't work
ip route add default via 10.64.64.64 table hsdpa-connection

Install Vtun and edit config as necessary.

/etc/vtund.conf

options {
  port 5000;            # Listen on this port.

  # Path to various programs
  ppp           /usr/sbin/pppd;
  ifconfig      /sbin/ifconfig;
  route         /sbin/route;
  firewall      /sbin/tables;
  ip            /sbin/ip;
}

# Default session options
default {
#  type tun;            #tun, ether, tty(default), pipe
#  proto tcp;           #udp, tcp
  compress no;          # no, yes, zlib, lzo
  encrypt no;          #yes, no
  stat yes;             #yes, no: check /var/log/vtund/SessionName_X
  speed 0;              # By default maximum speed, NO shaping
}

# PPP tunnel example.
DSL-PPP-Tunnel {
  passwd  yourpassword;     # Password
  type  tty;             # PPP tunnel.
  proto tcp;            # UDP/TCP protocol
#  compress  lzo:9;      # LZO compression level 9
  encrypt  no;         # Encryption
#  keepalive yes;       # Keep connection alive
#  persist yes;

    srcaddr {
        iface eth0;     # Use first up-/ downstream-device
    };

  up {
        # Connection is Up
        ppp "file /etc/ppp/options.vtund";
  };

  down {
        # Connection is down
  };
}

HSDPA-PPP-Tunnel {
  passwd  yourpassword;     # Password
  type  tty;             # PPP tunnel.
  proto tcp;            # UDP/TCP protocol
#  compress  lzo:9;      # LZO compression level 9
  encrypt  no;         # Encryption
#  keepalive yes;       # Keep connection alive
#  persist yes;

    srcaddr {
        iface ppp0;     # Use first up-/ downstream-device
    };

  up {
        # Connection is Up
        ppp "file /etc/ppp/options.vtund";
  };

  down {
        # Connection is down
  };
}

Here we set the options for PPP.

/etc/ppp/options.vtund

noauth
lock
debug
dump
#logfd 1
logfile /var/log/vtund.log
passive
updetach
#To Enable PPP compression Comment the following line. --START--
noccp
novj
novjccomp
nopcomp
noaccomp
#To Enable PPP compression Comment the following line. --END--

# Set the tunnel endpoint as the default gateway for the entire machine
defaultroute
# Enable multlink so this connection can be used in a bundle
mp

/etc/default/vtun

# Defaults for vtun initscript
# sourced by /etc/init.d/vtun
# Created by the maintainer scripts

#
# This is a POSIX shell fragment
#
# Should the standalone server be started?
# RUN_SERVER=no
# SERVER_ARGS="-P 5000"

# Client sessions to start. Up to ten instances can be configured.
#
# Session name
# CLIENT0_NAME=viper
#
# Destination host
# CLIENT0_HOST=vtun-server.somewhere.com.au
#
# Optional parameters
# CLIENT0_ARGS=
#
# CLIENT1_NAME=
# CLIENT1_HOST=
# CLIENT1_ARGS=

CLIENT0_NAME=DSL-PPP-Tunnel
# Real IP address of dedicated server running vtun, adjust accordingly
CLIENT0_HOST=188.165.0.43

CLIENT1_NAME=HSDPA-PPP-Tunnel
# Real IP address of dedicated server running vtun, adjust accordingly
CLIENT1_HOST=188.165.0.43

/etc/sysctl.conf

net.ipv4.ip_forward=1

/etc/rules-save

# Generated by iptables-save v1.4.10 on Thu Aug 11 20:02:56 2011
*filter
:INPUT ACCEPT [308:33472]
:FORWARD ACCEPT [150:16924]
:OUTPUT ACCEPT [259:26848]
COMMIT
# Completed on Thu Aug 11 20:02:56 2011
# Generated by iptables-save v1.4.10 on Thu Aug 11 20:02:56 2011
*nat
:PREROUTING ACCEPT [87:6487]
:INPUT ACCEPT [2:412]
:OUTPUT ACCEPT [37:2646]
:POSTROUTING ACCEPT [37:2646]
# Rule to Masquerade/NAT the local network change accordingly
-A POSTROUTING -s 192.168.1.0/24 -o ppp1 -j MASQUERADE
COMMIT
# Completed on Thu Aug 11 20:02:56 2011

/etc/rc.local

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Start HSDPA connection
pon

# Load ftp connection tracking module
modprobe nf_conntrack_ftp

# Restore  previous iptables config
iptables-restore /etc/rules-save

# Restart the local DNS caching server if you have one
service dnsmasq restart

exit 0

At the server end we have to setup Vtun, PPP and source NAT.

/etc/vtund.conf

options {
  port 5000;            # Listen on this port.

  # Path to various programs
  ppp           /usr/sbin/pppd;
  ifconfig      /sbin/ifconfig;
  route         /sbin/route;
  firewall      /sbin/tables;
  ip            /sbin/ip;
}

# Default session options
default {
#  type tun;            #tun, ether, tty(default), pipe
#  proto tcp;           #udp, tcp
  compress no;          # no, yes, zlib, lzo
  encrypt yes;          #yes, no
  stat yes;             #yes, no: check /var/log/vtund/SessionName_X
  speed 0;              # By default maximum speed, NO shaping
}

DSL-PPP-Tunnel {
  passwd  yourpassword;     # Password to authenticate the vtund client
  type  tty;            # PPP tunnel.
  proto tcp;            # Use UDP or TCP protocol
#  compress  lzo:9;      # LZO compression level 9
  encrypt  no;         # Encryption
  keepalive yes;        # Keep connection alive

  up {
        # Connection is Up
        ppp "file /etc/ppp/options.vtund";

  };

  down {
        # Connection is down
  };
}

HSDPA-PPP-Tunnel {
  passwd  yourpassword;     # Password to authenticate the vtund client
  type  tty;            # PPP tunnel.
  proto tcp;            # Use UDP or TCP protocol
#  compress  lzo:9;      # LZO compression level 9
  encrypt  no;         # Encryption
  keepalive yes;        # Keep connection alive

  up {
        # Connection is Up
        ppp "file /etc/ppp/options.vtund";

  };

  down {
        # Connection is down
  };
}

/etc/ppp/options.vtund

noauth
lock
#debug
#dump
#logfd 2
#To Enable PPP compression Comment the following line. --START--
noccp
novj
novjccomp
nopcomp
noaccomp
#To Enable PPP compression Comment the following line. --END--

# Enable multilink bundling
mp

#Set local and remote IP addresses for the PPP connection
192.168.10.1:192.168.10.2

/etc/default/vtun

# Defaults for vtun initscript
# sourced by /etc/init.d/vtun
# Created by the maintainer scripts

#
# This is a POSIX shell fragment
#
# Should the standalone server be started?
RUN_SERVER=yes
# SERVER_ARGS="-P 5000"

# Client sessions to start. Up to ten instances can be configured.
#
# Session name
# CLIENT0_NAME=viper
#
# Destination host
# CLIENT0_HOST=vtun-server.somewhere.com.au
#
# Optional parameters
# CLIENT0_ARGS=
#
# CLIENT1_NAME=
# CLIENT1_HOST=
# CLIENT1_ARGS=

/etc/rc.local

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o eth0 -j SNAT --to 188.165.0.43

exit 0

http://www.opennet.ru/soft/vpn_table/vtund-ppp1.html
http://www.dslreports.com/forum/r20456553-MLPPP-Guide-on-Linux

33 thoughts on “Bonding/teaming internet connections”

  1. “2. A dedicated server of some sort on the internet with Ubuntu” (Can this be the same computer as in 1, so we can do all this in one computer with ubuntu.)

    As an ordinary user, is it possible to setup my computer(with ubuntu in it) to utilize my DSL & 3G internet connections without setting up a server. If you can guide me about using my two different internet connections (as one being used in firefox and other in chrome or opera).

    1. Short answer, no. You have to have a server, of some sort, on the internet, to be the other end of the tunnel/vpn. It also doesn’t work particularly well with links that drop lots of packets like 3G, aka HSPA, connections do. However, I am working on that.

  2. Hi John,

    Firstly, this is a great article and i’d like to thankyou dearly for it. I have a question, im not sure if this is possible much but, would it be possible to Link 2 Internet Connections coming into the eth0 and eth1 interfaces (two gateways obviously) . If so how would everything above have to be modified in order to get that done?

    Sean

  3. I’m interested in using this sort of setup for an RV. I have all the major requirements, but I’m wondering if you know of any resources to help me make it more robust: easily support connecting/disconnecting more 3g/4g devices via tethering as their available (for collaboration among ~10 band and crew members), more fine-tuned load-balancing to make use of those flaky 3g connections you warn against.

    1. I’m working on that side of things. The main problem with this setup is tunnelling TCP with TCP connections. It’s never going to work well, causing “ack wars” and tunnel disconnects. l2tp overcomes these problems. In terms of fine grained control of load balancing, I’m also working on that. In a nutshell you have to tell the kernel PPP code what speed each connection so that it divides the traffic appropriately instead of just dividing it by 2/3/etc. I’m am only at dirty hack stage, and still haven’t got it working properly. Once I have I imagine there would have to be some sort of automated test every 5 minutes, when idle, to determine the relative link speeds and adjust accordingly.

    1. I’m sorry. If you don’t understand what’s going on here, you’re probably not going to have much success with it. The tunnel connection is between a PC/server on the LAN and a dedicated server on the internet. We are using policy routing and PPP multi-link to split the traffic evenly over 2 different internet connections – in this case a standard DSL connection (your router box as you put it) and a mobile “broadband” connection (HSDPA dongle). It needs to be adapted for your own setup, obviously. It sounds to me like you need to read up on some more fundamental network technologies like NAT/masquerading before you attempt it.

      Even if you do get it working, you need to be aware that tunnelling TCP within TCP connections is bad, and will almost certainly result in regular “ACK” wars between the inner and out layers of the tunnel, and regular connection restarts. To get reliable tunnels you would therefore need to use a UDP solution such as l2tp. It also needs to be born in mind that the kernel multi-link PPP code, by default, splits the traffic exactly in half, not taking account of any difference in speeds between the connections, and so throughput will only ever be twice what the slowest connection is capable of at any given time. If you happen to have two internet connections which are the same speed you will be in luck. Otherwise, there may be a way of telling the kernel PPP code how to split the traffic, and I have been in contact with the guy that wrote the PPP fragmentation code with a view to doing the same, but haven’t as yet made that breakthrough.

      To address your specific questions:-

      1. I don’t understand the question.
      2. We are not concerned with public DSL IP and HSDPA IP directly. Traffic destined for the DSL connection goes out of an interface with IP 192.168.1.254 and is routed via 192.168.1.252, which is a DSL router, traffic originating from 192.168.1.254 is forced to go through this router via policy routing (see /etc/network/interfaces). My mobile broadband provider gives out addresses in the 10.0.0.0 range, so we don’t need to know specifically what that IP is either, just simply tell policy routing that anything originating from an address in the 10.0.0.0 range should be routed via the HSPA interface (probably ppp0 if you have only the one HSPA connection, see /etc/ppp/ip-up.d/bond-route).
      3. That is a rule to masquerade traffic on the local network through our tunnel PPP interface, so that other machines on the LAN can take advantage if they have this machine (192.168.1.254) set as the default route.
      4. Defaultroute in the “first” /etc/ppp/options.vtund (which is actually being used twice BTW, once for each tunnel session) is 192.168.10.1 or, in other words, the server end of the tunnel. You don’t need to specify an IP for the default route as vtun is intelligent enough to get the tunnel endpoint from the config already specified in the server /etc/ppp/options.vtund.
      5. 188.165.0.43 is the address of my dedicated server, adjust to your own.

      Clear? Probably not, but then you don’t sound as though you have the requisite knowledge to understand what’s going on here.

          1. hi there john,,i’m working on this i have the server and all recruitment’s will try to get it to work if its possible to contact me via email or skype please as there is some things i’ll need some help with

            thanks and have a nice day..

            my skype:QatarMo

  4. ok,,here it goes i got it working and both lines connect to the server but my problem is that there is no internet i only can ping my local address’s and the server on the other end of the tunnel,,trying to ping any net address(googld etc) does not seem to work,,cant seem to find what i need to do

  5. ok im a noob

    this has to be added to the server

    /etc/sysctl.conf
    net.ipv4.ip_forward=1

    its working now and when i check whats my ip its the servers ip address now to test for speed’s,,

    thanks a lot for this great guide and if its possible contact me please…

    1. You’re welcome. Sorry I neglected to put the sysctl stuff in there. Note you can also apply that file without rebooting i.e.

      sysctl -p

      FYI a much better way of doing the same thing is through using OpenL2TP, it creates a more stable tunnel because it uses UDP. However it still won’t work well with HSPA links because the kernel multilink fragmentation code balances packets evenly, meaning you only get a multiple of the slowest link.

      1. will i have this running now with 2 adsl’s and i tested and i get double the uplink etc,,don’t really know what FYI is and if you will put up a guide on how to set it up or guide me to a link,,what i want really is the best way to combine 2 or more adsl’s and get the max bandwidth,,

        thanks..

          1. For Your Information.

            Doesn’t seem to be doing much for your download. If it works for you fine. I just found I got a lot of tunnel disconnects.

            I will write a Openl2tp how to at a later stage.

            1. well let me explain my setup,,i run everything from a vmware workstation running Ubuntu server 12.04,,with virtual Ether net’s,,i have tested adding a HSPA+ to the mix but then the disconnects happened,,tel you do a guide on the Openl2tp i’ll try playing around with this see if i can improve the current setup,,

              thanks again..

              1. You won’t get it working any better. As I’ve explained in other comments here the fundamental problem is running TCP in TCP tunnels. Basically the inner and outer layers will get into an “ack war” which is what causes the disconnects. The reason a TCP in UDP tunnel works better is because then there is only one layer trying to take care of the connection as UDP is “connectionless” i.e. it doesn’t know or care if the data gets there and doesn’t attempt retries, hence the inner and outer layer don’t interfere with each other.

                Trying to run a TCP in TCP solution is the networking equivalent of running around with your shoe laces tied together. ;)

                  1. So, that’s roughly twice what each connection is capable of on it’s own?

                    How often does it disconnect? I found every 3 or 4 minutes, but that may have been due to the HSPA connection I was using.

  6. I’d be really interested to see your config for the openl2tp variant. I’m trying to bond two symmetrical annex m dsl lines, but due to inconsistencies in the way that the linux kernel bonding mechanism handles tun devices across different kernel versions/distributions, I’ve hit a brick wall.

    It is theoretically possible to use a tun device as a slave on Centos 6.5 (using xmit_hash_policy of layer 3+4) , which I can get working on my VPS, however I’m struggling to get it working on my CPE equipment, which is an openwrt trunk running kernel 3.14.3. I can access the bonded remote interface via an openvpn tunnel, but I cannot add my tun devices to the bond on my local end, so I’m currently sitting with a setup that is bonded only on the remote end of the link.

    I came across this post of yours that offers an alternative, namely vtun and MLPP. The tcp nature of the vtun tunnel is problematic, so I’d want to use openl2tp and getting a look at your config would be a great help. Feel free to email me if you’d prefer that to typing a reply in your comments section.

    1. I’d like to be able to help, but those configs have since been lost in the fullness of time (and reinstalls). All I can say is, it’s reasonably easy once you have the routing setup correctly, as in this article. I can try to point you in the right direction if you have specific questions about OpenL2TP configuration options. The only other thing I will say is that you “might” have to fiddle with the mtu, as in my experience trying to use the same mtu across providers didn’t work. You may well be better off going with a deliberately low mtu setting (like 1480).

    1. The kernel PPP muli-link code will only balance links symmetrically. Therefore, total bundle speed is limited to a multiple of the slowest link speed. In practice, this often means that bundled 3G connections are slower and certainly not any faster than one 3G connection on it’s own. This seems to be a limitation imposed by mobile cell towers, as much as the kernel code. I feel that even if relative weights were assigned to each link, the tower would still be a bottleneck for overall throughput.

  7. John,

    Is it possible succesfully bond multiple 4G connections to acheive higher throughput speeds? I have multple Verizon Wireless lines. The lantency ranges of course from 60ms to 90 ms etc.. I can reach speeds of 50Mbps to 70Mbps. I’ve looked at bonding systems from Peplink, and Viprinet. But I’m wondering If I try and bond 4 lines that are 4G could I expect speeds over approx 150Mbps even all the lines hit at least 50Mbps.

    Thanks for your help I’m hoping you can shed some light on the challenges that are faced, and if this is really possible. I’ve seen a lot of conflicting information about this.

    1. The way I did it subsequently (with OpenL2TP as opposed to Vtun, as TCP tunnel in TCP suffers from ack wars) you will be restricted to a multiple of the slowest link speed. In reality, while this can mean you have periods of what you might hope for in terms of throughput, it often means bonding (it’s actually multilink-PPP using policy routing and OpenL2TP) is the same speed or even slower than one link on it’s own. Things to take into consideration are:

      1. The maximum throughput of the local tower.
      2. Whether you are using a different provider for each connection (mtu of one connection provider can effect bonding as a whole).
      3. It should be possible to assign a weight to each PPP connection in the link and therefore negate the multiple of slowest link rule, but I didn’t get any further than contacting the kernel PPP code maintainer to confirm it could be done.
      4. The limitations of wireless technologies and radio frequencies.

Leave a Reply