Understanding PL-VINI and IIAS
NOTE: As part of the Abilene upgrade to NewNet, the PlanetLab Abilene nodes are currently being decomissioned and replaced by VINI nodes. As a result, the abilene_pl.rb configuration cannot actually be run at this time. The following is still useful for understanding virtual networks built with the PL-VINI/IIAS tools. We will provide a new VINI Abilene configuration once the VINI rollout is complete.
As described here, the PL-VINI/IIAS control and data planes can be configured in a number of different ways. This page looks at several configurations and explains the functionality each provides. All are variations of the Abilene virtual network configured using config/abilene_pl.rb, which assigns OSPF link costs to each virtual link identical to those used to manage the real Abilene network.
In all PL-VINI configurations, four things remain the same: the way that network interfaces are created in UML, the mapping of DNS names to IP addresses, the NAPT setup, and the IP addresses assigned to OpenVPN clients.
UML network interfaces
Virtual network interfaces in UML are created as follows.
- eth0 is used for communicating with the PlanetLab machine's tap0 interface and the OpenVPN clients. It always has an IP address one more than that of tap0 interface; for example, if tap0 has an address of 10.1.89.1, then eth0 will be 10.1.89.2. It is attached to a /24 network.
- eth1 is used to send packets through the NAPT egress and out onto the network. It has an IP address of 192.168.X.99 (where X is assigned by the configuration scripts) and is attached to a /24 network. External destinations associated with the NAPT egress are not directly reachable on this /24 network and so a next hop of 192.168.X.1 is associated with this interface. Packets sent to this next hop traverse the NAPT.
- eth2..ethN represent tunnels to adjacent nodes in the topology. Each virtual link has its own 192.168.X.0/24 network (where X is assigned by the configuration scripts); the endpoints have IP addresses of 192.168.X.2 and 192.168.X.3.
So, for example, this is what ifconfig displays on the Washington D.C. node in the Abilene virtual network, which directly connects to nodes in Atlanta (on eth2) and New York (on eth3):
bash-2.05b# ifconfig
eth0 Link encap:Ethernet HWaddr FE:FD:0A:00:DA:02
inet addr:10.0.218.2 Bcast:10.255.255.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:5
eth1 Link encap:Ethernet HWaddr FE:FD:C0:A8:7C:63
inet addr:192.168.124.99 Bcast:192.168.124.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:5
eth2 Link encap:Ethernet HWaddr FE:FD:C0:A8:66:03
inet addr:192.168.102.3 Bcast:192.168.102.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:5
eth3 Link encap:Ethernet HWaddr FE:FD:C0:A8:70:03
inet addr:192.168.112.3 Bcast:192.168.112.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:5
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Mapping names to IP addresses
The scripts generate an /etc/hosts file that maps an interface's IP address to a name. The format of the name is the interface name plus the node's label attribute. For example, if the PL-VINI node running on machine planetlab1.wash.internet2.planet-lab.org has a label of wash, then its eth2 interface is assigned a name of eth2.wash. This mapping of IP address to names is solely for the convenience of the experimenter, and can be changed without impacting PL-VINI.
External destinations via NAPT
The eth1 interface on each node is used to contact any external destinations for which that node has been configured as an egress. Any TCP, UDP, or ICMP packet sent out on this interface to the associated next hop address will traverse the NAPT (Network Address and Port Translation) gateway in Click and be sent on the network to its destination. For example, we can manually add Princeton's network as an external destination, and then download the CS department's welcome page, as follows:
bash-2.05b# route add -net 128.112.0.0 netmask 255.255.0.0 gw 192.168.121.1
bash-2.05b# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.112.0 * 255.255.255.0 U 0 0 0 eth3
10.0.114.0 * 255.255.255.0 U 0 0 0 eth0
192.168.104.0 * 255.255.255.0 U 0 0 0 eth2
192.168.121.0 * 255.255.255.0 U 0 0 0 eth1
128.112.0.0 192.168.121.1 255.255.0.0 UG 0 0 0 eth1
10.0.0.0 tap0.nycm 255.0.0.0 UG 0 0 0 eth0
bash-2.05b# wget http://128.112.136.35/index.php/Main_Page
--12:34:17-- http://128.112.136.35/index.php/Main_Page
=> `Main_Page'
Connecting to 128.112.136.35:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[ <=> ] 10,476 --.--K/s
12:34:18 (2.19 MB/s) - `Main_Page' saved [10,476]
XORP and Quagga automatically add entries such as these to the routing table to reach external destinations configured with the add_nat_dests Node method.
OpenVPN clients
OpenVPN clients participate in an experiment by connecting to an OpenVPN server running on one of the experiment nodes. The OpenVPN server running on a node assigns IP addresses to clients from the /24 of the local tap0 interface. For example, if tap0 has an IP address of 10.1.89.1, then the OpenVPN server will be 10.1.89.5 and the first client to connect will be assigned 10.1.89.6. Note that the OpenVPN server does not respond to pings on its IP address.
The OpenVPN server pushes out routes to 10.0.0.0/8 and to its configured external destinations (i.e., reachable via NAPT on eth1) to its connected clients. This means that the clients should be able to reach all of these addresses as long as the experiment provides inter-node connectivity. For example, the addresses will be reachable when running OSPF using Quagga or XORP.
Global variables:
$router = "none"
With this configuration, each UML kernel only has connectivity to its direct neighbors. For example, here is the routing table on the Washington D.C. node:
bash-2.05b# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.102.0 * 255.255.255.0 U 0 0 0 eth2 192.168.112.0 * 255.255.255.0 U 0 0 0 eth3 192.168.124.0 * 255.255.255.0 U 0 0 0 eth1 10.0.218.0 * 255.255.255.0 U 0 0 0 eth0 10.0.0.0 tap0.wash 255.0.0.0 UG 0 0 0 eth0
Only the directly connected interfaces on the Atlanta and New York neighbors are reachable:
bash-2.05b# traceroute 192.168.102.2 traceroute to 192.168.102.2 (192.168.102.2), 30 hops max, 38 byte packets 1 eth4.atla (192.168.102.2) 17.754 ms 17.476 ms 17.464 ms bash-2.05b# traceroute 192.168.112.2 traceroute to 192.168.112.2 (192.168.112.2), 30 hops max, 38 byte packets 1 eth3.nycm (192.168.112.2) 5.955 ms 5.746 ms 5.539 ms
Note that the 10.0.0.0/8 route is inactive in this configuration.
Global variables:
$router = "Quagga"
Quagga produces no output to the console. A few seconds after UML boots up, Quagga will insert destinations it learns of from OSPF into the local routing table. For example, on the Washington D.C. node:
bash-2.05b# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.0.141.0 eth3.nycm 255.255.255.0 UG 20 0 0 eth3 10.0.218.0 * 255.255.255.0 U 0 0 0 eth0 10.0.217.0 eth3.nycm 255.255.255.0 UG 20 0 0 eth3 10.0.221.0 eth3.nycm 255.255.255.0 UG 20 0 0 eth3 10.0.245.0 eth3.nycm 255.255.255.0 UG 20 0 0 eth3 10.0.248.0 eth4.atla 255.255.255.0 UG 20 0 0 eth2 10.0.253.0 eth3.nycm 255.255.255.0 UG 20 0 0 eth3 192.168.100.0 eth4.atla 255.255.255.0 UG 2022 0 0 eth2 192.168.101.0 eth4.atla 255.255.255.0 UG 1433 0 0 eth2 192.168.102.0 * 255.255.255.0 U 0 0 0 eth2 192.168.103.0 eth3.nycm 255.255.255.0 UG 1193 0 0 eth3 192.168.108.0 eth3.nycm 255.255.255.0 UG 2643 0 0 eth3 192.168.109.0 eth4.atla 255.255.255.0 UG 3915 0 0 eth2 192.168.110.0 eth3.nycm 255.255.255.0 UG 1741 0 0 eth3 192.168.111.0 eth3.nycm 255.255.255.0 UG 4041 0 0 eth3 192.168.104.0 eth3.nycm 255.255.255.0 UG 933 0 0 eth3 192.168.105.0 eth3.nycm 255.255.255.0 UG 2380 0 0 eth3 192.168.106.0 eth3.nycm 255.255.255.0 UG 3675 0 0 eth3 192.168.107.0 eth3.nycm 255.255.255.0 UG 4475 0 0 eth3 192.168.116.0 eth3.nycm 255.255.255.0 UG 2390 0 0 eth3 192.168.117.0 eth4.atla 255.255.255.0 UG 2032 0 0 eth2 192.168.118.0 eth3.nycm 255.255.255.0 UG 1203 0 0 eth3 192.168.119.0 eth3.nycm 255.255.255.0 UG 1751 0 0 eth3 192.168.112.0 * 255.255.255.0 U 0 0 0 eth3 192.168.113.0 eth3.nycm 255.255.255.0 UG 4531 0 0 eth3 192.168.114.0 eth4.atla 255.255.255.0 UG 856 0 0 eth2 192.168.115.0 eth3.nycm 255.255.255.0 UG 943 0 0 eth3 192.168.124.0 * 255.255.255.0 U 0 0 0 eth1 192.168.120.0 eth4.atla 255.255.255.0 UG 3925 0 0 eth2 192.168.121.0 eth3.nycm 255.255.255.0 UG 243 0 0 eth3 192.168.122.0 eth3.nycm 255.255.255.0 UG 3685 0 0 eth3 192.168.123.0 eth3.nycm 255.255.255.0 UG 4485 0 0 eth3 10.1.88.0 eth4.atla 255.255.255.0 UG 20 0 0 eth2 10.1.89.0 eth4.atla 255.255.255.0 UG 20 0 0 eth2 10.0.117.0 eth3.nycm 255.255.255.0 UG 20 0 0 eth3 10.0.114.0 eth3.nycm 255.255.255.0 UG 20 0 0 eth3 130.207.0.0 eth4.atla 255.255.0.0 UG 20 0 0 eth2 128.252.0.0 eth3.nycm 255.255.0.0 UG 20 0 0 eth3 10.0.0.0 tap0.wash 255.0.0.0 UG 0 0 0 eth0
Note that now all active IP addresses in the 10.0.0.0/8 network are now reachable. For example, we can traceroute to the Seattle node and see the UML network interfaces that the packet traverses:
bash-2.05b# traceroute eth0.sttl traceroute to eth0.sttl (10.0.141.2), 30 hops max, 38 byte packets 1 eth3.nycm (192.168.112.2) 5.986 ms 5.783 ms 5.556 ms 2 eth3.chin (192.168.104.2) 29.076 ms 28.730 ms 28.921 ms 3 eth3.ipls (192.168.103.3) 33.610 ms 33.873 ms 33.654 ms 4 eth4.kscy (192.168.110.3) 43.849 ms 44.023 ms 44.059 ms 5 eth2.dnvr (192.168.105.2) 55.928 ms 56.551 ms 56.620 ms 6 eth0.sttl (10.0.141.2) 93.567 ms 95.746 ms 93.567 ms
There are two ways to configure the IP data plane with XORP: with IP forwarding performed by UML or by Click. Each case is discussed separately below.
IP forwarding in UML
Global variables:
$router = "XORP" $click_forwarding = false
This configuration looks much like the Quagga case, except that XORP spits out lots of messages when it is starting up. XORP has successfully initialized once it reports: No more tasks to run; at this point it will start inserting entries into the routing table. For example, on the Washington D.C. node:
bash-2.05b# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.100.0 eth4.atla 255.255.255.0 UG 2022 0 0 eth2 192.168.101.0 eth4.atla 255.255.255.0 UG 1433 0 0 eth2 10.0.117.0 eth3.nycm 255.255.255.0 UG 1193 0 0 eth3 192.168.102.0 * 255.255.255.0 U 0 0 0 eth2 192.168.103.0 eth3.nycm 255.255.255.0 UG 1193 0 0 eth3 192.168.112.0 * 255.255.255.0 U 0 0 0 eth3 192.168.113.0 eth3.nycm 255.255.255.0 UG 4531 0 0 eth3 10.0.114.0 eth3.nycm 255.255.255.0 UG 233 0 0 eth3 10.1.88.0 eth4.atla 255.255.255.0 UG 2022 0 0 eth2 10.0.245.0 eth3.nycm 255.255.255.0 UG 2380 0 0 eth3 10.1.89.0 eth4.atla 255.255.255.0 UG 846 0 0 eth2 192.168.108.0 eth3.nycm 255.255.255.0 UG 2643 0 0 eth3 192.168.124.0 * 255.255.255.0 U 0 0 0 eth1 10.0.218.0 * 255.255.255.0 U 0 0 0 eth0 192.168.109.0 eth4.atla 255.255.255.0 UG 3915 0 0 eth2 192.168.110.0 eth3.nycm 255.255.255.0 UG 1741 0 0 eth3 192.168.111.0 eth3.nycm 255.255.255.0 UG 4041 0 0 eth3 10.0.248.0 eth4.atla 255.255.255.0 UG 3915 0 0 eth2 10.0.217.0 eth3.nycm 255.255.255.0 UG 933 0 0 eth3 192.168.104.0 eth3.nycm 255.255.255.0 UG 933 0 0 eth3 192.168.105.0 eth3.nycm 255.255.255.0 UG 2380 0 0 eth3 10.0.141.0 eth3.nycm 255.255.255.0 UG 4475 0 0 eth3 192.168.106.0 eth3.nycm 255.255.255.0 UG 3675 0 0 eth3 10.0.253.0 eth3.nycm 255.255.255.0 UG 3675 0 0 eth3 192.168.107.0 eth3.nycm 255.255.255.0 UG 4475 0 0 eth3 10.0.221.0 eth3.nycm 255.255.255.0 UG 1741 0 0 eth3 130.207.0.0 eth4.atla 255.255.0.0 UG 846 0 0 eth2 128.252.0.0 eth3.nycm 255.255.0.0 UG 1193 0 0 eth3 10.0.0.0 tap0.wash 255.0.0.0 UG 0 0 0 eth0
One difference between the XORP and Quagga configuration is that XORP does not advertise a route to the local eth1 interface, and so there are a few less entries in the routing table. Now all active address in the 10.0.0.0/8 network should be reachable; for example, we can traceroute the Seattle node from Washington D.C.:
bash-2.05b# traceroute eth0.sttl traceroute to eth0.sttl (10.0.141.2), 30 hops max, 38 byte packets 1 eth3.nycm (192.168.112.2) 6.021 ms 5.720 ms 5.701 ms 2 eth3.chin (192.168.104.2) 28.995 ms 29.215 ms 28.846 ms 3 eth3.ipls (192.168.103.3) 33.577 ms 33.850 ms 34.095 ms 4 eth4.kscy (192.168.110.3) 44.403 ms 44.652 ms 44.340 ms 5 eth2.dnvr (192.168.105.2) 55.832 ms 55.892 ms 55.766 ms 6 eth0.sttl (10.0.141.2) 93.517 ms 94.374 ms 94.823 ms
IP forwarding in Click
Global variables:
$router = "XORP" $click_forwarding = true
The key difference between this configuration and the previous one is that now Click forwards the IP packets instead of UML. This reduces overhead on the forwarding path, leading to lower latency and increased throughput.
The UML kernel's routing table looks exactly the same as before, but UML is not actually forwarding the packets. Rather, XORP duplicates the kernel routing table entries it inserts into Click. You can telnet into Click on the $click_port you specified in the experiment configuration and inspect its routing table directly. On the Washington D.C. node:
bash-2.05b# telnet tap0.wash 13653 Trying 10.0.218.1... Connected to tap0.wash. Escape character is '^]'. Click::ControlSocket/1.1 read _xorp_rt4.table 200 Read handler '_xorp_rt4.table' OK DATA 961 10.0.218.2/32 - 4 10.0.218.1/32 - 5 192.168.124.99/32 - 6 192.168.102.3/32 - 7 192.168.112.3/32 - 8 10.0.218.0/24 - 0 192.168.124.0/24 - 1 192.168.102.0/24 - 2 192.168.112.0/24 - 3 10.0.114.0/24 192.168.112.2 3 10.0.117.0/24 192.168.112.2 3 10.0.141.0/24 192.168.112.2 3 10.0.217.0/24 192.168.112.2 3 10.0.221.0/24 192.168.112.2 3 10.0.245.0/24 192.168.112.2 3 10.0.248.0/24 192.168.102.2 2 10.0.253.0/24 192.168.112.2 3 10.1.88.0/24 192.168.102.2 2 10.1.89.0/24 192.168.102.2 2 128.252.0.0/16 192.168.112.2 3 130.207.0.0/16 192.168.102.2 2 192.168.100.0/24 192.168.102.2 2 192.168.101.0/24 192.168.102.2 2 192.168.103.0/24 192.168.112.2 3 192.168.104.0/24 192.168.112.2 3 192.168.105.0/24 192.168.112.2 3 192.168.106.0/24 192.168.112.2 3 192.168.107.0/24 192.168.112.2 3 192.168.108.0/24 192.168.112.2 3 192.168.109.0/24 192.168.102.2 2 192.168.110.0/24 192.168.112.2 3 192.168.111.0/24 192.168.112.2 3 192.168.113.0/24 192.168.112.2 3
The first column in this table shows the destination, the second is the next hop, and the third is an index that corresponds to the Ethernet interface (e.g., index 3 corresponds to eth3). Above, entries with indices ≤ 3 are mirrored from the UML kernel routing table; entries with higher indices are used internally by the Click forwarder.
The output of traceroute shows that Click is actually forwarding the packets. Hops beginning with "fea" are responses from Click's forwarding engine. Again, on the Washington D.C. node:
bash-2.05b# traceroute eth0.sttl traceroute to eth0.sttl (10.0.141.2), 30 hops max, 38 byte packets 1 fea.wash (10.0.218.3) 0.692 ms 0.628 ms 0.476 ms 2 fea.nycm (10.0.114.3) 5.500 ms 5.376 ms 5.241 ms 3 fea.chin (10.0.217.3) 27.735 ms 27.769 ms 27.795 ms 4 fea.ipls (10.0.117.3) 32.174 ms 32.159 ms 32.022 ms 5 fea.kscy (10.0.221.3) 41.719 ms 41.928 ms 42.146 ms 6 fea.dnvr (10.0.245.3) 53.108 ms 52.990 ms 53.059 ms 7 fea.sttl (10.0.141.3) 89.924 ms 90.080 ms 89.781 ms 8 eth0.sttl (10.0.141.2) 90.538 ms 90.359 ms 90.457 ms
Comparing this traceroute with one where UML forwards the packets, we note two things. First, the path reports two extra hops, corresponding to the Click forwarder on the Washington D.C. and Seattle nodes. On every node, an IP packet first passes through the Click forwarder before being forwarded up to UML or out to a tunnel; traceroute makes this behavior explicit. Second, forwarding in Click instead of UML shaves off about 0.5ms latency for each virtual router that the packet traverses.