CounterPath has recently been adding new services and devices to our Boston Pop Infrastructure. We have had some situations where we exceeded the number of physical interfaces with respect to the number of bonded [protected] interfaces we would like to deploy on an individual server. This led us down the path to review, and improve upon, our standard configuration for our Linux servers.
Let’s begin with a description of our environment without going into painful detail. CounterPath primarily uses Cisco network equipment for our routing and switching infrastructure. We run all gigabit Ethernet inside of our infrastructure and use IBM BladeCenter E chassis for hosting our NCG and other servers. We find that the IBM hardware is superiorly built and has served us very well over time. The BladeCenter environment provides maximum redundancy, with a high level of space and power efficiency, which is not achievable via racking and stacking pizza box servers.
At some point in the past CounterPath moved away from providing each blade with four network interfaces to two. This shift was primarily because of the lack of a need to provide NCGs with separate physical administrative network interfaces. In addition, there was a small issue with particular flavors of RHEL and their selection / ordering of network interfaces that caused us some trouble a few years ago. We always have the ability to add these network interfaces back as our BladeCenter chassis' support, and are loaded with four ethernet switch modules, but we have removed the daughter card that provides each blade with eth3 and eth4 physical interfaces.
Traditionally CounterPath always bonded eth0 and eth1 on our blades to create a bond0 interface. This provides redundant and resilient network access for each of our blade servers. As we have evolved, we have had a few situations where we were looking to deploy NCGs or other network servers with either multiple IPs on the same network subnet / VLAN or interfaces on multiple subnets / VLANs.
CounterPath has been able to keep many systems in the original configuration of bonding two physical ethernet interfaces into a single bond interface when they need only one network connection. We did however change the way in which we bonded interfaces to provide a better availability for each individual server. We added sub-interfaces to that bond interface for when we have a situation where one server needs two or more IP addresses within the same subnet or VLAN. We also added virtual interfaces to that bond interface for when we have a situation where one server needs to be connected to multiple IP subnets or VLANs.
There are some tricks and tips however that will help others along the path. Here is a quick list:
- Miimon is a much better method of supporting bonding of physical interfaces. We switched to this, and away from arp, as our standard configuration for bonding. Both of these methods watch the availability of the primary interface, but miimon (while it requires some driver support) is much faster (hundreds of milliseconds vs seconds in our experience).
- Sub-interfaces (for multiple IPs on the same network) and virtual interfaces need only be created on your bond interfaces when you are running a bonded configuration.
- Switches that support both VLAN trunking and link state grouping are critical. VLAN trunking (as it sounds) allows you to carry traffic from multiple VLANs to servers over all of their physical interfaces. This allows your systems to configure virtual interfaces to take advantage of the multiple VLANs that are on each physical interface.
While researching solutions, I found some great resources from both the IBM and Linux communities. Here is a quick list of articles and documents that contained critical insights:
I also found these additional resources for BladeCenter Configurations to be valuable and thought I would share: