Today I’m writing about a recent experience with network modeling and testing in order to support a state-wide enterprise network routing protocol migration (moving an EIGRP domain – about 95 routers – entirely to OSPF). I was responsible for developing a High Level Design for OSPF areas, developing detailed configuration scripts and overseeing the deployment teams in order to minimize user impact during the migration (this business is a critical 24x7x365 operation).
My analogy for this type of project is “changing the tires on the bus while it’s still moving”. I therefore decided it was important to build a model of the core network (which included several sites, including 2 data centers and several network hubs) due to the complexity. This project also included 5 separate phases – it was important to have the ability to confirm the state of the network at the end of each phase.
What follows is a detailed account of how I configured GNS3 for modeling and testing configurations. I won’t get into the specifics of the EIGRP, BGP, or OSPF design here, though that could be good information for another post (or several). This post also doesn’t go into detail on overall methodology. I may cover these topics at a later time.
What is the background of the network?
- Approximately 1,000 routes (with summarization)
- Built almost entirely of point-to-point connections (T1, DS3, Ethernet) with a high level of redundancy between major sites and data centers.
- Utilized Bi-directional redistribution between the Data Center and the rest of the network (this was primarily due to the Data Center running only OSPF (Juniper) and the WAN devices running EIGRP (Cisco).
What do I mean by modeling and testing the network?
- By modeling the network, I’m referring to recreating network device configurations running on actual instances of Cisco IOS and Juniper JUNOS, though these instances happen to be running inside virtualized hypervisors. It’s possible to do this with physical devices as well, but it would be very time consuming (and costly) to do so.
- In my case, I was able to run a simulated version of an actual production core network entirely on a single PC, albeit a reasonably high-end one (more on that later).
What did the model of the network look like?
Why model the network?
In a word – Accuracy.
It’s nearly impossible to initiate migrate a large network accurately based on what you have in your head. Use of actual routing operating systems (running in a virtualized environment) allows for of routing algorithms do the work for you – with little or no risk.
- Network Elements Modeled:
- EIGRP/OSPF Routing Metrics
- Network Connections
- Base Routing Configurations
- Specific things evaluated and mitigated during this exercise:
- Effect of Changing administrative distance in EIGRP and OSPF
- Effect of redistributing static routes into EIGRP and OSPF during the migration
- Effect of moving summarization points in the network
- Effect of altering EIGRP/OSPF and redistribution with BGP
- Effect of temporary redistribution required during the migration
- Effect of traffic flow changes
- Effect of removing the EIGRP routing protocol
What kind of PC hardware was used?
Here are the pertinent specs.
- Type/Model: Lenovo ThinkPad T520
- CPU: Intel Core i7-2760QM @ 2.40 Ghz (Quad-Core)
- Memory: 16 GB
- Primary Disk: Samsung SSD 830 – 256GB
- Secondary Disk: 500 GB 7200 RPM (spindles)
I ran this set of models on a single computer for convenience, but this was not absolutely required. I know of other network teams using several computers to build even larger network models.
What network modeling software was used?
Below are some of the specific software versions I used on this project:
- GNS3: 0.8.2-BETA2 (a big thanks to the development team). There are newer versions available now.
- VMware Workstation v7.1.5 (for running Olive instances) – These instances were connected to GNS3 through the virtual network editor. While it is possible to run JUNOS inside GNS3 as well, I chose not to go that route since I had the VMware instance already up and running.
How was the network modeling software configured? And how did you get 3X routers to work simultaneously on one machine?
It definitely took some trial and error – but below are the settings and router configuration tweaks that I used.
Cisco Routers (GNS3)
IOS version: c7200-spservicesk9-mz.124-24.T6
- GNS3 General Configuration
- Platform/Model: c7200/7200
- Midplane: vxr
- NPE: npe-400
- Memory and Disk
- RAM Size: 512 MB
- NVRAM Size: 128 KB
- PCMCIA disk0 Size: 64 MB
- I tended to use FE interfaces for the modules. For simulation purposes, I would alter the bandwidth/delay so that EIGRP metrics would be represented correctly in the model.
- Special Configurations:
- idlepc = 0x606e0038
- idlemax = 200
- I used this on every router in the model except a few of the central devices. Idlemax restricts the maximum CPU that each instance can use, but was definitely helpful in scaling the model to this size. More details on this can be found here. Many thanks to @0x86DD for pointing this out. This must be configured directly in the .net configuration file as follows:
workingdir = working
udp = 11100
image = D:....
ram = 512
idlepc = 0x606e0038
idlemax = 200
sparsemem = True
ghostios = True
- Disabled Bi-Directional Forwarding Detection (BFD) – BFD is processor-intensive on a virtualized platform and wasn’t central to the goal of this project.
- Set all BGP timers to very high values – My customer had a tendency to use relatively short BGP keepalive and hold timers (under 10 seconds) which did appear decrease the stability of the BGP connections somewhat in this scenario.
- Used the default EIGRP and OSPF Timer Values
JUNOS Routers (VMware Workstation)
- JUNOS version (Olive): 10.1R1.8
- Memory: 512 MB
- Processors: 1
- Hard Disk: 8GB (Thin Provisioned)
- Network Adapters: Custom VMNet adapters as needed
It’s worth noting that features such as Memory Virtualization make running this many VMs possible with only 16GB of RAM.
What was the CPU/Memory Utilization of your system while running all of these devices simultaneously?
In general, when all of the systems were running normally, CPU Utilization would run between 18 to 35% and Memory Utilization was around 45% (of 16GB). This is inclusive of general office tasks that the machine was performing (this is my primary work machine).
Screenshot of Task Manager below with all of the systems running):
Why didn’t you use a sanctioned network modeling platform for Cisco routers?
Because there wasn’t one at the time of this writing. Hopefully this will change soon.
Network modeling and testing for large complex network migrations is critical to minimizing the impact to network operations by avoiding misconfigurations and more accurately predicting what will happen after each change. It is extremely difficult – if not impossible – to accurately script configuration changes without doing this type of legwork.
While this blog entry provided some specifics for GNS3 configurations, there are other available tools for network modeling that could be applicable to your environment:
- Junosphere Lab (VJX Series)
- Virtualization on actual network platforms, such as JUNOS Logical Systems (great blog post here), or Cisco Virtual Device Contexts.
- Routing Analytics tools, such as Packet Design Route Explorer. Route Explorer has the ability to test the effect of routing/link changes without having the model the configurations directly. Route explorer may still not be able to handle redistribution scenarios since it evaluates routing tables separately and does not account for platform-specific conventions such as Administrative Distance (Cisco) or Route Preference (Juniper)
Lastly, many thanks to the GNS3 team for developing this exceptionally useful network simulation tool.