[AF] Aronyme Soup, Routings, and things

[Lamar Owen]

On Thursday 10 May 2007, Rich Wood wrote:
 > ------ At 12:04 PM 5/10/2007, Lamar Owen wrote: -------
 >
 > >I would see topics on this list as being possibly technical but
 > > nonbroadcast (like how to set up BGP and HSRP on Cisco IOS 12.4 on a pair
 > > of 7401ASR's with Stateful NAT failover and an OC3 APS 1+1 WAN link, to
 > > pull a topic off the top of my head
 >
 > What'd he say? Once I figure it out I'm sure it'll be much more
 > fascinating than any other topic we've dealt with on AF.

:-)

Sorry for the acronym soup.  That's:
How to set up the Border Gateway Protocol  and Hot Standby Router Protocol on
Cisco Internetwork Operating System 12.4 on a pair of 7401 Application
Specific Routers with Stateful Network Address Translation failover and an
Optical Carrier 3 Automatic Protection Switching 1+1 Wide Area Networking
link.

Long form translation (and, yes, I do mean LONG!):

The Internet connection for PARI is at a co-location facility where we have a
full rack and a 100 megabit FastEthernet Internet connection.  We have an OC3
link from our site to the co-location facility, about 35 miles away.  OC3's,
in case you're not familiar, pump data at 150 megabits per second (the line
rate is 155Mbps, the missing 5Mbps is used for Synchronous Optical NETwork
(SONET) framing).  OC3's are typically provisioned with two data paths on two
fiber pairs; a 'working' circuit, which is the primary, and a 'protect'
circuit, the secondary.  If anything happens to the working circuit, the
protect switches in automatically as per Telcordia's GR-253 specification.
This 'working plus protect' is the '1+1' mentioned.  Working to protect and
back switching takes a few milliseconds at the most.

What that means for me is that it takes four routers to terminate both ends of
the pipe.  I have a Cisco 12012 on the working pair on this end, a Cisco 7507
on the protect pair on this end, and matching Cisco 7401ASR's on the
co-location facility's end.

The 12012, incidentally, is about the size (and takes about the power) of a
Harris 'Gates 2.5' AM transmitter, and costs at retail about what fifty Gates
2.5's would cost, at around $1 Million (no, I didn't pay that; ours cost much
much less), assuming a Gates 2.5 can be had for $20,000 or so; even on eBay
12012's are going for $134,000 or so.  The 7507 is eight rack units, and
would run at retail ten Gates 2.5's. The 7401's were put at the co-lo because
they take much much less power, being 1RU devices.  At $22,500 per 7401, that
is a mighty expensive one rack unit, though!  The 12012 has a total of 2
gigabytes or so of RAM in the chassis, the 7507 has a little over a gigabyte,
and the 7401's have 512MB.  Fairly beefy routers.

Also at the co-lo is a big honking server; a Dell PowerEdge 6950 with 4 dual
core 2.8GHz Opterons and 32GB of RAM.  I'm in process of getting 15-20TB of
disk on that end to connect to it.  A matching 6950 and storage array are on
this end of the OC3; the two Dell's are running VMware ESX server and back
each other up.

Providing a 'port expander' function for the seriously port-limited 7401s is a
Cisco Catalyst 5505 switch with nine gigabit ethernet ports and 24 10/100
ports.  One 10/100 port is the Internet connection.  The 7401's have two
Gigabit ethernet ports each, along with the OC3 port adapter, and run 802.1Q
VLAN trunking to the 5505, which 'splits out' the VLAN's to separate access
ports.

Now, with APS switching of the OC3, and with router redundancy and failover
being key issues at both ends, the Cisco Hot Standby Router Protocol is in
use at the co-lo so that the various virtual machines running on the 6950 can
get to the internet or back to my LAN on this side of the OC3 if either of
the two 7401's were to fail.  That is, the two routers work together and
provide a single 'virtual' IP address in addition to their native IP
addresses; the various virtual machines use that virtual IP address as their
default gateway.  Whichever 7401 has a valid OC3 connection is the one that
is the HSRP primary router.

Likewise, the routing has to be robust from the two 7401's to and from the
internet provider's router; this is where the Internet's core routing
protocol, Border Gateway Protocol (BGP), comes into play.  BGP is what makes
the Internet possible given the hundreds of millions of IP addresses in use
out there.  So my two 7401's advertise my /24 address space to the ISP's
router, which then advertises that on out to the Internet, stripping my
private use autonomous system number (ASN) off in the process.  While I
thought about getting a publicly valid ASN, the cost is fairly high, and
a /24 is too small to advertise over the Internet anyway, with typical BGP
filters getting rid of a prefix longer than a /20.  BGP provides robust
failover on that side.  The rest of the Internet's routers now have an ASN
path to my /24, through my ISP's ASN.

The 7507 and the 12012 shepherd this end of the OC3, and I'm using Open
Shortest Path First (OSPF) as my internal routing protocol to my LAN, which
is running on Cisco Catalyst 5500 series ethernet switches and Catalyst
8540MSR combined Layer 3 Ethernet and ATM (asynchronous transfer mode)
switches, into my internal ATM cloud (which I'm transitioning to switched
Ethernet, slowly).

However, as I only have 256 IP addresses in my /24, and more than 256 devices
on my LAN, I have to, like most folks, run Network Address Translation (NAT)
on my border routers (the two 7401's).

Why not on the 12012 and 7507, you might ask?  Well, 12012's and 7507's to get
APS working are limited to Cisco IOS 12.0S; 12.0S doesn't do NAT.  The 7401's
can run a 12.0, 12.1, 12.2, 12.3, 12.4, and even 12.4T IOS; I chose 12.4 for
a couple of reasons: it's stable, having reached General Deployment (GD)
status, and it does HSRP-aware NAT for both static NAT mappings and dynamic
pooled NAT mappings.  Earlier IOS versions did a subset of this, but it
wasn't until 12.4 that it all came together.

See, if I have a router failover event, all the existing NAT sessions have to
be maintained; with 12.4 and the 7401's if one router were to fail the other
keeps a synchronized copy of the NAT table, and will failover automatically,
maintaining all NAT sessions.  Also, the 7401's have a hardware NAT assist
engine.  And APS on the 7401's works on 12.4 IOS; win-win all the way around.

And when you're pulling a max of 40GB per hour of Internet traffic, failover
is a big concern!

Figuring out routing in this way is quite similar to audio routing in a studio
or RF work at the transmitter, and requires the same troubleshooting gestalt;
part of the reason I enjoy my job as much as I do.
-- 
Lamar Owen
Chief Information Officer
Pisgah Astronomical Research Institute
1 PARI Drive
Rosman, NC  28772
(828)862-5554
www.pari.edu

Content-Type: text/plain; charset="us-ascii"