[BC] Disaster planning

[Chris Gebhardt]

Broadcast List USER wrote:
> I don't know if I totally get the connectively problem.
> 
> Our Dallas office had both Southwestern Bell and Verizon, with Verizon  
> being the primary phone and Internet carrier, though they are the CLEC  
> there.
> 
> Manhole fade (flooding) took us out.  TOTALLY.
> 
> UNLESS you have satellite (and we did) plus LEC, plus CLEC, I don't  
> know how you can stay up.  The VSAT terminal did not keep the phones  
> up, or the Internet.  It kept the programming up.
> 
> While the sat feed stayed up, but it wasn't broad enough.
> 
> I think you REAlLY need to think outside the box, then think outside  
> THAT box.

Without drawing into to many mundane details, here's a quick look at 
what we do:

INTERNET:  We take multiple carriers on multiple fiber laterals.  That 
is, we have different providers that come in on different physical fiber 
that is routed in through opposite corners of the building.   That's 
fairly common.  Better datacenters do that as SOP.

We take a step further and use licensed microwave shots from the roof of 
our building to another facility a little over a mile away, which has 
its own fiber entrances.  In regular use, both buildings are linked 
together and there is always traffic passing across the MW links.  The 
routers decide what the best path for any given traffic is - out the 
fiber, or out the MW and the other facility's fiber - based upon how 
many hops it would take.   That also allows us the flexibility to be 
able to run each facility independently (ie: if the MW shot goes down) 
or run each facility off the other one (ie: if that facility loses all 
direct connected fiber).   In broadcasting terms, this would be roughly 
analogous to having a MW and telco-provided STL up simultaneously.

PHONES:  We could easily place an ASTRISK box onsite and handle all our 
own switching and trunking off a PRI.  But if we encounter an outage of 
some sort, we need to focus our attention on that, while still being 
able to make and receive phone calls in the front office.  So we 
outsource our phones.  Our main numbers are ported to a SIP provider. 
Under normal circumstances, those numbers simply forward to numbers that 
are provided by our phone provider.  In this case, the cable company's 
"digital phone" product.   If an outage were to occur that caused us not 
to be able to use our main PBX connected to the cable then we can, with 
about 6 clicks, change the phones to start rolling to our company cell 
phones.  We can ring all the cell phones simultaneously, or we can ring 
them in priority order and if the first phone isn't answered, the call 
rolls to the 2nd etc.   Any which way we work it, it's transparent to 
any callers.

The phone idea is something that any business could really put into 
practice, and the cost is pretty low.   The Internet redundancy is 
something that is a pricier and more complicated endeavor.   There are 
SOHO routers that feature multiple WAN uplinks and promise to choose the 
best route, but my experience with those units, while not extensive, 
hasn't been stellar.   Besides, as Chip mentioned, if both your 
providers come in off the same physical path, all you have is provider 
redundancy, not path redundancy.   A guy with a backhoe could wipe it 
all out.   If you have a telco provider and a cable provider at your 
location, I suppose that would be one way to go, unless both providers 
share poles.  (At our facility, it just so happens that the phones run 
on N-S poles and cable runs E-W).

Chris Gebhardt
VIRTBIZ Internet Services
chris at virtbiz.com | (972) 485-4125