[BC] Internet attacks

[Chris Gebhardt]

Jonathan E. Hardis wrote:
  > I'm sure that you are absolutely right.  Attackers only hit stale IP
> addresses and never bother to get fresh addresses from DNS.
> 
> Also, about those Government idiots -- why, I bet they load-balance 
> their servers -- the fools that they are -- instead of letting a set 
> of backup machines sit idle and unused until just this sort of attack 
> happens.

Oh, careful not to over-simplify the matter!   Actually, with a lot of 
DDoS attacks, it doesn't make a difference in the world if you unplug 
the server, change the IP, move your DNS.   That might help for the 
low-level script-kiddie.

We have seen and fought off and assisted our downstream customers in 
working through these sorts of issues.   Take our most recent 
experience, for example.   A customer network was being attacked.  No 
reason.   The target was an IP that wasn't even assigned to anything in 
the middle of their /27 CIDR.  No forward (A or CNAME) records to it. 
Just running on one of their boxes.   They're on a 100Mbps FastE port 
from us and their port was fully saturated.  (Typical consumption for 
them is less than 1Mbps).  Our upstream traffic was higher than that (we 
take GigE connections to our peering providers).

So one of our early actions was to disable the port.   That stopped the 
port traffic, but not the upstream traffic.  In fact, it began to 
increase as the botnet grew.   Pretty soon it was starting to really 
chew up router processing and impact network performance.  We run dual 
Cisco GSR12012 gigabit switch routers, which are fairly substantial 
pieces of equipment.   We're not hanging off little 2600 or 7200 gear 
over here like a lot of ISPs our size do.

We then pulled the customer's gateway out of routing, so we were no 
longer routing for their /27 anymore.   No IP, no gateway, no problem, 
right?   Not so fast.   The routers were still being flooded with 
requests.   By now, we're starting to exhibit packet-loss on the 
network.   The only way to shed the load would be to stop advertising 
the /24 of which the /27 was a member.   But that would knock out a 
handful of other customers and take them from an impairment to a full-on 
outage.

See, the customer that was "targeted" may not have even been of interest 
to the attacker at all.   It was simply an IP somewhere in the middle of 
one of our ARIN allocations.  The operator of a botnet knows how it will 
be recieved, and knows that he can target anything in a range and have a 
good shot at causing enough collateral damage so that the target cannot 
operate.

We got on with our peering providers and set up a BGP community with 
them to null-route the affected /27.   That way, the route gets dropped 
at a distributed level upstream from us, before the traffic can be 
concentrated.   The packet-loss went away and so did the traffic.

Analysis of the traffic found that it was coming from all over the 
world.  Lots of big-name universities were involved (think Ivy League, 
both sides of the pond) which is where most of the traffic originated 
from as well as various DSL and cablemodems around the world, systems 
located in other datacenters, you name it.   The real problem was the 
packets that they were sending were constructed in such a way as to 
maximize the CPU load needed to process them while being sent on a 
minimum of actual bandwidth.   Big things come in small packages.

I'm not coming to the defense of Gov't IT managers.   I have no special 
love for anybody in particular over there.   However, I do not doubt 
that they're bringing in some pretty sharp minds.   If they're treating 
it as a priority from a management standpoint, you can be assured that 
some smart folks are really racking their brains and reworking 
infrastructure to fix it.

Also... from a forensic standpoint, keep in mind that in some cases it's 
helpful to just keep on collecting data...

Chris Gebhardt
VIRTBIZ Internet Services
chris at virtbiz.com | (972) 485-4125