Does Greatly Increased Network Traffic on EC2 Instances Decrease EBS Performance?
I’m sort of throwing this out there. BitBucket, a source code hosting service based on Mercurial and running on EC2 and off EBS volumes, had a long downtime recently, and it was apparently caused by a DDoS attack flooding their servers with spoofed UDP traffic.
Putting aside the initial oddness with Amazon’s support and that having such an insanely long downtime sucks big time, one thing is quite interesting to consider.
Access to their EBS volumes was horribly slow while the network traffic peaked out. The network traffic increase seemed to correlate directly to the decrease of EBS volumes, which in turn are read from and written on over the network. The facts at hand leave me to assume that EBS volume access happens over the same network interface, be it virtual or physical, as the normal network access to the EC2 instance.
Sure, we’re talking about an extraordinary peak in traffic, but I’m quite curious how the balance works out when you have say daily peaks during the evening hours, which involve heavy I/O on your EBS volumes, for whatever reasons.
You could and you should argue that this shouldn’t happen a lot, BitBucket is quite a special case. You should keep as much data on external storage like S3, but services like BitBucket can’t rely on that, they need the data on disk, the same is true for databases.
It’s hard to think of a simple and universal solution for this, as is always the case for DDoS attacks. The traffic needs to be capped above the level of the instances, which is exactly what Amazon did in this case.
In the end I hope I’m wrong with the assumption ventured in the subject, or that at least that it will be fixed in the near future.
