Today, July 13th, Clockwork’s services were briefly affected by a distributed, denial-of-service (DDoS) attack on our primary datacenter. I’d like to explain what happened and answer some questions you may have.
Unknown criminals directed a DDoS attack at our primary datacenter, beginning on July 12 around 8:30 pm. Clockwork engineers were paged by our monitoring systems; we investigated and reached out to our Internet service provider (ISP). We received confirmation of the attack’s nature from our ISP. By 10 pm, our ISP’s network operations team had mitigated the majority of the attack; by 1 am the following morning, it was completely deflected.
This afternoon, shortly after noon, there was a brief time period while our ISP worked with their upstream providers to renew the defensive measures. The onslaught was still underway, and there was another, brief interruption in our services while our ISP blocked the attack again.
What is a DDoS?
A distributed denial-of-service attack (DDoS) works by overwhelming its target’s systems with requests. It is like a quiet restaurant when its servers are swamped if three tour buses pull up simultaneously: normal customers can’t get through because of the unexpected rush. A DDoS attack makes websites appear offline by overwhelming them with fake traffic.
Internet criminals and malcontents use DDoS attacks as a political or extortion tool. They can happen to anyone. As a service provider, Clockwork cannot control what these criminals do—we can only control how we respond.
How does Clockwork keep me safe today?
First and foremost, we protect you by partnering with the some of the best local Internet service providers in the business, ipHouse and Atomic Data. ipHouse is the provider for our primary datacenter and we use Atomic for our disaster recovery (DR) facility.
Second, we monitor our systems closely so we can be informed when action is necessary to protect our customers and their businesses. Our experienced, dedicated team of system administration professionals are on call 24/7/365 should our monitoring detect a network event such as this attack.
Third, we maintain and test a disaster recovery plan in the event our primary facility is no longer available and is unlikely to become available for an extended time. When we declare an event to be a disaster, we can migrate our hosting services to our DR datacenter (managed by Atomic), a process that can take up to 24 hours.
Why didn’t you declare a disaster?
I judged both events (last night’s initial attack and today’s issue) to be short-lived and manageable by ipHouse’s network operations team. I made the decision not to declare a disaster, since I believed the attack would be mitigated faster than we could switch to our DR facility. Now that the events have been mitigated, I believe this was the right decision for these circumstances.
How will you improve your ability to handle attacks like this one?
I am pleased to say we are in the process of moving from a primary/DR facility model to one with multiple primary facilities. We’ve rolled out new infrastructure in the last two months to prepare for these changes, adding new servers, fast disk, and improved networking hardware to our primary facility. We are now looking ahead to deploying the same combination of powerful equipment to a second primary facility.
That means we will be able to shift our services between multiple, always-on facilities in events like this one. The delay and risk inherent in executing a disaster recovery procedure will be lessened, allowing us to move websites between facilities safely and seamlessly. In other words, we will be able to dodge the attack rather than have to take the punch.
We’re here for you, our customers. I want you to feel comfortable and confident in the midst of the inevitable storms that occur on the Internet. Clockwork wants to be a trusted partner at your side, ready with help and advice.
VP, Director of Technology, Clockwork