Sunday’s early morning wake-up call from Cloudflare certainly wasn’t fun. But before we get into any of that if you’re thinking about trying Cloudflare you should. It’s a fantastic service that is the future of front line website defense and it’s available now.
For Fruition, the outage was first detected when our Pingdom alerts and PagerTime pages started beating us over the head at 3 am. Not getting any hardware alerts is a good thing. Those didn’t come but sometimes it all just blows up before the alerts get sent out. After doing some digging through the Splunk server it looked like everything was fine. A quick ssh into the server via my phone from my now disgruntled better half and a quick short key to show what’s running on the server and everything looked fine. Yet, several dozen client sites were still down. That meant crawling out of bed for the laptop.
We’ve been through DNS outages before. Several years ago Ev1servers in Houston had an explosion that destroyed their DNS servers. That sucked for them because they had an explosion and an entire datacenter knocked out. That sucked for me because I was in Mexico at my best friends wedding sitting on the beach (that was before you had to get armored cars and a seal team to go to Mexico).
So back to what caused the Cloudflare outage? Well, I wish I knew. Cloudflare has some short tweets about it and what looks like a very detailed blog post that you can see a snippet of on Google but their blog provider Posterous is also down. I can’t say I know anything about Posterous but if Cloudflare is using them they can’t be half bad? For our clients that were down Sunday morning we’re of course sorry and we’ll keep updating you via our regular channels and do a summary of Cloudflare’s problem when we know more. For those of you that are thinking about trying out Cloudflare do it. It’s great and you won’t regret it.
President/Founder, Tru Family Dental
Marketing, Dependable Cleaners
President, Frame Destination
President & Founder, Family Travel Association