Many of you noticed — and we’re grateful that you did — that our website and email were down for a few days last week and over the weekend. Service is now back. I’d like to run you through a timeline of what happened, and how we continue to be affected by the outage.
- On Wednesday, Oct. 12, around 1pm EDT, all of our services through our Web host, Site5, went down without warning or explanation. Initial tech support triage said we’d be back up in 4 hours and that the entire server — USCENTRAL416 — was down for “maintenance.”
- During the downtime, all services, including the Web server (HTTPD) and email (IMAP/SMTP) were “hard down” — i.e., the URLs wouldn’t resolve and emails were bouncing back to recipients.
- Over the next few days, Caffeinated Press leadership attempted to get status updates or uptime estimates from Site5. Emails went unanswered. Tweets were answered mostly by bots that asked us to message them with a ticket number (which we did, to no avail). John even called them, only to get a generic non-answer … after spending an hour on hold. In short: We had no info about what was going on, or why, or when service would be restored.
- We provided some updates using Facebook and Twitter.
- Service was finally restored in the mid-afternoon of Monday, Oct. 17.
- We experienced roughly 120 hours of continuous service disruption. We still don’t know why.
After we were restored, we discovered that Site5 had apparently reverted to the last saved backup, which meant that all data on all servers supplied after roughly 6 a.m. on Monday, Oct. 10, was irretrievably lost. That includes emails for three days before the service disruption.
Here’s what you need to know:
- Caffeinated Press is doing fine and we’re not going anywhere. Except, that is, to a new and more reliable hosting partner.
- Any communication sent to us by email between 6a on 10/10 and 3p 10/17 is permanently deleted, beyond our ability to recover. This data includes routine messages as well as submissions through our Query page for both Caffeinated Press and The 3288 Review.
- We apologize for anyone’s confusion about our going-dark. In the absence of both Web and email, we had only recourse to social media, which isn’t necessarily everyone’s cup of tea.
Thanks for your patience as we’ve worked through this most frustrating experience with Site5. Although I’ve been with them for a decade, the level of service quality has declined markedly in the last year, and this most recent experience with our shared server is the last straw. We’ll be migrating away over the next week or so.