We run our company on Google Apps for Business, and we’ve never had any problems. Until now. On Friday morning we came in to work to find that all our calendars are completely, utterly messed up. We lost data, ownership changed randomly, and some of us lost access to our own calendars. We’re still coming to terms with what happened, and it’s hard to explain without getting into specific detail, so I’ll just give you taste of the damage. Sorry for all the names of people you don’t know, but it’s the only way I can keep this straight:

• My work calendar disappeared from my account, and is now owned by Chris’s personal Google Apps account.
• I now own the Boardroom calendar, which is a resource calendar.
• Chris’s work account now owns his personal calendar.
• Debré’s work calendar disappeared from her account, and is now owned by Philip’s work account.
• Philip’s personal calendar disappeared from his account, and is now owned by Angela’s work account.
• Some events have gone completely missing from calendars — they’re just not there any more. Those events also disappeared from our clients’ calendars, which is a huge embarrassment and inconvenience because we don’t know when we’re supposed to meet with who.

And on and on it goes. Google Apps for Business promises 24/7 customer support, with a guarantee that “We’re always available to help via phone or email.” We discovered this issue on Friday morning, so I called immediately. They opened a case, I sent some screen shots, and then we waited.

Later in the day I got an email saying that “Your case will now be further analyzed by the next tier of support.” And then they went dark. I kept sending more information as we found it, but nothing happened. I phoned again on Friday evening and Saturday morning, but by then, phone support couldn’t do anything because the case was transferred to a “calendar specialist”.

On Saturday evening I received an update from Google Apps Support:

I looked at the calendars you mentioned and see the strange names that you mentioned. However, determining exactly what happened will require some in depth logs analysis. This will take some time.

And after that — nothing. Now it’s Monday morning and our ability to run our business is crippled since we don’t have access to our meeting schedule. And since we know that some events have gone missing, we can’t even trust what remains.

On the Google Apps Calendar page it says this (my emphasis added):

Google Calendar is designed for security and reliability with features like encrypted connections to Google’s servers, simultaneous replicated storage for your calendar appointments, built-in disaster recovery and fine-grained sharing, which lets you share your calendar with people in and out of your organization.

So if there’s built-in disaster recovery, why can’t they just restore our calendars to the state it was in Thursday night? Why has it been three days and we’re not getting regular updates and progress reports? Why promise 24/7 customer support if you can’t deliver it? And yes, this includes both phone and email support of their “core services”, which includes Google Calendar.

I’m writing this to hopefully accomplish three things:

• Get some more attention on the issue so that Google can fix it and let us get on with our business.
• Ask if anyone has experienced this problem before — if so, please get in touch.
• Warn you about something you most likely already know: your data is not safe anywhere.

That’s the story so far. Google, please help. We just want our calendars back, and we’d love to know what happened.

Update 12/10/2012: The Internet works! On Monday evening I received a call from a senior Google Calendar employee, and he spent 30 minutes on the phone with me to help troubleshoot and get to the bottom of the issue. Our calendars aren’t fixed yet, but I’m confident that Google is now on top of it and will give us regular updates on what’s going on. At this point it looks like a 3rd party application had a sync issue with GCal, and that instigated a weird chain reaction. I’ll update again with more information once it’s all been fixed.

Update 12/12/2012: We’re mostly back up and running. It sounds like BusyCal had a conniption during one of its regular syncs, and used an API call that created the whole mess. I’m told that the API call that was used doesn’t give them much logging, so we’re not able to figure out exactly what happened. That’s too bad, but at least we’re mostly restored (still some events missing, but we’ll live with that). They ended up undeleting all events that were deleted when the issue happened. I think we could have done that on Friday, and still not sure why it took so long to sort out, but since this post went out Google became really involved and responsive, so I’m happy with that. And that, as they say, is that.