Vox Down Time

Jul 24, 2007 at 9:41 PM
42 comments

Vox was unavailable between approximately 2pm and 9:30pm PDT on Tuesday, July 24, 2007. We want to apologize for any inconvenience or frustration this caused.

Around 2pm PDT there was a power outage in the SOMA neighborhood of San Francisco which affected over 30,000 people in the San Francisco area, as well Six Apart's co-location facilities. This power outage caused downtime in several large networks, including all of Six Apart's services (LiveJournal, Vox, TypePad and Movable Type) and other networks such as Craigslist, Technorati and Yelp.

Once power was restored, Six Apart staff began an ordered process of restoring data to each of our sites. This process lasted over a span of several hours, as there is a step-by-step process to bring machines back up and make sure the network is running.

During the downtime we were able to send a brief email to Voxers who are signed up for email notifications. Once the Six Apart website was back up, we posted updated information there.

Your data is safe. Again, we apologize for the inconvenience caused by the window of time where you could not access Vox. We greatly value your presence on Vox and will bring you any further information as it comes in.

42 comments

Comments

Perhaps there might be value in battery power after all...

France was down as well ;)

Glad you're back!!!!! :)

Thank you for your hard work to get back online. I missed you! us! whatever this strange relationship is... :)

Glad everything came through ok.

I saw that the effort to get typepad up and running 4-7:30 pm...I'm assuming that took precedence over vox because of the paying customers? :-P

So dish! Was there really a pack of angry customers rioting outside 365 Main? Did a drunken employee really run through yanking connections out of racks? Did a major data/coloc center really fail to have any backup power when the public grid went south? (You know, Powell is probably close enough to run over there with a spear and an angry look...I now have visions of the 6A version of Lord of the Flies...)

Geeze... makes you wonder how people powered their blogs before "power".

I missed you Vox. <3

Vox! You're back! I missed you! I thought something was wrong with my internet connection...but its good to have you back.

[this is good]

An EDS engineer once told me, "You know your product was successful if, when you turn it off, people cry and feel a sense of loss."

I missed you, my lovely blogging platform!

[this is good]

Thank you very much for getting VOX up and running so quickly! I actually wrote you guys a little note on my blog: http://marsofel.vox.com/library/post/thank-you-sixapart.html

Oh, and the email you guys sent around wasn't unappreciated, either -- it was the first time a site had ever contacted me regarding downtime, and I really was thankful for it.

Keep up the fantastic work!

[this is good]

*HUG* *smile* *HUG!* Welcome back!

I missed you.

[this is good]

Thank's 6A for all your hard work!!!!

[this is good]

Yay Vox!

I can't believe you guys/girls are apologizing... THANK YOU FOR THE RAPID RESPONSE & RESTORATION OF SERVICE!!

I want you all to take a day off with double pay.

Vox is back! :D (i was getting scared!)
i missed u :(
thanks team :D

Oh hell yeah I was frustrated. I warn you that I might be looking for some monetary compensation for the extreme emotional distress I suffered as a result of this power outage.

Good job getting back up and running. Yay Vox!

Welcome back, and thanks for trying to keep us informed.
given the fact that you are paying for uninterrupted services I hope you are demanding accountability over the fact that your server's host didn't turn their back-up generators on for 45 minutes, at the very least you should get a rate cut, otherwise I say take your business elsewhere.

Giggleblythe replied to Lord Charon’s comment:
Jul 24, 2007
Reply

I agree with this... whoever is hosting vox, livejournal et.al. really needs a butt kicking.

So no back up plan when the electric goes out in San Francisco?

As a paying TypePad blogger I can confirm the above that "we" took presidence over Vox. There are many business blogs on TypePad and the campers were not at all happy. Even the SixApart status board was not accessible. In all honesty I think even under the most extreme conditions there should be a failsafe for the place we are urged to turn to to find out what's going on when we lose access to our blogs.

I think apologies are appropriate. SixApart has always had politeness going for them.

Things have changed a lot though. I guess that was inevitable.

I was sure it was me....

Thanks guys.

Man, thanks to you people, I actually had to be productive this afternoon ... I even made it to my afternoon meeting on time!

Power outages - especially when the back-up batteries don't matter - suck! Tell me that the 6A employees who weren't able to assist in the restoration all sat around with candles, holding hands and singing kumbaya!

Well, even if it isn't true, it's a great mental image. Thanks for all you do to keep our blogosphere a happy place! :-D

Oh the joys of power outages! I remember a similar thing happening when i was on placement. Power went out, 8 servers were offline. Bringing them up one by one and making sure the network was still ok was a mission that lasted all day!

Anyways, glad to see everything's sorted now! :D

No worries mate! I did think that it was me when I couldn't get to my Perm LJ or my darling Vox ~ thank heavens for off site accounts keeping track.

On the other hand, I now can blame you guys for FORCING me into having a twitter account!

[this is good]

You guys are the BEST!!! Dealing with power outages in any situation is tough - I appreciate so much what it took to keep everything together there, and I'm sooooooooooooooooooooooooooooooooooooo happy we're all back!

(Now I'm going to my account page and make sure I'm set up for email notification.)

Welcome back.

As a data center hosting manager, I'm curious who your Co-Lo provider is? I work with a facility on the east coast who had two outages within a week last year. Not fun to deal with.

glad to see vox up and running again! i do have one concern though. i didn't receive an email notification about the down time, and i'm signed up for notifications.

Me too. I'm signed up for everything but neighborhood updates (and I don't think Vox downtime counts as a neighborhood update) and didn't receive any email notification from Vox. I had to google to find out what the problem was, and only then was pointed to the sixapart status page, when it was up and announcing the issue.

thank you all for working so hard and bringing vox up again so quickly-- when you were gone, I felt sad and disconnected. that's a pretty amazing effect for virtual connectivity-- what you have made is powerful.

of course I had to wonder who broke the internet. ;)

Thank you for getting us back online! Funny, despite having another blog elsewhere, where I could have got things off my chest, I actually missed Vox.

[this is good]

I slept through the whole thing :P but kudos to you guys for all your efforts. Thanks.

Thank you for getting the word out about the power outages in your area. It is refreshing to have a company who cares about free and paid customers alike. I have no problem with paid customers getting priority. As a free user of VOX I appreciate all that has gone into making this a great site to blog on.

In perspective there were others affected by this power outage who were probably doing some very important things that impact the lives of their families, friends and neighbors. Lets hope no one was hurt as a result of the power outage.

[this is good]

it happens, hell my power got shut OFF at home yesterday! at least yours is back on and vox is back!

[this is good]

I agree with what Marisa said... absolutely ACE customer service. Communication to the best of your ability when your avenues were seriously constrained. Keep up the stellar work. If I left my heart in San Francisco, then Vox has clearly taken possession of it since...

Its Ok, I forgive you!

Me, again - if there's ever another outage (knock wood??) here's someone who may help (not that you really needed it): the Power Grid Angel

Well, maybe not...anyway -thanks again for getting back online. ::smoochies::

it's kind of weird the devotion you inspire.

do you have bad dreams at night?

No problem. Glad SF is safe - at least your blackout was a blip in time, parts of the UK are still desprately submerged. Glad your servers aren't under water.

I guess I'll go back and delete the rant on mah blog about this....NO! don't read it...

WAIT!!

LOL I live next to the notorious SOMA district PG&E sub station.
3 years ago a transformer "not a robot" exploded and caused the longest holiday weekend blackout. Power for most of the Market street businesses were out.
But I found out one thing about my home.
My home is directly wired to the substation and therefore even if the whole neighborhood is out of power, My home is the only home on the block that has power.

If my memory serves isn't PG&E the utility that was featured in the movie "Erin Brockovich" that was sued for contaminating ground water? I am glad everything is back to normal with VOX.

You guys should get some UPSes. :P
At least it was a widespread outage and no fault of your own, like your generator running out of bio-diesel... just sayin'.

Anyway, I'm glad everything came up gracefully. Good job!