Rank: Member
Joined: 11/25/2003(UTC)
Posts: 370
|
Looks like a big Outage at Resposio as of about 12PM.
Anyone know what's up? |
|
|
|
|
|
|
Rank: Member
Joined: 11/5/2003(UTC)
Posts: 1,786
|
Had a couple of customers report this too. No other info.
|
|
|
|
|
|
Rank: Member
Joined: 12/23/2003(UTC)
Posts: 909
|
|
|
|
|
|
|
|
Rank: Member
Joined: 1/10/2005(UTC)
Posts: 714
Thanks: 14 times
Was thanked: 1 time(s) in 1 post(s)
|
IM working from home today... anyone actually give them a call just to see whats going on? I dont have their number here.
|
|
|
|
|
|
Rank: Member
Joined: 11/5/2003(UTC)
Posts: 1,786
|
Yes, may not by a problem at Resposio but their sites are still unreachable at the moment from here.
|
|
|
|
|
|
Rank: Member
Joined: 12/23/2003(UTC)
Posts: 909
|
|
|
|
|
|
|
|
Rank: Member
Joined: 1/10/2005(UTC)
Posts: 714
Thanks: 14 times
Was thanked: 1 time(s) in 1 post(s)
|
hmmmm still not working for me getting errors everywhere and my login info doesn't work.
|
|
|
|
|
|
Rank: Member
Joined: 11/25/2003(UTC)
Posts: 370
|
Looks like a complete power outage took place. So some processes may need attention on your server.
The previous system shutdown at 12:01:11 PM on 5/12/2011 was unexpected. (In logs on all dedi servers) |
|
|
|
|
|
|
Rank: Member
Joined: 1/10/2005(UTC)
Posts: 714
Thanks: 14 times
Was thanked: 1 time(s) in 1 post(s)
|
yup Noah is all over it...
|
|
|
|
|
|
Rank: Member
Joined: 11/5/2003(UTC)
Posts: 1,786
|
So the colo lost complete power? Generator issues? Weather? I wonder how many systems had to fail for this kind of situation to happen. I know that several years ago during a hurricane in Richmond some data centers had trouble getting fuel for generators and did lose power for a short time.
|
|
|
|
|
|
Rank: Administration
Joined: 4/2/2004(UTC) Posts: 2,393 ![United States]( "United States") Location: Hummelstown, PA Thanks: 6 times
Was thanked: 163 time(s) in 158 post(s)
|
Yeah, this one affected us as well...been getting client calls all afternoon. Ugh! |
Aaron Sherrick
BV Commerce
Toll-free 888-665-8637 - Int'l +1 717-220-0012 |
|
|
|
|
|
Rank: Member
Joined: 12/23/2003(UTC)
Posts: 909
|
I'm pretty sure that 99.99% of us only call the power company when there's an outage... of which, 99.99% of us have no patience during the outage. |
|
|
|
|
|
|
Rank: Member
Joined: 11/25/2003(UTC)
Posts: 370
|
Setting up some new Amazon S3 and Azure Blob accounts here over the next few days. |
|
|
|
|
|
|
Rank: Member
Joined: 11/25/2003(UTC)
Posts: 370
|
After playing around with a few firewall issues I just completed setup of backup on one of Resposio's dedi servers. Not that you would ever need it or not that we will see another such outage in years it is always good to know your data is in your hands at a remote loaction. Anyway, I used CloudBerry backup and explorer applications along with Amazon S3 Simple Storage. Just starting the process with another using Microsoft Azure Blob storage. I know their are plenty of online services out there for backup but the combo of using the Cloudberry apps with Amazon S3 or Azure seem simple and not so bad on cost. |
|
|
|
|
|
|
Rank: Member
Joined: 11/6/2003(UTC)
Posts: 1,903
|
Well we are still waiting on the facility report 1 week after the event. It was quite an issue.
To our customers, first and foremost I am sorry for the issue and will continue to do anything possible to avoid downtime, now and in the future.
At four hours this was our worst outage in 14 years. We have actually moved complete cages between facilities in less time.
What happened? - From what we have received in a general outline. At around noon time Public Service Of NH (PSNH NH's main power company) requested the facility go to backup power. The datacenter is the largest power user on the grid and occasionally gets requests to go to backup power for various reasons to alleviate the grid for PSNH. Probably 10 times a year +/-. This is pretty normal at the three facilities we use, they are all huge power users.
Everything power related in the Bedford, NH facility is redundant, the main power in, the UPS battery banks, and the generators (the generators ALWAYS have a minimum of 7 days fuel on hand). When the power switch took place the main controller in the DC blew a capacitor (literally it blew up), normally this would be planned for and the redundant controller would handle the fail over. Well in this case the secondary controller is in the same cabinet (built this way by the manufacturer) due to the damage caused by the capacitor the secondary controller could not come u as well. Because they are the main power controllers, they were also unable to go back to line power.
It took the facility engineers 1 hr to find the cause. The facility had a complete controller spare in the building, but they did not have two of them. It took another hour to get the second spare on site (brought in from another facility), and then approx. 2 hrs for the rebuild.
Over the weekend they isolated the two controllers into separate cabinets to avoid the same thing happening again in the future.
Some of this info will probably be better explained after we get the full PMR.
To all of our BV customers that worked through it with us, I would like to say THANK YOU. When something like this happens and it is out of our/your control to fix, there is a feeling of helplessness and desperation that comes out pretty quick. I say to the guys all the time - panicking never helps - lol, it almost always makes things worse.
Lets hope for at least another 14 years without another event like it! |
Noah |
|
|
|
|
|
Rank: Member
Joined: 11/5/2003(UTC)
Posts: 1,786
|
Also keep in mind that Amazon's Cloud service was down for several days in the northern virginia facility just last month. Even the biggest hosting providers have outages from time to time. Sounds like a rare event indeed.
|
|
|
|
|
|
Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.