| When | Who | What |
|---|---|---|
| 2009-01-06 19:12:25 | ben | A summary of today's events:-
All our ADSL connections from the 20CN (ATM) BT network come into a single rack. Although they come in on two BT Centrals, all traffic goes through a single ATM box that translates between BT's ATM and our IP network. Today we suffered a very unfortunate sequence of events. Firstly power to the entire rack was tripped. When that was restored, we diagnosed a faulty power supply. A replacement was sourced and fitted, but we then found the ATM box had failed. A replacement for that was sourced and fitted, but the engineers had trouble configuring it. Simultaneously an alternative ATM box was sourced and fitted (plan B). The plan B box finally started working about 18:00. Incidentally the plan A box also started working at 18:20. As you may be aware, for many years we have been improving our infrastructure to use multiple redundant systems and interconnects, so as to avoid a single failure taking down the network. Sadly this ATM box was one of the last remaining single points of failure. We are in the process of moving all ADSL over to BT's new 21st Century Network, which is delivered by IP over multiple redundant links. Sadly delays by BT mean this hasn't happened yet, so we still need the ATM box. Just to clarify, today's outage only affected 20CN ADSL connections. ADSL connections on 21CN were unaffected, as were all our other services (server colo etc.) Not surprisingly, we have had several clients ask us how they can avoid such outages in the future. The simple answer is to have multiple redundant internet connections, with no single point of failure. For example, you could have a backup connection using 3G. If you are interested, please do talk to us about your requirements, as we are continually developing new solutions (and as always can offer bespoke solutions if needed). |
| 2009-01-06 19:09:55 | ben | The ATM box has now been stable for about an hour. If you still have problems, please try power-cycling your ADSL router. If that doesn't help, please contact us. |
| 2009-01-06 18:11:03 | ben | The ATM box is now up and running, and ADSL lines are reconnecting. Due to the number of lines trying to connect, you may find your ADSL is not yet stable. I will post another message when things are stable |
| 2009-01-06 17:25:07 | ben | Another brief update. "Plan A" ATM box is still being worked on, the engineers sound hopeful that they are close to sorting it. "Plan B" ATM box is currently being configured, again the engineers are sounding optimistic. The race is on to see which plan works first.
We will of course carry on working this evening until it is fixed. We will also be answering support phone calls (020 7517 4902) and emails support@watchfront.net , so please don't hesitate to contact us if you need to. Once again thank you for your patience |
| 2009-01-06 15:52:41 | ben | A brief update - plan A (replacement ATM box, problems with config) is still being worked on, and plan B (alternative replacement ATM box) hardware should be arriving at the data centre very shortly. |
| 2009-01-06 14:48:53 | ben | As a Plan B we are also sourcing an alternative replacement ATM interface, in case we continue to have trouble getting the current replacement working. |
| 2009-01-06 14:29:22 | ben | Unfortunately the engineers are having trouble configuring the new hardware platform. Difficult to give an accurate time estimate at the moment, it really depends how quickly the engineers can find the cause of their troubles.
Thank you for your patience, and please rest assured we are doing all we can to rectify the problem. |
| 2009-01-06 12:00:25 | ben | Replacing the power supply has not helped, so now replacing the entire ATM interface. Hope to be back up in a couple of hours. More news shortly. |
| 2009-01-06 09:34:56 | ben | Sadly, there has been a hardware failure in just about the only component in the rack that is not dual redundant. We are doing everything we can to replace the part as soon as possible, but I'm afraid it is likely to be hours rather than minutes before services are restored.
I do apologise for this |
| 2009-01-06 09:25:43 | ben | All our ADSL is currently down, due to power supply failure in the rack housing the BT Centrals. We are working to rectify this, and hope to have some better news shortly. Apologies for any problems this may cause |
| 2009-01-06 08:10:48 | ben | We are currently suffering an outage affecting ADSL, and are investigating. More news shortly |
Old Status messages are moved to our Status History page
