Explanation of yesterday’s outages

There were 2 unconnected events yesterday which both disrupted service. Firstly, between approx 10:00 and 11:45, one of our interconnectivity providers, Datahop, suffered some failures which resulted in some, but not all traffic between our network and ‘The Internet’ being dropped. Typically, this manifested itself as websites being unavailable and certain outside monitoring services reporting services being down when they were in fact working.

Secondly, at approximately 13:30, our wholesale provider of BT ADSL connectivity, A&A, suffered a fibre outage to BT. Whilst there are backup fibres in place, services apparently did not automatically switch over and instead flapped up and down. Manual intervention saw service restores by approximately 14:00 and the automatic fail-over mechanisms are being reviewed.

As a side effect of the ADSL flapping, a very very large number of SMS and email notifications were generated whilst the outage persisted. These gradually worked their way through various queues, but had the unfortunate effect of continuing to alert users about the outage, for a further 3 hours after it had finished.

We apologise once again for the disruption.

A different problem has arisen with ADSL connections dropping this is an issue outside of our control and we are chasing for a solution. Sorry for the problems today we will post information as soon as we receive it.

The issues we saw earlier this morning have now settled down. We apologise for any issues caused.

The problem customers re seeing is due to an issue with a link provided by Datahop here is what they have advised us. “At 11:04 today we experienced a number of physical hits to the dark fibre connectivity between Telehouse North and GlobalSwitch 2. This caused multiple interface flaps which in turn caused multiple ring topology re-convergence events, that may have caused hits to your connectivity.” We appologise for any inconvenience.

A few customers are reporting, connectivity issues. We are investigating.