The BGP blip we experienced this afternoon was due, we suspect, to a subtle bug affecting some of our older linux-based routers (in this case derek). As mentioned before, we are in the process of replacing these with new FireBrick FB6000 routers – we actually have the FB6000s to replace derek and clive, and are in the process of configuring them. In the meantime we are keeping a very close eye on the old routers.

Routing to Amsterdam is now recovering, and services returning to normal. This appears to have been a strange BGP blip, which we are investigating.

The problem in Amsterdam appears to be an odd routing problem that is affecting some people and not others. Those affected will have trouble reaching servers hosted in our Amsterdam DC, which include outgoing smtp server and one of our DNS relays.

Apologies for any problems this may be causing.

We are currently suffering some sort of problem with our data centre in Amsterdam, and are investigating

You may have had trouble calling us during the last half an hour. A power outage at Telehouse North may have affected some incoming calls. Apologies for that.

As an aside, that power outage affected many other ISPs and could have caused some problems on the internet at large, however it did not directly affect our services.

An update on yesterday’s routing problem – due to a sequencing problem, BGP on a particular router in our core cluster (derek) got into a state whereby it was announcing routes that it couldn’t fulfil. This resulted in some traffic, particularly ADSL traffic, being black-holed. Left to its own devices, routing would have recovered in about 10 minutes. However we decided stop and restart BGP on derek, so routing was restored in about 3 minutes.

To avoid this happening again, the short term solution is a subtle change to the config to avoid this particular sequencing problem. The longer term solution is to replace the routers with FB6000 routers, which we are hoping to do in the next few weeks

Apologies for any problems this may have caused

We have just suffered a brief outage due to a problem with our core routers. Routing is now back to normal and we are investigating.

The Amsterdam connectivity problems of this morning were triggered by a brief outage in the Packet Exchange network that provides connectivity between our Amsterdam and Maidenhead data centres. Unfortunately this triggered a failure of OSPF in our Amsterdam core routers, meaning that our Maidenhead routers were announcing routes to servers in Amsterdam that they couldn’t actually provide. This meant that the servers in Amsterdam were available to networks that routed via Amsterdam transit, but not those that routed via Maidenhead.

We are in the process of replacing our core routers with FireBrick FB6000s which should not suffer from this sort of problem.

The connectivity problem in Amsterdam has been resolved, we will post more details here shortly. Apologies for any problems this may have caused

We are seeing some sort of connectivity issue with our data centre in Amsterdam, and are investigating. More news shortly