Today's outage (February 3, 2009)

[Rik]

Firstly, IDNet would like to apologise for the network outage today. Our LNS server (that terminates broadband connections) failed and that event caused a cascade of problems: the core routers were complaining about not being able to see the routes to broadband customers which meant that our office leased-line to London couldn't access our Status Page, to update it; Our phone system couldn't cope with a 1,000 fold increase in our usual call volume (we'll be taking this up with our 0800 provider too); And when we managed to get the LNS server back up one of our Centrals (the new one) didn't respond which meant that the remaining Centrals quickly became congested. We do not yet fully understand what caused the problem with the LNS server and we are still investigating.

And all this just when we thought we were getting on top of the latency issue...

If anyone is still experiencing poor performance please reboot your router as we have blocked any new logins through to the 'old' Centrals in order to utilise the capacity in the new Central.

Again apologies to all and we can assure you that we will be taking steps in order to ensure that this does not happen again.
Regards
Simon

Original message is here.