BT Failure

[Lance]

Taken from the network status page:

2010-10-20 11:24: We have yet another BT service failure this morning affecting around 50&% of lines. BT are aware and are investigating.

[Lance]

2010-10-20 12:31: BT have identified a fault with one of our two Host Link connections to them. The fault investigation is ongoing and also why those affected customers have not automatically failed over to the working connection.

[Lance]

2010-10-20 13:30: ADSL Max lines on the 20C broadband network are successfully failing over to the working BT Gigabit link but ADSL2+ lines on the 21C network are not. BT are diagnosing.

[Lance]

2010-10-20 14:49: We suspect that BT are filtering our ADSL2+ customers' login authentications. We have informed BT of this and have asked to them to check through their filter configurations.

[Lance]

2010-10-20 15:44: BT Openreach are tracing the fault on the failed Gigabit circuit and BT Operate are working on the ADSL2+ login fault. We still have no estimated time to fix for either fault. We are chasing them very hard on this.

[Rik]

 
Updated: 20 October 2010 16:42:02
Both of these faults have been escalated to the highest levels within BT due to the toally unacceptable length of time it is taking to find and fix the problems.

[Rik]

 

Message: BT have now resolved the authentication problem so that customers on the WBC 21CN network are able to login through our functioning Gigabit link. We are still awaiting a fix for the other Gigabit link.

[Steve]

 

2010-10-20 19:30: BT have now broken both our circuit into them!

[Rik]

 

UPDATE: 20 October 2010 19:00
BT broke all circuits so that ADSL Max customers were disconnected, and fibre,
ADSL2 customers etc were disconnected again.

No explanations yet received from BT, but connections began to be restored around 19:40.

[Rik]

 

Updated: 20 October 2010 21:11:17
Status:In Progress
Message: The outage this evening was caused by an Openreach engineer testing the wrong circuit - he took the live circuit off line, tested it and declared it operational! We're sending them back to fix the circuit that is still down.

[Steve]

 


Service: Announcement
Posted: 21 October 2010 00:38:51
Updated: 21 October 2010 00:40:07
Status:In Progress
Message: Openreach have located a break in the fibre cabling for our failed circuit and are repairing it.

[Steve]

 

Service: Announcement
Posted: 21 October 2010 07:12:51
Updated: 21 October 2010 07:15:12
Status:In Progress
Message: Both circuits are fully operational and full resilience was restored at 03:15.

[Simon]

Thanks for the updates, Steve.

[Rik]

Sixteen hours for the fix - it's not really good enough from BT is it.