Regarding Today's outage and next steps

[Rik]

  Too near reality for comfort, Mitch.

[Den]

I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours. 

[Rik]

It does paint a sorry picture, doesn't it.

[zappaDPJ]

Not to mention MAC requests I'd imagine, Mitch.

I'm still here

And to be fair my connection hasn't dropped once since the last major outage... wood... touching... 

Thanks for the update Simon.

[Rik]

 

We glued your cables in, Zap.

[zappaDPJ]

Thanks

[pctech]

I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours. 

Oh dear,

Alarm engineers seem to know more about the kit than BT, then.

[Steve]

I suppose cos ones an engineer and the other's a Bloody Technician

[pctech]

 

[Technical Ben]

Thanks for updating us as much as you can.
So the files were deleted off an old system/copy, but the backup/updating part of the computer sent the request to all the servers? 
I've seen a couple of company wise programming/system errors getting released. It's not a pretty picture. I often wonder if they bother with in house testing. Some of them were big errors. 

[Rik]

I think BT regards its customers as the testers, Ben.

[zappaDPJ]

Ironically I did have an outage last night at around 12.15am. Luckily it was very short lived and given the timing I'd guess at yet more exchange work.

[Rik]

BT can't help themselves.

[pctech]

Thanks for updating us as much as you can.
So the files were deleted off an old system/copy, but the backup/updating part of the computer sent the request to all the servers? 
I've seen a couple of company wise programming/system errors getting released. It's not a pretty picture. I often wonder if they bother with in house testing. Some of them were big errors. 

If a Cisco router can't find a startup config it will search the network for a TFTP server containing a confg file.

[Simon_idnet]

We've still not received an Incident Report from BT covering the outage on the 20th Oct - 4 weeks ago now! We have been informally advised that the repair work undertaken on the 10th Oct was incomplete such that the fibre break on the 20th exposed the faulty repair of the 10th (we'd already worked that out). We've been chasing weekly for a full report but apparently the only two guys who can perform a full audit are busy with the current BRAS upgrade programme because it is "not going very well at all" ...!

[Rik]

I think we'd noticed the latter, Simon. They're probably camped out in Edinburgh.

Meantime, IDNet's reputation is damaged by BT's incompetence but BT doesn't care. 

[esh]

Haha, good grief. That is just amazing. Good to know the BRAS upgrades are going as well as your fixes 

You get top marks for effort

[zappaDPJ]

Incredible!

Thanks for the update Simon.

[pctech]

Meantime, IDNet's reputation is damaged by BT's incompetence but BT doesn't care. 

Bang on Rik, sadly IDNet is just another BT BTnet customer.

[Rik]

To be treated with total disdain.

[pctech]

No matter what they may say, I reckon BT Retail probably gets the best treatment.

[Rik]

I wouldn't be at all surprised, Mitch. Proving it might be more difficult, of course.

[Simon_idnet]

We've finally had an Incient Report from BT covering the outage on 20th October.

There are some glaring inaccuracies regarding the timeline which we will be taking up with BT Management as, at the end of the day, our main issue is with the length of time that it took them to resolve the fault. Frustratingly, due to BT personnel availability, this meeting cannot now take place until the New Year.

To recap, one of the Gigabit links suffered a fibre break. All traffic should have failed over to the other Gigabit link but only 20CN (ADSL Max) connections successfully failed over. None of the 21CN (ADSL2+ and FTTC) traffic that had been on the broken fibre circuit managed to fail over. This was because the BT software configuration had not been reinstated correctly after the outage on the 10th October.

On a positive note, BT have now audited all other ISPs connections to check that no one else has been misconfigured similarly and they have also reviewed their configuration proceses and training and have implemented a revised template which includes quality checks throughout the setup.

[Rik]

Sounds like progress, Simon, but it's small consolation for the grief it caused you. Are they also going  to train engineers not to disconnect the working circuit?

[g7pkf]

I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours. 

Does not surprise me I was in telecoms moved to alarms for a few years classic i remember was getting a call to a direct line site.

BT eng had dissed it and said to me he was going to trace it back as it had no dial tone so was faulty.

took me about 10 minutes to compose myself (stop laughing) and point out the error of his ways.

direct lines are what they say they go direct from customer to central monitoring station albeit provided by BT but do not have any BT equipment on them.