Regarding Today's outage and next steps

Started by .Griff., Oct 20, 2010, 22:38:57

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

Rik

 ;D Too near reality for comfort, Mitch.
Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

Den

I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours.  :slap:
Mr Music Man.

Rik

It does paint a sorry picture, doesn't it.
Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

zappaDPJ

Quote from: Rik on Oct 26, 2010, 16:28:00
Not to mention MAC requests I'd imagine, Mitch.

I'm still here ;D

And to be fair my connection hasn't dropped once since the last major outage... wood... touching...  :pray:

Thanks for the update Simon.
zap
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

Rik

Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

zappaDPJ

zap
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

pctech

Quote from: Den on Oct 26, 2010, 18:30:32
I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours.  :slap:

Oh dear,

Alarm engineers seem to know more about the kit than BT, then.



Steve

I suppose cos ones an engineer and the other's a Bloody Technician :whistle:
Steve
------------
This post reflects my own views, opinions and experience, not those of IDNet.


Technical Ben

Thanks for updating us as much as you can.
So the files were deleted off an old system/copy, but the backup/updating part of the computer sent the request to all the servers?  :slap:
I've seen a couple of company wise programming/system errors getting released. It's not a pretty picture. I often wonder if they bother with in house testing. Some of them were big errors.  ???
I use to have a signature, then it all changed to chip and pin.

Rik

I think BT regards its customers as the testers, Ben. :(
Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

zappaDPJ

Ironically I did have an outage last night at around 12.15am. Luckily it was very short lived and given the timing I'd guess at yet more exchange work.
zap
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

Rik

Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

pctech

Quote from: Technical Ben on Oct 26, 2010, 21:28:35
Thanks for updating us as much as you can.
So the files were deleted off an old system/copy, but the backup/updating part of the computer sent the request to all the servers?  :slap:
I've seen a couple of company wise programming/system errors getting released. It's not a pretty picture. I often wonder if they bother with in house testing. Some of them were big errors.  ???

If a Cisco router can't find a startup config it will search the network for a TFTP server containing a confg file.


Simon_idnet

We've still not received an Incident Report from BT covering the outage on the 20th Oct - 4 weeks ago now! We have been informally advised that the repair work undertaken on the 10th Oct was incomplete such that the fibre break on the 20th exposed the faulty repair of the 10th (we'd already worked that out). We've been chasing weekly for a full report but apparently the only two guys who can perform a full audit are busy with the current BRAS upgrade programme because it is "not going very well at all" ...!

Rik

I think we'd noticed the latter, Simon. They're probably camped out in Edinburgh. :(

Meantime, IDNet's reputation is damaged by BT's incompetence but BT doesn't care.  :shake:
Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

esh

Haha, good grief. That is just amazing. Good to know the BRAS upgrades are going as well as your fixes  :whistle:

You get top marks for effort :)
CompuServe 28.8k/33.6k 1994-1998, BT 56k 1998-2001, NTL Cable 512k 2001-2004, 2x F2S 1M 2004-2008, IDNet 8M 2008 - LLU 11M 2011

zappaDPJ

Incredible! :shake:

Thanks for the update Simon.
zap
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

pctech

Quote from: Rik on Nov 17, 2010, 15:47:11
Meantime, IDNet's reputation is damaged by BT's incompetence but BT doesn't care.  :shake:

Bang on Rik, sadly IDNet is just another BT BTnet customer.


Rik

Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

pctech

No matter what they may say, I reckon BT Retail probably gets the best treatment.


Rik

I wouldn't be at all surprised, Mitch. Proving it might be more difficult, of course.
Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

Simon_idnet

We've finally had an Incient Report from BT covering the outage on 20th October.

There are some glaring inaccuracies regarding the timeline which we will be taking up with BT Management as, at the end of the day, our main issue is with the length of time that it took them to resolve the fault. Frustratingly, due to BT personnel availability, this meeting cannot now take place until the New Year.

To recap, one of the Gigabit links suffered a fibre break. All traffic should have failed over to the other Gigabit link but only 20CN (ADSL Max) connections successfully failed over. None of the 21CN (ADSL2+ and FTTC) traffic that had been on the broken fibre circuit managed to fail over. This was because the BT software configuration had not been reinstated correctly after the outage on the 10th October.

On a positive note, BT have now audited all other ISPs connections to check that no one else has been misconfigured similarly and they have also reviewed their configuration proceses and training and have implemented a revised template which includes quality checks throughout the setup.


Rik

Sounds like progress, Simon, but it's small consolation for the grief it caused you. :( Are they also going  to train engineers not to disconnect the working circuit? ;D
Rik
--------------------

This post reflects my own views, opinions and experience, not those of IDNet.

g7pkf

Quote from: Den on Oct 26, 2010, 18:30:32
I was called out by a customer this morning, BT engineer was on site and had disconnected the wrong pair of wires thus disconnecting the RedCare from the alarm system and was trying to say that the fault on the line was caused by the RedCare equipment. I reconnected the line and pointed that the fault was on the other line and he had disconnected the wrong one, he had been there for four hours.  :slap:

Does not surprise me I was in telecoms moved to alarms for a few years classic i remember was getting a call to a direct line site.

BT eng had dissed it and said to me he was going to trace it back as it had no dial tone so was faulty.

took me about 10 minutes to compose myself (stop laughing) and point out the error of his ways.

direct lines are what they say they go direct from customer to central monitoring station albeit provided by BT but do not have any BT equipment on them.