Regarding Today's outage and next steps

[Rik]

Take a look here, John, top item:

http://threestore.three.co.uk/payg/?modem=1

[zappaDPJ]

On the whole, I would agree. However, I would like some resilience, though. Two outages in ten days has caused me work problems. I am thinking about a backup ISP or maybe investing in a 3G wireless router.

This is the key for me as well. I realise that for many people occasional downtime presents no more than a minor inconvenience but for others it can be quite devastating. The recent outages have have had repercussions on my daughter's school work. Her computing course is completely dependent on having access to her school's on-line services and her tutors will not accept loss of connectivity as an excuse for failure to complete a task on time. My wife brings work home which requires Internet access. Last night's outage had her travelling back to do 30 minutes work, a three and a half hour round trip. Both my wife and daughter are looking firmly in my direction as to why this is happening and I think most people here know how much I personally rely on the Internet.

I guess I've been hit harder than most, there appears to be never ending exchange work and other work on the infrastructure that continually takes me out, especially after midnight. There should at least be a system in place for warning people of planned outages, there certainly used to be. In the 5 years or more I was with Pipex (pre Tiscali), I was always informed via email of any planned work that might effect my line and in all those years I had one solitary outage. Why is there no system in place to do this now?

I don't know what to do any more, I'm at my wits end to be honest.

[Rik]

Why is there no system in place to do this now?

There is, Zap, if BT tell ISPs in time. Too often of late, they've done the work then told ISPs the next day that it was planned, but notifications weren't sent out.

[Technical Ben]

There is, Zap, if BT tell ISPs in time. Too often of late, they've done the work then told ISPs the next day that it was planned, but notifications weren't sent out.

Been there. Our systems use to go live on a Monday/Tuesday. Our weekly memos were sent out Wednesday.  We did occasionally get a weeks notice.

[Rik]

  The world conspires against us, Ben.

[Sarah]

. . .Too often of late, they've done the work then told ISPs the next day that it was planned, but notifications weren't sent out.

This is a bit of a diversion but I have been reading with interest of a puzzle in Cambridge city centre  -  a few days ago Anglian Water dug up a road to deal with a sewer collapse, and found a mysterious "subterranean chamber":
Chamber of secrets at site of sewer work

"It is this, rather than a collapsed sewer, that caused a depression to appear in the road. . .

Whoever built the concrete chamber also sliced a sewer pipe in half – taking away six metres of the tube – and then attempted to repair it using a hose.

Water company bosses said the chamber had been built "right on top of a Victorian clay pipe" and that it had been constructed "recently".

John Clare, a spokesman for Anglian Water, said: "It's very mysterious. We don't know who built this. The engineers are baffled. The sewer has not collapsed as we thought – it has been effectively removed by a third party."

Today:  Mystery of cavern solved:

"BT has now admitted that it was responsible for the cavern – which is full of fibre optic and phone lines, as well as several feet of water. It was built during the construction of the Grand Arcade in 2004."

It seems BT forgot to tell anyone ... 

Look at the link for a photo of the hose BT used to connect the two sections of severed sewer

[Steve]

Marvellous story Sarah. Are we surprised though at the guilty party.

[Sarah]

This is a big story as the site in question is probably the busiest bus, taxi and pedestrian intersection in the centre of the old city.  Any subterranean earthworks here are a public spectacle, and everyone takes an interest

[Technical Ben]

That's certainly criminal damages, no?

[Rik]

It's unbelievable that BT behaved in such a cavalier way - they seem to think they are outside all man made controls.

[Technical Ben]

It's unbelievable that BT behaved in such a cavalier way - they seem to think they are outside all man made controls.

And those above man. 

[Rik]

 

[Simon_idnet]

We've had an Interim report from BT concerning the outage on the 10th. We are forbidden from publishing it. My interpretation is that BT were running an internal clerical programme to delete old configuration data that was no longer active. Our config was deleted in the management database and that this dataset change was automatically pushed out to the routers that our Broadband Gigabit Host Links are connected to. We have been advised that this programme has been halted whilst a root cause analysis is undertaken.

We understand that humans can make mistakes though with proper training and supervision followed by sanity checks something this calamitous should have been avoided. The big issue here is the length of time that it took to be resolved despite the information that we provided during the course of the outage. We are arranging a meeting with BT Management to review the two outages and we expect them to follow this with a review of their fault investigation procedures.

Simon

[Glenn]

Thanks for the update, Simon

[pctech]

I thought they are likely to be prevented from making it public.

[Rik]

Thanks for that, Simon. Classic BT forethought.

[pctech]

BT= Ballsup Telecom ?

[Rik]

That's the polite version.

[pctech]

Indeed, I really feel quite sorry for the IDNet staff, I imagine they got an earful from a lot of customers.

[Rik]

Not to mention MAC requests I'd imagine, Mitch.

[pctech]

Sadly yes Rik.

[davej99]

Seems to me these outages could happen to any ISP. IDNET have done a good keeping us in the picture. See no point in beating them up or moving ISP. Certainly see no reason to dish it out to Tech Support - ever - it does just not help anyone. Would never dream of using essentially a BT domestic service for mission critical work, or really any homeworking that relies on it. I barely noticed the various outages and frankly consider them a plain fact of life with a fragile technology and not worth the excitement generated.

[Baz]

have they already started with the new apprentice scheme and let some kid out of school loose on the system unassisted


thats what they cant tell us 

[Rik]

It feels like it, Baz.

[pctech]

'Pressing that big red button makes the connection go faster, innit'