Loss of service reports

[Rudds]

What I know at this point.

1) IDNet rent a hostlink (chunk of fibre and terminations etc) from BT, which connects their customers to their network via BT's network. These hostlinks are supplied over two sets of fibre, one live, one there as a fallback should the first fail. That fallback should be transparent to customers, ie it should happen before you notice a lost connection.

2) A week or so back, someone at BT wiped IDNet's configuration from the system, effectively BT ceased to recognise them as a customer, so it was impossible to connect as an IDNet customer.

3) Today, the primary hostlink failed. The secondary took over, but would only connect ADSL Max customers due to a configuration problem. That took BT several hours to fix and, last time I heard, they still hadn't repaired the primary link. Connections started coming back late this afternoon, then, around 7, BT broke all connections. These started to return around 7:45.

As yet, BT have not given a satisfactory explanation for any of this.

Hey Rik thanks for the explanations and update mate.
Your efforts are truly apreciated.
Back up to around 3 megs now

[Bill]

Thanks, Simon. If you get someone to kick, can we all join in please?

A picture would do... as long as it's good enough to make a wax model from

[Technical Ben]

And as said before. AFAIK most ISPs have to negotiate around the net anyway. So you will, at some point, probably cross a pipe/server owned by BT (just like having to pass their office along a road). Then things start to go wrong.

[JohnH]

In the meantime we'll be pursuing them yet again concerning the events of today.

Beyond an abject apology, are BT obliged to compensate you for loss of service?

[Dan]

Another customer here who's not happy. I don't want compensation I want uptime and reliability, then speed. That's why I'm happy to pay the little bit extra.

I work remotely, currently while I like idnet and their service is superb for individual problems, I have no confidence in the idnet/bt supply. I'm looking for an explanation and reassurance. If it's bt at fault an apology and acknowledgement of the inconvenience caused would help.

Last chance saloon...

[Dan]

Hope our posts here can be used as evidence of the impact to bt.

[Rik]

We've made sure IDNet are aware of this thread.

[Conrad]

Rik, we are all BT customers by proxy, so I think we should all be able to send complaints to BT.

Why not have some kind of reduced bandwidth stand-by with say Clueless & Witless (or any other network provider) for a 'non-elegant' failover for when things go completely gummage? I'd rather have a really slow and rubbish connection than no connection at all.

[adamb]

A 1 month rolling customer contract cannot be good for idnet right now. I know they have it in place because they are confident that people will not want to move because of the service. If BT keep messing up then it's bound to affect business which is why the ISP should be compensated. My comments earlier about extra bandwidth compo was tongue in cheek but I completely echo the comments made a few minutes ago. I am happy to pay a little extra for a service which is reliable and fast. Maybe we should all move to South Korea where they will soon have 1GB broadband

[Rik]

Rik, we are all BT customers by proxy, so I think we should all be able to send complaints to BT.

Believe me, I've been to the top and merely got told to talk to IDNet.

Why not have some kind of reduced bandwidth stand-by with say Clueless & Witless (or any other network provider) for a 'non-elegant' failover for when things go completely gummage? I'd rather have a really slow and rubbish connection than no connection at all.

The cost would be significant, would customers pay extra for the increased reliability. Bear in mind that BT have been responsible for two major outages this year, but that between them they only accounted for some 13 hours of downtime for the worst hit users, less for many. I make that a downtime of about .15% (though my maths may be wrong ). Paying for a complete extra network to cover such a scenario would be a significant cost, and technically I'm not sure it could be done.

[gyruss]

I wonder if BT evr have this happen to their OWN customers? If they can get it right on their own connections, they have no right to mess up IDNets!

absolutely spot on i'd say.. the conspiracy theorists among us might end up thinking that the outages were more 'by design' than by fault, the precision and and effect of the faults causing total disruption at the level it has has certainly had the desired effect with the 'rival ISP's customer base'  after all, check the replies above.. noones happy about this in the slightest, me included.

but i'm not paranoid, i'll put my faith in IDNET once more i think.. but i urge idnet to please.. please begin communicating more effectively to its customer base after these 'events' so there is less conjecture.  We realise that you've had problems getting information out of BT, but ultimately further down the line your customer base feels we are not getting informatio either and its took this long for us to be told you've not had any info from BT yet aswell.

[esh]

Never ever put something down to conspiracy when incompetence is possible.

Also I would not ever consider a 1 month contract to be a Bad Thing. I pretty much only ever sign up to communications providers of any sort that have less than a 3 month contract for remote offices and the like. It's obviously different for major installations but for this sort of end user... yeah, I feel far far more comfortable with 1 month contracts.

Edit: just for the record, I have very high latency right now. But this might be things settling down.

[sobranie]

sobranie: As I understand the backup hostlink is hardware independent of the primary hostlink. So, theoretically, if it were working properly, it should fall-over elegantly without problems as Rik said. There shouldn't be need for a third hostlink unless the primary fails, and the secondary fails, which would be very unlikely. A configuration error will obviously wipe out *both*, because neither knows how to route the data. No amount of hostlinks will fix a configuration fault.

The primary and backup both failed as far as I'm concerned Esh.  Unlikely .... yes but in view of the fact thatI've had absolutely nil connx for most of the day I must presume they BOTH fell over elegantly or otherwise, hence my thoughts on having a 3rd option which is not dependant on the same hardware as that which failed.
So, the basic question is ... How much would this cost and would punters be prepared to pay extra for access to this link.

[g7pkf]

Both link's same time???

Doubt it probarly more than likely someone deleted config, this would/could happen simultaneously to both router's.

That or BT are being there normal pain in the AR****

[esh]

They may *appear* to have both failed but this isn't true, and more host links would not help.

Primary link failed. 20CN -> secondary. 21CN -> stuck on primary, hence down (BT configuration error). There was also some issue with 21CN authentication which appeared to be some other fault, unsure if it was related. If your DSL will not authenticate, then again, your data doesn't even get to the hostlinks.

[Rik]

The primary and backup both failed as far as I'm concerned Esh.  Unlikely .... yes but in view of the fact thatI've had absolutely nil connx for most of the day I must presume they BOTH fell over elegantly or otherwise, hence my thoughts on having a 3rd option which is not dependant on the same hardware as that which failed.
So, the basic question is ... How much would this cost and would punters be prepared to pay extra for access to this link.

That's the key question, Rick. It would be possible to build a network with enough redundancy to survive all but nuclear attack, but would anyone pay for it?

I hear what you say about the failure today, but one link did switch transparently, but only Max customers. Anyone on 21CN, fibre or ADSL2+ couldn't log in. My instinct is that the issues of the 10th weren't fully resolved and today's events showed up where BT had gone wrong.

Ultimately, we may all be paying the price for an excess of savings at BT. Too few staff, in particular too few experienced staff, may lay behind all this.

[g7pkf]

Well mines now back up. which is good.

an explanation from BT would be nice.

[Rik]

Wouldn't it just. I think they're out buying a new load of scapegoats right now. 

[g7pkf]

Or there planning a takeover.

[esh]

I assume this latency is going to clear up. I will monitor things until the morning.

[.Griff.]

I'm getting the bloody "BT Wholesale Service Information" on some websites now.  

(Yes I have restarted the modem/router and yes I have flushed DNS)

[Rik]

I assume this latency is going to clear up. I will monitor things until the morning.

I'm only seeing a very small increase, 14 to 15ms, which could easily be down to the time of night - I really must go and eat soon.

[Rik]

I'm getting the bloody "BT Wholesale Service Information" on some websites now.   

(Yes I have restarted the modem/router and yes I have flushed DNS)

Any examples, Griff?

[gingerjedi]

Are you saying that ADSL Max customers were not affected? Because thats not my experience.

My connection wasn't available when I got in at about 5 and it went down again an hour later, I was moved from ADSL 2 back to Max when I had the HR fault, IDNET confirmed this to me only a few of weeks back.

[Bill]

It would be possible to build a network with enough redundancy to survive all but nuclear attack, but would anyone pay for it?

That's how (and why) the original version of the internet (ArpaNet) was conceived... and if BT had implemented their infrastructure properly it would have the same resilience.

You don't need dozens of pieces of hardware to achieve it, just properly configured (!) links between connection points to provide alternate paths.