Regarding Today's outage and next steps

[pctech]

AAISP didn't seem to beat around the bush (neither do I) and I used to find reading their status page and the Director's own blog very amusing but yes they have toned it down.

I don't think BT care whether its good or bad publicity as the fact is they are in a win win situation as even LLU operators rely on them to service the copper loop, the so-called 'last mile'

[psp83]

At the end of the day no one really knows how the ISP talks to BT behind closed doors.

Take AAISP for example, what they say on the status messages could be a hard man act for the public but behind closed doors they could be kissing BT's arse!

Same could go for IDnet, IDnet could be taking the more gentle approach in the public but could be beating the cr*p out off the BT management as you read this 

[pctech]

I wouldn't mind 5 minutes with BT management and a baseball bat.

[karvala]

Not after the last 2 weeks. Unless you like that kind of fun!  

On a side note. This is bad publicity for BT too. I'm in two minds about it really. Part of me says "IDNet can be honest, without embarrassing BT by exposing their faults" as this probably does not strengthen their relationship with BT. As BT would want to cover over (in a professional, not underhanded) such errors, fix them quickly, and learn from their mistakes. After all, we don't like it when people point out our faults as we walk down the street.
However, the other part of me says "But BT need to pull their socks up, perhaps making it clear what the fault is, is the only way to get them to do this". This will no doubt damage the relationship with BT. Sometimes the truth hurts, unless your willing to accept it. After all, companies that turn complaints into successes do better than those who don't even get complaints in the first place.

Basically, I'd understand if IDNet just said "BT has had a fault, we are investigating". But they currently go into a lot of detail. Which we are thankful for. I'm just not sure if BT are.  

Appeasing BT would be absolutely the wrong thing to do, period.  It means they wouldn't take their responsibilities towards IDNet seriously, or I might say even less seriously than they do now.  It's imperative that IDNet stand up to BT and hold them to their commitments.  I deal with consumer rights issues on a daily basis, and I can tell you that there are two quite distinct corporate attitudes.  One, from a company such as IDNet, is a genuine concern for their customers, a desire to keep people happy.  When dealing with those companies, a flexible co-operative approach works very well, and is much better than blowing off steam and creating a bad atmosphere.  The other, from a company such as BT, is very little concern for their customers (direct or indirect).  This comes about due to a combination of poor management, unrealistic expectations and targets which force shortcuts to be taken, and a corporate structure that allows any one individual to avoid responsibility.  Those companies operate on a line-of-least-resistance approach, and a co-operative and friendly attitude when dealing with them is taken as evidence that you can walked over without too many adverse consequences for them; they need to screw someone, and you're likely to make less fuss than the next man, so you're going to get it, regardless of whether or not you deserve it.  In that situation, you have to find where the company is vulnerable, and you have to apply pressure there which unambiguously conveys the message that it's not in their best interests to screw you over; it will cost them more to do that than to do it right.

I think if anything, IDNet have been too co-operative towards BT historically.  That's understandable, because Simon and Tim are basically nice guys who want to maintain good relations with their business partners.  I think it's becoming evident even to them, however, that goodwill counts for nothing when dealing with a company like BT.  It's only how much pain they believe you can cause them that counts in the end.

[karvala]

AAISP didn't seem to beat around the bush (neither do I) and I used to find reading their status page and the Director's own blog very amusing but yes they have toned it down.

I don't think BT care whether its good or bad publicity as the fact is they are in a win win situation as even LLU operators rely on them to service the copper loop, the so-called 'last mile'

They care about it only insofar as the dear old regulator is only ever motivated to do something by overwhelming public opinion applied via government pressure.  Enough Daily Mail headlines, or irate punters in a DTI minister's constituency surgery, and the regulator is forced to overcome his natural inertia and take some action against BT, which they prefer not to happen.  And make no mistake, the regulator will do nothing whatsoever without such pressure.

[dlorde]

On another note this forum is a godsend and Rik, Simon and the other guys (sorry if I didn't name you's) do a fantastic job really on Idnets behalf so heres a big thank you to them from me and I guess many others on this forum, THANKS GUYS!

WHS - I may not use the forum very often, but the contributions of Rik, Simon and the rest of the crew are the other major reason for me staying with IDNet. A big thank you from me too 

[sn]

No, it means that providers outside of BT provide a link to the net.

If you could expand on your one liner I would be grateful.

[JohnH]

But it gives a choice for those who may need it perhaps for reasons of speed or reliability.

But if you are on a small rural exchange only served by BT IPstream resellers, there's not much can be done, I take it. 

[pctech]

What Dorset was saying was that, BE, like BT can provide a link to deliver traffic to an ISP but if BE don't have presence in an exchange (A point of presence) then it would require a BT ADSL link to take the traffic from the customer to a POP and then onward to the ISP.

My understanding as regards BE/O2's configuration is that, if the customer's exchange does not have a POP, the traffic is delivered to the core part of their ISP network via a BT hostlink, just like IDNet which would suggest the BT backhaul to a local POP isn't provided under a wholesale agreement.

[Rik]

I agree, Ben, I was just trying to emphasise that, because they don't publicly rant and rave doesn't mean that IDNet don't lean heavily.

[Technical Ben]

I agree, Ben, I was just trying to emphasise that, because they don't publicly rant and rave doesn't mean that IDNet don't lean heavily.

Oh, I'm sure they do. I was just hoping BT don't throw a hissy fit over it all. ISP:"Go clean up your phone exchange/link" BT:"But I don't wanna!  "
Oh, and karvala, I've seen the "poor management, unrealistic expectations and targets which force shortcuts to be taken, and a corporate structure that allows any one individual to avoid responsibility." That kills a company and it's staff. Drains it of it's real resources. Sad so many take that approach.

[armadillo]

Looking at other suppliers is fine but there is a huge caveat: act in haste, regret at length.

I have just migrated away from an ISP which changed wholesale supplier after some episodes of unreliability with their original supplier. The service then deteriorated in aspects apart from reliability of backbone to the point where I felt no option but to leave.

I am prepared to accept some outages. Always, I think it is right to judge an ISP by how they deal with problems rather than slam them for the occurrence of problems. I have had several ISPs who either did not respond adequately or who lied about it ("there is a fault affecting a very small minority of customers" etc.)

Faults will occur on any system. The only way to guard against faults is to employ multiply redundant configurations. Several concurrent wholesalers with duplication of routes. All the hi-tech kit duplicated several times with any one copy capable of carrying all traffic without degradation.

That can be done but not at £20 to £50 per month per customer.

If someone needs a guaranteed always-on service, they can pay for a dedicated adsl line with a SLA, and pay dearly.

So I think it is important not to overreact and kick BT too hard. Yes, they make mistakes. No they should not. Yes they should put some procedures in place to reduce the probability of a recurrence. But whatever they do and whoever supplies a service, no collection of humans can ever guarantee a 100% success record. The costs of going from 95% to 100% escalate dramatically with each increment of reliability and 100% is not attainable.

There is nothing in the laws of probability to say that a failure rate has to be evenly spread over time. Faults can arrive together, like buses.

I am not apologising for BT or idnet. I hate BT's help call centre as much as anyone. And I am just as annoyed when I lose my internet connection. Kneejerk reactions are never the way to go. By all means do the research. But ask customers for their opinion after, say, six months of faultless service and then see how keen people are to jump to a different provider.

[Rik]

Nicely put. 

[JohnH]

On the whole, I would agree. However, I would like some resilience, though. Two outages in ten days has caused me work problems. I am thinking about a backup ISP or maybe investing in a 3G wireless router.

[Rik]

How about a PAYG mobile dongle, John?

[pctech]

Like all good ISPs IDNet have redundent upstream connections. (at least one American transit provider which is Cogent I believe plus links to several internet exchanges) so failure in an upstream link would result in automatic route re-selection via another link.

Unfortunately the same is not true regarding bringing customer traffic in, while BT can (in theory) re-route traffic between two nodes on its network in the case that for example, someone puts a digger through a fibre, the physical hand-off to the ISP is only as resilient as the number of links they rent so if they rent one and it goes down then thats tough until BT fix it.

As has been explained in IDNet's case they had two so the other should have taken the strain.

Now I wonder if the two links were fed by the same piece of kit and whether it maybe best to connect at another physical location in addition to what they have now?

[Rik]

Assuming BT offer them that option, Mitch? We shouldn't lose sight of the fact that for many customers, the switchover happened seamlessly as it was meant to (until some engineer decided to disconnect the working circuit!). Those that couldn't connect, 21CN customers, were, I suspect, victims of October 10 - I have a gut feeling that the configuration wiping wasn't fully corrected and, as a result, showed itself again yesterday. Purely my opinion, I have no information from IDNet on the issue yet - they are still waiting for explanations from BT.

[pctech]

Yes true, I don't know the physical topology of IDNet's set up but just maybe if they had one router in another DC (BT have presence in most if not all) then maybe the engineer would have thought before travelling as its a bit of a no brianer if two connectors are side by side?

[Rik]

I honestly don't know the answer, Mitch, but BT on both outages insisted on following flowcharts rather than engaging brains.

[Steve]

Perhaps the flowchart is the only method possible for the current crop of BT engineer, especially if those with knowledge and experience have now left for one reason or the other.

[pctech]

Sounds just like work.

[Rik]

Perhaps the flowchart is the only method possible for the current crop of BT engineer, especially if those with knowledge and experience have now left for one reason or the other.

Sadly, I think you've hit the nail on the head, Steve. The experience and skills have left the building, they were too expensive for BT.

[JohnH]

How about a PAYG mobile dongle, John?

What I was thinking was along the lines that Dorset suggested. Something like a Draytek router that I can use for my regular connection, but that kicks in using a 3G modem if there is an outage.

However, the further problem is that here, mobile signals are a distant dream, but what seems possible is to invest in a Vodafone Sure Signal and use it with the router and a mobile dongle.

I keep hearing about PAYG dongles, but I can't actually find one - they all seem to be contract. Given that I only want it for resilience, I don't really want a contract.

This aside, do you think this is all do-able?

Alternatively, I could just broadband enable my second phone line and put a cheap connection on it - but dealing with Plusnet, BT or Talk Talk etc. in order to achieve it, is really anathema to me. Also, I would have to mess around between routers when I want to use the second connection, albeit hopefully rarely.

[pctech]

John, a sure signal requires a working fixed ADSL connection so would kind of defeat the object.

[JohnH]

 

Why did you have to let a little detail like that get in the way of a good theory.