lost service at 7:20pm

[Rik]

Only because it fell under Scottish jurisdiction. 

[David]

Thats because a prod isnt classed as a probe up there 

[Rik]

It depends what you prod.

[David]

And where.... this was not in the street 

[Simon]

'Up there'?  Up where, David?   

[Rik]

Pompeii?

[David]

ohhhhhhhhhhhhhhhhh misssus 

[Rik]

 

[Ray]

 

[hells bells]

My connection went down last night and I phoned out of hours. Simon got back to me quite promptly (thanks for that Simon!)
Through past experience I know how long BT engineers can take with these things so I went to bed with a good book instead. 

Checked the status this morning and saw everything was sorted and I know that Simon and the crew will be smacking knuckles to get answers out of BT about this.

Just so you know, we know its not IDNet at fault here and we still appreciate the good service you give us!
Keep up the good work!

[hells bells]

Unfortunately my connection speed didn't double over night though! 
There again, it didn't decrease either! 

[ou7shined]

Well my connection's back to normal. 

[Simon_idnet]

We have now discovered that one of our Host Link circuits was down due to a BT PEW (Planned Engineering Works) that they hadn't bothered to warn us about!

More to the point, the service should be resilient such that customers automatically fail-over to the other Host Link circuit. We have now discovered that the failure of the fail-over was all our fault, according to BT - there's a surprise! Apparently the configuration of our routers that talk to their routers is wrong and should have been setup differently - I suppose we are guilty of not being clairvoyant!

This is the best bit: when one circuit fails BT see that no traffic is being sent down that circuit and their "intelligent" load-balancing algorithms see this and so try to balance the traffic by sending all new customer logins to the "under-utilised" circuit - that one that's broken!!

The way around this is a complicated bit of BGP trickery which lets our routers force BT's load-balancer to send all traffic to us via the circuit that is not broken. A very clever and helpful chap at BT Operate has assisted us in implementing their BGP fix on our Cisco LNS routers. The configuration has now been verified and we will be testing it during a quite period.

[David]

  must you ? v

Thanks Simon 

[Simon]

Thanks for the update, Simon.  I guess you need to polish your crystal ball.  

[zappaDPJ]

I suppose we are guilty of not being clairvoyant!

Mystic Meg to the rescue! I've heard she can foretell planned engineering works better than BT 

Thanks for the update Simon

[Technical Ben]

...

This is the best bit: when one circuit fails BT see that no traffic is being sent down that circuit and their "intelligent" load-balancing algorithms see this and so try to balance the traffic by sending all new customer logins to the "under-utilised" circuit - that one that's broken!!
...

(emphases mine)
I laughed out loud (very much so) to that little snipped. Better than any jokes I've heard in ages.... oh wait, it's not a joke.  

PS. BT need a "If [router down] = true, then [swap to other router] = true" switch or something in their coding. 

[Rik]

BT - how to cock up the world in one easy lesson.

[David]

Buzby has a lot to answer for 

[Glenn]

I think he's dead by now, David.

[Rik]

And buried.

[David]

Not another one...

[Rik]

There's always another one.

[David]

Cheerful Charlie 

[Gary]

Cheerful Charlie 

Yep he is bi-winning