Increasing consistent small packet loss

[psp83]

It does look like BT are being busy with upgrades lately..

Code Select Expand

Start: 25/03/2014 02:01
Cleared: 25/03/2014 06:00
Duration: 3 Hours 58 Minutes
Message: This Planned Engineering Work is required on the 21C Network within the (Devizes) region for (capacity upgrades).
Area Codes: 01225 01380 01706

[Gary]

Do excuse the odd look, its because I'm using a Mac and users of Firefox and Safari see slow speeds on the 'TBB' download figures because of flash in these browsers. I wont download Chrome just for a speedtest, so the results are abit odd but give the general impression. So apart from the drop in speed a few weeks back, things are looking ok at the moment.

[Bill]

There's an interesting comment on TBB forums from a rep for another (big) ISP (nameless, but not a million miles away from BT) tonight:

BTW now provide us with a list of which customers are on which SVLAN within the BTW network so our quality monitoring can now look for congestion down at the SVLAN level.

And for my exchange:

BT already have case open at Didcot for degraded service, I've added the affected Plusnet customers to the case to help, we can definitely see problems on 3 SVLANs, a 4th shows up on the SVLAN status report as being upgraded and a 5th could be affected but we don't have enough customers on that one to be 100% certain.

Link

So much for "We cannot see any problems"

[Gary]

And for my exchange:

Link

So much for "We cannot see any problems"

Being totally dumb here, Bill. Who said they couldn't see any problems? On a side note I had at about 1:30pm today a dip to 40Mbps  A nice huge blob of red once again on the TBBQM's too, and looking at a Zen one they had no issue at all at that time. WTF is going on!!! 

[Technical Ben]

BT. This entire problem may have partly Stemmed from BT not being honest/able to see their own network failures. There probably was some congestion (lack of bandwidth) on the side of IDNet, which they have both fixed and understandably have been on a wild goose chase, as BT have not played fair. So I'll give IDNet more of the benefit of the doubt as it's now come to light that other ISPs are getting similar problems and they are hopefully now all banding together to sort BT out with a big push. 

A nice huge blob of red once again on the TBBQM's too, and looking at a Zen one they had no issue at all at that time.

If the new info that has come to light is true, then it's a randomish problem that means we will see the packet loss on one user, but not else where. Smaller ISPs might be more effected than larger ones, so finding the one customer on Zen who was also effected would be rather hard. Whatever the fault with BT is (the technical stuff is mentioned on the blog post over on ThinkBroardband), it is now apparent it's not just IDNet it's effecting, though it was effecting them more than others.

[Gary]

while I agree partly with you, the timing is classic IDNet, they have had this glitch at about this time for 6months now, so excuse me if I see a pattern rearing it's head as well as all the BT stuff which worries me greatly.  Small ISP's don't seem the place to be anymore, and for FTTC they offer very little benefit over larger ISP's in all honesty, as long as you choose wisely of course as when things go wrong its a potential nightmare. Oh and that blob is way to familiar, it was on both TBBQM's on craigs, and IDNets dedicated one as well, and not on any others in the bigger leagues, so as if by magic I'm back to why to square one again wondering what I'm paying for. Sorry bad day and I'm sick of the jumble of excuses BT issues and general avoidance I have seen at times in the last six months over this. 

[Steve]

I think like Gary, I can see 5 IDNet TTBQMs all with this afternoon's blip and one non IDNet TTBQM without it. I agree it's not conclusive but it is consistent with the original issue. IDNet do not appear to have any congestion issues outside of normal working hours.

[Bill]

Being totally dumb here, Bill. Who said they couldn't see any problems?

As Ben said- BT.

In emails to/from support so I can't link to them.

I think may may have decided that admitting to them was easier than trying to fix the DNS server in their test domain so I could run the TAP3 tests they were insisting on... whatever, they still haven't fixed that either!

[Gary]

I think like Gary, I can see 5 IDNet TTBQMs all with this afternoon's blip and one non IDNet TTBQM without it. I agree it's not conclusive but it is consistent with the original issue. IDNet do not appear to have any congestion issues outside of normal working hours.

Thats the issue, Steve, as you said while not conclusive for myself this is way to familiar, but never appears at weekends, which makes me very suspicious and less likely to blame BT network issues for this event tbh as this was part of the original issue which I don't think can truly be passed off to BTW or BTOR.

[Bill]

but never appears at weekends

Saturday:



Sunday:



Last weekend as it happens but it doesn't really matter- they're typical since Christmas.

[Gary]

Your graphs are congestion/packetloss though are they not Bill? I don't see any at weekends at the 1:30pm slot and none in the evening at any point, the TBBQM's I can see on other sites from IDNet don't show anything like your graphs either, so I think todays glitch is IDNets issue, sadly yours isn't. I see no issues outside iDnets working hours at all. 

[Bill]

Ah, OK... I hadn't kept up with the thread drift

[Bill]

I hope I'm not tempting fate by posting this, but by this time of day my BQMs (see sig) are normally looking a bit sad, but they're not... even a speedtest looks quite good:



A touch of congestion maybe (the low single-thread result) but that's the best I've had in the evening for a long while!

Maybe one of the slight glitches earlier was me being fixed (so to speak )


Note to Zappa- hang in there buddy, there is yet hope!

[psp83]

Being totally dumb here, Bill. Who said they couldn't see any problems? On a side note I had at about 1:30pm today a dip to 40Mbps  A nice huge blob of red once again on the TBBQM's too, and looking at a Zen one they had no issue at all at that time. WTF is going on!!! 

so finding the one customer on Zen who was also effected would be rather hard. Whatever the fault with BT is (the technical stuff is mentioned on the blog post over on ThinkBroardband), it is now apparent it's not just IDNet it's effecting, though it was effecting them more than others.

I don't know if Gary is talking about my TBBQM but if he is, remember I was a IDnet customer with the same issue, packet loss during the day... I migrated and the problem went away....... So a change of ISP fixed my problem, everything else is the same... This points the finger at IDnet still suffering from peak time (business hours) congestion some days.

I hope everything get fixed for you all soon though.

[Technical Ben]

I kind of agree PSP, but for different reasons. It would seem it's been down to the way BT is managing ISPs. So as always, IDNet gets the short end of the stick. But as the poor management seems to stem from BTs integration fo the networks, it's creeping up on the other ISPs as well now.

For example, this is not IDNet related, but is noticed by all now: http://www.thinkbroadband.com/news/6360-no-sign-yet-of-bt-wholesale-congestion-improving.html

Thats the issue, Steve, as you said while not conclusive for myself this is way to familiar, but never appears at weekends, which makes me very suspicious and less likely to blame BT network issues for this event tbh as this was part of the original issue which I don't think can truly be passed off to BTW or BTOR.

As far as I can tell, either starting with the ISP and it's own allocation of bandwidth, there seems to be a problem with BT not properly allocating or directing that traffic. So, a company like IDNet will be hit more often than others, as from what I can tell, they like to manually and finely manage their own network. A larger network/ISP or one that "buys in bulk, ships out" might be lesser effected, as random blips from the current bug with BT will show less. It's not necessarily that IDNet constantly have too little bandwidth, but that currently on the Exchange level, something at BT headquarters is not showing IDNet when and how they need to correct any problems for customers.

Especially as other companies have now reported a similar problem, hopefully enough of them going to BT and saying "No, our customers are still reporting problems, fix it" will get the job done. 

Your graphs are congestion/packetloss though are they not Bill? I don't see any at weekends at the 1:30pm slot and none in the evening at any point, the TBBQM's I can see on other sites from IDNet don't show anything like your graphs either, so I think todays glitch is IDNets issue, sadly yours isn't. I see no issues outside iDnets working hours at all. 

If it's exchange limited (which I think the technical details on TBB blog mention), then everyone will get slightly different results. As some of us IDNetters are on the same exchange (or virtual circuit etc), then we are getting "false positives" to the problem being with IDNet. Where as bigger companies would see lots of customers getting random (as on different exchanges/virtual circuits) packet losses, ours instead matches up. At least that's my theory so far. It's entirely assumption though.

I really hope it is solved, as we would not want it getting worse, and I'm sure people are working really hard to correct it (I've been in jobs where we all did the best we could in circumstances we had no control over).

Finally, there seem to be some strange "rules" from BT that don't quite filter down to users in the most apropriate way. Such as the new "3% packet loss is expectable" or "FTTC speeds for 90% of the time, but not during congestion". :/

[joe]

it's not just iDNet that's for sure



BT speed test 13.11Mbps (80/20 service) attached

[Technical Ben]

Is that FTTC from another ISP? I just read this on TBB forum:

The Service Level Agreement (SLA) is that users should achieve the above throughputs for 90% of the busy 3 hour period.

Which means, it's "by design" that FTTC speeds fluctuate wildly? Wow.   

[joe]

Ben,

Yes it is. The packet loss occurs less frequently but is not absent altogether.

[Bill]

The packet loss occurs less frequently but is not absent altogether.

Up until yesterday I would have agreed with that- I've almost always had a scattering of red dots on the BQM throughout the day. Not enough to bother about.

It's early days yet, but since whatever was done yesterday to remove my evening horribleness there isn't a trace of red, not a solitary pixel... so it can be done!

[Gary]

Which means, it's "by design" that FTTC speeds fluctuate wildly? Wow.   

I dont think you can say that, ADSL2+ can do the same, there has to be elasticity in any connection to allow for congestion, but its not designed to do it wildly, and up till the 'Mavericks' issue it didn't at IDNet. When packetloss knocks out ppp as it did yesterday it aint good. At least Bill is recovering.

[Technical Ben]

Up until yesterday I would have agreed with that- I've almost always had a scattering of red dots on the BQM throughout the day. Not enough to bother about.

It's early days yet, but since whatever was done yesterday to remove my evening horribleness there isn't a trace of red, not a solitary pixel... so it can be done!

I've had similar experiences now. Some days perfect, others it's back. Which again makes me wonder if it's exchange congestion, or a bug in BTs management of Exchanges or the Virtual equivalent (which would mean it would only effect certain ISPs at a time, as each is on a different Virtual circuit though on the same Exchange)? Though again, it could be at the ISP level I suppose.

[zappaDPJ]

Note to Zappa- hang in there buddy, there is yet hope!

Well... ...strange things have occurred which I think are worth documenting. Here are three fairly randomly chosen BQM snapshots from the last two months which are typical of what happens every day plus one from Tuesday (18th).







As you can see the pattern is predictable and very similar in nature to what Bill was seeing on his BQM. Two early afternoon engineer's visits failed to find a fault which was predictable because the problem is clearly peak time congestion. Then, as if by magic, suddenly a loss of sync occurred in the early hours of Wednesday morning!



You don't need to be a rocket scientist to see the result. It's too early to be sure but it appears that whatever caused the loss of sync has fixed my problem. Perhaps an upgrade at my exchange? 2.00am would be about the right time for it. Somewhat unexpectedly I had a third, totally unannounced visit from an Engineer today. He replaced the entire BT box and supplied a new modem. To be honest I'm not sure what's going on, IDNet hadn't booked this engineer and it was only by chance that I was in when he called.

Anyway, long story short, my problems may be over but nobody really knows why. Does that sound familiar? It must be magic

[Bill]

Anyway, long story short, my problems may be over but nobody really knows why. Does that sound familiar? It must be magic

I suspect somebody in BT knows perfectly well why, but they're not saying

And "over-subscribed SVLAN" probably comes into it somewhere... interesting that you had a ~20 minute loss, that's what I've seen mentioned on tbb as the usual sort of time to sort an SVLAN out, but I didn't get one. Odd.

Glad you're back in business

[Bill]

Whether IDNet have any problems of their own or not I can't say, but today's update at the end of this tbb news item is interesting.

To me, it looks as close to a mea culpa as you'll ever get from BTW.

[Baz]

Whether IDNet have any problems of their own or not I can't say, but today's update at the end of this tbb news item is interesting.

To me, it looks as close to a mea culpa as you'll ever get from BTW.

what as in a " we've made a balls up but wont say so "