Anyone else down this morning?

[jameshurrell]

I monitor three IDNET lines all in different exchanges and all in different parts of the UK (2x 20century lines and 1x 21stcentury) and all three of them were down at 7am. My own line shows that the router was connected fine, but I had no external internet access and the TBB monitor was showing the line was not responding and hadn't been since around 6am. I rebooted my router and the line came back up fine and internet access established.

Need to remotely reboot the other two routers to get them back online I suspect as they are still down now.

Anyone else seeing this?

[Bill]

Yes, same here.

It was seriously weird, in spades... IPv6 worked perfectly, IPv4 to anything at IDNet (except POP3 email) was fine, outside IDNet but withiin the UK didn't work, outside the UK it was fine.

BQMs showed a red spike at about 05:45 on IPv6, the IPv4 one went solid red at that time.

Support weren't aware of any problems, but I didn't want to reboot the router in case I lost everything... I was going to wait until Simon (IDNet) got to the office and badger him, but saw your post, bit the bullet and rebooted the router... now everything seems fine again 

[joe]

Yes, I am. Checked TBBQ this morning and had no internet connection.

Rebooted Fritz and whilst TBBQ is unchanged I can now talk to the outside world.

IDNet support don't know of any problems - they witnessed my reconnect and said all OK before the reconnect as far as they are concerned.

[lozcart]

It was the same for me when I left home this morning, I could access idnetters and IDnet web site but nothing else. Now at work and checked my TBBM and this is showing a red line since 5.45am, I will contact home and get them to reboot the router.

[J!ll]

Me too.

[Bill]

whilst TBBQ is unchanged

Even that seems to have sorted itself out now

[zappaDPJ]

A similar if not the same problem occurred yesterday at around 4.30pm. Very few people seemed to notice it probably because of the time of day and because it lasted for no longer than 10-15 minutes.

In that case there was no loss of sync, but intermittently what appeared to be a total loss of throughput. It could be seen on all the IDNetters staff BQMs but the odd thing was that while for some it was a momentary blip, for others it displayed as a more prolonged issue. It's the same in this case. Yesterday I had the full duration outage while this morning merely a small blip.



Other BQMs I can see show a two hour outage.

[Steve]

I rebooted modem and router this am due to lack of time, so I'm unsure whether a router reboot would have solved it for me. I had to reboot the router yesterday evening as well.

[Simon]

My connection seems fine, but I've dropped an enquiry into IDNet to find out what happened.

[Bill]

@ Zappa- that's a bit worrying, rather implies it might happen again


edit- just checked- I had that spike ~4:30 yesterday, but only on IPv6...

[Gary]

Same here, could not get online this morning, router showed I was connected all stats ok, it was like a stale session, rebooted router and lost some speed but back online now. 

[Simon]

I'm told the outage was outside of IDNet's network, so either with BT, or one of their peering partners.  As already seems apparent, a router reboot restores connectivity.

[Gary]

The joys and mysteries of BT  Cheers for the info, Simon.

[andrue]

Hmm. My FTTC connection seems to be down. Called Support and apparently the only solution is to reboot the router, I was hoping they could just reset the session. Well that's not going to happen until this evening :-/

[mervl]

Had the momentary blips showing on TBBQM both at 1630 yesterday and just before 0600 this AM. Wasn't using th'internet at either time, but some monitoring software I have running on the one connected PC shows nothing so as it checks for a connection every two minutes any outage must have been less than that; but the Fritz router doesn't show any loss of sync at all. Previously it'd looked as though a loss of connectivity on the BTw network beyond the cab OR on IDNet's servers showed up as a resync at the PoP - so I'm thinking from the recent posts a peering problem but why would routers respond differently? I'd have thought there are only two types of reyncs that the Fritz VDSL modem/router can detect: 1. at the DSLAM and 2. with IDNet's servers (and it reports separately for them).

[Bill]

It wasn't a loss of sync in either case (for me, at least), that shows up in the router log.

There was nothing whatever in the log on either occasion, a peering/routing problem would seem to be the answer.

Though why it only affected my connection to IPv4 sites in the UK I wouldn't even pretend to understand

[Steve]

Peeering issue on IPv4 network.   Another wild guess 

[Bill]

 Fair enough 

But non-UK sites were OK... I know little about peering, but would they be handled differently to UK routes?

[zappaDPJ]

@ Zappa- that's a bit worrying, rather implies it might happen again


edit- just checked- I had that spike ~4:30 yesterday, but only on IPv6...

Those are my thoughts as well. IDNet support can't see any evidence of the issue and as of yesterday afternoon they were not aware of any planned work. Hopefully someone out there is aware of what did occur and it's not just a symptom of something about to fail.

[Glenn]

Fair enough 

But non-UK sites were OK... I know little about peering, but would they be handled differently to UK routes?

I could get to my WRRacing forum this morning which is hosted in USA, but as Bill has already stated, no UK sites.

[Bill]

Those are my thoughts as well. IDNet support can't see any evidence of the issue and as of yesterday afternoon they were not aware of any planned work. Hopefully someone out there is aware of what did occur and it's not just a symptom of something about to fail.

The other matter of concern to me is that my TV wouldn't respond to either the remote or the (soft) on/off button this morning... I had to power-cycle it.

First the TV, then the internet, I wonder what the third item is going to be? 

[andrue]

I'd have thought there are only two types of reyncs that the Fritz VDSL modem/router can detect: 1. at the DSLAM and 2. with IDNet's servers (and it reports separately for them).

That's why I hoped/expected IDNet could just bounce my PPP session. I doubt very much that the modem is the issue - it prolly didn't even notice and is still syncd like it has been for the last two months or more. It seems to me that it should be perfectly possible for IDNet to fix the issue remotely but..meh. Part of me accepts that it's my router and my problem but to be honest I'm also jaded enough now to think that they just can't be bothered.

So I have no personal email until this evening. Prolly won't miss much anyway but irritating nonetheless. Come lunch I'll have to resist the temptation to drive home just to reboot it. Maybe if I keep telling myself the chances of getting any email are low and anyway most servers will retry and won't give up after a mere eight hours of outage

[psp83]

This is my TBBG



I started to notice problems from 2am 

[Bill]

Maybe if I keep telling myself the chances of getting any email are low and anyway most servers will retry and won't give up after a mere eight hours of outage

You shouldn't lose any email afaics, except possibly if you run your own mailserver?

It will have reached (eg) IDNet's mailservers and should stay there until you read it.

[mervl]

It seems to me that it should be perfectly possible for IDNet to fix the issue remotely but..

But as it's not a local network problem and as my router maintains the connection, and yours goes down doesn't that mean to this ignoramus that your router needs to be "fixed" (a reboot): and IDNet have even less chance of doing that remotely than you? Unlucky perhaps, or are some routers better at hanging on or restoring? (The horrendously expensive and over-spec'd Fritz! does seem to be good at it, to my untrained eye). As you say the modem doesn't come into it - that just maintains the connection to the local network DSLAM.

[cavillas]

Not sure if this gives any light on the mater in hand. https://btbusiness.custhelp.com/app/service_status_consumer/ss_cat/2468,2470

[andrue]

You shouldn't lose any email afaics, except possibly if you run your own mailserver?.

Yes, I do. Much preferred 99% of the time but just occasionally..not. Mind you it's only the fourth outage in a decade which isn't bad. I caused one. ISPs caused another two and the fourth was HDD failure. Actually my SSD had a firmware bug that manifested itself after six months of 24/7 running. Luckly Crucial had issued a fix a month previously

But as it's not a local network problem and as my router maintains the connection, and yours goes down doesn't that mean to this ignoramus that your router needs to be "fixed" (a reboot): and IDNet have even less chance of doing that remotely than you?

It could do I suppose - depends what state the router is in. If it's somehow crashed or locked up then clearly they can't help. I think it's more likely though that the session it has is bad and it just needs to establish a new one. I suppose it's possible that IDNet don't have access to 'dead' sessions but I doubt it. I'll have a look tonight and let you know what I see. The chap I spoke to just didn't seem interested in a lengthy conversation. He just offered 'reboot your router' as the likely solution to any and all problems and short of pushing him (which it doesn't really warrant) that was that. I can't really blame him for it but it's the kind of anonymous unsympathetic attitude I associate with bigger ISPs.

[Ruerin]

Was down also, router needed to restart to reconnect fully.

I thought it was my hardware acting up again as I couldn't access here to check!

[Steve]

I'm just wondering, bad thing really. PPPOE is established with the BT radius server, ok I guess they can reset their end but do they have access to the BT radius server to drop your connection. My suspicion is no they don't, isn't that one reason why we get stale sessions.

[Reya]

My connection went for Dennis the Menace red and black stripes (rather apt, as I was gnashing my teeth until I could get to the forum on my phone's 3G connection to see what was up) -



Thanks for the router reboot tip. Worked a treat! 

[andrue]

I'm just wondering, bad thing really. PPPOE is established with the BT radius server, ok I guess they can reset their end but do they have access to the BT radius server to drop your connection. My suspicion is no they don't, isn't that one reason why we get stale sessions.

You know that does make sense when you put it like that. Ah well. In less than an hour my mail server will be back online one way or the other. Meanwhile I also have a collection of stripes to show:

[Steve]

It might make sense but could be utter tosh.

[karvala]

Glad to come across this thread; was wondering what had happened this morning.  Just to join in the TBB fun:-



BTW, if anyone can explain the regular periodic max latency (yellow) spikes (I'm on FTTC), I'd be curious to know the cause.  No big deal, though.

[andrue]

The good news is I'm back. I suspect Steve was correct - my router was sat there happily connected to all intents and purposes.

The bad news - had another response from Support. Apparently I shouldn't expect an uncongested service. Well fair enough - sorta. Except that I fail to see how they can blame a four day drop in throughput that occurs on the 19th of every other month on  congestion. As I've replied to Simon - is he really claiming that everyone in my town goes mad for the internet on the 19th of every other month and that this madness lasts for four straight days without a break then just stops?

Either he doesn't know what he's doing or (more likely I think) he's just trotting out the first excuse he can think of to try and get rid of me.

[Clive]

I couldn't access any UK based websites such as my bank or the BBC yet other websites were perfectly fine.  Since I was visiting the bank this morning I thought I may as well complain about their website being down and they clearly thought I was a half wit.   

[zappaDPJ]

 

[Simon]

Always check here first before going to the bank.  We may have found something else to spend your money on. 

[Steve]

Karvala - During the night do you have anything running on your computer? It looks like a a pattern of periodic but short lived heavy use.

[mervl]

Could it just be worth keeping an eye out for how different routers seem to perform? I wouldn't recommend it because of price and it's too highly spec'd for home use but my router/modem just seems not to suffer stale sessions - it hangs on (usually) even during what TBBQM appears to show as a momentary loss of connectivity; or drops the connection and always reconnects as soon as the network fault clears, until then it just keeps on trying. The only time I can recall suffering apparently with a stale session necessitating a reboot was strangely with IDnet's supplied WNR1000, even though my former nameless (and shameless) ISP supplied equipment built of bricks from the electronic stone age, but it worked faultlessly in the same way! My other Fixed Wireless connection has a different Netgear cable router that hangs on through thick and thin, but kills two birds with one stone as it doesn't have to suffer a radius server (and I don't get a bill either!.)

[andrue]

Could it just be worth keeping an eye out for how different routers seem to perform? I wouldn't recommend it because of price and it's too highly spec'd for home use but my router/modem just seems not to suffer stale sessions - it hangs on (usually) even during what TBBQM appears to show as a momentary loss of connectivity; or drops the connection and always reconnects as soon as the network fault clears, until then it just keeps on trying.

I've been toying with the idea of changing my router. I'm interested in playing with IPv6 and although my router is supposed to support it now it doesn't seem to. But I want to be sure the next one supports IPv6 since that would be part of the reason for changing. Probably I'll leave it until April. That's when my IDNet contract period ends so it seems like a good time to review everything.

[pctech]

All I would say in defence of any ISP is that I doubt BT is forthcoming with exchange backhaul load data and any they do give out is probably out of date historical.

The only folks with a realtime view are likely to be Openreach's main Network Operations Centre and best of luck trying to get them to talk to you even if you manage to get hold of a number to get through.

[andrue]

All I would say in defence of any ISP is that I doubt BT is forthcoming with exchange backhaul load data and any they do give out is probably out of date historical.

The only folks with a realtime view are likely to be Openreach's main Network Operations Centre and best of luck trying to get them to talk to you even if you manage to get hold of a number to get through.

Oh you're probably right. I think it's something odd in the DLM or profile handling so almost certainly within BT's network. I wouldn't mind if they'd at least investigate it though. It's the inane responses and disinterest that annoys me. I chose to pay a premium for IDNet because I thought I'd be getting quality support. Maybe it's just a lack of customer awareness but personally I'd find 'It's a fault in BT's network and not in our control' to be more acceptable than some cock-and-bull about congestion and suggestions that second-best is all I should really expect from a residential connection.

[pctech]

Hmmmm, I'd agree with you.

Ofcom should really be holding BT to a higher standard by making it accountable to ISPs and ultimately their end users.

Only thing I can suggest is if people are on ADSL ask to be switched to the Telefonica connection if available or move to an ISP whom I don't care to mention who are well known for giving BT grief on behalf of their customers at least until the fault is investigated.

[adrinux]

Me too. (Not fttc though) just plain old idnet via BT wholesale.
Still down, sort of.

I've been trying to debug this all day. I've been using a TP-Link WR1043ND router flashed with OpenWRT for the last year or so, paired with a TP-Link 8616 modem. No problems. But today the net was down, a simple router/modem reboot didn't fix it. To cut a long story short:

I can see the modem is in sync, adsl light is green etc.
Authentication won't happen - I get an error 'timeout waiting for PADO packets' and 'unable to complete PPPoE discovery'.
I've tried swapping most of the cables, connecting directly to the bt socket etc, made no difference.

What did work: getting my old Netgear DG834 out of the loft and plugging it in. Instant connection. This made me think it was the modem, so I put the DG834 into modem only mode and connected it to the TP-Link WR1043ND, but it still won't go through authentication, same errors. So I think I've narrowed it down to something to do with the WR1043ND.

But I don't get it. It was working fine at 1am and broken by 9am, I changed nothing. Reverting to saved settings (from may) didn't help.

I'm really getting the impression that something has changed, some setting needs tweaking on the WR1043ND and in its PPPoE stuff, but damned if I know what.

Left the DG834 plugged in for now so the wired computers have access for the evening, but I have no wireless with the DG834.

I need to get to the bottom of this though, any suggestions welcome.

[andrue]

Authentication won't happen - I get an error 'timeout waiting for PADO packets' and 'unable to complete PPPoE discovery'.
I've tried swapping most of the cables, connecting directly to the bt socket etc, made no difference.

http://en.wikipedia.org/wiki/Point-to-point_protocol_over_Ethernet#Server_to_client:_Offer_.28PADO.29

I'd guess at it being related to today's outage but it might be a coincidence. You could try re-configuring your login details in case the router has scrambled them internally. Any such scrambling might not appear on the UI. I had a bug like that once several years ago in something else I was working on. The UI had a group of radio buttons and a numeric value indicating which was active. The code treated 0 and 1 as the first two but the third was activated by 'anything else'. The back-end engine however only accepted 0, 1 or 2 (I wrote it which is why ). So a value of 255 showed to the user as button 3 but the application blew up when my engine objected.

[Simon]

Sorry, can't help on the technical front, but

[Steve]

 
Same solutions as Andrue really but mine more desparate.


Sorry I've absolutely no idea, I assume you've tried resetting the router, has rebooting the router lost some manually entered setting, obviously I'm clutching at straws.

[Simon]

Oh, just a literal stab in the dark, but I don't suppose it's at all possible that the "flashed with OpenWRT" has been removed / reset by any chance, if that would make a difference?

[mervl]

Can't help with the technical side of things  but I think it's an easy cop out to blame BT. OK the 20CN network is outdated and ADSL DSLAMS and the local loop can be dodgy; but on 21CN unless there's known local congestion (because everyone suffers) - and which affects even LLU services in some parts - BT Wholesale seem to me to be well on top of things on the backhaul. BT and Sky are investing heavily, though I'm not so sure about Telefonica and Voda (C&W) though a reducing customer base must ease the pressure. To try and ensure reliability I chose the best EU equipment I could afford - and I haven't regretted it, in fact I've come to the view it's been worth it - and that it makes more of a difference than choice of ISP if you want to optimise your connection.

After all isn't the point of automation to keep human hands off?

[Technical Ben]

Um. Is it my connection then, dying router, or anyone else experiencing more disconnects? I have a sync and "connection" to something but the internet shows as non existent. Then it starts working again.

[jane]

Just adding my experience with a TP-Link WD8960N on adsl2+. The internet was down this morning as per the other posts here. All the router reported was a failure to ping the primary DNS server which I had set to OpenDNS. I just changed the router setting back to IDNet (i.e. obtain DNS servers automatically) and the internet came back up without having to reboot. So it was probably a stale session but it had a less drastic solution thankfully.

[Simon]

Isn't this about the time BT work usually starts?  Could this be affecting anyone, from Alf's link above?:

More details
Estimated time to resolve:
Due to suspected Cable Thefts & damage caused by 3rd parties, a small number of our customers in the below areas, may experience a loss of telephone and/or broadband services. We hope to have service restored as quickly as possible and apologise for any inconvenience this may be causing.

Prees- 0194884 ( estimated clear 11/01/2013)
St Fergus- 0177983,01779872,01779875,01779876 (estimated clear 08/01/2013)
Wakefield - 01924 (estimated clear date 09/01/2013).

Otford- 01959522,01959523,01959525,01959526 (estimated clear date 17/01/12)

Midland - 01261 (estimated clear date 11/01/13)
Wanstead - 0208530, 0208989 ( estimated clear date 07/01/13)

Cosham - 02392 (estimated clear date 07/01/2013)
Ballygown - 028975 (estimated clear date 07/01/13)
Bewdley - 0129940 ( estimated clear date 11/01/13)
Barnsley - 012267 (estimated clear date 09/01/13).

https://btbusiness.custhelp.com/app/service_status_consumer/ss_cat/2468,2470

[karvala]

Karvala - During the night do you have anything running on your computer? It looks like a a pattern of periodic but short lived heavy use.

No, that's the mystery of it; the computers are all shut down at night, only the router is left on.  I also operate a security white list by MAC address and check the logs, and definitely no one else is accessing the network at any time, so it's not any activity at my end.

[adrinux]

re: Thanks for the support and welcomes.
As for the suggestions: It's nice to see you're all as much in the dark as me

I remembered something else I'd tried - hooking up the computer directly to the modem and trying a pppoe connection. Both OSX and windows 7 failed to connect that way. At least with the tp-link modem. Will try that with the netgear tomorrow.

What that means though is that the problem can't lay with the wr104nd. There just seems to be this dichotomy between combined-modem-router works and separate-modem-router fails.

fwiw OSX's system.log shows this at connection time:
pppd[821]: PPPoE connecting to service '' [access concentrator '']...
kernel[0]: PPPoE domain init
pppd[821]: PPPoE connection failed, No route to host

Reaching the point where I think I can call idnet support.

[J!ll]

Down again this morning.

[Glenn]

Mine was working fine at 5.30 before I left for work, it looks OK now too.

[andrue]

Mine looks find this morning according to TBBQM.

[Steve]

Adrinux , Have you rechecked your modem settings, obviously it is syncing but not necessarily passing on the connection information. I have used a Draytek 120 previously without issue acting as a PPPOE to PPPOA bridge

[adrinux]

Steve: Repeatedly. I've also, in effect, tried two different modems (the TP-Link modem I was using, and the Netgear DG834 in modem only mode).

Any combination that requires PPPoE to PPPoA fails to negotiate the connection:
Win7 + modem
OSX + modem
tp-link wr1043nd router + modem
and I've tried all of those combos with both modems.

But the old Netgear DG834 combined modem/router which presumably just does PPPoA works. I have no idea why.

I'm missing wireless and gigabit ethernet by sticking with the DG834

Bottom line is the combo of TP-Link WR1043ND wireless router and 8616 ADSL modem have been working fine for over a year, then yesterday they just stopped working - no storms, no power cuts, no software changes, no hardware changes.

I've also tried several different ethernet cables and bypassed the telephone extension cable, so I'm convinced now that the problem lies outside my home network.

(and thanks Andrue, http://en.wikipedia.org/wiki/Point-to-point_protocol_over_Ethernet#Server_to_client:_Offer_.28PADO.29 was interesting)

[Steve]

As a temporary solution to give you WiFi and gigabit LAN can you use the TP router in 'bridge'  mode. ie DHCP server off, This will give you a gigabit switch, a WiFi access point. The Netgear will provide NAT, firewall etc and WAN input.

[adrinux]

Yep, that's plan C. I'd far rather keep the firewall and routing on the TP in place because it's up to date with my current networking setup. What's on the Netgear is a year out of date. Toying with the idea of putting the TP in the Netgear's DMZ...what other options are there? Static routes?

But really I just want it fixed.

[adrinux]

I'm up!

After another extensive bout of googling and forum post reading yesterday evening I came across a suggestion to switch encapsulation from VC-MUX to LLC when having to use PPPoE (as you do with a separate bridging modem like the TP-Link 8616).

And lo and behold the simple change worked. So now up and running and back to normal! 

Damn annoying I went through 2 days of frustration to figure that out though 

I also have no idea why LLC works and VC-MUX doesn't.

[Simon]

Glad to hear its sorted.