Short Outage - Early Friday morning - Anyone else?

[JB]

I'm just attempting to check if one of my routers is developing a problem.

Did anyone else have a short outage about 3.00am this morning (Friday 27th May)? Especially in the north west.

I realise this may have been local to me or even the router itself but I'd be interested to know if anyone else had similar.

[jameshurrell]

Yep I monitor three lines, two in shropshire, one in bedfordshire, all three had a short outage in the early hours. Looks like an idnet blip because there is a line on the same exchange in bedfordshire with a different isp that was unaffected.

[Rik]

Are the lines 20CN or 21CN?

[Glenn]

I had the same in Berkshire

[Ray]

I had a brief loss of PPP at about 2:30am.

[tehidyman]

I am in Cornwall, presume this is the same phenomenon.

[Ray]

Yes, that's exactly the same as my graph.

[Glenn]

and mine too

[dmgeurts]

Same here in Hertfordshire from 2:20 till 2:35AM. My own monitoring showed it wasn't BT as it could reach the IDnet PE (first hop outside my network when doing a traceroute). I'm suspecting IDnet had a routing issue last night.

[Bill]

And another... South Oxon, FTTC.

[sobranie]

Ditto here on (01326)

[Bill]

Looking at the router log I get repeated instances of

Code Select Expand

  May 27 02:25:50  daemon  pppd[2622]: PPP: Start to connect ...
  May 27 02:25:50  daemon  pppd[2622]: PPP server detected.
  May 27 02:25:50  daemon  pppd[2622]: PPP session established.
  May 27 02:25:50  daemon  pppd[2622]: Using interface pppewan_1
  May 27 02:25:50  daemon  pppd[2622]: Connect: ppp_ewan_1 <--> eth0
  May 27 02:25:50  daemon  pppd[2622]: Couldn't increase MTU to 1500.
  May 27 02:25:50  daemon  pppd[2622]: Couldn't increase MRU to 1500
  May 27 02:25:50  daemon  pppd[2622]: Couldn't increase MRU to 1500
  May 27 02:25:50  daemon  pppd[2622]: PPP LCP UP.
  May 27 02:25:54  daemon  pppd[2622]: Remote message: CHAP authentication success, unit 4846
  May 27 02:25:57  daemon  pppd[2622]: Received bad configure-nak/rej:  03 06 d4 45 35 78 81 06 3e 06 26 7d 83 06 3e 06 26 7d
  May 27 02:26:00  daemon  pppd[2622]: Received bad configure-nak/rej:  03 06 d4 45 35 78 81 06 3e 06 26 7d 83 06 3e 06 26 7d
  May 27 02:26:03  daemon  pppd[2622]: Received bad configure-nak/rej:  03 06 d4 45 35 78 81 06 3e 06 26 7d 83 06 3e 06 26 7d
  May 27 02:26:06  daemon  pppd[2622]: IPCP: timeout sending Config-Requests
  May 27 02:26:06  daemon  pppd[2622]: Couldn't increase MTU to 1500.
  May 27 02:26:06  daemon  pppd[2622]: Couldn't increase MRU to 1500
  May 27 02:26:06  daemon  pppd[2622]: Connection terminated.
  May 27 02:26:06  daemon  pppd[2622]: Doing disconnect

Authentication server problem?

[jameshurrell]

Are the lines 20CN or 21CN?

Hi Rik, two lines are 20CN and one is 21CN. All had the outage.

[dmgeurts]

Looking at my PPPoE logs I noticed the following:

Code Select Expand

ppp: [wan_link0] Name: "bras-red10.l-nws"

instead of:

Code Select Expand

ppp: [wan_link0] Name: "telehouse-gw4"

Next to that I saw that while my PPPoE connection bounced a few times I was given wrong IP addresses. Incidentally the IP address I was given belongs to BT (ripe query on 81.146.178.2). So all in all something went wrong on IDnet's or BT's network (or the Radius realm forwarding) disrupting the connectivity to IDnet's gateway machines, hence connecting me to BT's network instead:

Code Select Expand

May 27 01:23:35 ppp: [wan] 172.16.68.167 -> 81.146.178.2
May 27 01:23:35 ppp: [wan] IPCP: LayerUp
May 27 01:23:35 ppp: [wan] IPCP: state change Ack-Sent --> Opened
May 27 01:23:35 ppp: [wan] IPADDR 172.16.68.167
May 27 01:23:35 ppp: [wan] IPCP: rec'd Configure Ack #7 (Ack-Sent)
May 27 01:23:35 ppp: [wan] IPADDR 172.16.68.167
May 27 01:23:35 ppp: [wan] IPCP: SendConfigReq #7
May 27 01:23:35 ppp: [wan] 172.16.68.167 is OK
May 27 01:23:35 ppp: [wan] IPADDR 172.16.68.167
May 27 01:23:35 ppp: [wan] IPCP: rec'd Configure Nak #6 (Ack-Sent)
May 27 01:23:35 ppp: [wan] IPADDR 0.0.0.0
May 27 01:23:35 ppp: [wan] IPCP: SendConfigReq #6
May 27 01:23:35 ppp: [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
May 27 01:23:35 ppp: [wan] IPCP: rec'd Configure Reject #5 (Ack-Sent)
May 27 01:23:35 ppp: [wan] IPCP: state change Req-Sent --> Ack-Sent
May 27 01:23:35 ppp: [wan] IPADDR 81.146.178.2
May 27 01:23:35 ppp: [wan] IPCP: SendConfigAck #183
May 27 01:23:35 ppp: [wan] 81.146.178.2 is OK
May 27 01:23:35 ppp: [wan] IPADDR 81.146.178.2
May 27 01:23:35 ppp: [wan] IPCP: rec'd Configure Request #183 (Req-Sent)
May 27 01:23:35 ppp: [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
May 27 01:23:35 ppp: [wan] IPADDR 0.0.0.0

[esh]

Looks like a total mess. Here is some excerpts.

Code Select Expand


May 27 02:23:10 mpd: AUTHPROTO CHAP MD5
May 27 02:23:10 mpd: Using authname
May 27 02:23:29 last message repeated 7 times
May 27 02:23:36 mpd: Name: "ESR8.Miltonkeynes3"
May 27 02:23:51 mpd: MESG: Authentication failure
May 27 02:23:59 mpd: MESG: Authentication failure
...
May 27 02:27:25 mpd: Name: "ESR8.Miltonkeynes3"
May 27 02:27:25 mpd: Using authname "<<omitted>>@gw5"
May 27 02:29:27 mpd: ENDPOINTDISC [LOCAL] 74 65 6c 65 68 6f 75 73 65 2d 67 77 32
May 27 02:29:27 mpd: Name: "telehouse-gw2"
May 27 02:29:59 mpd: Name: "ESR8.Miltonkeynes3"
May 27 02:29:59 mpd: ENDPOINTDISC [LOCAL] 74 65 6c 65 68 6f 75 73 65 2d 67 77 32
May 27 02:29:59 mpd: MP MRRU 1524
May 27 02:29:59 mpd: Name: "telehouse-gw2"
May 27 02:30:31 mpd: Name: "ESR8.Miltonkeynes3"
May 27 02:31:24 mpd: Name: "telehouse-gw4"
May 27 02:31:47 mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
May 27 02:31:47 mpd: IPADDR 0.0.0.0
May 27 02:31:59 mpd: Name: "ESR8.Miltonkeynes3"
May 27 02:32:00 mpd: ENDPOINTDISC [LOCAL] 74 65 6c 65 68 6f 75 73 65 2d 67 77 32
May 27 02:32:00 mpd: Name: "telehouse-gw2"
May 27 02:32:28 mpd: Name: "ESR8.Miltonkeynes3"
May 27 02:32:28 mpd: Name: "telehouse-gw4"
May 27 02:32:30 mpd: <<IP>> is OK
May 27 02:32:30 mpd: IPADDR <<IP>>
May 27 02:32:46 check_reload_status: rc.newwanip starting


[Rik]

Hi Rik, two lines are 20CN and one is 21CN. All had the outage.

Thanks, James, that eliminates one variable.

[Rik]

I understand that a brief 'blip' caused connections to fail over to a second radius server, this became briefly over-loaded, leading to the inability to connect for a short period. That cleared itself and most people had re-connected within 15 minutes or so. There may be one or two who have stale sessions which a router re-boot should clear.

[Steve]

 Mine never reconnected without a reboot,although it was repeatedly trying for 4 hours.

[Rik]

Mine was fine, I wouldn't have known anything had happened if it wasn't for this thread.

[esh]

Are these radius servers under BT's authority?

[Rik]

I'm not sure, tbh. There definitely are BT radius servers, but whether they're reciprocated on the IDNet side I don't know.

[Steve]

I know that there was an issue with the MK BRAS yesterday but that was at midday.

[Rik]

There's 11 MUXes down in Yorkshire this morning due to a cable fault.

[Steve]

Or missing cable 

[Rik]

 

Shame if they thought they were getting copper and ended up with fibre.