Google down - routing issue at Telehouse?

JamesAllen · Apr 25, 2013, 14:30:04

Now, hopefully this will fix itself but I've just found that Google is 'down' and on doing a trace route I notice a failure on one of the nodes out of Telehouse. Here it is:

Tracing route to www.google.com [173.194.41.144]
over a maximum of 30 hops:

1 <1 ms 1 ms 1 ms 192.168.0.3
2 31 ms 31 ms 30 ms telehouse-gw2-lo1.idnet.net [212.69.63.51]
3 * * * Request timed out.
4 * telehouse-gw6-gi3-401.idnet.net [212.69.63.182] reports: Destination net unreachable.

Other sites are working (though I assume some will also be affected by this), for instance Facebook:

Tracing route to star.c10r.facebook.com [31.13.80.17]
over a maximum of 30 hops:

1 1 ms 1 ms 2 ms 192.168.0.3
2 31 ms 31 ms 31 ms telehouse-gw2-lo1.idnet.net [212.69.63.51]
3 * * 31 ms telehouse-gw6-gi4-400.idnet.net [212.69.63.246]

4 31 ms 32 ms 31 ms ge-2-25.r02.londen03.uk.bb.gin.ntt.net [213.130.48.161]
5 31 ms 31 ms 31 ms ae-4.r23.londen03.uk.bb.gin.ntt.net [129.250.5.40]
6 38 ms 38 ms 38 ms as-0.r20.parsfr01.fr.bb.gin.ntt.net [129.250.5.9]
7 39 ms 38 ms 38 ms ae-0.r21.parsfr01.fr.bb.gin.ntt.net [129.250.2.181]
8 41 ms 39 ms 39 ms po-2.r01.parsfr02.fr.bb.gin.ntt.net [129.250.4.178]
9 40 ms 39 ms 39 ms 82.112.96.146
10 60 ms 39 ms 40 ms ae1.bb02.cdg1.tfbnw.net [204.15.22.198]
11 39 ms 38 ms 39 ms ae2.pr02.cdg1.tfbnw.net [204.15.23.15]
12 39 ms 46 ms 55 ms po126.msw01.02.cdg1.tfbnw.net [31.13.27.93]
13 38 ms 39 ms 38 ms 31.13.80.17

Anyone else seeing this?

Bill · Apr 25, 2013, 14:31:58

Similar here- I can't connect to tbb, and a few other sites as well.

JamesAllen · Apr 25, 2013, 14:34:05

Thanks for confirming Bill - glad it's not just me, though didn't think it could be with the specific node failure in the trace route.

Hopefully IDNet will get this resolved quickly. Wonder if the turn it off and back on again on that node will sort it

Technical Ben · Apr 25, 2013, 14:35:47

Seems to be falling over in some form.

jameshurrell · Apr 25, 2013, 14:36:18

Yep, Google and Thinkbroadband down, plus my client's VPN IP address and OWA address (cannot do any work

). i've rebooted here, but made no difference.

JamesAllen · Apr 25, 2013, 14:38:42

Not sure if this is a good or bad thing, but bing.com is working so we can at least search while Google isn't accessible.

Quite amazed as I haven't been on Bing for ages - they have totally ripped off Google in almost every way down to the layout.

patc57 · Apr 25, 2013, 14:39:32

Confirmed, seems major link/route is down, as I can't reach 3 out of 5 of my US based servers. A 100 packet test MTR to one currently looks like this:

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| FTTC-IDNET - 0 | 100 | 100 | 0 | 1 | 2 | 1 |
| telehouse-gw4-lo1.idnet.net - 0 | 100 | 100 | 25 | 31 | 117 | 117 |
| Destination network unreachable. - 100 | 22 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 22 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 22 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 22 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 22 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 22 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 23 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 24 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 22 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| Destination network unreachable. - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 21 | 0 | 0 | 0 | 0 | 0 |
|________________________________________________|______|______|______|______|______|______|

miriam_idnet · Apr 25, 2013, 14:48:02

Hi All,

We are investigating an external routing/peering problem as we speak, keeping the status page updated.

Regards,

Miriam
IDNet Support

Bill · Apr 25, 2013, 14:51:41

Thanks Miriam

Technical Ben · Apr 25, 2013, 14:59:21

One of the benefits of having the forum on the companies servers, is we get to it before any of these routings? all this time I thought it was magic that kept the forums working when everything else crashed.

karvala · Apr 25, 2013, 15:09:26

The BT Bletchley core is down according to the internet traffic report.

EDIT: Actually, scrap that; I think they're using an outdated server name; looks there is never any response on that.

jameshurrell · Apr 25, 2013, 15:09:29

Fixed for me now.

JamesAllen · Apr 25, 2013, 15:12:42

I'm back up. Thanks for sorting that out quickly IDNet.

karvala · Apr 25, 2013, 15:13:00

Me too; nice and fast.

Bill · Apr 25, 2013, 15:16:03

Ditto here

One or two US sites took a little while to wake up, but they got there!

Bill · Apr 25, 2013, 15:22:22

Interestingly, because the tbb main site is dual-stack and I use IPv6 I could access my BQMs during the whole outage, and they show very different effects:

brian_idnet · Apr 25, 2013, 15:23:36

Hi All,

This should now be resolved.

Regards,

Brian

zappaDPJ · Apr 25, 2013, 15:56:18

It does seem to be. Thanks for the update Brian