Poor throughput to NL

[benjanyan]

I've been suffering quite poor throughput to certain services in the Netherlands for about 2-3 weeks. This has been quite noticeable as I run my own VPN from a rented VPS in the Netherlands and it affects this.

I was previously able to max out my connection most of the time but at certain times of the day, it slows to a crawl (200-300KB/s) but probably only about two thirds of the time so it's not a particularly consistent issue. The issue has appeared and gone away in the time I've been writing this.

Now, it'd be easy to blame contention on my VPS node but this seems to affect everything to their datacentre (e.g. looking glass). It also appears to affect other datacentres as well as I see the same sort of performance on Speedtest.net (without my VPN) to a sizeable chunk of NL-based servers.

My only other connections that I can test from are a second VPS based in the UK and my phone's 4g connection. Both are performing as expected unlike via my IDNet connection. I've raised this with both my VPS provider and IDNet but haven't really made much progress in tracking the issue down. One's gone quiet after saying that they can't see anything wrong their end, the other thinks its an datacentre issue.

I think the easiest way to demonstrate this is with speedtest.net below.

To the first server it selects in the UK. Looks great (I'm on 55/10).


Middelburg also looks fine:


Amsterdam, uh oh:

But my EE 4g connection performs much better.

Tracert:

Code Select Expand

Tracing route to ams.host.speedtest.net [185.80.220.177]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  nozomi [192.168.1.1]
  2     6 ms     6 ms     6 ms  212.69.63.54
  3     6 ms     6 ms     6 ms  212.69.63.126
  4     6 ms     6 ms     6 ms  redbus-gw7.idnet.net [212.69.63.94]
  5    13 ms    13 ms    14 ms  80.249.212.69
  6    13 ms    13 ms    13 ms  ams.speedtest.net [185.80.220.177]

Similar to one in Ede:

Again EE 4g is more like it:

Tracert to the server:

Code Select Expand

Tracing route to snelheid.routit.net [37.153.234.250]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  nozomi [192.168.1.1]
  2     6 ms     6 ms     6 ms  212.69.63.54
  3     6 ms     6 ms     6 ms  212.69.63.126
  4     6 ms     6 ms    13 ms  lonap.openpeering.nl [5.57.80.6]
  5    16 ms    16 ms    16 ms  telecity-cr.openpeering.nl [217.170.0.243]
  6     *        *        *     Request timed out.
  7    16 ms    16 ms    16 ms  jointtransit.routit.nl [217.170.19.76]
  8    15 ms    15 ms    15 ms  be104.core4-a.tc2.routit.net [37.0.80.83]
  9    17 ms    16 ms    16 ms  be34.core3-a.tc2.routit.net [37.0.80.0]
10    16 ms    16 ms     *     1-1-vms01-tc2.routit.net [84.246.25.66]
11    13 ms    12 ms    12 ms  rt234bb153-37-250.routit.net [37.153.234.250]

Looking at the tracerts, there seems to be a different route to each one once it's past the first two hops after my router. I can't tell who's responsible for these first two either.

Anyone else able to replicate this issue on their own IDNet connections or have any idea what I can do to find out more? Cheers!

[zappaDPJ]

How odd. Apart from a little loss in the upstream my results are almost identical to each other, Maidstone being the nearest test server to me.

[benjanyan]

Yes it's pretty odd.

Currently behaving itself again:

[nowster]

Probably a glitch in the BGP routing tables, or a link from London to Amsterdam playing up.

[benjanyan]

The Amsterdam speedtest server seems to be behaving now whereas I still see the issue using wget. As of 10:30pm, the effects seem to be easing off again so no examples in this post.

I'm testing with:
http://lg.nl.ramnode.com/static/100MB.test
http://lg-dro.liteserver.nl/128MB.test

I'm also using TB as a control:
http://ipv4.download.thinkbroadband.com/100MB.zip

I've tested these from my IDNet connection and another VPS I have in the UK which always seems to be okay.

I'm mainly fussy about this due to having a VPN connection to a VPS hosted with liteserver in NL. I suspect it upsets a UDP OpenVPN connection more than a TCP HTTP download.

I'm getting sort of fed up with it now... I don't know what to do other than migrate in the hope that it goes away.

[benjanyan]

In the interest of backing my above post up with some numbers now here are some wget tests at 5:30 that it's starting to act up again.

Code Select Expand


yuki_n@user-PC ~
$ wget 'http://lg.nl.ramnode.com/static/100MB.test'
--2018-03-04 17:31:28--  http://lg.nl.ramnode.com/static/100MB.test
Resolving lg.nl.ramnode.com (lg.nl.ramnode.com)... 176.56.238.3, 2a00:d880:3:1::787:d6bd
Connecting to lg.nl.ramnode.com (lg.nl.ramnode.com)|176.56.238.3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100000000 (95M) [application/octet-stream]
Saving to: '100MB.test'

100MB.test          100%[===================>]  95.37M  1.01MB/s    in 79s

2018-03-04 17:32:47 (1.21 MB/s) - '100MB.test' saved [100000000/100000000]


yuki_n@user-PC ~
$ wget 'http://lg-dro.liteserver.nl/128MB.test'
--2018-03-04 17:33:03--  http://lg-dro.liteserver.nl/128MB.test
Resolving lg-dro.liteserver.nl (lg-dro.liteserver.nl)... 185.31.172.235, 2a01:6340:1:20:3::10
Connecting to lg-dro.liteserver.nl (lg-dro.liteserver.nl)|185.31.172.235|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 128000000 (122M)
Saving to: '128MB.test'

128MB.test          100%[===================>] 122.07M  3.82MB/s    in 32s

2018-03-04 17:33:35 (3.87 MB/s) - '128MB.test' saved [128000000/128000000]


yuki_n@user-PC ~
$ wget 'http://ipv4.download.thinkbroadband.com/100MB.zip'
--2018-03-04 17:33:47--  http://ipv4.download.thinkbroadband.com/100MB.zip
Resolving ipv4.download.thinkbroadband.com (ipv4.download.thinkbroadband.com)... 80.249.99.148
Connecting to ipv4.download.thinkbroadband.com (ipv4.download.thinkbroadband.com)|80.249.99.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/zip]
Saving to: '100MB.zip.1'

100MB.zip.1         100%[===================>] 100.00M  6.13MB/s    in 16s

2018-03-04 17:34:03 (6.12 MB/s) - '100MB.zip.1' saved [104857600/104857600]


And from my UK based VPS (this at least shows the remote server is not necessarily at fault although routing may vary)

Code Select Expand

yuki_n@ruuko:~$ wget 'http://lg.nl.ramnode.com/static/100MB.test'
--2018-03-04 17:34:32--  http://lg.nl.ramnode.com/static/100MB.test
Resolving lg.nl.ramnode.com (lg.nl.ramnode.com)... 176.56.238.3, 2a00:d880:3:1::787:d6bd
Connecting to lg.nl.ramnode.com (lg.nl.ramnode.com)|176.56.238.3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100000000 (95M) [application/octet-stream]
Saving to: '100MB.test'

100%[======================================>] 100,000,000 36.0MB/s   in 2.6s

2018-03-04 17:34:34 (36.0 MB/s) - '100MB.test' saved [100000000/100000000]

yuki_n@ruuko:~$ wget 'http://lg-dro.liteserver.nl/128MB.test'
--2018-03-04 17:34:41--  http://lg-dro.liteserver.nl/128MB.test
Resolving lg-dro.liteserver.nl (lg-dro.liteserver.nl)... 185.31.172.235, 2a01:6340:1:20:3::10
Connecting to lg-dro.liteserver.nl (lg-dro.liteserver.nl)|185.31.172.235|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 128000000 (122M)
Saving to: '128MB.test'

100%[======================================>] 128,000,000 89.4MB/s   in 1.4s

2018-03-04 17:34:43 (89.4 MB/s) - '128MB.test' saved [128000000/128000000]

yuki_n@ruuko:~$ wget 'http://ipv4.download.thinkbroadband.com/100MB.zip'
--2018-03-04 17:34:53--  http://ipv4.download.thinkbroadband.com/100MB.zip
Resolving ipv4.download.thinkbroadband.com (ipv4.download.thinkbroadband.com)... 80.249.99.148
Connecting to ipv4.download.thinkbroadband.com (ipv4.download.thinkbroadband.com)|80.249.99.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/zip]
Saving to: '100MB.zip'

100%[======================================>] 104,857,600 58.7MB/s   in 1.7s

2018-03-04 17:34:55 (58.7 MB/s) - '100MB.zip' saved [104857600/104857600]


So I'm seeing about 1/6th my connection speed to RamNode and 4/6th to LiteServer. The number seems to vary quite a lot. Tests to ThinkBroadband look okay and retesting from a remote UK based VPS show significantly more than my connection can handle.

And 15 minutes later, RamNode has increased to full speed but LiteServer have dropped right down (taking over 5x longer than the first test). ThinkBroadband remains sort of the same.

Code Select Expand


yuki_n@user-PC ~
$ wget 'http://lg.nl.ramnode.com/static/100MB.test'
--2018-03-04 17:45:42--  http://lg.nl.ramnode.com/static/100MB.test
Resolving lg.nl.ramnode.com (lg.nl.ramnode.com)... 176.56.238.3, 2a00:d880:3:1::787:d6bd
Connecting to lg.nl.ramnode.com (lg.nl.ramnode.com)|176.56.238.3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100000000 (95M) [application/octet-stream]
Saving to: '100MB.test.1'

100MB.test.1        100%[===================>]  95.37M  5.94MB/s    in 16s

2018-03-04 17:45:59 (5.84 MB/s) - '100MB.test.1' saved [100000000/100000000]


yuki_n@user-PC ~
$ wget 'http://lg-dro.liteserver.nl/128MB.test'
--2018-03-04 17:46:04--  http://lg-dro.liteserver.nl/128MB.test
Resolving lg-dro.liteserver.nl (lg-dro.liteserver.nl)... 185.31.172.235, 2a01:6340:1:20:3::10
Connecting to lg-dro.liteserver.nl (lg-dro.liteserver.nl)|185.31.172.235|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 128000000 (122M)
Saving to: '128MB.test.1'

128MB.test.1        100%[===================>] 122.07M   641KB/s    in 3m 2s

2018-03-04 17:49:06 (686 KB/s) - '128MB.test.1' saved [128000000/128000000]


yuki_n@user-PC ~
$ wget 'http://ipv4.download.thinkbroadband.com/100MB.zip'
--2018-03-04 17:49:10--  http://ipv4.download.thinkbroadband.com/100MB.zip
Resolving ipv4.download.thinkbroadband.com (ipv4.download.thinkbroadband.com)... 80.249.99.148
Connecting to ipv4.download.thinkbroadband.com (ipv4.download.thinkbroadband.com)|80.249.99.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/zip]
Saving to: '100MB.zip.2'

100MB.zip.2         100%[===================>] 100.00M  6.09MB/s    in 17s

2018-03-04 17:49:27 (5.92 MB/s) - '100MB.zip.2' saved [104857600/104857600]


[benjanyan]

I've sent a couple of emails to IDNet support about this and had another reply today. Brian's been pretty fast to respond but I'm unfortunately not any closer to seeing the issue resolved.

The graphs across our network to Netherlands has plenty of available bandwidth and has been running at about 50% capacity over the weekend with no issues seen so may suggest the issue is outside of our visibility I'm afraid.

I guess I'll either have to wait and see if it goes away by itself or just migrate with the assumption that it'll go away because I'm out of ideas.

[nowster]

I'm only getting 1.5MB/s out of lg-dro.liteserver.nl. My link's capable of 20MB/s. It sounds like the problem might be at the other end.

[benjanyan]

I'm only getting 1.5MB/s out of lg-dro.liteserver.nl. My link's capable of 20MB/s. It sounds like the problem might be at the other end.

Thanks. Not sure about the other end. Maybe somewhere between.

I'm seeing about 2.9MB/s on my IDNet connection yet 86.6MB/s from my UK VPS (shared gigabit) to lg-dro.liteserver.nl so the remote server is capable.

[nowster]

And just now I'm saturating my link with that liteserver.nl test.