Latency issues in the midst of upload increase

[PhilipH]

Hello all,

Rogers started increasing the upload speed on my node last weekend, they started rebooting the modems every morning at 5am for a few days and there were an increase on the uncorrectable code words which made me think something is being worked on, then I noticed the an increase to the upload speeds up to 175mbs. Of course with all the tinkering that you guys do as of late you mess up the stability of the connection, now I have waves of increased latency throughout the day which is a problem for my work connection and online gaming. 

Now I understand that the upload increase is a result of a midsplit, however how do you do a midsplit when the OFDMA channel is not used? I even have doubts that OFDM is even used on the Dowsntream side, please see all the details below:

Traceroutes:

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 1.341 ms 1.317 ms 1.257 ms
2 pool-99-246-168-1.cpe.net.cable.rogers.com (99.246.168.1) 46.327 ms 46.257 ms 46.380 ms
3 24.156.150.225 (24.156.150.225) 42.466 ms 45.554 ms 45.705 ms
4 3032-cgw01.wlfdle.rmgt.net.rogers.com (209.148.232.37) 43.852 ms 42.090 ms 45.750 ms
5 209.148.235.214 (209.148.235.214) 45.342 ms 45.474 ms 45.603 ms
6 * 72.14.216.54 (72.14.216.54) 14.577 ms 12.863 ms
7 192.178.98.191 (192.178.98.191) 21.928 ms 192.178.98.117 (192.178.98.117) 24.377 ms 192.178.98.195 (192.178.98.195) 23.628 ms
8 142.251.68.27 (142.251.68.27) 24.087 ms 172.253.69.113 (172.253.69.113) 21.713 ms 23.287 ms
9 dns.google (8.8.8.8) 23.932 ms 23.359 ms 23.638 ms

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 1.362 ms * *
2 pool-99-246-168-1.cpe.net.cable.rogers.com (99.246.168.1) 36.999 ms 37.123 ms *
3 24.156.150.225 (24.156.150.225) 33.618 ms 35.168 ms 40.092 ms
4 3032-cgw01.wlfdle.rmgt.net.rogers.com (209.148.232.37) 38.241 ms 39.780 ms 39.732 ms
5 * * *
6 * * 72.14.216.54 (72.14.216.54) 12.825 ms
7 192.178.98.195 (192.178.98.195) 18.220 ms 192.178.99.35 (192.178.99.35) 18.289 ms 192.178.98.121 (192.178.98.121) 18.385 ms
8 216.239.40.255 (216.239.40.255) 18.345 ms 216.239.50.119 (216.239.50.119) 18.163 ms 216.239.49.189 (216.239.49.189) 18.118 ms
9 dns.google (8.8.8.8) 16.244 ms 17.768 ms 17.876 ms

Ping results:

ping pool-99-246-168-1.cpe.net.cable.rogers.com
--- pool-99-246-168-1.cpe.net.cable.rogers.com ping statistics ---
60 packets transmitted, 60 received, 0% packet loss, time 59079ms
rtt min/avg/max/mdev = 7.805/17.605/72.239/13.824 ms

PingPlotter - I know you do not accept this, but it clearly shows the period of increased latency and packet loss:

https://imgur.com/a/5DMWSyQ

Modem stats:

Channel IDLock StatusFrequencySNRPower LevelModulation

Downstream	Channel Bonding Value
7	8	9	10	2	3	4	5	6	1	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	0	0	33	34
Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked	Locked			Locked	Locked
591000000	597000000	603000000	609000000	849000000	855000000	861000000	579000000	585000000	279000000	615000000	621000000	633000000	639000000	645000000	651000000	657000000	663000000	669000000	675000000	681000000	687000000	693000000	699000000	705000000	711000000	717000000	723000000	825000000	831000000	837000000	843000000			352800000	922800000
38.605377	38.983261	38.983261	38.983261	40.946209	38.983261	38.983261	38.605377	38.983261	37.636276	38.983261	38.983261	38.605377	38.605377	38.605377	38.983261	38.605377	38.983261	38.605377	38.983261	38.983261	38.605377	38.983261	38.983261	38.983261	38.605377	38.983261	38.983261	40.366287	40.366287	40.366287	40.366287			39.53 dB	39.95 dB
2.799999	2.799999	2.900002	3.099998	3.799999	3.500000	3.500000	3.099998	3.000000	7.199997	3.099998	2.900002	3.000000	2.900002	2.500000	2.500000	2.400002	2.200001	2.200001	2.200001	2.299999	2.400002	2.400002	2.299999	2.299999	2.299999	2.599998	2.500000	3.299999	3.700001	3.500000	4.000000			7.699997 dBmV	2.500000 dBmV
QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256	QAM256			OFDM	OFDM

Upstream	Channel Bonding Value
Channel ID	7	8	5	6	11
Lock Status	Locked	Locked	Locked	Locked	Not Locked
Frequency	32300000	38700000	21100000	25900000	45700000-84950000
Symbol Rate	5120 KSym/sec	5120 KSym/sec	2560 KSym/sec	5120 KSym/sec	0 KSym/sec
Power Level	30.270599	30.270599	28.510300	29.020599	41.391663
Modulation	64QAM	64QAM	64QAM	64QAM	OFDMA
Channel Type	US_TYPE_ATDMA	US_TYPE_ATDMA	US_TYPE_TDMA_ATDMA	US_TYPE_ATDMA	US_TYPE_OFDMA

CM Error Codewords
Channel ID	7	8	9	10	2	3	4	5	6	1	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	0	0
Unerrored Codewords	290943348	289546336	289550829	289532664	289555360	289559848	289523516	289564850	289528144	289569447	289537103	289573928	289578617	289541781	289583112	289587674	289596690	289601332	289601685	289601804	289606234	289592197	289610891	289624488	289629070	289619918	289633464	289638028	289615248	289642655	289647254	289651738	0	0
Correctable Codewords	0	0	0	1	0	0	0	0	0	12	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Uncorrectable Codewords	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

Last year I went through something similar for FIVE MONTHS until it got fixed in November and now it seems it is happening again as I said every time when I see Rogers tinkering on the node I am getting stressed out it will be a mess...

**Labels Added**

[-G-]

@PhilipH  The upload speeds in my neighbourhood just got upgraded as well.  I checked the gateway stats and can confirm that the symbol rate on the upstream D3.1 OFDMA channel is 0, so perhaps this is normal, or a bug?

If my math is right, this channel must be active to attain upload speeds exceeding 100 Mbps.

Small latency spikes are also to be expected, and some packet loss as well.  We'll have to wait and see if Rogers has done their network engineering right to cope with this modest speed bump.

[PhilipH]

Thanks for replying, what I am experiencing is that the upload speed is not constant at 175mbs or 150mbs that was supposed to, it fluctuates, currently is at 60mbs when I run Rogers hosted speedtest. I do not think those are small latency spikes as it interferes with the real time network applications that I am using (funny way to say online gaming lol).

Do you see any latency spikes on your side? 

Also after the ordeal I went through last year with the high latency spikes for month to no end and Rogers closing my tickets mere hours of opening  them and refusing to help made me very twitchy when I see my PingPlotter graphs to the node explode. I personally value low latency and stability over speed.

[-G-]

---

@PhilipH wrote:

Thanks for replying, what I am experiencing is that the upload speed is not constant at 175mbs or 150mbs that was supposed to, it fluctuates, currently is at 60mbs when I run Rogers hosted speedtest. I do not think those are small latency spikes as it interferes with the real time network applications that I am using (funny way to say online gaming lol).

---

Latency spikes would be shown by variances in your ping round trip times.  It's all about how buffering, packet prioritization and network queuing affects traffic flow and how the network is designed and engineered to perform under load.

If you are seeing variances in upload speeds, I would guess that this is due to TCP/IP congestion control kicking in.  If the TCP/IP protocol senses that the network is under load, it will back off and transmit at a lower  rate to prevent a "network meltdown".

Do you see any latency spikes on your side? 

---

Only minor, so far.  Right now, things seem to be performing well.  We'll have to see how things are this evening.

FYI, here are my current speed test results from Rogers, Ookla and the Waveform Bufferbloat test.

[57]

When I tested a few weeks back, I had upload speeds around 55.  I was surprised when I tested today and got 176...  I'm located near Yonge & Lawrence in Toronto. See sample screenshots below:

ST1

ST2

[PhilipH]

I agree when it works, I see similar results as yours, however when I have the ping spike waves then both download and upload speeds take a hit. My initial feeling was the OFDMA is not being utilized and everything goes through the D3.0 channels which is causing the issue, I do understand the OFDMA is needed for these higher upload speeds but I am not convinced it is a UI bug. And how about the Downstream channels, how can I tell they are using OFDM? When looking at the Codewords section they are not shown there, which makes me wondered if they are used at all as well?

[-G-]

@PhilipH  It looks like you have an Arris (XB7) gateway, and the stats (particularly the error stats) that display on those can be are usually wonky (displaying either zeros or totally inaccurate values) even if your service is perfectly fine.

Have you tried performing a Rogers speed test using the HomeConnect app?  https://communityforums.rogers.com/t5/Internet/Test-Your-Internet-Speed-Using-the-Ignite-HomeConnect...

I would send a private message to @CommunityHelps and ask them to assess the health of your gateway's connection to the Rogers network.  If there is a problem in your area, they can dispatch a tech to troubleshoot or ask their support teams to investigate further.  If they say your connection is fine and that there are no issues affecting your area, you might need to focus your troubleshooting on your in-home network or the computer that you are using for testing.  It is conceivable that there could be other upstream issues affecting your Internet connectivity but the @CommunityHelps team can definitely confirm whether or not your Ignite Gateway's connection is healthy.

[PhilipH]

Issue was gone for a few days until today Rogers started tinkering again this is where we are at now - https://imgur.com/a/yMaWhnY 

Tracing route to google.com [172.217.165.14]
over a maximum of 30 hops:

1 <1 ms 1 ms 1 ms Docsis-Gateway.phub.net.cable.rogers.com [192.168.1.1]
2 83 ms 129 ms 117 ms pool-99-246-168-1.cpe.net.cable.rogers.com [99.246.168.1]
3 13 ms 15 ms 10 ms 24.156.150.225
4 12 ms 18 ms 12 ms 3032-cgw01.wlfdle.rmgt.net.rogers.com [209.148.232.37]
5 * 13 ms 17 ms 209.148.235.214
6 * * * Request timed out.
7 14 ms 23 ms 13 ms 192.178.98.121
8 14 ms 11 ms 10 ms 216.239.40.255
9 13 ms 10 ms 22 ms yyz12s06-in-f14.1e100.net [172.217.165.14]

Trace complete.

Ping statistics for 99.246.168.1:
Packets: Sent = 91, Received = 91, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 7ms, Maximum = 78ms, Average = 15ms

 @CommunityHelps can you please assist? This latency is unacceptable.

[PhilipH · Accepted solution]

Alright, it has been a while and here is how the whole story has ended, I am posting this for anyone who might be experiencing the same issues.

It took me some time but one point I figured that packet loss and increased latency was coincidental with dropping of the upstream OFDMA channel to status "Not locked" and going back to the unreliable in my area Docsis 3.0 channels and until it was able to lock the OFDMA Docsis 3.1 upstream channel the latency was all over the place. This going "not locked" to "locked" was happening a lot in the evening and in the severe cases could last for hours and making the Internet completely unusable. At one point it started losing Upstream completely bringing the modem offline.

Of course I started calling Technical Support, to their credit most of them saw the issue and did not give me the usual run around to get me off their backs, although everyone was pushing to complete their satisfactory survey, which is completely annoying. I got two field techs sent to my place, very nice guys, one of them put a filter, gave me a new modem but of course they both saw the "flaps" and created tickets for the maintenance team. Meanwhile the "flaps" and the loss of upstream started to happen a lot more often, I was keep calling and calling, the phone people were horrified at one point to see how bad the upstream was.

Last Saturday as I was driving, I saw a Rogers truck at one of the big green boxes which I assumed was the CMTS but it was an amplifier box, I stopped and spoke to the person, finally someone was sent to look into the whole thing. He said the cable under the road was damaged, he came back a few days later and I spoke to him again, they brought a whole crew and marked the road with orange and red spray, although after that the issue was gone once they finished working in the afternoon. So bottom line is the network maintenance team know what they are doing but to engage them properly takes a HUMONGOUS effort, why I have to call and call and field techs to create those internal tickets and after months it gets fixed when I started calling literally every night and I was just about to escalate to the office of the president!? That was extremely bad show, and I shared those concerns with the person that called me to check why I have been calling so much. The whole thing makes me horrified to think if I was with TPIA, there would have been no chance of fixing this!