cancel
Showing results for 
Search instead for 
Did you mean: 

CODA-4582 - Open Issues for Investigation

RogersDave
Retired Support
Retired Support

*** This post was last edited May 2, 2017 ***

 

Good morning Community,

 

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

 

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

 

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

 

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

 

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

 

 

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

 

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

 

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

 

Update April 22: This problem seems resolved in firmware 2.0.10.27

 

 

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

 

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

 

Workaround: manually select higher channels (149-153-157-161)

 

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

 

 

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

 

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

 

 

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

 

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

 

 

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

 

This problem has been reported to Hitron for investigation.

 

 

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

 

Investigation continues with Hitron.

 

 

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

 

 

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

 

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users


 

Future Planned Improvements

The following are items that we are working on in parallel of the above.

  • Improvement in WiFi speeds
  • Improvement in latency / bufferbloat

 

 

Dave

 

*Edited Labels*

2,620 REPLIES 2,620

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

@Mythen

 

It happens to me on twich, I have to refresh the page often it just freeze. I think it's due tu bad routing to twitch servers, I know in Atlantic the routing is very bad for gaming to anything US east. Not sure how's the routing to twitch but I suspect it's part of the problem.

Re: CODA-4582 - Open Issues for Investigation

Mythen
I plan to stick around

@JohnBeaudin.  Its driving me nuts.  

@RogersDave  Any ideas on how we can fix the routing if that is the issue.  Pretty please? 

 

 

Re: CODA-4582 - Open Issues for Investigation

coder_1990
I plan to stick around

On 27 here, not much improvement either maybe aside from 5ghz lock and signal strength. Other than that, speeds are all over the place. Much packet loss still going on. Latency seems slightly improved. But it seems inconsistent.

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

@Mythen

 

I am not sure about twitch , but I did some test with one of my neighbor on Bell.

 

I did a traceroute to steam servers , and the routing from Bell is much better.

 

Bell 28ms

Rogers 65ms

 

Rogers is basically going from my location to Toronto and then to Chicago or new york.

Bell is going from my location to Montreal and then Chicago and that result in a much better ping.

 

That's the routing to Steam servers, now is Rogers doing the same routing for anything else going to Toronto first, I am not sure Mythen.

 

All I know is that the routing can ben improved and Dave already confirmed he wants to work on that later when he's less busy. Not sure if that's part of the twitch problem however.

 

I did a tracert to twitch and he'res my result.

 

3 20 ms 31 ms 23 ms 69.63.255.117
4 27 ms 30 ms 23 ms 209.148.229.145
5 47 ms 47 ms 48 ms 209.148.229.241
6 95 ms 61 ms 63 ms 209.148.229.225
7 49 ms 50 ms 49 ms 209.148.229.222
8 61 ms 59 ms 61 ms ce-0-2-0-1.r06.chcgil09.us.bb.gin.ntt.net [128.24
9 64 ms 63 ms 64 ms ae-5.r20.chcgil09.us.bb.gin.ntt.net [129.250.3.5]
10 125 ms 121 ms 124 ms ae-1.r20.sttlwa01.us.bb.gin.ntt.net [129.250.3.42
11 121 ms 121 ms 123 ms ae-28.r05.sttlwa01.us.bb.gin.ntt.net [129.250.2.4
12 110 ms 113 ms 112 ms ae-4.amazon.sttlwa01.us.bb.gin.ntt.net [198.104.2
13 109 ms 106 ms 109 ms 52.95.52.182
14 132 ms 109 ms 110 ms 52.95.53.139
15 127 ms 124 ms 123 ms 54.239.42.231
16 112 ms 117 ms 122 ms 54.239.42.194
17 155 ms 158 ms 208 ms 52.93.13.4
18 111 ms 112 ms 111 ms 52.93.12.253
19 143 ms 148 ms 135 ms 52.93.12.36
20 138 ms 127 ms 123 ms 52.93.12.67
21 112 ms 109 ms 116 ms 52.93.15.229
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete. (www.twitch.tv)

Re: CODA-4582 - Open Issues for Investigation

Mythen
I plan to stick around

Here's my traceroute to twitch.tv as well

 

 

 

3 11 ms 15 ms 10 ms 67.231.222.253
4 39 ms 34 ms 46 ms 69.63.250.201
5 51 ms 62 ms 54 ms 209.148.229.241
6 63 ms 63 ms 63 ms 209.148.229.225
7 58 ms 61 ms 56 ms 209.148.229.222
8 80 ms 82 ms 82 ms ce-0-2-0-1.r06.chcgil09.us.bb.gin.ntt.net [128.241.3.189]
9 82 ms 79 ms 84 ms ae-5.r20.chcgil09.us.bb.gin.ntt.net [129.250.3.5]
10 84 ms 83 ms 85 ms ae-0.r25.nycmny01.us.bb.gin.ntt.net [129.250.2.167]
11 88 ms 84 ms 81 ms ae-2.r08.nycmny01.us.bb.gin.ntt.net [129.250.6.209]
12 91 ms 94 ms 91 ms ae-0.akamai-onnet.nycmny01.us.bb.gin.ntt.net [129.250.194.62]
13 90 ms 88 ms 85 ms a23-217-168-126.deploy.static.akamaitechnologies.com [23.217.168.126]

Trace complete.

 

Re: CODA-4582 - Open Issues for Investigation

RyzenFX
I'm a reliable contributor
I've been noticing this same issue with Twitch.tv for the past 2-3 weeks now. Randomly during a stream, it would buffer or the stream would pause and not re-buffer. Based on what you guys mentioned, the buffering could definitely be caused by the routing Rogers uses.


It's so hard to watch Twitch streams with so much buffering- hopefully we can get this issue actioned ASAP.

EDIT: I just went through your trace routes and FOR SURE there's a routing problem. The severs you're hitting respond back with a ping >= 100ms which is a huge difference compared to Bell's routing.

Re: CODA-4582 - Open Issues for Investigation

Mythen
I plan to stick around

@RyzenFX That's exactly my issue with twitch. sometimes it buffers and starts other times it just freezes and I have to refresh   

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

Same thing, I wonder if other people here experience the same issue with twitch? it's not new from .27 it was doing it on .26T2 as well.

Re: CODA-4582 - Open Issues for Investigation

RyzenFX
I'm a reliable contributor
ae-1.r20.sttlwa01.us.bb.gin.ntt.net

^that is the server in which my ping jumps from a 48ms to >90ms. This should be the 9th server in the traceroute.

We're all hitting twitch servers in seattle, hosted by Amazon, I'm not entirely sure if routing is the problem.

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

@RyzenFX

 

Sorry for the confusion, Twitch I hit the same thing as you, Chicago was a reference to steam games and not the twitch issue.

Re: CODA-4582 - Open Issues for Investigation

Mythen
I plan to stick around

@JohnBeaudin Which steam games are you referring to?

 

 

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

@Mythen

 

Any steam games that are hosted by VALVE, 

 

I am playing the most populars one like CSGO and DOTA2.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@Datalink wrote:

The laptop could have a 100 Mb/s port.  That wouldn't be unusual depending on the age of the laptop. 


I can't remember the laptop, but I was looking at a brand new one a few months ago that only had a 10/100 ethernet port. Pretty sure it was a ~$600 one from Best Buy.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@gp-se wrote:

change your DNS servers, it sounds like routing through Rogers is causing your issue. You can use Open DNS, Google DNS, or download DNS Bench and see what servers are the fastest for you.


Sorry I'm confused -- How would DNS cause increased latency and packet loss?

Re: CODA-4582 - Open Issues for Investigation

Triple_Helix
I plan to stick around

.27 is still working very well here!

 

Wi-Fi Hardware: Killer Wireless-n/a/ac 1525 Wireless Network Adapter

 

Image and video hosting by TinyPic

Ethernet Hardware: Killer e2200 Gigabit Ethernet Controller

 

Image and video hosting by TinyPic

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@RyzenFX wrote:
I've been noticing this same issue with Twitch.tv for the past 2-3 weeks now. Randomly during a stream, it would buffer or the stream would pause and not re-buffer. Based on what you guys mentioned, the buffering could definitely be caused by the routing Rogers uses.


It's so hard to watch Twitch streams with so much buffering- hopefully we can get this issue actioned ASAP.

EDIT: I just went through your trace routes and FOR SURE there's a routing problem. The severs you're hitting respond back with a ping >= 100ms which is a huge difference compared to Bell's routing.

I'm having trouble understanding how this could be a routing problem, as opposed to a really bad platform/client.

 

A video stream there is ~1.5mbit from what I can see. On a 250mbit connection in 100ms you can load 16 seconds worth of the stream.  A reasonable buffer (let's say 32MB) should be able to hold nearly 3 minutes of video. You could literally have a 2 minute latency, and still be able to watch streams without interruption.  100ms of latency is not going to affect anything.

 

A really quick google search shows literally thousands of results of people complaining of buffering issues on twitch.

 

I think it's more likely to be a really poorly designed platform that can't buffer properly, can't  and fails easily with lost or delayed packets, rather than a problem with Rogers' network.  It's possible that the current modem is exacerbating a poorly designed client.

Re: CODA-4582 - Open Issues for Investigation

Datalink
Resident Expert
Resident Expert

@Telek, fwiw, I use OpenDNS and find that my routing to various targets can be reduced by 50% in some cases, thats in the number of servers and ping time to the target.  Doesn't make any sense at all, but that is what I've noticed.  

 

Running GRC's DNS Benchmark is a good idea.  That can be downloaded from www.grc.com .... Freeware .... Utilities .... DNS Benchmark.

 

You can modify the DNS server list to drop some servers and add others.

 

When it comes to buying laptops these days, consumers have to do their homework and look very carefully at both ethernet port rates and wifi capabilities of the laptop.  Its pretty common to find 100 Mb/s ports and 2.4 Ghz wifi, both of which will cause a great deal of heartache considering the cost of some of these laptops.  its too bad that the manufacturers still cheap out when it comes to both of those items. 

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@Datalink wrote:

@Telek, fwiw, I use OpenDNS and find that my routing to various targets can be reduced by 50% in some cases, thats in the number of servers and ping time to the target.  Doesn't make any sense at all, but that is what I've noticed.  


That is very interesting! I wonder if, in the event where DNS is setup with multiple addresses (as almost all sites are), some DNS servers will geographically prioritize the order of the results?

 

Routing has absolutely nothing to do with DNS, so it must be the way that the results are returned.

 

It would be useful to compare the DNS lookup results of servers which do result in differnet routing.

 

FWIW, I've always used Google's DNS servers.  One of the first things that I change.

Re: CODA-4582 - Open Issues for Investigation

sdmeller
I've been here awhile

Hi guys,

For the last month and half I'm having strange issue with my modem(s).

I was using Rocket modem in a bridge mode with Asus RT-68U for more than a year without issues.

Suddenly it started to loose internet connection 5-10 times per day. Mostly it was happening at evenings and at night but sometimes during the day. WiFi part was working flawlessly, just no connectivity to internet.

To test it further, I disconnected the RT-68U and started to use build-in router - same issues.

Connected my laptop with wire to modem - same problem.

When it loosing internet, after 3-5 minutes it's reconnecting itself and working fine. Rebooting modem helps too.

I called Customer Support, they decided to check signals strength (which were OK) and told me that they "boosted" them a bit. It didn't help. They told me that it probably a modem issue and advised to replace it for new CODA.

So I replaced the Rocket for CODA, left it as modem+router (no RT-68U) and... exactly the same issue - 5-10 disconnects/reconnects per day. Then it was on firmware 24. 

Called Rogers again, they told me that maybe my area (North York) is affected by "spikes" at evenings and opened investigation. Few days later I got an automated call, saying that "nothing wrong was found".

Over last two weeks the number of issues decreased to 2-4 times per day. I checked and saw that firmware version I have now is 26T2, probably it's related.

When it's disconnects I can see WAN Up Time: 000 days 00h:00m:00s

DOCSIS WAN is empty (no open downstream and upstream ports).

After few minutes it reconnects itself.

Did someone experienced this issue. Any advice?

Thanks.

 

 

 

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@sdmeller wrote:

Hi guys,

For the last month and half I'm having strange issue with my modem(s).

...

Suddenly it started to loose internet connection 5-10 times per day. Mostly it was happening at evenings and at night

...

Called Rogers again, they told me that maybe my area (North York) is affected by "spikes" at evenings and opened investigation. Few days later I got an automated call, saying that "nothing wrong was found".

...

When it's disconnects I can see WAN Up Time: 000 days 00h:00m:00s

DOCSIS WAN is empty (no open downstream and upstream ports).


If you can show us your event log and signal strengths (make sure to remove all MAC addresses otherwise the forum mods will get angry) that might help.

 

It sounds like a cabling or otherwise signal issues. If multiple modems are failing to stay registered to the network, there's something wrong upstream of your modem.

 

Do you have a cable amplifier in the mix anywhere? Can you try connecting the modem directly to your incoming line in your house/condo and see if the problem goes away?

 

Also, I'm 99% sure that the "let us file a ticket and you'll get a call back" is really just "Let me set a random delayed timer that calls you and says that the problem has been resolved"... lol

Re: CODA-4582 - Open Issues for Investigation

Datalink
Resident Expert
Resident Expert

@Telek, yup, absolutely agree, "Routing has absolutely nothing to do with DNS".  But, when I ran Pingplotter or WinMTR with OpenDNS or Rogers DNS, I would see a large reduction in the number of servers and final time to the target when OpenDNS was used, as opposed to using the Rogers DNS.  Makes no sense to me at the moment.