CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[lethalsniper]

@crazyshot

[crazyshot]

your numbers certainly do look much better than mine a bit. but anyways rogers escalated my problem so I am happy about that

also the red here isnt as dark as the other one, is this packet loss or something much different?

[Datalink]

@crazyshot your ping plots look fine.  There isn't any indication of packet loss on those plots.  The difference is the red is that @lethalsniper's plot has packet loss at the destination target, yours doesn't.  The colour scheme that pingplotter uses to display time bands is green, yellow, red.  Those colour levels can be set to other ping times from what I remember.  You have three oddball high ping times on the plot, but that doesn't look like a continuing issue.   

In the case where the first hop shows packet loss, that is due to a time to live issue with the modem that for some reason results in packet loss indications showing up for the first hop.   There isn't a problem with the first hop to the modem however.   To confirm that you can simply ping the modem by entering 192.168.0.1 into the address bar to ping the modem.  

Bring the time right down by entering 0.1 or higher in the ping interval entry window.   You will see occasional packet loss at that low time which I'm in the middle of sorting out. It appears to be generated by some but not all of the IPV6 router advertisements transmitted from the modem.  On my list of things to do is step up the time to see where the packet loss stops.  

The packet loss that you are currently seeing is indicated when you go beyond the modem. I've seen this testing other modem as well, so, that is something that requires attention from Hitron.

[crazyshot]

@Datalink Hmm okay. Anyways Rogers escalated the issue and their network engineers said they were unable to find an issue even though there's clearly packet loss on my end. I'm actually going crazy, what should I do now.

[RyzenFX]

@crazyshot I'm experiencing packet loss to the league servers aswell. I also opened a ticket w/ Rogers and they were unable to find an issue aswell. This leads me to assume that the packet loss we're experiencing could be caused by the Puma 7 UDP packet loss bug. We're going to need to wait for @RogersDave's return so he can update us with the progress that was made with the issue.

In the meanwhile, you can run the network diagnostic tool provided by League of Legends. Run the file it tells you to download and it will create a network log titled Riot-lol-results-pc.txt. Upload the file and send them the required information. 

[crazyshot]

@RyzenFX do you mind PM'ing me the link? I can't seem to find it. Also can I see your WinMTR results, I just want to compare it to mine.

[RyzenFX]

Here's the link to upload your network logs: http://na.leagueoflegends.com/en/tools/network-logs

I can't provide a PingPlotter result as of now, but my results are pretty similar to Lethalsniper- no packetloss except 100% packetloss on the last node (the League server). I can also provide you with more IPs that hit the League servers in Chicago. I extracted the IPs from the text file that was the diagnostic file made. 

192.64.170.15

192.64.173.40

104.60.131.4 

[crazyshot]

thank you so much, i will do this. so far I've been getting bounced around by Rogers and Riot hopefully this will change that

[lethalsniper]

its now been 2 days  with lag and high latency spikes is this a node issue @RogersDave @Datalinkits been 2 days now with nothing but lag spikes and game delay around the same time

[Alex4161]

@RogersDave And others

I had an interesting issue with my modem yesterday that I wanted to share with everyone.  I have the .23 firmware and the 2.4ghz off and only the 5 ghz turned on. Yesterday evening I was not able to have any device connect to the 5ghz channel stating the password was incorrect.  Only after power cycling the modem did all the devices connect.  The funny thing was I had already power cycled the modem a day or so ago.

i believe there was another member who had a similar experience. @RogersDave When you return, you might want to have a look at this issue as I believe it could be a bug related to the change in password fix that was done (ie no special characters). On my router I just have letters and numbers .

thanks

[Jeffj]

Woke up to missing channls again this morning, and it again coinsides with very low freezing temps over night. 😞 hopefully they update me on my ticket today.

[crazyshot]

There's a Rogers store near my place so I figure why not just replace this modem, I'll come back with an update later today. 

[Jeffj]

---

@Jeffj wrote:

Woke up to missing channls again this morning, and it again coinsides with very low freezing temps over night. 😞 hopefully they update me on my ticket today.

---

Just spoke to ROgers, The tech they sent to inspect the equipment in my Back alley reported damage but didnt have the hardware and equipment to fix it at the time and a Tech with the proper hardware is schedualed to come out and repair it between 4 and 6 PM tomorrow. So i guess My saga is almost over.  2 modems later  they finally found the issue outside. 

[crazyshot]

Well I went to swap the modem, the packet loss is still present but I am not lagging at the moment and I hope it stays this way. Guess I gotta wait on @RogersDave

[yyzsam]

What FW are you on, still .13 since it's new from the store?

[crazyshot]

@yyzsam yes I am on .13

[yyzsam]

Now that I am on .23 it appears that my DLs are maxing out at 270 vs 330 before.

[Bond007]

Hi Dave, 

My modem is on .23 firmware, and I am still experiencing issue connectivity issues accessing Shared Drive over Wifi when accessing WD NAS devices DS5100 / DS6100 and PR4100 all behave the same. Before the CODA-4582, I was on Rocket Wifi CGN3 modem and no issue, so it is something to do with the modem.

Amongst, other issues that you have listed in your POST.

Please advise if there is something that I can do from my end. 

Also, it would nice to know an ETA when to expect updates.

Thanks,

[lethalsniper]

once again tonight i,m getting high ping/latency on hop 2 which is causing issue when gaming @Datalink 

[crazyshot]

@lethalsniper don't worry I am also getting packet loss on hop 2

edit: Idk about anyone else but I can't see the image you posted

[crazyshot]

So even with my replacement modem I am still getting packet loss to the League servers. The modem doesn't seem to be the issue anymore unless I got really unlucky and have 2 bad modems. Rogers basically told me that I am the only one in the world with this problem as they can't seem to replicate the issue. They are sending out a technician to my place today to see if they can replicate it, hopefully they fix it.