CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[Breadwinka]

@prateeck7 @RogersDave I have the same issue if I start downloading I get pretty bad lag too. Also my Speeds have been very inconsistant lately.

https://www.dslreports.com/speedtest/8280603

https://www.dslreports.com/speedtest/8034708

[Alex4161]

One thing I would suggest to @amd7674 and others is if you find congestion issues on the network, contact Rogers support and report the problem. If you don't tell them, the odds are it won't get fixed.

I did this a few days ago and they determined high latency and packet loss issues that caused upload/download speed concerns. They assigned the ticket to their Network division and I did find improvements in performance a few days later.

[Hybrid_Noodle]

@RogersDave did some changes this evening and after multiple reboots of modem and router, the Netgear appears to be working properly. May have just been something stuck in the background that got cleared.
Disregard my stated issue.

[RogersDave]

---

@Hybrid_Noodle wrote:
@RogersDave did some changes this evening and after multiple reboots of modem and router, the Netgear appears to be working properly. May have just been something stuck in the background that got cleared.
Disregard my stated issue.

---

No problem, I'll remove this from the log.

[NBomb]

Hey @Datalink or anyone else - do you have any experience w/ Ubiquiti Edgerouters?

I've narrowed down my router as being the bottleneck here, (BTW people, a good speed test for real throughput is to download a DVD img torrent of Knoppix Linux. It's free, legal, and big enough to not finish too quick to see results), and I see a lot of people talking about using one of those as a main router, with something consumer as a wifi AP.

Secondly, how can I reduce the bufferbloat going on? On my DSL, the method would be to enable QoS to limit throughput to like 98% of max, but that's not practical @ gigabit speeds.

[hoob]

I've been following these CODA issues as I seem to be affected by the "wifi interference" issue. Right now my modem is running 19 and hasn't been reset in weeks, so I decided to do some reference checks.

All tests are done 10h30-11h00 on Saturday 07JAN, using the Rogers Speed Test page to have a consistent point of relative reference. I am running in Gateway Mode:

Modem running .19, no recent resets or restarts:

* Wifi 5 and 2.4 20/40 MHz: 17D, 19U, 8ms ping, 52ms jitter

* Wifi 5 and 2.4 20 Mhz: 19D, 19U, 4ms ping, 8ms jitter

* Wifi 5 and 2.4 off: 18D, 20U, 9ms ping, 7ms jitter

* Wifi all off: 17D, 20U, 8ms ping, 3ms jitter

Modem running .19 after 5 minutes quiet time unplugged while I cleaned the cat litter box:

* Wifi all off: 349D, 12U, 11ms ping, 7ms jitter 

* Wifi 5 and 2.4 off: 383D, 11U, 10ms ping, 12ms jitter

* Wifi 5 and 2.4 20 Mhz: 283D, 11U, 10ms ping, 7ms jitter

* Wifi 5 and 2.4 20/40 MHz test#1: 506D, 20U, 7ms ping, 417ms jitter

* Wifi 5 and 2.4 20/40 MHz test#2: 499D, 20U, 5ms ping, 2ms jitter

* Wifi 5 and 2.4 20/40 MHz test#3: 459D, 17U, 8ms ping, 5ms jitter

* Wifi 5 and 2.4 20/40 MHz test#4: 338D, 15U, 11ms ping, 14ms jitter

* Wifi 5 and 2.4 20/40 MHz test#5: 353D, 16U, 9ms ping, 6ms jitter

 * Wifi 5 and 2.4 20/40 MHz test#6: 341D, 17U, 11ms ping, 6ms jitter

Key observations:

* Download speeds before the reset were very sick, upload speeds were typical

* After the reset, upload speeds took a noticeable drop (from around 20 to around 11)

* After the reset, download speeds increased from 30 to around 300+

* Increased speed by insane inconsistent jitter with 2.4 40/20 enabled

* Speeds with CODA modem are still notedly below what I got with the previous 3552 modem, even with wifi disabled

[Jeffj]

---

@NBomb wrote:

Hey @Datalink or anyone else - do you have any experience w/ Ubiquiti Edgerouters?

I've narrowed down my router as being the bottleneck here, (BTW people, a good speed test for real throughput is to download a DVD img torrent of Knoppix Linux. It's free, legal, and big enough to not finish too quick to see results), and I see a lot of people talking about using one of those as a main router, with something consumer as a wifi AP.

@Secondly, how can I reduce the bufferbloat going on? On my DSL, the method would be to enable QoS to limit throughput to like 98% of max, but that's not practical @ gigabit speeds.

---

Knoppix?  man i thought every one abandoned that for Kali long ago. 

[hoob]

..... and...... just like that, doing nothing in particular, 30 minutes later my connection takes a nosedive:

* Wifi 5 and 2.4 20/40 MHz test#7: 21D, 15U, 10ms ping, 3ms jitter

* Wifi 5 and 2.4 20/40 MHz test#8: 19D, 16U, 8ms ping, 8ms jitter

* Wifi 5 and 2.4 20/40 MHz test#9: 19D, 16U, 11ms ping, 9ms jitter

* Wifi 5 and 2.4 20/40 MHz test#10: 18D, 18U, 8ms ping, 657ms jitter

* Wifi 5 and 2.4 20/40 MHz test#11: 19D, 17U, 8ms ping, 9ms jitter

* Wifi 5 and 2.4 20/40 MHz test#12: 18D, 17U, 8ms ping, 86ms jitter

Since this is a dramatic drop off in modem speed after reset, @RogersDave if you are working today and want to take a look before I reset the modem let me know via PM. You have my MAC from a PM weeks ago.

[prateeck7]

@NBomb I already verified my speed downloading Knoppix and Ubuntu, as suggested before on this forum, from torrent. My speed is 8MB/sec (megabytes) at max which is same as I am getting at Testmynet. 

If I can get 250mbps+, I am a fairly happy person. I am not expecting 400mbps+ speed with this modem like before; and I never expected even close to real Gigabit speed with any other modem. That may, just, be not possible.

[NBomb]

I don't plan to actually use it, but it's a good large filesize w/ enough seeds to max out a connections's throughput. 🙂

[DaveFro]

---

@NBomb wrote:

Hey @Datalink or anyone else - do you have any experience w/ Ubiquiti Edgerouters?

I've narrowed down my router as being the bottleneck here, (BTW people, a good speed test for real throughput is to download a DVD img torrent of Knoppix Linux. It's free, legal, and big enough to not finish too quick to see results), and I see a lot of people talking about using one of those as a main router, with something consumer as a wifi AP.

@Secondly, how can I reduce the bufferbloat going on? On my DSL, the method would be to enable QoS to limit throughput to like 98% of max, but that's not practical @ gigabit speeds.

---

I have a Ubiquiti ER-X as well as a few AP-AC-Lites. I have spend some time optimizing and now have full wired speed and pretty good wifi speed. Wired I was running around 400, but turns out that hardware offload for NAT wasn't on by default, turned it on, now I get just about full gigabit throughput from wan. I did it via cmd line, but it can also be done using config tree under system > offload, enable hwnat and ipsec. I think ER-L should have hardware offload enabled by default since offload was just added to ER-X maybe one or two firmware versions ago. Let me know if enabling hardware offload help.

[DaveFro]

I guess I should have put the speeds in...

WAN-LAN using modem in bridge mode on wired connection with hardware offloaded NAT: 930/48

LAN-LAN on wireless AP-AC-Lite 5ghz AC (standing beside the AP): 410/430

WAN speed isn't always that high, during congested periods it can go down to about 700/30, still pretty good...

Wired setup is: Modem (bridged) > ER-X > cheap gigabit switch > computer

Wireless setup is Modem > ER-X > cheap switch > Access points

[JohnBeaudin]

I was in the middle of a game (Dota2), I got disconnected, I then noticed all the lights off on my modem and then it restarted.

My first reaction was maybe a new firmware push but no still on .19 , just wondering if the reboot come from you guys or if it's an anomaly?

[traghipp]

Since being upgraded to firmware 19 I am experiencing speed degradation the requires a modem reboot every 24hrs. The reboot resolves the issue. Anyone else experiencing this?

This is on a wired Ethernet connection using cat6 cable.

[Dragus]

---

@traghipp wrote:
Since being upgraded to firmware 19 I am experiencing speed degradation the requires a modem reboot every 24hrs. The reboot resolves the issue. Anyone else experiencing this?

This is on a wired Ethernet connection using cat6 cable.

---

Yep that looks like my problem. Wasn't sure the first time I rebooted, thinking it was pure coincidence. But after reading RogersDave first post it was on my mind.

But the 2nd time I rebooted, I did a before and after....

before:

test #1) beanfield

test #2) Start communications

After reboot:

1) Start communications

2) Nexicon

Upload is still about 1/2 of my old Gigabit modem before switching to the CODA.

I have some other speed tests from the afternoon with flunctuations from 100-500Mb/s on the same hosts. So the connection really starts to get flaky then it eventually ends up to a crawl.

Have a Tech coming around 10am-noon so I guess I'll see if he can do anything about it. Must just be a firmware fix at this point.

[Dragus]

Tech guy came, said the last tech guy connected my cable wire to the wrong one (the white ones).

Not sure if that was the problem considering the posted speeds after rebooting the modem is pretty good.

He did give me a new CODA modem so lets hope it was just a bad batch I got the first time.

But if it comes back, I guess I can be a guinea pig with my logs so that Rogers can find a fix for this speed degradation issue.

[traghipp]

I can now report that since updating to firmware 19, everyday between 3:30pm and 3:40pm my download speed drops to under 10mbs which requires a modem reboot. The issue occurs exactly 24hrs after the rebooting modem.

[DaveFro]

---

@traghipp wrote:
I can now report that since updating to firmware 19, everyday between 3:30pm and 3:40pm my download speed drops to under 10mbs which requires a modem reboot. The issue occurs exactly 24hrs after the rebooting modem.

---

Are you using the built in router, or running the CODA in bridge mode?

I am running in bridge mode and don't have the same reboot issue, and am running 19 firmware.

Maybe the issue is with the router, and not the modem?

Currently the modem has been up for 2 days and 19 hours, and I'm still getting great speeds. I did see some slowdown last night, but very minor, like 700 instead of 930. I assume that is just because everyone on the block is using lots of bandwidth at that time, but speeds went back to 930 this morning.

[traghipp]

I'm using built in router with computer connected by cat6 Ethernet cable

[Alex4161]

@RogersDave

I have also been noticing speed degration that is resolved with a router reboot.  What is even more interesting is that I see that the router seems to have rebooted on its own at times.  I see that my current LAN Up Timeis  000 days 12h:51m:20s. I have not rebooted the modem but the logs show the following events (i've removed my mac addresses)

1 01/07/2017 22:07:10 68010300 error DHCP RENEW WARNING - Field invalid in response v4 option;;CM-QOS=1.1;CM-VER=3.1;
2 01/08/2017 03:39:24 90000000 warning MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;;CM-QOS=1.1;CM-VER=3.1;
3 01/08/2017 14:07:02 82000200 critical No Ranging Response received - T3 time-out;;CM-QOS=1.1;CM-VER=3.1;

I know that there are latency and packet losses in my area (case number is C118816087) that has been logged since Jan 4).

[wh4tfi]

I am experiencing the issue of non-accessible samba shares over wireless. Wired connection no problem. Also connections with wifi-controlled devices (like lights) are sluggish and intermittent. 

Worst case is a Raspberry Pi digital music player that uses a shared drive on another computer as file source. The file server is a wired connection on my LAN, the RPi is wifi. (This worked fine before I switched to Rogers Internet a few weeks ago.)

I also have another wifi-connected network music player (Logitech Squeezebox) that uses the same file server, but connects via a dedicated port on that server, and this still works normally.