CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[VivienM]

---

@RyzenFX wrote:

@gp-se Maybe we could see that in the near future because Rogers will be releasing another DOCSIS 3.1 modem by mid 2017. Rogers will be using a modem that is compatible with Comcast's X1 platform- from what I've read online, Technicolor supplies Comcast with D3.1 modems, so there may be a good chance that a technicolor D3.1 could be what we're using next. 

---

I just thought of something interesting. I was looking at Shaw's Blue Sky forums earlier today (Blue Sky is their X1 product), and someone pointed out that since it's a completely different system including a completely different billing/provisioning system, basically, when you move over, they cancel your account in the old system, create a new account in the new system, and then they replace almost all your hardware, including all cable boxes (sorry, old ones can't work anymore), your eMTA for your phone, and more or less your Internet modem/gateway...

That makes me wonder if people in X1land could end up with the new gateway, whatever it is, while people in legacy land get different hardware...

[gp-se]

I really hope we get the Technicolor Gateway, I've had enough of the Intel Puma and it's baggage. Rogers must realize that the Intel Puma Chipset issues are costing them a lot of tech support hours, engineering hours, and of course costing them customers who are frustrated and leaving. The amount of Modem swaps alone is crazy, just from reading these forums, people are going though like 4-5 modems.

Intel doesn't seem to care to fix all the Puma 6 issues, they teased us with ipv4 ICMP fixes, then disapeared, thinking that would be enough. Now Intel released the Puma 7, which is looking like it suffers from the same issues, just not to the same extent, however the issues are still there. The Cable Companies need to show Intel they can't get away with this, change vendors!

[RyzenFX]

I would really REALLY appreciate it if we can be updated with the progress for the fixes of the modem pleaseee. We haven't heard anything for weeks, and I think we would be happy to hear some progress being made. 

Besides that, Hitron needs to be pushed even harder from Rogers to have these bugs fixed soon. I think I heard alot of the bugs would be fixed by March, but we haven't been updated with any of the progress.. I'm clueless at this point.

@gp-se I agree with you, Rogers should change vendors and move to the Broadcom-based modem, however would they be able to get the modems at the same price as they did for the CODA-4582's? Assuming that the new gateway being released mid-year is broadcom-based would prices go up? 

[VivienM]

---

@RyzenFX wrote:

@gp-se I agree with you, Rogers should change vendors and move to the Broadcom-based modem, however would they be able to get the modems at the same price as they did for the CODA-4582's? Assuming that the new gateway being released mid-year is broadcom-based would prices go up? 

---

Given the huge support costs that the 4582 has probably entailed, who cares about hardware costs?

If you look at Rogers' history, it looks to me like the main reason they've gone with Hitron is not cost but time to market, i.e. Hitron had a product with the newest thing (e.g. 24-channel D3 back in 2013, this thing in late 2016) shipping before the other manufacturers did. And really, to the extent that most of the code/bugs come from Intel, what good would it have done to add an Arris/Technicolor/whatever Puma 6 product 6-18 months later? (I don't think Broadcom has a 32-channel D3.0 product, not sure about 24-channel...)

I find very interesting, btw, that in the U.S., Puma 6 issues seem to have mostly been discovered in the past 3-6 months (and mostly centered around some Arris standalone modems). Puma 6 CGN3-family gateways have been widely deployed in Rogersland for 3 years!

[RyzenFX]

I think my modem might be faulty or there could be another issue at hand. When data is being transmitted from the 2.4/5G band, I hear a noise coming from the DOCSIS tuner where the Coax goes into the modem.

Can anybody try running a speedtest over wifi and listen closely to their modem? When the speedtest is running you should hear a noise that is synchronized to the data being transferred to the speedtest sever.

Please let me know!

[VivienM]

---

@RyzenFX wrote:
I think my modem might be faulty or their could be amother issue at hand. When data is being transmitted from the 2.4/5G band, I hear a noise coming from the DOCSIS tuner where the Coax goes into the modem.

Can anybody try running a speedtest over wifi and listen closely to their modem? When the speedtest is running you should hear a noise that is synchronized to the data being transferred to the speedtest sever.

Please let me know!

---

Do you have a black dot 4582?

[RyzenFX]

Yup, black dot from the second batch that was tested before being shipped to stores.

Do you have a CODA in gateway mode in which you run the test I described?

[VivienM]

---

@RyzenFX wrote:
Yup, black dot from the second batch that was tested before being shipped to stores.

Do you have a CODA in gateway mode in which you run the test I described?

---

No, mine is bridged... (and not a black dot model, which is costing me 8-9 D3.0 channels)

[StephKitty]

My internet connection just died on me about half an hour ago. I contacted Rogers and they told me that it was a whole quadrant that was out.  It's back up now, but it seems my Docsys 3.1 light is no longer light blue, and is now showing just a D3.0 connection.  

What gives?  did they  disable D3.1 tonight or something?

[Double_K]

---

@gp-se wrote:

I Wish Rogers would give us this:

http://www.technicolor.com/en/solutions-services/connected-home/broadband-devices/cable-modems-gatew...

It uses a Broadcom Chipset. 

---

@gp-se - source? 

[lethalsniper]

getting packet loos on hop 2 and why am i getting mx of 31ms when pinging my modem use to be at 1ms for max @RogersDave 

Ping statistics for 192.168.0.1:
Packets: Sent = 500, Received = 500, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 31ms, Average = 1ms

[Datalink]

@lethalsniper if you're going to inquire about hop #1, then perhaps you could show the plot for hop #1 instead of the final hop #18.  At the present time I can't fathom why your average time to the modem is 1.2 ms.  I see 0.6 ms or less as an average and I've run enough ping tests to the modem to consider that as being accurate. Perhaps you need to start looking at your own equipment  to determine what the problem is in this regard.  As for the 5 ms max time, perhaps something on your pc fired up and called home when the ping test was running. That would push up the time for one or two pings.  Its very difficult to run a clean ping test these days due to all of the applications that are running on a pc, calling home on a regular looking for updates.  As an example, and I've said this to you before, Microsoft also uses its own ping test to establish whether or not the pc has internet connectivity and present that fact in the form of the icon that you see in the task bar.  Perhaps that was responsible for that 5 ms ping.  Without running a ping application such as Hrping that logs the time and ping response, and examining a log from Wireshark to correlate the event, asking an open ended question of "why is my ping time so high (at 5 ms which is nothing)" is basically useless.  Nobody could begin to guess the cause of what might have been a singular one second event displayed in a 10 min timeframe.  You're going to have to present more information, and perhaps, in the course of doing that, you might just answer your own question.  

[lethalsniper]

Ping statistics for 192.168.0.1:
Packets: Sent = 500, Received = 500, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 31ms, Average = 1ms

[Datalink]

Is that 31 ms time the very first ping, which is not abnormal as its the very first ping, or somewhere in the middle which indicates an application issue?  The number alone doesn't mean anything. 

[lethalsniper]

its all over i,m getting pings of 7,9,20,12,max 31, no there is no programs working when pinging router and a plus side i would ping the tp-link router when i had thecoda on bridge mode and ran the same test as now with max being 1ms

[Kpare]

Don't be concerned with jitter when you ping the IP gateway. Just remember that the router's function is to forward packets, not to respond to them. The process to respond to ping requests is low priority for the router so you may see lots of variances in ping response times. If you are getting lots of drops or response time is varying by 100s of ms, then that is cause for concern.

I've worked with Cisco Nexus 7000 switches which can respond to a ping at 10 ms from the PC on the local LAN, yet pinging a the server one hop away is sub 1 ms.  But once again the process on the Cisco switch to respond was not on the main bus and was treated as low priority. 

The ultimate test using ping is always the end node. That ea.com service has avg 80 ms response and no drops, which looks good. If that is the server you are establishing IP session with,  then that is the result you should be monitoring.   

[Datalink]

There are always programs running in the background.  Windows 10 is by far the worst version of Windows for that as far as I'm concerned.  In order to ping the modem, you would have to disconnect the router, run the modem in gateway mode and then ping 192.168.0.1.  Are those times from a direct connection to the modem with the modem in Gateway mode, or are they with the router connected to the modem, running the test to 192.168.0.1 which pings the router instead of the modem?

[JohnBeaudin]

At first I had hopes to fix the latency issue on the CGN3 series.. 6 months later I had hope with a new modem CODA 4582.. and now that what's next?

The new modem  that won't have a PUMA chipset, ( Mid 2017) 

I wonder if Rogers will stay silent until then, would be nice to have some updates.

[lethalsniper]

These tests are done hard wired to the coda modem on gateway mode ( not a 3rd party router )

[RyzenFX]

Any progress updates w/ the 4582 so far?

[JohnBeaudin]

Only one person can provide us updates, we all waiting for his return so we can have updates.