CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[lethalsniper]

Okay thanks but what was your max on hop 2,3,4,5 ?

[JohnBeaudin]

Hop 2 Max 15.8

Hop 3 Max 26.2

Hop 4 Max  30.1

Hop 5 Max 49.1

[lethalsniper]

Still better then mine I'm getting 100 + yesterday I got in the 200s

[toolcubed]

Anyone else have horrendous bufferbloat results on the dslreports speed test? During the test (which I've done multiple times both wired and wireless), pings spike to 2000+ms and when the test finishes my bufferbloat is rated "F". I thought it was just the nature of the speed test but then I decided to run the same test at my in-laws place where they have Bell Fibe (FTTN, VDSL, 25/10), and I always ended up with a bufferbloat rating of between A and B...pings remained relatively low, below 40ms...no major spikes at all. So the terrible results I get at home appear to be pointing to issues with the Rogers network. I know Bell in general (DSL or fibre) will almost always have much better latency than Rogers/cable but 2000+ms spikes??? Somethin's definitely not right...

[JohnBeaudin]

BlufferBoat of D-F is normal for Rogers

on Bell you will get A+ A-

Apprently the blufflerbloast is caused by the speedboost but I think dslreports speedtest just fail to properly blufferboat because of the speedboost. 

[soundwave80]

I used to get D and F for bufferbloat when i used the CODA, since i switched back to the CGNM 3552 i now get A's and B's
Something is definitely wrong with these modems.

[prateeck7]

@JohnBeaudinNow lets not just assume "its normal for Rogers" everytime complains about the issue, alright?

I never got 900mbps with CGN but I was able to get above 800mbps constantly. But most importantly the internet was stable and Bufferbloat too. So, it is NOT normal because it takes me more than 2 mins to download 1GB while I get ping spikes of more than 5000ms constantly since I got this CODA modem. It's like I am on a dial up connection.

I am sick of Rogers being defended. I used to defend Rogers too. But I am frustrated. Sorry for venting frustation here. But I get so many time outs when doing a simple google search. Sometimes it takes more than 20 secs to send or open my email alone. Youtube is stuck most of the time. For the past week I have been using LTE heavily. And this issue exasperated since my area was upgraded to Docsis 3.1. I thought it would improve the performance NOT ruin it.

So far Rogers is just taking a chill pill and taking advantage of us being patient.

[JohnBeaudin]

Not defending Rogers, but I am used to high latency in online games and issues with hiltron modems, I hope one day all this will be fixed but it's only hope at this point.

[prateeck7]

Don't even talk about gaming. It is impossible to play for me right now. Any chance, I can get out of contract with a Rogers wihtout paying penalty? I have heard I can make a complaint to CRTC and get out that way.

[JohnBeaudin]

There's no need to bash Rogers, it won't help we are all aware of the issues with their network and I ma pretty confident that they aware as well.

They are truly trying to improve our experience, I understand it cant lead to frustration sometimes, but all we can do is be patient and wait.

[prateeck7]

There is every reason to bash Rogers. I am not bashing there employee who I have had pleasant experience with and I can't blame them.

I am done being patient and wait. We live in 21st century. It should not take more than 20 secs to send a simple text email or google search something.

Patience is the reason they are taking it slow to solve this issue. They are aware and I am sure they are trying to improve our experience. But they need to push it hard and fast. Not take it easy.

Edit: Took me about 15 secs to post this reply. I could post faster with Dial Up.

[VivienM]

---

@prateeck7 wrote:

Patience is the reason they are taking it slow to solve this issue. They are aware and I am sure they are trying to improve our experience. But they need to push it hard and fast. Not take it easy.

---

I've said this before, but a big part of the problem is that Rogers likes to push the envelope, unlike U.S. MSOs that are happy selling lower-speed packages with ancient technology.

This CODA-4582, for example - unless @Datalink has heard otherwise, I believe it's the ONLY DOCSIS 3.1 gateway, or at least the only Puma 7 one (I have heard of D3.1 modems, not sure about D3.1 gateways), currently actively deployed by a cable operator in North America.

Meanwhile, U.S. operators are barely running 24-channel bonded D3.0... which Rogers was doing over 3 years ago. Most of their customers are still running 8/16-channel D3 modems. It would not surprise me if Rogers was the only MSO in North America currently running D3.1 in production...

The other problem, too, is Rogers' perpetual optimism. Someone very optimistic decided to have a big campaign to get 3552 gigabit customers to replace their 3552s with the 4582s ASAP so they could reallocate some network capacity from 3.0 and be able to launch 3.1... but then, it turns out the 4582 has more bugs than the 3552. I think they probably wish they had delayed that campaign by 2-3 months now...

(Also, one suggestion to everyone at Rogers who reads this: why not consider having a 'public' hardware beta program like the firmware beta program? the logistics of getting beta hardware to testers might be a bit challenging, but I think you could find a couple dozen people around here, at least, willing to try out the new devices...)

[RyzenFX]

I'm pretty frustrated as well b/c of how much problems I'm experiencing. I really wish Rogers & Hitron took the time to nail out the bugs before creating this mess.

My problems with the CODA-4582:

• UDP Packet loss (PLEASE get this fixed it's super annoying in latency sensative games)
• dowlink + uplink LEDs on D3.0 are incredibly bright, especially at night. (please dim it)
• after reboot/reset of modem LAN port negociates at 0.5-1mbps w/ computer (I would have to restart my computer every time for the modem to negociate at 1Gbps)
• 2.4Ghz + 5Ghz range performance is bad; needs to be improved greatly

I understand that Hitron and Rogers a using all their resources to fix the speed slowdown bug, but why hasn't Hitron done some extensive testing before shipping the modems out to Rogers?  Also, why not reach out to the community for some beta testers before the modems are shipped to all the consumers??? Comast had done this with before deploying D3.1 along with their technicolor line of modems.

I think it's time to get an official update on the progress with this modem. We've waited a good amount of time, and it would really help to give the community some ETAs on when some of the problems will be adressed.

Don't forget Rogers will be launching a new D3.1 modem by mid-2017 which will be obtained by Comast.... so hopefully things are much better on that modem.

[JohnBeaudin]

I have to agree with Ryzen here the UDP packet loss really needs to be fixed.

Hopefully it will be on the list for new firmware updates @RogersDave 

[VivienM]

---

RyzenFX wrote:I understand that Hitron and Rogers a using all their resources to fix the speed slowdown bug, but why hasn't Hitron done some extensive testing before shipping the modems out to Rogers? 

---

One thing I would love to understand is how much of the code on these things is Intel vs Hitron. If most of these bugs are Intel's bugs, then... that's an additional level of complexity.

But we're back at the fundamental problem here: I think Rogers decided to rush the CODA-4582 into the field because they thought it would be better than the Puma 6 gateways with their latency issues. This may have also been Hitron's opinion if they had no luck getting a fix for Puma 6 problems from Intel (it's worth noting they were on this issue months before Arris). Anyone remember how excited @RogersDave sounded when he announced the 4582?

[JohnBeaudin]

I remember indeed.

And with the current silence for weeks I think it's becoming more and more clear that they might not be able to fix the latency issue at all, or not any time soon.

[RyzenFX]

---

@VivienM wrote:

---

RyzenFX wrote:I understand that Hitron and Rogers a using all their resources to fix the speed slowdown bug, but why hasn't Hitron done some extensive testing before shipping the modems out to Rogers? 

---

One thing I would love to understand is how much of the code on these things is Intel vs Hitron. If most of these bugs are Intel's bugs, then... that's an additional level of complexity.

But we're back at the fundamental problem here: I think Rogers decided to rush the CODA-4582 into the field because they thought it would be better than the Puma 6 gateways with their latency issues. This may have also been Hitron's opinion if they had no luck getting a fix for Puma 6 problems from Intel (it's worth noting they were on this issue months before Arris). Anyone remember how excited @RogersDave sounded when he announced the 4582?

---

If I remember correctly, Dave told me that the UDP packet loss is Intel's bug. Intel's IoT devision which is responsible for fixing the problem is located in Israel, so conversations b/w Rogers, Intel, and Hitron are extremely slow... I remember when Dave was excited to announce the 4582. Those were the good days, but now it's back to fixing lots and lots of bugs.

[crazyshot]

@JohnBeaudin do the other modems Rogers offers have latency issues? I am down to switch cause at this point I don't even care anymore. 

[RyzenFX]

@crazyshot  Yes, the other modems (CGN3 series, and CGNM) series are all plagued by the Puma 6 latency problem. However, it hasn't affected my gameplay in League of Legends at all. 

[VivienM]

---

@JohnBeaudin wrote:

I remember indeed.

And with the current silence for weeks I think it's becoming more and more clear that they might not be able to fix the latency issue at all, or not any time soon.

---

Are you talking about the Puma 6 latency issue, or latency issues with the 4582?

I'm not sure about the Puma 6 issue; I'm sure Arris and other manufacturers with lots and lots of Puma 6 inventory in the U.S. must be pushing Intel for a fix on that one.

[JohnBeaudin]

The modems that don't have the UDP issues are no longer in productions.