CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[4kgigabitguy]

---

@Eaglzfan wrote:
Yes both 2.4 and 5

---

Sign up for the rocket modem firmware trial.... When I got the beta firmware this issue went away

Let me know how it goes

4kgigabitguy

[Eaglzfan]

Ok thank you. Where do I go to do that?

[4kgigabitguy]

---

@Eaglzfan wrote:
Ok thank you. Where do I go to do that?

---

view the first post on this thread http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/thread-id/3315...

[Eaglzfan]

Got it. Thanks again for your time

[RyzenFX]

@4kgigabitguy @RogersDave @Datalink I just switched my CGNM-3552 back to the CODA-4582. I ran some test games of LoL, however it's packet loss is nowhere near the packet loss I averaged with the 3552; it's worse. I averaged  6.7 packets lost/min. However, I do have a suspicion that the CMTS is not communicating with the modem properly (i.e CMTS is not giving the modem enough upstream signal strength, which needs to be adjusted). 

Upstream Signal Strength:

Upstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	30596000	ATDMA - 64QAM	29.250	1	6400000
2	38595785	ATDMA - 64QAM	33.250	3	3200000
3	23700000	ATDMA - 64QAM	29.250	2	6400000

I understand the upstream signal strength looks normal for this modem because it uses less signal strength, but like I said above, although the modem is interpreting that the signal strength is fine, there needs to be adjustments at the CMTS level.

Despite both modems being on DOCSIS 3.0 upstream, the upload graph varies for both modems. Why does it show a constant upload speed graph for the 3552 than on the 4582? There's definitley something wrong here with the CODA-4582, or CMTS. 

@4kgigabitguy What are your upstream signal strengths? I would like to compare yours to mine. 

[gp-se]

@RyzenFX my upload levels are at 26.250 currently, and I've seen them as low at 25.500 

I still manage to get 30-40mb/s upload speeds during off peak hours, so the signal level being "low" shouldn't be an issue. The CMTS provides only the required power level for the connection, so if the levels are low it should be fine because the CMTS determines that level is all that is required.

[RyzenFX]

@gp-se Hm... But shouldn't the upload graph of the CODA-4582 look nearly identical to the CGNM-3552? This tells me something is wrong on the upstream side. You can see that the upload speed is dipping on the CODA-4582 graph, but why?

[Double_K]

@RyzenFX  Assuming all other things are equal (which is probably a bad assumption), the 3552 upload graph *appears* to indicate some sort of traffic shaping - hence the smoother graph at lower speed.  It appears the outbound wan queue on the 3552 may be buffering your outbound traffic.

Testing on DSLReports speed test would show you bufferbloat on the upload, which would help with this theory.

[JohnBeaudin]

A few people also tested the 3552 for gaming vs the CODA and it's a lot better on the 3552, all theory points in the direction that the 3552 is handling better so far, CODA is still on beta and it should out perform the 3552 easily, however we might not see that until a few firmwares fix. but once that's done we should finally see what the CODA is capable of.

[RyzenFX]

@Double_K I ran a speedtest on DSL Reports, here is my result:

This makes  sense, the bufferbloat that is affecting the CODA-4582 is not cosmetic at all (it was previously called cosmetic), but it is an actual problem. This bufferbloat problem is affecting the modem on the downstream and could possibly be at the source for the packet loss I'm seeing in League of Legends and random ping increases in other games as well. I've read that Bufferbloat causess packet delay variation, which causes the packet loss I'm seeing in league of legends. This bufferbloat problem is only attributable to the downstream. Multiple users have already complained of this problem earlier.

This bufferbloat can be attributable to the speedboost feature on all the Ignite plans EXCEPT for Ignite Gigabit. It makes a lot of sense. For instance, @4kgigabitguy is on the Ignite Gigabit where  speedboost is not enabled and gets 0.9 packets lost/minute in League of Legends. In my case, I'm receiving 6.7 packets lost/min on the 100u plan. If you read here, the Casa Systems CMTS could also be at the source of the problem. In Comcast D3.1 land, they aren't experiencing any bufferbloat issues which makes sense, because they're on a Cisco CMTS (or because they don't have a speedboost feature). So the problem has to be attributable to the speedboost feature + the CMTS.

@RogersDave @Datalink What do you think of this? I think it makes sense, bufferbloat can be due to the speedboost feature and CMTS, but I'm really curious what both of your opinions are.

[4kgigabitguy]

@RyzenFX

Here are my upstream levels(my levels tend to flucate but I haven't noticed degraded performance that would point to signal issues)... Previously I was using the CGN3ACSMR & captured some logs & tests prior to my switch..Will share later today

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	30596000	ATDMA - 64QAM	43.500	4	6400000
2	38595609	ATDMA - 64QAM	46.500	6	3200000
3	23700000	ATDMA - 64QAM	43.250	5	6400000

I am going to play another match of LoL today & will provide results again. I also plan to test out CSGO today, I am welsome to any suggestions on the best way to capture logs/performance for CSGO.

Upgrading from regular ps4 to ps4 pro today so looking forwarding to playing some games online as well!!

4kgigabitbuy

[Double_K]

@RyzenFX I was actually referring to "bufferbloat" on your upload - which would be your modem/gateway.  Assuming that result is on the CODA-4582, your results show an upload bufferbloat of 95ms average, with a 555ms sample.  Did you run a similar test on the 3552?  If so, what did it show for upload?

For the download, Dave has previously indicated that the bufferbloat for download is related to the SpeedBoost for non-Gigabit customers.  To prove this, go to the DSLReports Speedtest preferences, and in the Advanced Settings, set your down speed to the advertised speed (ie. 100) of your connection.  You'll notice your download bufferbloat get alot better, but your download speed will be limited to 100Mbps for the entire test.

[4kgigabitguy]

@RyzenFX

If you just got the CODA, did you check what firmware it came with? I am running the trial version. Have you put in a request to get the beta yet? I would try testing with that one as I didn't have the best performance with .24

Will do a deeper review on your posts as possibly there is another factor that is causing you to not get the same results as myself.

[RyzenFX]

@4kgigabitguy Thanks for replying. Your upstream signal strength is much stronger than mine, and when looking back at your speedtests (specifically the 2015 macbook air) on the Rogers speedtest site, your upload graph looks really similar to my upload graph on the CGNM-3552. Also, your bufferbloat looks good on the upload and download, and ideally that is what I should be getting. I'm on .24, but hopefully I'll get upgraded to .26 tomorrow (sent a PM to Dave requesting upgrade). I'll let you know how things go.

Image removed due to privacy reasons - RogersMoin

@Double_K There could be bufferbloat on the upstream aswell, I don't want to keep that out of the problem.  Unfortunately, I was not able to run a similar test on the 3552 prior to switching, but I'd imagine that the bufferbloat for both upstream and downstream would be a lot lower. If you can look at the upload graph above, that is what I SHOULD be seeing on my CODA-4582.

[Telek]

---

@RyzenFX wrote:

@4kgigabitguy @RogersDave @Datalink I just switched my CGNM-3552 back to the CODA-4582. I ran some test games of LoL, however it's packet loss is nowhere near the packet loss I averaged with the 3552; it's worse. I averaged  6.7 packets lost/min. However, I do have a suspicion that the CMTS is not communicating with the modem properly (i.e CMTS is not giving the modem enough upstream signal strength, which needs to be adjusted). 

Despite both modems being on DOCSIS 3.0 upstream, the upload graph varies for both modems. Why does it show a constant upload speed graph for the 3552 than on the 4582? There's definitley something wrong here with the CODA-4582, or CMTS. 

@4kgigabitguy What are your upstream signal strengths? I would like to compare yours to mine. 

---

FWIW, I have 34.5, 35.5 an 38.5 for upload on D3.1 and get pretty much full 50mbit upload. However has been pointed out, these have no meaning from the client side. I could be at 25 or 40.

We already know that the CODA modem has a lot of problems -- I think it's safe to assume that the problem lies on the modem side.

[Telek]

---

@JohnBeaudin wrote:

A few people also tested the 3552 for gaming vs the CODA and it's a lot better on the 3552, all theory points in the direction that the 3552 is handling better so far, CODA is still on beta and it should out perform the 3552 easily, however we might not see that until a few firmwares fix. but once that's done we should finally see what the CODA is capable of.

---

This is working on the assumption that the hardware is fit. We know that the chipset has major issues. We know that the hardware design itself has issues (hence the need for extra shielding, and known problems with WiFi interference).  I don't think that it's safe to assume that this modem will ever be better, and FirmWare fixes will likely only be to work around hardware limitations.

[Telek]

---

@RyzenFX wrote:

This bufferbloat can be attributable to the speedboost feature on all the Ignite plans EXCEPT for Ignite Gigabit. It makes a lot of sense. For instance, @4kgigabitguy is on the Ignite Gigabit where  speedboost is not enabled and gets 0.9 packets lost/minute in League of Legends. In my case, I'm receiving 6.7 packets lost/min on the 100u plan. If you read here, the Casa Systems CMTS could also be at the source of the problem. In Comcast D3.1 land, they aren't experiencing any bufferbloat issues which makes sense, because they're on a Cisco CMTS (or because they don't have a speedboost feature). So the problem has to be attributable to the speedboost feature + the CMTS.

---

I thought there were a lot of people here on the gigabit plan also complaining about UDP and packet loss issues. I don't think it's limited to just those below gigabit.

Occam's razor here -- Cisco equipment is probably a LOT better...

[Telek]

---

@RyzenFX wrote:

@4kgigabitguy Thanks for replying. Your upstream signal strength is much stronger than mine, and when looking back at your speedtests (specifically the 2015 macbook air) on the Rogers speedtest site, your upload graph looks really similar to my upload graph on the CGNM-3552. Also, your bufferbloat looks good on the upload and download, and ideally that is what I should be getting. I'm on .24, but hopefully I'll get upgraded to .26 tomorrow (sent a PM to Dave requesting upgrade). I'll let you know how things go.

Didn't you say that you're also on D3.0?  You're comparing your results to D3.1 -- don't think that's really fair.

[gp-se]

---

Telek wrote:

Occam's razor here -- Cisco equipment is probably a LOT better...

---

Cisco hardware is definitely better, but it costs a lot more. Regardless they just bought all this CASA hardware so we're stuck with it for a few years until they switch to FTTH

[RyzenFX]

@Telek I already mentioned that I'm on D3.1 downstream and D3.0 upstream- same configuration as @4kgigabitguy, and ideally I should be seeing similar results to him, but that's not the case. 

(I'm comparing D3.0 upstream results from the 3552 to the D3.0 upstream results on the CODA-4582; it's a fair comparison. I should be seeing really similar upload graphs between the town modems, but once again it's not the case.) 

 The buffer bloat on the CODA-4582 is much worse than 3552, as pointed out here. I've stated that the bufferbloat could be the reason for all the high packet loss in League of Legends, and random ping spikes in other games. I'm not sure if this is attributable to the Puma 7 chipset, but I'm confident that the Speedboost feature and Casa Systems CMTS is playing a big role in this problem.

[Telek]

---

@RyzenFX wrote:

@Telek I already mentioned that I'm on D3.1 downstream and D3.0 upstream- same configuration as @4kgigabitguy, and ideally I should be seeing similar results to him, but that's not the case. 

(I'm comparing D3.0 upstream results from the 3552 to the D3.0 upstream results on the CODA-4582; it's a fair comparison. I should be seeing really similar upload graphs between the town modems, but once again it's not the case.) 

---

Gotcha -- sorry, my bad.

---

@RyzenFX wrote:

The buffer bloat on the CODA-4582 is much worse than 3552, as pointed out here. I've stated that the bufferbloat could be the reason for all the high packet loss in League of Legends, and random ping spikes in other games. I'm not sure if this is attributable to the Puma 7 chipset, but I'm confident that the Speedboost feature and Casa Systems CMTS is playing a big role in this problem.

---

Sorry, I'm trying to follow. Since we see people on pure gigabit that also have the same problem, without speedboost, what's leading you to believe that it is a major factor?