CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[RyzenFX]

Yeah, it's so much better at handling UDP. Ping is low and packet loss is low.  Other than that, the CODA is better at everything else. 

[lethalsniper]

Oh that's good and isn't there suppose to be another firmware update for the CGN3552 which supposedly better for gaming ?

[lethalsniper]

Which firmware are you on with the CGN3552 ?

[markopas]

When I run the ipv6-test website I have never gotten better than 04/20 - I think I even got that while gateway mode and bridge mode directly connected to the CODA.  Do I really need to be overly concerned with IPv6 at this point in time?

[rjmaxim]

---

@markopas wrote:

When I run the ipv6-test website I have never gotten better than 04/20 - I think I even got that while gateway mode and bridge mode directly connected to the CODA.  Do I really need to be overly concerned with IPv6 at this point in time?

---

Markopas, no need to worry about IPv6.......for most transition from IPv4 to 6 will be transparent. IPv6 is a greatly needed protocol, particularly as we get into more IoT (Internet of Things) devices...IPv6 allows for significantly more IP adresses (Trillions) vs the 4.3 Billion in IPv4 plus much better security. Both Google and Facebook are totally on IPv6....but we don't notice it. It is more important for Rogers as a Service provider to have modems enabled for IPv6 traffic......unless you have a specific need to communicate with IPv6 only appliances. Here's more info on the subject > https://www.internetsociety.org/ipv6-frequently-asked-questions

[JohnBeaudin]

CGN3ACR is getting this end of March

4.5.8.29 (Target end of March 2017)

• This is a partial candidate list
• Fix: uplink speed
• Improvement: ICMP Latency
• Improvement: TCP Latency

I am curious to see after they get that update.. how much thier ping will be improved in online games.. if it's worth it I might go back to that modem because so far the Coda is laggy in online games.

[Double_K]

---

@JohnBeaudin wrote:

CGN3ACR is getting this end of March

4.5.8.29 (Target end of March 2017)

• This is a partial candidate list
• Fix: uplink speed
• Improvement: ICMP Latency
• Improvement: TCP Latency

I am curious to see after they get that update.. how much thier ping will be improved in online games.. if it's worth it I might go back to that modem because so far the Coda is laggy in online games.

---

@JohnBeaudin Depends on if your game uses TCP or UDP (or both), but just because ping (ICMP) may be improved, doesn't mean you won't be laggy due to UDP latency.

Using traffic shaping, a provider/device manufacturer could prioritize your ICMP traffic giving you the illusion of improved ping times, yet still have UDP packet latency causing "lagginess".

For most games, you want to focus on improvement in UDP latency.

[JohnBeaudin]

@Double_K

Almost all games I play use UDP, but yeah as a gamer I am looking for what would be best for me, since the CODA is not performing well latency wise in online games I am trying to see if something else would work better.

anyhow, let's see how it goes.

[Max24]

Since DOCSIS 3.1 was enabled in my area early last month my ping has been very unstable during the afternoon & evenings. During the night and early mornings my ping is pretty stable, I've been trying to diagnose the issue but haven't really been able to find out much. My guess is it's a network congestion issue as my game uses TCP (Windows PC). Below is a recent ping plotter:

[lethalsniper]

Yea that goes for me to , every night nothing but high ping and latency . Is the CGNM-3552 getting that firmware ?

[RyzenFX]

Hi @Max24 it appears that the packet loss is coming from the 3rd hop (67.231.221.225). Before we jump to any conclusions, can you please post another pingplotter graph to google.ca. This should help localize the issue (i.e if there's anything wrong on Rogers' end, or if it's a CMTS issue.)

[RyzenFX]

@lethalsniper All the PUMA 6 modems will be receiving the fix which should fix alot of it's latency issues. This includes the CGNM-3552, CGN3ACSMR, CGN3ACR, and CGN3.

[JohnBeaudin]

Pinging google.ca on CODA 4582

Target Name: google.ca

Date/Time: 3/13/2017 7:07:58 PM - 3/13/2017 7:17:58 PM

Hop Sent PL% Min Max Avg Host Name / [IP]
1 11 0 0.77 
2 11 0 10.40 15.41 12.23 2607:f798:80c:c1::1 [2607:f798:80c:c1::1]
3 11 0 19.05 25.51 21.49 2607:f798:10:8008:0:690:6325:5113 [2607:f798:10:8008:0:690:6325:5113]
4 11 0 18.01 24.16 20.47 2607:f798:10:1e7:0:241:5614:7017 [2607:f798:10:1e7:0:241:5614:7017]
5 11 100 0 0 0 [-]
6 11 100 0 0 0 [-]
7 11 0 36.96 70.77 41.35 2607:f798:10:296::2 [2607:f798:10:296::2]
8 11 0 36.29 42.86 38.79 2001:4860:1:1:0:32c:0:8 [2001:4860:1:1:0:32c:0:8]
9 11 0 36.46 61.08 43.92 2001:4860::1:0:713a [2001:4860::1:0:713a]
10 5 0 52.18 118.52 77.64 2001:4860::c:4000:d64b [2001:4860::c:4000:d64b]
11 5 0 50.11 54.92 52.64 2001:4860::1:0:cb45 [2001:4860::1:0:cb45]
12 10 0 48.33 54.06 50.82 2001:4860:0:1010::1 [2001:4860:0:1010::1]
13 10 0 74.89 153.58 108.97 2001:4860:0:1::1f31 [2001:4860:0:1::1f31]
14 11 0 47.71 54.17 49.61 google.ca [2607:f8b0:4009:802::2003]

I get 100% Packet loss on hop 5 and hop 6

[RyzenFX]

@JohnBeaudin

The 100% packet loss shouldn't be something you should be worried about because it doesn't transfer to the rest of the servers. From what I can see the rest of the hops and the final hop (google.ca) shows 0% packet loss. You should be fine.

[mhs2]

I don't know if this is related to the recent outage in Ontario, but I'm only getting 200 down/ 4-5 up on the gigabit package CODA 4582 black dot.  Also poor ping times, packet loss, and I can't use my voip phone.

[RyzenFX]

@mhs2

Can you log into the modem and under BASIC, navigate to DOCSIS WAN and copy and paste your downstream overview, OFDM channel readings, and upstream overview? Maybe it can be a signal problem.

[mhs2]

Downstream Overview
Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Signal noise ratio (dB)
1	609000000	256QAM	0.800	34	38.605
2	363000000	256QAM	0.200	10	38.983
3	369000000	256QAM	1.700	11	38.983
4	375000000	256QAM	1.800	12	38.983
5	381000000	256QAM	0.600	13	38.983
6	387000000	256QAM	2.100	14	38.983
7	393000000	256QAM	1.100	15	38.983
8	399000000	256QAM	1.000	16	38.983
9	405000000	256QAM	2.700	17	40.366
10	411000000	256QAM	1.100	18	38.605
11	417000000	256QAM	2.200	19	38.983
12	423000000	256QAM	3.300	20	38.983
13	429000000	256QAM	1.700	21	38.983
14	435000000	256QAM	3.000	22	40.366
15	441000000	256QAM	2.700	23	38.983
16	447000000	256QAM	2.100	24	38.983
17	555000000	256QAM	0.300	25	38.983
18	561000000	256QAM	0.600	26	38.983
19	567000000	256QAM	1.000	27	38.983
20	573000000	256QAM	-0.100	28	38.605
21	579000000	256QAM	0.700	29	38.983
22	585000000	256QAM	1.400	30	38.983
23	591000000	256QAM	1.100	31	38.983
24	597000000	256QAM	1.300	32	38.983
25	603000000	256QAM	0.500	33	38.983
26	357000000	256QAM	1.700	9	38.605
27	615000000	256QAM	0.800	35	38.605
28	621000000	256QAM	0.600	36	38.983
29	633000000	256QAM	-0.200	37	38.983
30	639000000	256QAM	-0.300	38	38.605
31	645000000	256QAM	0.200	39	38.605
32	651000000	256QAM	-1.300	40	38.983
OFDM Downstream Overview
Receiver	FFT type	Subcarr 0 Frequency(MHz)	PLC locked	NCP locked	MDC1 locked	PLC power(dBmv)
0	NA	NA	NO	NO	NO	NA
1	NA	NA	NO	NO	NO	NA
Upstream Overview
Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	23700000	ATDMA - 64QAM	30.250	2	6400000
2	38595805	ATDMA - 64QAM	34.250	3	3200000
3	30596000	ATDMA - 64QAM	30.250	1	6400000
OFDM/OFDMA Overview
Channel Index	State	lin Digital Att	Digital Att	BW (sc's*fft)	Report Power	Report Power1_6	FFT Size
0	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K
1	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K

[RyzenFX]

All your downstream and upstream signal levels look really good, have you tried rebooting the modem after issues haven been cleared? If that doesn't work, if you have time try calling into customer support and have them check if there are any issues that could be affecting your speeds. Have them also check the area node aswell.

[mhs2]

I tried using live chat and it was useless, I'll try calling.  Also tried a reboot

[RyzenFX]

Yeah your signals look really good, you should be hitting 940Mbps download and 50mbps/ 30Mbps upload, but it doesn't make sense why you aren't hitting them, unless there is an area noise issue or congestion issue.

[mhs2]

I forgot to mention that I had a tech come out a month or so ago and he said that my house needed to be rewired and they would wait until the spring to bury the line from outside as well.