CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[Aduly1998]

Hello dave, i am currently using the coda 4582 but i am not receiving the new firmware. Can you please push it to my router?

[Telek]

What plan are you on? Odd that you only have 8 channels connected (or did you just cut the other ones out?), yet DOCSIS 3.1.

The problem might be on your computer -- when you're getting the problem, try using nslookup and check to see what the DNS is returning.

[RogersDave]

---

@JohnBeaudin wrote:

Firmware will be ready in 2 weeks, but do you know when trial participants will be able to beta test .27? I can't wait to give that a shot.

---

I may not have been 100% clear on this one. I'm hoping to have the firmware 2.0.10.27 to deploy to trial participants in the next 2 weeks. Deployment to general modem population should come after once we are done with our internal testing and community testing.

---

@Aduly1998 wrote:
Hello dave, i am currently using the coda 4582 but i am not receiving the new firmware. Can you please push it to my router?

---

@Aduly1998, The current production firmware is 2.0.10.24. If you are interested in being part of the firmware trial program, send a private message to @CommunityHelps including your modem MAC address and serial number. They will send you further instructions.

Dave

[belowme]

@Datalink

Appreciate the information.

I've swapped my CODA 4 times in the past 2 weeks, each to one with a black dot.  The latest tech guy mentioned one with two black dots but when I went to the store to exchange they said they didn't have any but the ones they did have (1 black dot) were the latest build.

I'll get on the line with tech support and start this process and get back to you.

[Datalink]

@belowme, if you have already had a tech visit and still have the situation where you are missing the majority of DOCSIS 3.0 channels, when you are chatting with tech support to arrange a tech visit, request a senior tech.  

[belowme]

Thanks @Datalink 

Chatting with support now. They say I do have all DOCSIS 3.0 channels. 
I might have copied/pasted incorrectly.

Anything else I can provide that may hint at an issue?

[Datalink]

Ok, can you copy and paste the Downstream and upstream tables again.  Lets see if they paste in completely. 

[JohnBeaudin]

@RogersDave

Indeed I did misunderstood the 2 weeks, but not it's cyrstal clear, Thanks Dave!

[roy86]

same here I get disconnected from the internet whenever i get these logs. Called tech to check signal level and he said everything is good.  Modem is in bridge mode. Today I got disconnected after 4 days. Is this normal?

The DOCSIS event logs is shown here

No.	Time	Type	Priority	Event
1	04/07/2017 17:00:51	82000200	critical	No Ranging Response received - T3 time-out; CM-QOS=1.1;CM-VER=3.1;
2	04/07/2017 17:00:52	82000300	critical	Ranging Request Retries exhausted;CM-QOS=1.1;CM-VER=3.1;
3	04/07/2017 17:00:52	82000600	critical	Unicast Maintenance Ranging attempted - No response - Retries exhausted;CM-QOS=1.1;CM-VER=3.1;
4	04/07/2017 17:00:53	82000200	critical	No Ranging Response received - T3 time-out;CM-QOS=1.1;CM-VER=3.1;
5	04/07/2017 17:00:53	82000300	critical	Ranging Request Retries exhausted;;CM-QOS=1.1;CM-VER=3.1;

[Adva]

My only issue with this modem so far is random reboot.

On firmware .24 available to general public.

Got it installed satureday. It rebooted the same day once. Then it rebooted Tuesday early in the morning. Now it rebooted Friday afternoon.

[RyzenFX]

@Adva Firmware .26T2 fixes this issues but will not be released to the public b/c as Dave noted, it is an interim test build. Firmware .27 is going to be the network-wide release but Dave is expecting the firmware in the next-2 weeks, but this firmware will be distributed to the beta testers at first..

So maybe in 3-4 weeks .27 will be released to the general public. 

[Mythen]

last few days I am getting crazy lag spikes.  to the point where anything I watch online buffers for a moment or 2.  What would cause that?  Anyone else seeing this?

[belowme]

@Datalink @RogersDave

Here is the full table
I just rebooted too in order to access with a computer. Now part of my morning routine...


Downstream Overview
Port ID Frequency (MHz) Modulation Signal strength (dBmV) Channel ID Signal noise ratio (dB)
1 657000000 256QAM -2.400 17 38.983
2 561000000 256QAM -0.200 2 40.946
3 567000000 256QAM -0.600 3 38.983
4 573000000 256QAM -0.900 4 40.366
5 579000000 256QAM -1.200 5 40.366
6 585000000 256QAM -0.800 6 40.366
7 591000000 256QAM -1.100 7 40.366
8 597000000 256QAM -1.700 8 40.366
9 603000000 256QAM -1.300 9 40.366
10 609000000 256QAM -1.300 10 40.366
11 615000000 256QAM -1.500 11 40.366
12 621000000 256QAM -1.200 12 40.366
13 633000000 256QAM -0.600 13 40.366
14 639000000 256QAM -1.000 14 38.983
15 645000000 256QAM -1.200 15 40.366
16 651000000 256QAM -1.700 16 38.605
17 555000000 256QAM -0.500 1 40.366
18 663000000 256QAM -2.600 18 38.983
19 669000000 256QAM -2.200 19 38.983
20 675000000 256QAM -2.400 20 38.983
21 681000000 256QAM -2.700 21 38.983
22 687000000 256QAM -2.300 22 38.983
23 693000000 256QAM -1.800 23 38.983
24 699000000 256QAM -2.000 24 38.983
25 705000000 256QAM -2.400 25 38.605
26 711000000 256QAM -3.200 26 38.605
27 717000000 256QAM -4.000 27 38.605
28 723000000 256QAM -4.200 28 38.605
29 825000000 256QAM -5.300 29 37.636
30 831000000 256QAM -5.600 30 37.636
31 837000000 256QAM -5.100 31 37.636
32 843000000 256QAM -5.300 32 37.636

[4kgigabitguy]

@RyzenFX @Datalink @RogersDave @JohnBeaudin

Last night, I noticed my upload speed dropped dramatically on some speed tests which was odd. I've honestly have had a solid 50 Mbps upload on both wired & wireless ever since I've got Gigabit which has been over a month now. I was on a mission to find the root cause but saved it for the morning & played about 2 hours of GTA Online for PS4. Did not have any issues.

So today I began looking for the cause of the drop in upload. As of right now the best I have seen is 30Mbps upload &  I was even getting as low 10-20 Mbps upload. Did some reboots, checked the levels, the usual stuff etc. I am actually getting better results on DSL reports speedtest for bufferbloat with the lower upload speed(got an A for the first time, pasted below)....As you can also see on the Rogers speed check the upload graph is not as "smooth" as it was before...Chatted with Rogers online to see if they have any issues reports for my area & he mentioned that possibly there is. He mentioned it appears something is causing noise or congestion in my area & put in a ticket.

Went to test League of Legends right after & first thing I noticed by ping was sitting at 52MS...every game I have played in the past ping was sitting at 38MS. After the match I used Logs of Lag to check results. As you can see below the packet loss has increased & the ping is higher. It was still not enough to notice degradation in game performance as everything ran smooth. Figured I'd share as interesting to analyze these results compared to my previous ones.

Will dig into little further into this & report back.

[Mythen]

I have the 70u package. And we are seeing massive lag spikes. Like in the 2500 range. Causes twitch streams and youtube videos to buffer. Also affects my work connection. No idea how to fix this. Any ideas people

[4kgigabitguy]

---

@Mythen wrote:
I have the 70u package. And we are seeing massive lag spikes. Like in the 2500 range. Causes twitch streams and youtube videos to buffer. Also affects my work connection. No idea how to fix this. Any ideas people

---

Hey @Mythen

Can you provide some info on your setup ? What firmware is your modem running & how are the signal levels ? Also how long have you been experience the issue (just started or it's been like that for a while etc) ? How many devices are you running at home ? Also any troubleshooting steps you may have already taken.

This will be helpful in diagnosing the issue. I don't have 70u but I am using the CODA modem with Gigabit service & the family streams alot...they haven't been having buffering issues. 

[Mythen]

I'll post signal levels when I get home. But they look ok. I have the coda 4582 with the .26t2 firmware. Basically one PC directly connected to modem and another connect using powerline adapters. Not much wifi stuff. 2 pixel XL cellphones. 2 tablets and the 1st generation chrome cast. Not all are going at once. Mostly the two pcs. Issue has been happening last two weeks at least

[Windwalker]

So I signed up for 1000/50 in June 2016.   Download speeds have been all over the place, but my uploads were always fairly consistent at around 50.

Lo and behold, as of yesterday I can't get over 30 UP.  Was my profile changed on the system?  WTH!

Also, I have noticed that speeds for Gigabit are infuenced by how current a computer and processor I am using.  I just set up an i5 third gen computer and ran the speed test without antivirus installed.  The rogers speedtest resulted in 945/50 prior to yesterdays upload change.  As well the download graph was flat with no spikes or dropouts.

DSL reports yelded similar results.

@RogersDaveare you looking at the impact of antivirus software on speeds in your testing?

[RyzenFX]

@Windwalker To me it looks like it is the case of upload congestion. I've noticed a growing trend that many users are not able to hit 30Mbps/50mbps upload which only means that the upload usage is has increased dramatically within the past  6 months. You can call into Rogers and ask them to check if your node is affected by severe upload congestion/or possible noise on the upload side which can be limiting your speeds.

Hopefully D3.1 upload is coming in the next few months to help reduce the upload congestion. 

[RyzenFX]

@4kgigabitguy It probably could just be noise on the upload spectrum which could be reducing your speeds. This noise can become much more apparent in peak hours as a similar case happened to be a month ago. 

I do notice that your packet loss and bufferbloat is still quite low although you are affected by an uplink issue. 

What continues to puzzle me is that your upload bufferbloat is consistently low, whereas mine is consistently high. There's currently some work being done in the background to determine the cause of this issue, but I'm not sure how we both yield different upload bufferbloat results despite being on the same modem. 

[Mythen]

---

@4kgigabitguy wrote:

---

@Mythen wrote:
I have the 70u package. And we are seeing massive lag spikes. Like in the 2500 range. Causes twitch streams and youtube videos to buffer. Also affects my work connection. No idea how to fix this. Any ideas people

---

Hey @Mythen

Can you provide some info on your setup ? What firmware is your modem running & how are the signal levels ? Also how long have you been experience the issue (just started or it's been like that for a while etc) ? How many devices are you running at home ? Also any troubleshooting steps you may have already taken.

This will be helpful in diagnosing the issue. I don't have 70u but I am using the CODA modem with Gigabit service & the family streams alot...they haven't been having buffering issues. 

---

Downstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Signal noise ratio (dB)
1	615000000	256QAM	-4.300	11	36.387
2	561000000	256QAM	-4.200	2	37.356
3	567000000	256QAM	-3.700	3	37.356
4	573000000	256QAM	-3.900	4	37.356
5	579000000	256QAM	-3.700	5	37.356
6	585000000	256QAM	-4.000	6	37.356
7	591000000	256QAM	-4.100	7	36.610
8	597000000	256QAM	-4.100	8	37.356
9	603000000	256QAM	-3.900	9	36.610
10	609000000	256QAM	-4.500	10	36.610
11	555000000	256QAM	-4.300	1	36.610
12	621000000	256QAM	-3.900	12	37.356
13	633000000	256QAM	-4.100	13	36.610
14	639000000	256QAM	-3.900	14	37.356
15	645000000	256QAM	-3.900	15	37.356
16	651000000	256QAM	-3.500	16	37.356
17	657000000	256QAM	-3.800	17	37.093
18	663000000	256QAM	-3.800	18	36.610
19	669000000	256QAM	-3.800	19	36.610
20	675000000	256QAM	-3.800	20	36.610
21	681000000	256QAM	-4.300	21	37.093
22	687000000	256QAM	-3.900	22	37.356
23	693000000	256QAM	-4.000	23	36.610
24	699000000	256QAM	-4.200	24	36.387
25	705000000	256QAM	-4.300	25	36.610
26	711000000	256QAM	-4.900	26	36.387
27	717000000	256QAM	-4.800	27	36.610
28	723000000	256QAM	-5.700	28	36.387
29	729000000	256QAM	-5.000	29	36.387
30	735000000	256QAM	-5.400	30	36.387
31	741000000	256QAM	-5.300	31	36.387
32	747000000	256QAM	-4.300	32	36.175

OFDM Downstream Overview

Receiver	FFT type	Subcarr 0 Frequency(MHz)	PLC locked	NCP locked	MDC1 locked	PLC power(dBmv)
0	NA	NA	NO	NO	NO	NA
1	4K	251600000	YES	YES	YES	-7.400002

Upstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	38596000	ATDMA - 64QAM	44.000	3	3200000
2	30596000	ATDMA - 64QAM	38.250	1	6400000
3	23700000	ATDMA - 64QAM	38.000	2	6400000

OFDM/OFDMA Overview

Channel Index	State	lin Digital Att	Digital Att	BW (sc's*fft)	Report Power	Report Power1_6	FFT Size
0	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K
1	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K