CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[sbenninger]

I am on my second CODA now and both have the same issue with losing connection to the CMTS random times throughout the day. It is in bridge mode and for ~30s before the issue the latency shoots up then the 2 blue arrows go green then flashing to start to re-sync. Sometimes once a day up to 10 times a day. I am on the .27 firmware version but I have just signed up for the beta program to see if it helps...not sure how long it takes for the update to be pushed out?

Tech support checked all my signals, logs, area issues etc and could find no issues hence the suggestion to swap the CODA but the issue still remains.

One thing I was able to catch (perhaps its normal?) is that my WAN IP temporarily went to a 192.168.100.xx IP address right as the issue happened and after re-connection back to my normal 99.x.x.x address.

Thanks,

[Telek]

---

@sbenninger wrote:

I am on my second CODA now and both have the same issue with losing connection to the CMTS random times throughout the day. It is in bridge mode and for ~30s before the issue the latency shoots up then the 2 blue arrows go green then flashing to start to re-sync. Sometimes once a day up to 10 times a day. I am on the .27 firmware version but I have just signed up for the beta program to see if it helps...not sure how long it takes for the update to be pushed out?

Tech support checked all my signals, logs, area issues etc and could find no issues hence the suggestion to swap the CODA but the issue still remains.

One thing I was able to catch (perhaps its normal?) is that my WAN IP temporarily went to a 192.168.100.xx IP address right as the issue happened and after re-connection back to my normal 99.x.x.x address.

Thanks,

---

You're clearly losing signal, which means that there should be something in the logs. I find it hard to believe that there's no issues at all.

You might need to rule out on-premises issues. Since this happens frequently, and with two different modems, it's likely to be something in your location.

Try to connect directly to the incoming line, this is ideal and can eliminate almost all on-site issues. If you can do this, run a constant ping and check for drops throughout the day.

If necessary run through a single Y-splitter only, and try to have the modem directly next to the incoming line and splitter to avoid any potential issues with cabling in your residence. No amplifiers other other devices. Ensure that it's a high quality proper splitter designed for cable modems.

If you can prove that the issue still exists in those cases, you can get a technician to come out and evaluate.

[sbenninger]

Thanks for the suggestion. There is 1 splitter right near the modem that was installed 2.5years ago when we moved in. I will disconnect the cable box and run straight through to the modem for a day (if the fam can live without cable for a day) and see if that resolves anything. If I remove the splitter will the signal become too strong? At my old house I needed a -3db pad to knock down the signal with a direct line to the cable modem.

[Telek]

---

@sbenninger wrote:

Thanks for the suggestion. There is 1 splitter right near the modem that was installed 2.5years ago when we moved in. I will disconnect the cable box and run straight through to the modem for a day (if the fam can live without cable for a day) and see if that resolves anything. If I remove the splitter will the signal become too strong? At my old house I needed a -3db pad to knock down the signal with a direct line to the cable modem.

---

Check your 192.168.100.1 page and look for the modem signal levels. As long as they're under 8 you should be fine to remove the splitter.

[sbenninger]

---

@Telek wrote:

---

@sbenninger wrote:

Thanks for the suggestion. There is 1 splitter right near the modem that was installed 2.5years ago when we moved in. I will disconnect the cable box and run straight through to the modem for a day (if the fam can live without cable for a day) and see if that resolves anything. If I remove the splitter will the signal become too strong? At my old house I needed a -3db pad to knock down the signal with a direct line to the cable modem.

---

Check your 192.168.100.1 page and look for the modem signal levels. As long as they're under 8 you should be fine to remove the splitter.

---

For reference here are the logs that look to indicate a loss of signal:

No.         Time      Type      Priority Event

1              06/30/2017 09:35:52       90000000             warning                MIMO Event MIMO: Stored MIMO=-1 post cfg file MIMO=-1;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:97:de:05;CM-QOS=1.1;CM-VER=3.1;

2              06/30/2017 09:46:03       84000500             critical   SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:97:de:05;CM-QOS=1.1;CM-VER=3.1;

3              07/26/2017 18:02:03       90000000             warning                MIMO Event MIMO: Stored MIMO=-1 post cfg file MIMO=-1;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

4              07/26/2017 18:02:11       69010200             notice   SW Download INIT - Via Config file bac10800010664777d660640

5              07/26/2017 18:02:58       69011200             notice   SW download Successful - Via Config file

6              07/26/2017 18:38:57       90000000             warning                MIMO Event MIMO: Stored MIMO=-1 post cfg file MIMO=-1;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

7              07/27/2017 17:37:35       82000200             critical   No Ranging Response received - T3 time-out;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

8              07/27/2017 17:37:35       82000300             critical   Ranging Request Retries exhausted;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

9              07/27/2017 17:37:35       82000600             critical   Unicast Maintenance Ranging attempted - No response - Retries exhausted;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

10           07/27/2017 17:37:37       82000200             critical   No Ranging Response received - T3 time-out;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

11           07/27/2017 17:37:37       82000300             critical   Ranging Request Retries exhausted;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

12           07/27/2017 17:37:37       82000600             critical   Unicast Maintenance Ranging attempted - No response - Retries exhausted;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

13           07/27/2017 17:37:38       82000300             critical   Ranging Request Retries exhausted;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

14           07/27/2017 17:37:38       82000600             critical   Unicast Maintenance Ranging attempted - No response - Retries exhausted;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

15           07/27/2017 17:38:05       82000200             critical   No Ranging Response received - T3 time-out;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

16           07/27/2017 17:38:25       90000000             warning                MIMO Event MIMO: Stored MIMO=-1 post cfg file MIMO=-1;CM-MAC=x:x:x:x:x:x;CMTS-MAC=00:17:10:93:b3:0e;CM-QOS=1.1;CM-VER=3.1;

[sbenninger]

And signal levels with the splitter removed. I have a -3db attenuator if needed.

Downstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Signal noise ratio (dB)
1	615000000	256QAM	5.900	11	32.237
2	849000000	256QAM	7.800	2	31.915
3	855000000	256QAM	7.500	3	31.689
4	861000000	256QAM	6.700	4	31.072
5	579000000	256QAM	5.900	5	32.585
6	585000000	256QAM	6.200	6	32.676
7	591000000	256QAM	6.300	7	32.963
8	597000000	256QAM	6.200	8	32.963
9	603000000	256QAM	6.300	9	32.676
10	609000000	256QAM	6.200	10	32.585
11	303000000	256QAM	2.500	1	33.957
12	621000000	256QAM	5.900	12	32.237
13	633000000	256QAM	6.600	13	32.585
14	639000000	256QAM	6.800	14	32.585
15	645000000	256QAM	6.600	15	32.321
16	651000000	256QAM	6.600	16	32.321
17	657000000	256QAM	6.800	17	32.676
18	663000000	256QAM	6.800	18	32.676
19	669000000	256QAM	7.200	19	33.063
20	675000000	256QAM	7.200	20	33.063
21	681000000	256QAM	6.800	21	32.963
22	687000000	256QAM	6.600	22	32.963
23	693000000	256QAM	6.900	23	32.676
24	699000000	256QAM	7.700	24	32.676
25	705000000	256QAM	7.500	25	32.676
26	711000000	256QAM	7.300	26	32.676
27	717000000	256QAM	7.500	27	32.676
28	723000000	256QAM	7.400	28	32.321
29	825000000	256QAM	7.400	29	32.321
30	831000000	256QAM	7.100	30	31.994
31	837000000	256QAM	7.000	31	31.915
32	843000000	256QAM	7.100	32	31.915

OFDM Downstream Overview

Receiver	FFT type	Subcarr 0 Frequency(MHz)	PLC locked	NCP locked	MDC1 locked	PLC power(dBmv)
0	NA	NA	NO	NO	NO	NA
1	4K	290600000	YES	YES	YES	3.599998

Upstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	38596000	ATDMA - 64QAM	44.250	3	3200000
2	30596000	ATDMA - 64QAM	40.000	1	6400000
3	23700000	ATDMA - 64QAM	39.750	2	6400000

OFDM/OFDMA Overview

Channel Index	State	lin Digital Att	Digital Att	BW (sc's*fft)	Report Power	Report Power1_6	FFT Size
0	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K
1	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K

[Telek]

@sbenninger your log is full of errors -- that's pretty clearly a problem. Don't know why the tech would say there was no issue.

Your signal levels are great, no need for an attenuator. However your SNR is lower than I'd like to see (usually 38+).

Run it like this for a day, see if you have any problems.  Is it directly connected to the incoming line (not via cabling that's in the residence?)

[sbenninger]

Yes this line comes into the house by the electrical panel right from the rogers dmarc box and the modem is about 8ft away. The rest of the house is fed from separate feeds from the dmarc. There are rogers and bell green boxes right in front of my house by the road. 

For reference the SNR was ~34-35.5db prior to taking out the splitter. They have dropped with the direct feed. Perhaps I should add the attenuator to see if the SNR improves at a lower signal strength?

thanks for your assistance. 

[Telek]

You could try, but probably best to leave it as is and see if the condition improves.

[gp-se]

---

@sbenninger wrote:

Yes this line comes into the house by the electrical panel right from the rogers dmarc box and the modem is about 8ft away. The rest of the house is fed from separate feeds from the dmarc. There are rogers and bell green boxes right in front of my house by the road. 

For reference the SNR was ~34-35.5db prior to taking out the splitter. They have dropped with the direct feed. Perhaps I should add the attenuator to see if the SNR improves at a lower signal strength?

thanks for your assistance. 

---

as mentioned your SNR is low, call in and ask them to check other houses in the area. If everyone has bad SNR then perhaps you need a maintenance crew sent out to see where the noise is coming from. If the low SNR is just on your line then something is causing noise, perhaps ingress on a cable or something in the DMARC.

[AOD999]

I still have Loss of ofdm lock. Luckily I haven't seen it since a tech visit recently. Also have GUI issues where it fails to load the tables with pfsense after it provisions itself on the network.

[Telek]

So, any news on the new gateway that was supposed to be announced a couple of weeks ago?

[RyzenFX]

@Telek No news but I'd hold your horses on that one.

For all CODA-4582 users on firmware 2.0.10.27 the new network version is now 2.0.10.28T2. If you manage to reboot your modem, you will now receive this version on boot. I belive that this version only contain minor internal changes (nothing related to improvements) if I can recall what Dave said earlier. 

[Mythen]

@RyzenFX

Have you noticed that the latest version of the firmware .31 is not nearly as stable as the network wide version. My latency is all over the place

[RyzenFX]

@Mythen While I was on .31 I didn't really notice the latency problem that was present on .30. From my experience I believe that .31 still has a latency problem but not to the extent that was on .30.

While playing League of Legends there might have been some small hiccups where the ping would increase from 27ms to 31ms but that's all I really noticed on firmware .31. If you want you can reboot the modem to be put on .28T2 and compare the latency between the two versions. 

[Mythen]

I may do that. There is definitely something wrong with this version. The random lag spikes is back

[igorella1]

I have a strange problem to report:

I got CODA-4582 back in Dec 2016. It work reasonably well until June 2017 showing 130 mbps on the ethernet connection, and 185 on WIFI ( using 5GHz wide channel on both iPad Air Pro and LG G5) on my 100u plan. Very respectable.

Around mid-June my area experienced multiple Rogers cable outages, several technician vans became a fixture in the neighborhood. At one point my internet speed fell to below 1mbps. Reboot the modem. Speed back to normal. After 15 minutes down to 1 mbps. Several time.

Enter Rogers technician. Changed modem.   Normal speed. Refuses to wait for 15 minutes. After 15 minutes speed down to 1mbps.

Complaint to Rogers. Second tech came. This one changed a couple of connectors in the basement, plus a modem and left. Same story (1 mbps after 15 minutes)

Complaint to Rogers. Visit scheduled for Senior Tech. 2 vans arrived, one of the guys introduced himself as senior tech. Told me that my signal is too strong, added attenuator to a patch cable and left without waiting. 15 minutes and 1 mbps. I called him and demanded Rogers ID number. That is when he admitted that he is not a senior tech at all.

Complaint to Rogers. This time real live senior tech arrived. Very thorough. Took ownership of the issue. Replaced all the cables in the basement. Threw away the attenuator. Replaced the modem twice. Same picture. He then decided to test the speed at the green box with his PC and new (CODA-4582) modem. Same picture.  I talked him to test the same setup (modem plus his laptop) at the green box across the street. This worked like a charm. The problem was upstream from the box. So far so good. He strung up a temporary line and initiated workordersto bury the cable and for Rogers maintenance to fix the infrastructure problem. Everything worked for 2 to 3 weeks, survived several outages. Then somebody at Rogers removed the temp line, cancelled the burial order and reconnected me to the old green box. Problem reappeared if full force!

This time I went to the store and downgraded my modem (did not want to do it because of future proof and much poorer WIFI performance of the Rocket CGN3AMR modem. Result: stable fast connection. Speed of 130 on the ethernet, WIFI 185 on Android, but only 35 on iPad Air Pro next to the modem. 

If you think my description above is too verbose, here is a brief summary of the puzzle:

CODA-4582 on one green box works fine, on a different one, boots up with  high speed, but "chokes" within 15 minutes and slows down to below 1 mbps. Tested with 4 different CODA-4582 several times.

CGN3AMR works fine on the "problem" green box

[Telek]

@igorella1 so we know that, at least in the past, the CODA-4582 had a lot of problems where it would lock onto bad channels and refuse to release. This was a problem for many users with good connections in the past.

It's possible that since there's clearly a physical line issue, the CODA modem isn't switching away from bad channels still, whereas your CGN did. I would still push rogers to repair the line (escalate to the office of the president, you've been through more than enough). Since a senior tech diagnosed, found the problem, provided a patch that worked (which proves the problem), it's ridiculous to cancel that later on.

[Datalink]

@igorella1 I've sent a message to the moderators thru the @CommunityHelps address to review your post.  Hopefully that will start the ball rolling to see this thru to the end.  Please have a look at your avatar at the upper right when you're logged into the forum.  A number overlaid on that avatar will indicate a message waiting or other mention of your name in the forum.  Follow the avatar (as a link) to the message and profile services on the following linked pages to drill down to the message inbox. 

[igorella1]

Thanks for your help

[igorella1]

Thank you for taking the time and for encouragement