CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[notregistering]

I'm not sure if it is black dot, I can't see one so I'm guessing not. I got it beginning of January.

Are the signal levels okay? I think they are since between +10 and -10. It is on a splitter but the modem is on the -4 not the -7 and the 4k PVR box works fine.  I can also say this usually was fine on .19 just the speed degredation over time and i'd find 10 Mbps download max until a reboot from time to time. Upload generally was 30-35 when this happens.

[Datalink]

That black dot is pretty obvious, on the modems that have it, that is.  Can't miss it.  Whether your running DOCSIS 3.0 or 3.1, the external wifi interference will be a problem, so, a swap is in order followed by an update to .24 

The modem can actually live with low levels, at least in 3.0, not so sure about 3.1 yet, but, the missing channels won't do you any favours. 

[notregistering]

Hey Datalink,

so I sent you the images of pingplotter so you can see. It's not showing packetloss in that 10min timeframe but you can see where the 100% loss I mentioned is on that one hop.

I only tidied up the top to make sure of privacy on the public forums 😉

My 1st/2nd hop you see are my Cisco 3750G switch running that VLAN and then the /31 for Switch to PfSense connection.

And I hope to be back up within the hour on a swapped modem and start the process to get to .19 and to .24 (Since I assume @RogersDave will need to push this new firmware to a new MAC address now)

[Datalink]

Yup, got it.  One of the mods will have to approve the images before I can actually see them.  @CommunityHelps and Dave will need the MAC address and serial number for the new modem. 

[Hwaiting]

Does anyone know what determines which frequencies the channels run at?

Mine go all the way up to 843Mhz and my last 4 channels actually reside in the 800's which are -3dBs less than the other channels which are between the 500 and 700 range.  I see other people top out in the 600s.

On a side note, anytime I've checked signal strength at my cable boxes, I've always noticed the channels running at higher frequencies take a big hit in signal strength.  (i.e. SNE @ 789Mhz is -8dB but Global @ 165Mhz is +4dB).  I've always just assumed the higher frequency, the less it can penetrate or carry.

[Alex4161]

@Hwaiting

Yes the high frequency channels will be slightly weaker than the lower frequency channels. It should not matter as long as they operate within the -10 dBmV to 10 dBmV range.

Have a look at the Coax Cable Signal Loss chart in yellow over here: http://www.w4rp.com/ref/coax.html

Notice that for RG-6 wiring the signal difference within 100 feet for 200 Mhz vs 700 Mhz is about 3 dBmV which is what you are seeing and totally within spec. If your drop between signals is much more then there could be issues with the wiring, connectors, etc. and a call to Rogers Cable Repair might be an option.

[Telek]

---

@Alex4161 wrote:

Yes the high frequency channels will be slightly weaker than the lower frequency channels.

---

It's interesting looking at mine. I'm in a condo, so I'm about 80ft from the distro point.  My attenuation is ... interesting.

dB vs Frequecy

[Telek]

---

@Hwaiting wrote:

Does anyone know what determines which frequencies the channels run at?

There's basically a negotiation process when the modem starts up: checking which frequencies are currently in use, and utilizing ones which are not. So depending on how many devices are within your network segment, you can get frequencies anywhere in the supported range (which IIRC is 470-890, although technically the spec allows to just over 1000MHz which may be used in some situations).

[Alex4161]

@Telek

You are close to the Linex so you are experiencing positive slope of some sorts.  Generally the signal tends to drop down a bit and then equalizers are used to balance things out.  

Enjoy having STRONG signals with great s/n ratio 🙂

[Telek]

---

@Telek wrote:

---

@Hwaiting wrote:

Does anyone know what determines which frequencies the channels run at?

There's basically a negotiation process when the modem starts up: checking which frequencies are currently in use, and utilizing ones which are not. So depending on how many devices are within your network segment, you can get frequencies anywhere in the supported range (which IIRC is 470-890, although technically the spec allows to just over 1000MHz which may be used in some situations).

---

I stand corrected on the frequency range, but cannot edit that post anymore.

Just saw 357MHz used for downstream in someone's post, so it's gotta be at least that low.

[pacerdjl]

---

@tonytoronto wrote:

---

@Alex4161 wrote:

@RogersDave

One issue I have been seeing for some time is that if you dont use any of your wired or wireless devices for several hours you lose network connectivity. I turned my laptop on this morning connected to a cat 6 cable and did not get any internet access. I then tried to access the web on my google pixel and as soon as I did that the cat6 network connectivity was restored on my laptop.

Has anyone else had this issue?

thanks

---

  I have experienced that too, but in my case even after login in to modem webpage, the wireless and wired connections had no internet access. Had to reboot. This as now happened twice, and like you its always been first thing morning, no or litte activity during night.

---

@RogersDave

Yes. I have seen the exact same thing. Numerous times. CODA-4582 firmware .24 2nd gen black dot. Modem in standard gateway mode. Happens with wired devices, as well as 2.4G tablet, laptop. Randomly, usually after a few hours of inactivity. Service usually resumes within about 10 seconds once device or webpage access is attempted a few times. No entries in DOCSIS Event log (and modem doesn't reboot – has been up for almost 4 days now with no speed degradation!!). Almost like modem having trouble finding the right IP match for the device; so... DHCP issue???

Also, perhaps related to this, saw the following in the event log today:

20 02/21/2017 11:44:30 68010300 error DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;

[JohnBeaudin]

Looks like .24 is stable and will be a public release.

@RogersDave are you guys working on .25 already? 

[joker2168]

I am also having DHCP issues. Been running .24 since last week. Seems to happen after 24-48 hours. No errors but I can not get an IP. Settings are default. Have not changed anything in the DHCP area on the modem. 

[Telek]

---

@joker2168 wrote:

I am also having DHCP issues. Been running .24 since last week. Seems to happen after 24-48 hours. No errors but I can not get an IP. Settings are default. Have not changed anything in the DHCP area on the modem. 

---

DHCP internally or externally? What issues specifically are you seeing?

[joker2168]

DHCP on the LAN side. Devices unable to obtain an IP.

[Alex4161]

I sent Dave a Private Message about this issue to see if he can look at this and talk to Hitron.

This is an easy fix in that some networks have some heartbeat process incorporated that sends some data packet to the device to keep the connection active if there are periods of inactivity. Perhaps Hitron could look into something like this.

[joker2168]

Its a moot point for me now as I switched to Bridge mode with a EdgeRouter X. Just thought I would let Rogers know about the DHCP.

[soundwave80]

Is it too much to ask for a proper working firmware before pressuring me with email and text messages to swap modems?

[RogersDave]

---

@pacerdjl wrote:

Yes. I have seen the exact same thing. Numerous times. CODA-4582 firmware .24 2nd gen black dot. Modem in standard gateway mode. Happens with wired devices, as well as 2.4G tablet, laptop. Randomly, usually after a few hours of inactivity. Service usually resumes within about 10 seconds once device or webpage access is attempted a few times. No entries in DOCSIS Event log (and modem doesn't reboot – has been up for almost 4 days now with no speed degradation!!). Almost like modem having trouble finding the right IP match for the device; so... DHCP issue???

Also, perhaps related to this, saw the following in the event log today:

20 02/21/2017 11:44:30 68010300 error DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;

---

 Question to everybody, is this a new thing that you have observed only on 2.0.10.24 or something that existed before and need to be adressed by Hitron?

It has never been reported before so I just want to ensure it is reported correctly.

 Dave

[joker2168]

---

@RogersDave wrote:

---

@pacerdjl wrote:

Yes. I have seen the exact same thing. Numerous times. CODA-4582 firmware .24 2nd gen black dot. Modem in standard gateway mode. Happens with wired devices, as well as 2.4G tablet, laptop. Randomly, usually after a few hours of inactivity. Service usually resumes within about 10 seconds once device or webpage access is attempted a few times. No entries in DOCSIS Event log (and modem doesn't reboot – has been up for almost 4 days now with no speed degradation!!). Almost like modem having trouble finding the right IP match for the device; so... DHCP issue???

Also, perhaps related to this, saw the following in the event log today:

20 02/21/2017 11:44:30 68010300 error DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;

---

 Question to everybody, is this a new thing that you have observed only on 2.0.10.24 or something that existed before and need to be adressed by Hitron?

It has never been reported before so I just want to ensure it is reported correctly.

 Dave

---

For me just with .24 I had started to see this.

[notregistering]

Reporting back in.

Swapped my modem last night for a newer, black dot version. I was stuck on 2.0.10.13 until this afternoon when @RogersDave kindly pushed 2.0.10.24 to the modem for me. 

So far after 5 minutes of being back up, looking good! All pings are under 30ms now. Channels do not appear to randomly be missing either.  I see channels 9-40 with none missing in between.

I'm assuming it will take a bit to stabilize and will do extended testing before deciding that there is further issues. I am only pulling about 400-500 Mbps download, 38 Mbps upload. Prior to getting .24 on my original CODA, and then this new one that Dave pushed .24 to, I was able to get 850 Mbps down and 45-49 Mbps Up without much issue (save for the speed degradation over time on .19).

I can also report that while doing a speedtest, pings can get crazy and go into the thousands and timeout. Not sure if the CODA is getting clobbered with bandwidth / packets or why it can't seem to ping well while under heavy load.