CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[gp-se]

@Bond007 I was having issues getting my WD NAS to connect this morning, I ended up doing this to get it back online. 

-Shut down and disconnect the NAS from the Modem

-Factory Reset the Modem buy using a pen tip to push the button on the back of the modem, hold it down for 20 seconds. 

-Let the Modem boot up, then wait 5 mins

-Factory Reset the modem again holding the reset button on the back down for 20 seconds again.

-Let the modem boot up, run setup on the modem, configure WiFi and all that stuff.

-Plug the NAS back into the modem, then power on the NAS and allow it to boot up.

on the Modem I would suggest the following settings:

-SIP ALG: OFF

UPNP: OFF

2.4 Ghz Wifi Settings: 

Wireless Mode: Wireless N Only

Channel Width: 20mhz

WPS: OFF

Security: WPA 2 

Encryption: AES

5Ghz WiFi Settings:

Use the same settings for security, leave everything else alone. 

I hope that helps.

[crazyshot]

well, even the technician didn't even know why I was getting packet loss, so now I gotta wait for the supervisor to contact me

[toolcubed]

Alrighty, so now I've joined the club of the "speed degradation over time" and packet loss issues. Up until this week, internet has been rock solid. I'm using a black dot CODA with firmware .23, bridged and connected to an Asus RT-AC68U. I was consistently getting 200/12 on speed tests, while subscribing to the 100/10 plan. On Wed Feb 8 and Thurs Feb 9, I experienced service outages multiple times a day...Thurs was the worst when service was down almost the entire day. Ever since service was restored, i can't get anything faster than around 50Mbps. Tonight the fastest I get are 30/2. I also ran ping plotter and I'm getting some wicked packet loss in multiple hops including the first hop. I checked my DOCSIS stats and everything is peachy...signal levels are superb and no missing channels. I'm NOT currently served by D3.1. Prior to this week's outages, everything was solid. Now internet has been very wonky.

[timfolkins]

Welcome. It's a sad little club with very little on the update front for a little while.

[crazyshot]

@timfolkins @toolcubed do you guys mind running a trace route to these two IPs and let me know if you guys get packet loss on the 2nd hop?

192.64.170.252

104.160.131.1

They are the league servers btw, thanks!

[toolcubed]

I get 100% packet loss for the 1st IP on the last hop to that IP. For the 2nd address, no packet loss but pings avg around 60ms with occasional spikes to 120ms

[timfolkins]

about the same for me.

[crazyshot]

Well I just realized I get packet loss on the 2nd hop no matter what I am pinging to, even google. I always get exactly 50 percent loss on the hop. but when I am pinging the 2nd IP address itself it shows no packet loss..anyone knows what this could possibly mean?

[lethalsniper]

its been 4 days now with these horribl&now upload and jitter

[lethalsniper]

what with the bad upload and high jitter and ping

[lethalsniper]

like this ?

[lethalsniper]

this is just horrible hop2 hop3 hop 4 all rogers servers with high latency has somthing changed since last week ?because this is just horrible

[Bond007]

My modem is on .23 firmware and have been experiencing the same along with other issues.

I power cycle the modem number of times and may get lucky to get stable connection.

[timfolkins]

My modem is now slowing down in 2 hour increments. It was 7h, and then for a glorious 2 or 3 days it was almost a whole 24h... now... a measley 2h and all the line signal strengths are low.

I should have listened and never gone for this service. When will I learn. the 250 service was really steady for me.

[RyzenFX]

I understand that everybody is frustrated about having their speeds decrease over time. Rogers and Hitron engineers have been working on this issue for some time now, however apparently it seems to be a complex problem. I'm not aware of the progress that has been made, but hopefully we can get an update soon.

[borford]

I may have missed it, but have any of the firmware updates addressed Blackberry and Google devices listed below experiencing Wi-Fi intermittently dropping and quickly reconnecting? This was a known issue with firmware 2.0.10.13.

• BlackBerry Priv
• BlackBerry Q10
• Google Pixel XL,
• Google Nexus 6P

[prateeck7]

This is my Docis Event. What does it all mean? Can, anybody please interprete this?

No.	Time	Type	Priority	Event
1	01/01/1970 00:01:56	84000500	critical	SYNC Timing Synchronization failure - Loss of Sync;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
2	12/03/2016 12:19:42	90000000	warning	MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
3	12/03/2016 12:19:42	73040100	notice	TLV-11 - unrecognized OID;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
4	12/03/2016 12:19:48	69010200	notice	SW Download INIT - Via Config file silver_EMTA_Through_Auth_25Jun13_Thiru_NoCVC3_Coda.bin
5	12/03/2016 12:23:30	69011200	notice	SW download Successful - Via Config file
6	12/03/2016 12:25:18	90000000	warning	MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
7	12/03/2016 12:25:18	73040100	notice	TLV-11 - unrecognized OID;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
8	12/03/2016 12:25:18	68001102	critical	Config File Rejected due to Invalid or Unexpected TLV 11;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
9	02/09/2017 22:56:25	90000000	warning	MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
10	02/09/2017 23:26:21	68010300	error	DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
11	02/10/2017 02:11:39	84020200	warning	Lost MDD Timeout;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
12	02/10/2017 02:59:47	82000200	critical	No Ranging Response received - T3 time-out;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
13	02/10/2017 17:04:54	84020200	warning	Lost MDD Timeout;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
14	02/10/2017 17:07:02	90000000	warning	MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
15	02/12/2017 14:45:24	84020200	warning	Lost MDD Timeout;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
16	02/12/2017 14:45:32	82000200	critical	No Ranging Response received - T3 time-out;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;
17	02/12/2017 16:15:37	84020200	warning	Lost MDD Timeout;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=x:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;

Clear

Clear

[lethalsniper]

its only 11;30 on sunday and my speeds are already acting up @RogersDave

[lethalsniper]

anybody else getting high latency on hop 2,3,4 which are all rogers servers? @Datalink

[lethalsniper]

Host Name IP Address Hop Ping Time Ping Avg % Loss Pkts r/s Ping best/worst
hitronhub.home 192.168.0.1 1 1ms 0ms 0% 73 / 73 0ms / 3ms
* Unknown Host * 2 18ms 32ms 0% 73 / 73 8ms / 132ms
* Unknown Host * 66.185.89.101 3 24ms 29ms 0% 73 / 73 3ms / 93ms
* Unknown Host * 209.148.232.77 4 35ms 100% 0 / 73
* Unknown Host * 209.148.230.14 5 53ms 100% 0 / 73
* Unknown Host * 72.14.222.87 6 9ms 30ms 0% 73 / 73 9ms / 77ms
* Unknown Host * 209.85.255.232 7 25ms 100% 0 / 73
* Unknown Host * 216.239.46.162 8 88ms 100% 0 / 73
* Unknown Host * 216.239.50.235 9 61ms 100% 0 / 72
* Unknown Host * 108.170.244.1 10 50ms 100% 0 / 72
* Unknown Host * 216.239.41.161 11 39ms 40ms 0% 72 / 72 19ms / 118ms
ord37s07-in-f3.1e100.net 172.217.1.35 12 24ms 42ms 0% 72 / 72 19ms / 97ms

[JohnBeaudin]

@lethalsniper

pinging google.ca with Pingplotter

Hop 2 Avg 9.3

Hop 3 Avg 22.3

Hop4 27.5

Latency always been high on Rogers network thought, it's not something they fixed yet.