CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[Datalink]

isn't that what I said?   I'm sure it was, lol ....

[Vikentieff]

---

@Telek wrote:

What is it that you were doing that needed a factory reset as opposed to just a reboot?  I think this is there as an emergency bail-out situation. The last thing we want is to be on a beta firmware, find out that it's not so great, then have to suffer with it.

Keep in mind that you can backup your settings and restore them at any point, which may give you the same benefit that you find with a factory reset.

---

For this modem it is not an "emergency bail-out situation" -- it's a necessity, has always been, from the very beginning.  Also I've been in the beta since CGN days (which is like almost 2.5 years by now) and at the beginning you were able to do a reset without reverting to a public release fw. Now it's just silly because the modem's performance degrades rapidly, and after 2 updates it lags so badly it's unusable. 

[Datalink]

@Vikentieff are you running a VPN for extended periods of time by any chance.  Just trying to figure out why you would see a huge drop in performance.  I don't reboot my modem unless its had a very recent update or unless I switch from Bridge to Gateway mode or vice versa for test purposes.

[toolcubed]

Is anyone else experiencing packet loss with IPv6? My CODA is bridged and I'm using an Asus RT-AC68U router. IPv6 is configured properly on the Asus. Using ping plotter, there's no packet loss with IPv4 but with IPv6, there's packet loss across multiple hops including the very first one. I'm using Google to run ping plotter tests on both IPv4 and IPv6. Thanks.

[Datalink]

I haven't checked this with IPV6 but with IPV4 I'm seeing packet loss.  I'm coming to the conclusion that there's something strange going on with .27 when it comes to pingplotter returns from the CMTS.  I was playing with that just now in IPV4, pinging the CMTS and when the modem is in Gateway mode there is packet loss reported from the modem when the ping interval is less than 1 second.  Its also possible to see the occasional packet loss report from the CMTS, but that isn't usual from what I can see.  Remember this is only going as far as the CMTS.  When you go beyond the CMTS, say for example google.ca, at 1 second ping interval I might see the odd packet loss report from the modem.  If I reduce the ping interval below 1 second, something trips at some point and the packet loss becomes a regular occurance from both the modem and the CMTS, ending up around a continual 45% or so.  So that is definitely odd.  That didn't happen on previous modem firmware versions. 

I don't have time at the moment to repeat this in Bridge mode thru the router, but, I played very briefly with that last night and ended up with packet loss indicated from the modem and CMTS with a ping time less than 1 second.  I'd like to have a better look at that to confirm it, but, don't have time until later this evening. 

So, from what I can see, running 1 second intervals should take care of this for now.  The ultimate fix is to fix the modem.

If you ever doubt the pingplotter results, bring up a command prompt and ping the same address or any point along the way:  ping xxx.xxx.xxx.xxx -t    That will confirm or deny the pingplotter packet loss report.

Use Command C to bail out

[Mythen]

@Datalink. I've seen some of this as well. Where. I was testing ping plotter for twitch issues. If I use something like Google.ca that uses ipv6 it's not present. Check some of the ping plotter results I posted on the gaming thread

Mythen

[toolcubed]

I haven't tried using a command prompt yet but I consistently get packet loss with IPv6 regardless of address and regardless of interval. I haven't tried with the CODA in gateway mode. I get zero packet loss with IPv4...all is well. Interestingly enough, I disabled IPv6 on my Asus and not only do sites load quite a bit "snappier" but I also no longer get the occasional timeouts I was getting with IPv6 enabled. I think I'll leave it disabled for now 🙂

Oh, and the IPv6 packet loss isn't new to .27. I was seeing this several firmware versions ago.

[Datalink]

@toolcubed what are you using for the IPV6 DNS address in your Asus router?

And, are you using the AI Protection by any chance.  I think that uses site reputation and doesn't necessarily scan any or all IPV4 packets.  From what I've seen using speed tests with all or some of the components of the AI Protection enabled, it looks as though IPV6 packets are scanned, which will drop the throughput rate considerably.  Try running the speedtest at:

http://speedtest.xfinity.com/

With IPV6 enabled that should show both IPV4 and IPV6 results separately.  You have to run through one test before you can access the advance settings which is nothing more than site selection.  You can then choose Detroit as the nearest server.  Using that site, try different components of the AI Protection.  That will give you a pretty good idea of how much of a drop the AI Protection imposes, if in fact you are using it.

[toolcubed]

Thanks Datalink. AIProtection is disabled. I re-enabled IPv6 and ran thru the Xfinity speed test as you suggested. Results are more or less the same for both IPv6 and IPv4. Both are very good (slightly more on both download and upload than what I'm subscribed to...I have 100/10 but tests show 130/11). I then decided to do another IPv6 ping plotter test and there was NO packet loss. So either:

1. Packet loss on IPv6 is tied to peak times and congestion (I never ran a ping plotter test this early...it was before 7am ET).

OR

2. The disabling/enabling of IPv6 on my Asus fixed the problem. I'll keep testing and monitoring to see if/when the packet loss comes back.

IPv6 setting on the Asus:

Connection Type: Native
DHCP-PD: Enable
LAN Prefix Length: 64
Auto Config Setting: Stateless
Connect to DNS Server Automatically: Enable
Enable Router Advertisement: Enable

[Telek]

---

@Vikentieff wrote:

For this modem it is not an "emergency bail-out situation" -- it's a necessity, has always been, from the very beginning.  Also I've been in the beta since CGN days (which is like almost 2.5 years by now) and at the beginning you were able to do a reset without reverting to a public release fw. Now it's just silly because the modem's performance degrades rapidly, and after 2 updates it lags so badly it's unusable. 

---

But this really requires a factory reset? As opposed to just a reboot?  I haven't seen anyone complain here of problems that persist past a reboot.

[toolcubed]

OK, here's an update to my last post above.  Just did a couple more ping plotter tests to Google IPv6.  The first test showed packet loss on hops 1, 2, and 9.  The packet loss gradually went away on hops 1 and 2 but remained there for hop 9.  I then did another speed test from the xfinity site but it was normal on both IPv6 and IPv4 (130 down and 11 up).  I then did another ping plotter test to Google IPv6 and there was consistent packet loss on hop 9 only.  Still zero packet loss on IPv4.  These tests were performed around 10:45am ET.

[Telek]

---

@Datalink wrote:

I haven't checked this with IPV6 but with IPV4 I'm seeing packet loss.  I'm coming to the conclusion that there's something strange going on with .27 when it comes to pingplotter returns from the CMTS.  I was playing with that just now in IPV4, pinging the CMTS and when the modem is in Gateway mode there is packet loss reported from the modem when the ping interval is less than 1 second.  Its also possible to see the occasional packet loss report from the CMTS, but that isn't usual from what I can see.  Remember this is only going as far as the CMTS.  When you go beyond the CMTS, say for example google.ca, at 1 second ping interval I might see the odd packet loss report from the modem.  If I reduce the ping interval below 1 second, something trips at some point and the packet loss becomes a regular occurance from both the modem and the CMTS, ending up around a continual 45% or so.  So that is definitely odd.  That didn't happen on previous modem firmware versions. 

I don't have time at the moment to repeat this in Bridge mode thru the router, but, I played very briefly with that last night and ended up with packet loss indicated from the modem and CMTS with a ping time less than 1 second.  I'd like to have a better look at that to confirm it, but, don't have time until later this evening. 

So, from what I can see, running 1 second intervals should take care of this for now.  The ultimate fix is to fix the modem.

If you ever doubt the pingplotter results, bring up a command prompt and ping the same address or any point along the way:  ping xxx.xxx.xxx.xxx -t    That will confirm or deny the pingplotter packet loss report.

Use Command C to bail out

---

FWIW I almost always have computers on my network running continual V4 pings to google, and I never see packet loss issues, except for that time before the .24 firmware and we had that really bad performance degradation.  Using standard ping -t www.google.com command.  I'd expect that many of the ping losses here are actually from modem updates/normal reboots.

Also when using the ping -t command you can Control-Break to get current statistics without breaking the ping trace.

Readding the ping results (note to moderators: this does NOT contain private info)

Ping statistics for 172.217.0.164:
    Packets: Sent = 3904471, Received = 3903971, Lost = 500 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 3ms, Maximum = 3004ms, Average = 9ms

[dyaleyk]

I have been consistently having low speed issues on the 2.4GHz network. I have the 100u package and the most I can get when using the 2.4GHz network is somewhere around 25mbps while sitting right next to the CODA 4582 modem. If I were to move a few feet away, the speed drops to under 10. This has been happening with multiple devices.

I haven't used wired connection much but the few of the times I connected to run speed tests, it seemed fine. 5GHz network also seems fine for the most part.

I have tried rebooting as well factory resetting the modem but it hasn't helped. Channel is currently set to auto and seems to be find as per WiFi analyzing apps on my phone and laptop. I have several devices only capable of 2.4GHz so I need to use it.

Any advise?

[Datalink]

First thing I would do is, check/set the following 2.4 Ghz wifi parameters:

Wireless Mode:  802.11 n
Channel Bandwidth:  20/40 Mhz, although, for test puposes you could set this to 20 Mhz.  In a crowded wifi environment, I would set this for 20 Mhz.  
Wireless channel:  AUTO or, to an open channel if one existed, or to the channel that offers the least interference from neighboring routers and modems
WPS Enabled:  OFF
Security Mode:  WPA-Personal
Auth Mode:  WPA2-PSK
Encrypt Mode:  AES only

Save the setting and ensure that the Encrypt Mode stays on AES only.  If it changes on its own to TKIP/AES, change it back to AES only and save the setting again.  TKIP is no longer secure and from what I remember will cause the wifi data rates to cap at 50 Mb/s.  I'll have to look this up again.

Log into the modem and check/set the following 5 Ghz wifi parameters:

Wireless Mode:  802.11 a/n/ac mixed
Channel Bandwidth:  80 Mhz, although, for test puposes you could set this to 40 Mhz
Wireless channel:  149 to 165
WPS Enabled:  OFF
Security Mode:  WPA-Personal
Auth Mode:  WPA2-PSK
Encrypt Mode:  AES only

Once again, save the setting and ensure that the Encrypt Mode stays on AES only.  If it changes on its own to TKIP/AES, change it back to AES only and save the setting again.  

Reboot the modem if you had to make any changes, ADMIN ..... DEVICE RESET .... Reboot.

Did you use one of the following applications on your laptop to check the 2.4 Ghz wifi environment?

http://www.techspot.com/downloads/5936-inssider.html

https://www.acrylicwifi.com/en/wlan-software/wlan-scanner-acrylic-wifi-free/

http://www.nirsoft.net/utils/wifi_information_view.html

Ideally you would have a power separation in the neighborhood of 40 to 45 dBmW between your network and any network that overlaps the channel that you are using.  With inSSIDer, you would see that via the graphical display, and see it on the text data which can be sorted by using the column headers to sort up or down.  When that power separation decreases, you will end up with reduced data rates and possibly get to a point where running your network is almost impossible.  You have to take a critical look at the power levels and determine if there is any channel that offers the least amount of interference in terms of the power levels. 

Just to note the Hitron modems have never been noted for good wifi performance.  Having said that, there is an upcoming firmware version, V2.0.19.29 which will include improvements in wifi performance although I am not aware of the specifics.  That will hopefully be released to the trial group within the next month.  So, that might be enough to solve the issue.  Beyond that, if you are not able to resolve the issue thru a channel change, then perhaps you need to consider a router with external antenna which will give you better wifi performance.  That will also give you access to all of the settings that you typically find within a router but are not necessarily accessible in the modem.   

[lupinglade]

Could the constant ARP request bombardment from the Casa CMTS be causing the UDP packet loss issues? Up until the Casa was installed in my neighbourhood, everything was working great and there were no ARP storms.

On 2.0.10.27 and the UDP packet loss is still there all the time on any UT4 game server. As soon as a higher UDP rate is used (due to removing FPS restriction) there is consistent upstream UDP loss. Limiting the FPS to a lower rate will remove most of the loss - in this game the fps is directly tied to how many UDP messages go out. I can provide more info on the problem-causing rates if needed.

Also tried using a VPN (tunnel bear) and the packet loss goes away completely (since its tunnelled over TCP).

[ololo]

Does all coda 4582 modem updated to .27 version ? Or only for trial program user?

[Datalink]

@ololo if you don't have V2.0.10.27 loaded, as is seen in the Software (Firmware) Version on the modem's Status page, reboot your 4582.  It will load .27 during the boot process and reboot on its own.  I usually run a second reboot after all is said done.  Version .27 has been released as the latest production firmware version for all 4582s on the network, passing the final approval process after running on the trial modems for a period of time. 

[JohnBeaudin]

Other than improvements in wifi, is there anything else in the work with .29?

[lupinglade]

sh-4.2# iperf3 -c iperf.he.net -p 5201 -u -b 1M -i 1

Connecting to host iperf.he.net, port 5201

[  4] local 0000:0000:000:00:0000:0000:0000:0000 port 50133 connected to 2001:470:0:238::2 port 5201

[ ID] Interval           Transfer     Bandwidth       Total Datagrams

[  4]   0.00-1.00   sec   120 KBytes   983 Kbits/sec  15  

[  4]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec  16  

[  4]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec  16  

[  4]   3.00-4.00   sec   128 KBytes  1.05 Mbits/sec  16  

[  4]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec  16  

[  4]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec  16  

[  4]   6.00-7.00   sec   128 KBytes  1.05 Mbits/sec  16  

[  4]   7.00-8.00   sec   128 KBytes  1.05 Mbits/sec  16  

[  4]   8.00-9.00   sec   128 KBytes  1.05 Mbits/sec  16  

[  4]   9.00-10.00  sec   128 KBytes  1.05 Mbits/sec  16  

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams

[  4]   0.00-10.00  sec  1.24 MBytes  1.04 Mbits/sec  5.531 ms  60/159 (38%)  

[  4] Sent 159 datagrams

Constant packet loss, both directions.

[Datalink]

@lupinglade what version of iperf are you running, 3.13 as available off of the iperf.fr site, or 3.16 as available from neowin.com?

https://www.neowin.net/forum/topic/1234695-iperf-316-windows-build/#comment-596631033

[lupinglade]

iperf 3.1.7