cancel
Showing results for 
Search instead for 
Did you mean: 

CODA-4582 - Open Issues for Investigation

RogersDave
Retired Support
Retired Support

*** This post was last edited May 2, 2017 ***

 

Good morning Community,

 

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

 

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

 

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

 

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

 

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

 

 

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

 

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

 

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

 

Update April 22: This problem seems resolved in firmware 2.0.10.27

 

 

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

 

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

 

Workaround: manually select higher channels (149-153-157-161)

 

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

 

 

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

 

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

 

 

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

 

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

 

 

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

 

This problem has been reported to Hitron for investigation.

 

 

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

 

Investigation continues with Hitron.

 

 

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

 

 

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

 

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users


 

Future Planned Improvements

The following are items that we are working on in parallel of the above.

  • Improvement in WiFi speeds
  • Improvement in latency / bufferbloat

 

 

Dave

 

*Edited Labels*

2,620 REPLIES 2,620

Re: CODA-4582 - Open Issues for Investigation

@JohnBeaudin


@JohnBeaudin wrote:

For those of you who got D3.1 enabled, do you remember what time it got enabled (late at night perhaps?)?


Usually it's done at midnight.

 

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

Thanks @gp-se

Re: CODA-4582 - Open Issues for Investigation


@Mike2011 wrote:

@RogersDave

 

I checked this morning and I'm still on .24 firmware.

 

Is there something special that needs to be done to obtain the .26 ?


Fixed. I didn't have your latest MAC but I manually pulled it and pushed the firmware to your modem.

 

Dave

Re: CODA-4582 - Open Issues for Investigation

RogersDave
Retired Support
Retired Support

Hello Community,

 

Time for some quick updates on the status of the CODA. I have not updated the first post yet, too much going on but I’ll get to it eventually.

 

Based on the limited feedback received, it seems that firmware 2.0.10.26T1 resolved the issues around random reboots, at least for most users. I’d like to get additional feedback on this if possible.

 

I also had a discussion today with Hitron regarding modems with missing channels. In most cases for DOCSIS 3.1 enabled area, this will have very limited impact but it still requires a fix. It is not a physical problem with the modem but an issue where the modem decides too quickly at boot to exclude some channels. I should be getting another test build really soon with a fix for that problem.

 

This will leave us with the following 2 critical issues: UDP Packet Loss and WiFi performance and a few other minor issues.

 

Regarding WiFi performance, we made a configuration change on all CODA modems (takes effect after a reboot) to resolve Android phones (especially Pixel), frequently disconnecting from WiFi. This seems to have dramatically improved the performance and we are considering porting this fix to the CGN3 “AC” modems as well.

 

We are still validating if the difference in performance between lower channels and upper channels on 5 GHz is normal. Industry Canada allows much higher transmission power on channels 149-153-157-161 compared to 36-40-44-48 which is why I recommend manually setting the 5 GHz channel to the upper band. We are however validating that the lower band is operating as expected.

 

The UDP performance is still our top priority but we haven’t made much progress at this point. Multiple employees from Rogers and Hitron in Toronto, Denver and Taiwan tried to reproduce this problem without success. We are at the point where we will need assistance from the community to collect live logs in person from an impacted modem while the issue occurs. I will reach out in private to one or two of you who previously provided me logs for this issue. I need to run a series a tests firsts to select the best candidate for this and I will reach out in private to coordinate further testing.

 

Dave

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around
Thank you Dave, as always, for your amazing dedication and work on these issues! You have single handedly made this entire ordeal bearable.

Re: CODA-4582 - Open Issues for Investigation

Alex4161
I'm a senior contributor
@RogersDave

Thanks for the update and for all of your efforts. It is sad that Hitron would release Rogers new hardware with such garbage firmware. It makes you wonder the poor Quality Assurance processes that they have in their organization.

All of these issues not only inconveniences the customer base to no end, but can also create negative sentiment on the part of Rogers customers. We can be patient as Rogers addresses the issues but we have limits.

Getting reliable Internet should not be wishful thinking but a reality. Thanks for soldiering on and fixing all of the issues that Hitron has tossed your way.

Mahomed

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

Is there a specific program I can run in the background while I play games that would keep track so I can help you with the logs @RogersDave?

Re: CODA-4582 - Open Issues for Investigation

Wireshark is a good candidate.  But, the log files can become very large if the data capture runs for something like tens of minutes.  You probably don't have to let the data capture run for very long in order to record sufficient data. 

 

https://www.wireshark.org/

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

Thanks @Datalink I have 500 SSD + 4TB .. I will just install it on my 4TB I don't think I will run out of space I only have like 300gb used out of 4TB.

Re: CODA-4582 - Open Issues for Investigation

The file size becomes more of a problem if you are asked to send it somewhere.  Beyond a certain file size, and I don't know what that is, you would have to be able to upload the file as it would probaby blow up an email inbox.  So, if the file size isn't very large, you can simply attach it to an email.  

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

I got it installed, I can't wait to test this out, really wants to help out Dave, this UDP has to be fixed!

Re: CODA-4582 - Open Issues for Investigation

Take notes on what game you are running, whether the modem is in Gateway or Bridge mode with a follow on router and whether or not you have port forwarding enabled with the ports set as per the game requirements.  

 

After looking at a League of Legends data capture and from running UDP ping tests to pingplotter.com, I'm not convinced that the UDP problems are all attributable to the modem.  I think the data path from the start point to the end point might be playing a big part in the UDP problems, but, that's for Dave to confirm or deny. 

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

I know it's odd, for sure it's either due to Rogers modem or network.

 

I tested with friends with Bell, Videotron and Shaw(Alberta).

 

Tested all on same servers Chicago location.

 

All of my friends on Bell,Videotron and Shaw they're ping is much lower than mine but not only that, it stays stable it will fluctuate abount 1-5 ms in a whole 1 hour match, on the other hand mine can start at 43ms and usually end up at 75ms at the end of the match.. and I tested this many time , most of my tests done on CSGO, chicago location but I can reproduce it in any server.

 

I am not sure what it is @Datalink but all I know it's only happening on Rogers network.

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

So tonight I will use WiresShark and try to get some logs for Dave, hopefully it will help!

Re: CODA-4582 - Open Issues for Investigation

Its not just a matter of the ping time, its also a matter of whether or not the UDP packets are arriving at their end destination.  If you run a UDP test to pingplotter.com, you will see an ICMP destination unreachable response from pingplotter for those UDP packets that actually arrive at their server.  For the packets that time out along the way, you will see a Time to Live Exceeded response from the enroute servers.  Pingplotter processes both as a response and plots them.  The reality is that the packets that time out should probably be considered as failures as they don't arrive at the pingplotter server.  If you run a game which uses UDP, look carefully at the Wireshark Internet Protocol Version 4 Identification field number and see if the inbound packets are sequential and increment by "1" for every packet recorded.  I suspect that you might find that the packets are not sequential which to me suggests missing packets, but, I need Dave to confirm my thoughts on this one.  If there are missing packets that brings up a number of possibilities ranging from network timeouts enroute to issues with the modem.  Looking at Wireshark LOL and chat server records, the LOL is missing packets (I believe) while the chat server, which runs off of a separate server looks like it delivers all of its packets, based on the ID number.   So, thats rather interesting, but this requires confirmation at this point.  Why do a good number of LOL packets fail to arrive while the chat server packets all arrive?  So, as I indicated earlier, I'm beginning to think that there is more to this than just the modem.  

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

I agree with you @Datalink

 

and the games I run indeed use UDP, so I will pay close attention and I am almost certain that the log will confirm your theory.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@JohnBeaudin wrote:

I got it installed, I can't wait to test this out, really wants to help out Dave, this UDP has to be fixed!


You can dramatically reduce the size of the capture by setting proper filters.

 

The problem is that it's very difficult to notice UDP problems from the client side.

 

How does the system calculate the ping times? If it's essentially a pair of UDP packets that may also be difficult to track since they're both independent and based on the protocol of the application itself.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@Datalink wrote:

After looking at a League of Legends data capture and from running UDP ping tests to pingplotter.com, I'm not convinced that the UDP problems are all attributable to the modem.  I think the data path from the start point to the end point might be playing a big part in the UDP problems, but, that's for Dave to confirm or deny. 


All attributable? No. But as others have mentioned the AC modem didn't have nearly the same level of the problem as the CODA one does.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@Datalink wrote:

If you run a game which uses UDP, look carefully at the Wireshark Internet Protocol Version 4 Identification field number and see if the inbound packets are sequential and increment by "1" for every packet recorded.  I suspect that you might find that the packets are not sequential which to me suggests missing packets, but, I need Dave to confirm my thoughts on this one. 


It may be in practice, but by definition this isn't necessarily the case. You can have non-sequentual identifying numbers:

 

https://tools.ietf.org/html/rfc6864#section-4.1

 

>> The IPv4 ID field MUST NOT be used for purposes other than
      fragmentation and reassembly.
>> Originating sources MAY set the IPv4 ID field of atomic datagrams
      to any value.
>> All devices that examine IPv4 headers MUST ignore the IPv4 ID
      field of atomic datagrams.

 

The underlying problem is, and I suspect why they're having such a hard time identifying the cause -- UDP was designed specifically to be lossy, and have no verification of delivery. This makes it very hard to actually trace unless you control both ends.

 

Are there any games that can be tested where you can run your own server?  Grab a VPS and setup a server, and do the network tracing on both sides.  Use the server logs to determine the level of packet loss too.

Re: CODA-4582 - Open Issues for Investigation

Datalink
Resident Expert
Resident Expert

Yup, thats a little interesting allright.  The question is, with LOL, is the gapped ID intentional or the result of data path and/or modem issues.  Whats normal?  To know that, one would need a modem that works without any UDP issues to use as a comparison device, or as you suggest, run your own server which gives you access to both sides of the transmit/receive records.  

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around
Alternately, just try nping (part of nmap) in echo mode. Setup a server somewhere and flood UDP pings and see how much loss you get.

https://nmap.org/book/nping-man-echo-mode.html