cancel
Showing results for 
Search instead for 
Did you mean: 

CODA-4582 - Open Issues for Investigation

RogersDave
Retired Support
Retired Support

*** This post was last edited May 2, 2017 ***

 

Good morning Community,

 

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

 

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

 

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

 

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

 

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

 

 

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

 

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

 

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

 

Update April 22: This problem seems resolved in firmware 2.0.10.27

 

 

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

 

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

 

Workaround: manually select higher channels (149-153-157-161)

 

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

 

 

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

 

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

 

 

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

 

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

 

 

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

 

This problem has been reported to Hitron for investigation.

 

 

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

 

Investigation continues with Hitron.

 

 

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

 

 

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

 

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users


 

Future Planned Improvements

The following are items that we are working on in parallel of the above.

  • Improvement in WiFi speeds
  • Improvement in latency / bufferbloat

 

 

Dave

 

*Edited Labels*

2,620 REPLIES 2,620

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@Datalink wow, well done!

 

I'm very confused at your results though. You're highlighting 590mbit down but that seems to have 35-74% packet loss? Why settle at this number then?

 

Also why would 530mbit of UDP max out your high end i7 processor?  What sort of processing is it doing on this??

 

Also why would you need a 500Mb buffer size? That also seems crazy considering what's going on.

 

My 10 year old Core 2 Duo can handle gigabit TCP traffic without even causing any CPU load at all and buffers in the KB range...

Re: CODA-4582 - Open Issues for Investigation

Datalink
Resident Expert
Resident Expert

@Telek the iperf numbers are definitely strange.  I can repeatedly run up to 500 Mb/s UDP down without errors.  From 500 Mb/s up, the bandwidth and datagram numbers start to diverge between the received data and the server data, but, the loss result will indicate no losses.  And then you get to a tipping point and all of a sudden you see losses in the results.  I don't know why that happens, so, I have some reading to do and further testing to do.  My definition of success at this point with an iperf test is to end up with no errors and see a close association between the local data and the server data in terms of bandwidth and datagrams.  It doesn't make any sense to me at this point to see those last two data points diverge and yet see no loss declared by iperf.  So the numbers don't have to be exact, but, they should be pretty close unless someone with expertise in iperf tells me otherwise, as in "ah, its because of ........." as I nod my head in agreement.

 

What I was trying to highlight in that first section was the strange iperf behaviour where on one run I might see errors as you indicated, and yet, on the next run, it would turn out perfectly.  I didn't do anything different, same command, and yet the results are drastically different.  

 

If you look at the next section below that, running at 590 Mb/s, you can see that there are no errors. Thats the max Mb/s point that I saw today where I would see no lost datagrams.  If I went to 600 Mb/s down, I would end up with lost datagrams and you would see that in the results and in the % loss.  

 

In that section there are no errors, but, if you compare the bandwidth figures, bottom server data indicates 590 Mb/s, top receive data varies between 542 to 582 Mb/s.  The datagram numbers also have diverged at this point, but yet, iperf is still happy to say that there are no errors.  So, this is a little strange. 

 

"why would 530mbit of UDP max out your high end i7 processor?  What sort of processing is it doing on this??"  I wish I had that answer.  My immediate thought is that UDP doesn't follow the send and acknowledge routine.  Its a firehose with the iperf test, so, you either catch it or you don't.  Why is that any tougher than a much faster TCP/IP speedtest that I also run?  I don't have that answer at the moment.  One thing that I have to look at is the processor loading to see if the load stays on one processor or if it hops around.  But, as I indicated, I'm not sure that I can do anything about it with the windows version of iperf.

 

That buffer size is actually the socket buffer size.  What I found today is that if I didn't include that variable and increase the buffer size, I wouldn't have been able to run the UDP test at higher rates.  That leads to the question of whether or not an end user would have to increase that buffer size for games or any applications that use UDP, or, are they self adjusting?  

 

A Core 2 Duo, have one of those as well, what I'm typing on actually, and it runs rather well.  A little slower on a speedtest since Windows 10 but it still runs rather well.  My golden oldie Core 2 Quad Q6700 with its DDR2 runs 50 Mb/s up, 200 Mb/s down, UDP, without any errors.  Thats with the same settings as before, just throttled back on the download to get to an error free point.  That is also IPV4 only thru my Asus RT-AC68U.

 

If nothing else, today showed that there are more questions than answers.  The modem looks like it can run fast UDP up and down, the question is, is there an approximate 590 Mb/s cap on the modem, or is that just a pc problem?  That might take an overclocked CPU and 10 Gb/s nic card to determine that.

 

Maybe @RogersDave can explain some of the pitfalls of UDP.  

 

The one thing that was demonstrated was the UDP IPV4 performance versus the UDP IPV6 performance.  The IPV6 results are vastly superior to the IPV4 results, and the only explanation I can think of are either a NAT problem or some other issue that hasn't been identified as of yet.  Perhaps looking at this would benefit the Puma 6 modems as well. 

Re: CODA-4582 - Open Issues for Investigation

JohnyR
I plan to stick around

Hi @RogersDave 

 

Noticed that my net went down around 230-530am last night, and once it came back, all my signals went from being near 0 to closer to -7dBmV along with a changed IP and node. Do you know of any changes that happened in the Thornhill area?

Is -2.00 -> 7.8dBmV acceptable on the CODA? Upstreams are sitting at 29-32 (before was 32-37). Full signals here: https://pastebin.com/Lp6yimKJ

 

Thanks

Re: CODA-4582 - Open Issues for Investigation

DeeNice
I plan to stick around

>The one thing that was demonstrated was the UDP IPV4 performance versus the UDP IPV6 performance.  The IPV6 results are vastly superior to the IPV4 results, and the only explanation I can think of are either a NAT problem or some other issue that hasn't been identified as of yet.  Perhaps looking at this would benefit the Puma 6 modems as well. 

@Datalink Didn't you test in Bridge Mode using a 3rd party router capable of passing gbit from WAN to LAN? If the results were the same, that would suggest that it's not a NAT issue and perhaps something else. Not sure.. as I've said previously I'm a Systems guy who does networking as/where needed so i'm just guessing at this point. Weird as . . 

Re: CODA-4582 - Open Issues for Investigation

DeeNice
I plan to stick around
>Is -2.00 -> 7.8dBmV acceptable on the CODA? Upstreams are sitting at 29-32 (before was 32-37).

For reference, what are the acceptable values for that anyways?

Re: CODA-4582 - Open Issues for Investigation

LF1949
I plan to stick around

With firmware version .27 are there still restrictions to the characters that can be used for the WiFi SSID and login password (like no spaces etc.)?

 

Thanks.

Re: CODA-4582 - Open Issues for Investigation

JohnyR
I plan to stick around

I was told by an agent that -10 to +10 is acceptable on the downstream. Upstream should be 35-58 according to him as well. Not sure how true these values are, especially since the CODA handles it a bit differently than the Rocket modems.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@JohnyR wrote:

I was told by an agent that -10 to +10 is acceptable on the downstream. Upstream should be 35-58 according to him as well. Not sure how true these values are, especially since the CODA handles it a bit differently than the Rocket modems.


-12 to +12 is the max range that they will accept, although -10 to +10 is better -- 0 is "perfect".  36-40 is better for upstream, at least based on what we see most often on DOCSIS 3.0.

Re: CODA-4582 - Open Issues for Investigation


@JohnyR wrote:

I was told by an agent that -10 to +10 is acceptable on the downstream. Upstream should be 35-58 according to him as well. Not sure how true these values are, especially since the CODA handles it a bit differently than the Rocket modems.


@JohnyR, I can't be sure of easily of what happened during that 3 hour or downtime you experienced but I suspect it was a segmentation activation (when we break an area in half to increase capacity).

 

I checked your signal levels and everything is fine on the downlink but your uplink is out of spec. This also affects your neighbors so I'll have maintenance dispatched to make adjustments to the line amplifiers on the street.

 

For reference, 35 dBmV to 53 dBmV is the acceptable signal level for upstream for a DOCSIS 3.0 modem. The levels for a DOCSIS 3.1 modem are much lower because of a correction factor that is dependent on the channel bandwidth:

 

3.2 MHz channel: 32 dBmV to 50 dBmV

6.4 MHz channel: 29 dBmV to 47 dBmV

 

In your specific case, the channel at 23.7 MHz (6.4 MHz wide) is too low at 27.75 dBmV. The channel at 30.596 MHz (6.4 MHz wide) is right at the limit at 29.00 dBmV.

 

Dave

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

What would we do without you @RogersDave Heart

Re: CODA-4582 - Open Issues for Investigation

lupinglade
I plan to stick around
Really wish Rogers would offer a standalone modem, ideally nothing to do with Hitron.

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

Just staying away from the PUMA chipset would be enough for me -_-

Re: CODA-4582 - Open Issues for Investigation

DeeNice
I plan to stick around

I'd be happy to get anything that doesn't drop packets like a sieve. 

Re: CODA-4582 - Open Issues for Investigation

JohnyR
I plan to stick around

@RogersDave wrote:

@JohnyR, I can't be sure of easily of what happened during that 3 hour or downtime you experienced but I suspect it was a segmentation activation (when we break an area in half to increase capacity). 


Thanks for the detailed explanation Dave, always appreciated. Definitely looks like a segmentation since I've had numerous tickets submitted for nightly congestion (found that the upstream on the node was at 88.5% utilization). 

 

Do you have a ticket # I can reference for the maintenance you dispatched (thanks for doing that for me, btw).

Re: CODA-4582 - Open Issues for Investigation

Kyrluckechuck-2
I plan to stick around
Just throwing a quick message out there --
**NAT Loopback not working for wired clients** has been fixed for a couple of months, actually a week or two prior to '2.0.10.26T2' which supposedly fixed it. Completely forgot to mention that. 🙂

Re: CODA-4582 - Open Issues for Investigation

Vikentieff
I plan to stick around

Inability to do factory reset without getting downgraded to a public-released firmware is terribly inconvenient. I had .27 and after several beta updates the modem had gotten extremely laggy and ping was all over the place. After the reset it went back to 26T but most of the issues have stabilized right away, which proves my point. Please look into an option of allowing beta testers to do a factory reset without backrolling the firmware.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@Vikentieff wrote:

Inability to do factory reset without getting downgraded to a public-released firmware is terribly inconvenient. I had .27 and after several beta updates the modem had gotten extremely laggy and ping was all over the place. After the reset it went back to 26T but most of the issues have stabilized right away, which proves my point. Please look into an option of allowing beta testers to do a factory reset without rollbacking the firmware.


What is it that you were doing that needed a factory reset as opposed to just a reboot?  I think this is there as an emergency bail-out situation. The last thing we want is to be on a beta firmware, find out that it's not so great, then have to suffer with it.

 

Keep in mind that you can backup your settings and restore them at any point, which may give you the same benefit that you find with a factory reset.

Re: CODA-4582 - Open Issues for Investigation

Runnng a Factory Reset with trial firmware loaded in the past didn't result in a rollback.  There is probably a technical reason this time around as to why .27 wasn't protected when a Factory Reset was run.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@Datalink wrote:

Runnng a Factory Reset with trial firmware loaded in the past didn't result in a rollback.  There is probably a technical reason this time around as to why .27 wasn't protected when a Factory Reset was run.


Are you sure?

 


@RogersDave wrote:

If you are a trial participants, I will push 2.0.10.27 when it is available. The only difference for now is that if you perform a factory reset, you won't go back to .24, you will stay on 26T2 as this is the new baseline firmware. Dave


 Factory resets have always rolled back to the latest public baseline from what I can remember.  This was the case with .24 and .26 while in beta as well.

 

Re: CODA-4582 - Open Issues for Investigation

Datalink
Resident Expert
Resident Expert

Ah, yes, I am positively, absolutely, 100% sure, after running numerous trial versions preceding .24 and .26.  It wasn't standard procedure to roll back to a previous version if a Factory Reset was run.  If a roll-back to a previous version happens, its due to some technical reason specific to the version in question.  If Dave decides to make that standard procedure, ok, it is what it is.  Until the recent change, my standard procedure was to run a Factory reset after a trial version was loaded, coming out of the reset and reboot with the trial version intact on the modem, so, yes, I'm sure. 

Re: CODA-4582 - Open Issues for Investigation


@Datalink wrote:

Ah, yes, I am positively, absolutely, 100% sure, after running numerous trial versions preceding .24 and .26.  It wasn't standard procedure to roll back to a previous version if a Factory Reset was run.  If a roll-back to a previous version happens, its due to some technical reason specific to the version in question.  If Dave decides to make that standard procedure, ok, it is what it is.  Until the recent change, my standard procedure was to run a Factory reset after a trial version was loaded, coming out of the reset and reboot with the trial version intact on the modem, so, yes, I'm sure. 


@RogersDave explained this when he came up with the workaround for IPv6 bugs in early CODA-4582 firmwares.

 

Old way: modems booted up with whatever firmware and connected to the network. A day or two later, a script ran that checked all modems' firmware, and updated modems to the newest release firmware unless the modem was on an exception list. If you join the trial program, Dave puts you on the exception list and manually updates your firmware. If you factory reset, you keep your trial firmware.

 

New way: modems boot up, and if they have any version other the right one, they get immediately updated to the 'right' one. To avoid this happening with trial participants, Dave sets a setting in the trial participants' modems so that they won't be downgraded back down. When you do a factory reset... whoooooooooops... that setting is removed along with the others, so the modem reboots and promptly updates ('downdates'?) itself to the version that the CMTS mandates.

 

This whole thing was done so they could reenable IPv6 on the 4582 without the IPv6 bugs affecting new 4582s for a day or two...