cancel
Showing results for 
Search instead for 
Did you mean: 

CODA-4582 - Open Issues for Investigation

RogersDave
Retired Support
Retired Support

*** This post was last edited May 2, 2017 ***

 

Good morning Community,

 

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

 

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

 

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

 

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

 

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

 

 

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

 

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

 

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

 

Update April 22: This problem seems resolved in firmware 2.0.10.27

 

 

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

 

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

 

Workaround: manually select higher channels (149-153-157-161)

 

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

 

 

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

 

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

 

 

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

 

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

 

 

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

 

This problem has been reported to Hitron for investigation.

 

 

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

 

Investigation continues with Hitron.

 

 

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

 

 

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

 

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users


 

Future Planned Improvements

The following are items that we are working on in parallel of the above.

  • Improvement in WiFi speeds
  • Improvement in latency / bufferbloat

 

 

Dave

 

*Edited Labels*

2,620 REPLIES 2,620

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@MegnaGaming wrote:

 

Essentially every 10 minutes (exactly) there is a major latency peak followed by a double-peak 10 minutes later.

Have you tried on multiple devices? Are you plugged directly into the modem in gateway mode? Have you swapped out the modem before?

Re: CODA-4582 - Open Issues for Investigation

Jeffj
I plan to stick around

@MegnaGaming wrote:

Alright, now we're getting 19/20 - awesome.

 

@RogersDave @CommunityHelps

I have this crazy ongoing issue however that NO ONE seems to be able to solve it.

 

I was fighting with Rogers for months (since Sept) and between senior techs, T2 CSR and more, the only solution we found was to move from 100/10 to 250/20. I even ended up dealing with Natasha from The Office of the President.

 

After running some tests here, while I no longer notice the issue while hardwired, I still get it while connected via WiFi!

 

Essentially every 10 minutes (exactly) there is a major latency peak followed by a double-peak 10 minutes later. Ping Plotter image below.

 

https://i.gyazo.com/9932f1e48b0e2fc51a37cf5e0c8ae736.png

 

Is it possible this is something related to the firmware? It's SO very frustrating, especially when it causes buffering, latency issues, etc.


the fact you have had this issue since september when the CODA was not released would suggest its most likely and issue with the infrastructure, or your hardware not them modem or firmware.  😞

Re: CODA-4582 - Open Issues for Investigation

dantheman61
I plan to stick around

I have the CODA-4582 on version 2.0.10.19 around 8:00pm my seeds drops to 5mbps download a reboot fixes it and download is back up to 330mbps . This has happened 3 times now in the past 5 days.  Here are the modems logs.

DOCSIS Logs

The DOCSIS event logs is shown here

No.

Time

Type

Priority

Event

1

01/01/1970 00:01:23

82000200

critical

No Ranging Response received - T3 time-out;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

2

02/21/2017 18:22:05

90000000

warning

MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

3

02/21/2017 18:47:09

68010300

error

DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

4

02/22/2017 00:00:24

90000000

warning

MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

5

02/22/2017 00:28:26

68010300

error

DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

6

02/22/2017 05:07:10

69010100

notice

SW Download INIT - Via NMS

7

02/22/2017 05:13:56

69011100

notice

SW download Successful - Via NMS

8

02/22/2017 05:15:38

90000000

warning

MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

9

02/22/2017 18:05:04

84020200

warning

Lost MDD Timeout;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

10

02/22/2017 20:11:43

82000200

critical

No Ranging Response received - T3 time-out;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

11

02/23/2017 03:17:58

90000000

warning

MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

12

02/23/2017 03:18:02

84000700

warning

RCS Partial Service;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

13

02/23/2017 05:55:58

90000000

warning

MIMO Event MIMO: Stored MIMO=1 post cfg file MIMO=-1;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

14

02/23/2017 05:56:03

84000700

warning

RCS Partial Service;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=00:17:10:93:7e:1d;CM-QOS=1.1;CM-VER=3.1;

 

Re: CODA-4582 - Open Issues for Investigation

RyzenFX
I'm a reliable contributor

@dantheman61

 

It sounds like you're affecting by the speed degradation issue. What I will need you to do reboot the modem by unplugging the power cord, wait 30 seconds, and then replug the power again. This will push firmware 2.0.10.24 to your modem after you reboot. After it's done downloading, it will reboot again, and you should be on 2.0.10.24. This firmware will fix the speed degredation issue and your speeds should be constant. 

 

Let us know of the reuslts!

Re: CODA-4582 - Open Issues for Investigation

dantheman61
I plan to stick around

yes thank you I am now on  2.0.10.24

 

Re: CODA-4582 - Open Issues for Investigation

Triple_Helix
I plan to stick around

@RogersDave Looking at this thread and notice some were saying IPv6 was active so I did a factory reset and a reboot and now my speeds have gone down quite a bit. Smiley Sad  http://ipv6-test.com/

 

 

Image and video hosting by TinyPic

 

Image and video hosting by TinyPic

 

Downstream Overview
Port ID Frequency (MHz) Modulation Signal strength (dBmV) Channel ID Signal noise ratio (dB)
1 591000000 256QAM 3.900 31 40.366
2 363000000 256QAM 2.200 10 40.366
3 369000000 256QAM 2.800 11 40.366
4 375000000 256QAM 2.300 12 38.983
5 381000000 256QAM 1.700 13 40.366
6 387000000 256QAM 1.800 14 40.366
7 393000000 256QAM 2.000 15 38.983
8 399000000 256QAM 2.700 16 40.366
9 405000000 256QAM 2.800 17 40.366
10 411000000 256QAM 3.100 18 40.366
11 417000000 256QAM 3.400 19 40.366
12 423000000 256QAM 4.100 20 40.366
13 429000000 256QAM 3.700 21 40.366
14 435000000 256QAM 3.200 22 40.366
15 441000000 256QAM 3.200 23 40.366
16 447000000 256QAM 3.100 24 40.366
17 555000000 256QAM 4.400 25 40.946
18 561000000 256QAM 4.700 26 40.366
19 567000000 256QAM 5.000 27 40.366
20 573000000 256QAM 4.300 28 40.946
21 579000000 256QAM 4.500 29 40.946
22 585000000 256QAM 4.100 30 40.366
23 357000000 256QAM 2.300 9 40.366
24 597000000 256QAM 3.700 32 38.983
25 603000000 256QAM 4.200 33 38.983
26 609000000 256QAM 5.200 34 40.366
27 615000000 256QAM 5.400 35 40.366
28 621000000 256QAM 5.900 36 38.983
29 633000000 256QAM 5.500 37 38.983
30 639000000 256QAM 4.600 38 38.983
31 645000000 256QAM 4.100 39 38.605
32 651000000 256QAM 3.700 40 38.983
OFDM Downstream Overview
Receiver FFT type Subcarr 0 Frequency(MHz) PLC locked NCP locked MDC1 locked PLC power(dBmv)
0 NA NA NO NO NO NA
1 NA NA NO NO NO NA
Upstream Overview
Port ID Frequency (MHz) Modulation Signal strength (dBmV) Channel ID Bandwidth
1 30596000 ATDMA - 64QAM 39.500 1 6400000
2 38596000 ATDMA - 64QAM 42.750 3 3200000
3 23700000 ATDMA - 64QAM 39.500 2 6400000
OFDM/OFDMA Overview
Channel Index State lin Digital Att Digital Att BW (sc's*fft) Report Power Report Power1_6 FFT Size
0 DISABLED 0.5000 0.0000 0.0000 -inf -1.0000 4K
1 DISABLED 0.5000 0.0000 0.0000 -inf -1.0000 4K

Re: CODA-4582 - Open Issues for Investigation

Triple_Helix
I plan to stick around

@Telek wrote:

@Triple_Helix wrote:

@Triple_Helix wrote:

Okay just upgraded to the Gigabit sevice so here are my first results!

 

Wired results!

 

 

Image and video hosting by TinyPic

 


@RogersDave I called Rogers tech support and they tweeked things on there end and things look much better since going to the Gigabit! Any Idea when DOCSIS 3.1 will be released in North Oshawa? This is wired of course.

 

 


if that's DOCSIS3.0 then stay there for as long as you can, haha!  That's about the fastest that you'll see no matter what.

 

What did Rogers Tech support tweak to improve your speeds?  Any idea?

 

 


Don't know as they didn't say but it did get better over the few minutes during the call.

Re: CODA-4582 - Open Issues for Investigation

@Triple_Helix unless you have already truncated the IPV6 address in your post, go in and delete all after:

1 1 ms <1 ms 1 ms hitronhub.home [2607:   on the first trace line.

 

Otherwise, your IPV6 address will be hanging out in the wind so to speak.  

Re: CODA-4582 - Open Issues for Investigation

@prateeck7 you were inquiring about DNS servers.  I use OpenDNS for the purpose of web filtering for malware sites.  If you sign up for a freebie account you can specifiy what filtering what want to use.  There are a number of categories that I can't remember off the top of my head.  Load an OpenDNS address updater on your pc, so that that OpenDNS knows what IPV4 address is in use from your house and any DNS queries are checked.  Note that this is only for IPV4.  IPV6 filtering services are not available as of yet from OpenDNS, although its been on their list for a few years now.  OpenDNS does provide DNS service for IPV6 addressess and has done so for a considerable amount of time.   At the present time, I'm not in a hurry to switch over to IPV6, just a matter of enabling IPV6 in my Asus router when the time comes although when necessary, I run modem test sessions with it enabled.

 

Open DNS

 

208.67.222.222
208.67.220.220

 

2620:0:ccc::2      IPV6
2620:0:ccd::2      IPV6

 

google

 

8.8.8.8

8.8.4.4 

 

2001:4860:4860:0:0:0:0:8888    google IPv6:

 

rogers DNS:

 

64.71.255.204
64.71.255.198

 

IPV6:  don't have that available at the moment.

Re: CODA-4582 - Open Issues for Investigation

Jeffj
I plan to stick around

@Datalink wrote:

@prateeck7 you were inquiring about DNS servers.  I use OpenDNS for the purpose of web filtering for malware sites.  If you sign up for a freebie account you can specifiy what filtering what want to use.  There are a number of categories that I can't remember off the top of my head.  Load an OpenDNS address updater on your pc, so that that OpenDNS knows what IPV4 address is in use from your house and any DNS queries are checked.  Note that this is only for IPV4.  IPV6 filtering services are not available as of yet from OpenDNS, although its been on their list for a few years now.  OpenDNS does provide DNS service for IPV6 addressess and has done so for a considerable amount of time.   At the present time, I'm not in a hurry to switch over to IPV6, just a matter of enabling IPV6 in my Asus router when the time comes although when necessary, I run modem test sessions with it enabled.

 

Open DNS

 

208.67.222.222
208.67.220.220

 

2620:0:ccc::2      IPV6
2620:0:ccd::2      IPV6

 

google

 

8.8.8.8

8.8.4.4 

 

2001:4860:4860:0:0:0:0:8888    google IPv6:

 

rogers DNS:

 

64.71.255.204
64.71.255.198

 

IPV6:  don't have that available at the moment.


Rogers IPV6 DNS

 

Primary IPv6 DNS:
2607:f798:18:10:0:640:7125:5204
Secondary IPv6 DNS:
2607:f798:18:10:0:640:7125:5198

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

Soooo did the reboot tonight, after nearly 8 days of solid uptime 😄

 

Speed tested after reboot with 6ms 937.57 and 49.19.  Sweet!

 

So, ummm..

 

I have my own DHCP server. How should I go about setting it up for IPV6? (perhaps this should be a separate thread)

 

Pardon the ignorant question, but with IPV6 is there still natting, or essentially do all the devices get an external address now?

Re: CODA-4582 - Open Issues for Investigation

Jeffj
I plan to stick around

@Telek wrote:

Soooo did the reboot tonight, after nearly 8 days of solid uptime 😄

 

Speed tested after reboot with 6ms 937.57 and 49.19.  Sweet!

 

So, ummm..

 

I have my own DHCP server. How should I go about setting it up for IPV6? (perhaps this should be a separate thread)

 

Pardon the ignorant question, but with IPV6 is there still natting, or essentially do all the devices get an external address now?



You will want to set up your dhcp server to do IPV6 dhcp with prefix delligation. that will assign a public IPV6 address to any adapter that is asking for a IPV6 address. And yes, thus negating ipv4 natting. its a glorious thing for game consoles really. 

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

Hmmm. I enabled the DHCP server on the modem and turned off mine so that I could test out IPV6, which seemed to work perfectly. So far so good.  Shortly thereafter the modem management pages locked up though. I could ping the modem, I could use the internet, but both 192.168.1.1 and 192.168.100.1 (yes, I changed it to 1.1) refused to load from any computer. I had to reboot the modem to get it to come back.

Re: CODA-4582 - Open Issues for Investigation

RyzenFX
I'm a reliable contributor

Hello community!

Is anybody experiencing packet loss while playing League of Legends w/ the CODA-4582?

 

I'm currently trying to find an easy way to reproduce the issue for @RogersDave, because Hitron cannot reproduce this issue despite multiple tries through League of Legends, and it would help a lot if I can get some help from the community to reproduce this issue.

 

I'm able to reproduce this issue countless of times through League of Legends, however because Hitron cannot reproduce it, it makes it hard to fix the problem.

 

I have eliminated multiple variables that could cause packet loss such as: poor signal strength, poor signal levels, and the modem being over-loaded with WiFi devices. Despite having those variables being eliminated, the packet loss still exists. 

 

It would be great if people could post their experiences on League of Legends with this modem, as it would help me get a good picture of who's experiencing the issue and who's not. Also, league of legends network logs could be converted onto a chart on logsoflag.com, so if you can provide that aswell below, it will help.

Re: CODA-4582 - Open Issues for Investigation

Fwiw, UDP packet loss thru the modem is a known issue.  Its probably the same situation as the Puma 6 modem. 

Re: CODA-4582 - Open Issues for Investigation

@MegnaGaming, sorry this is taking so long.  You were asking earlier about the addition of the IPV6 ICMP rule, you should only have selected IPV6icmp out of the list of rules to add.  

Re: CODA-4582 - Open Issues for Investigation

RyzenFX
I'm a reliable contributor

@Datalink

 

I've read over at DSL Reports that the latest engineering code fixes the issue on Puma 6 and results look really impressive, but how would this be translated over to the Puma 7 code? 

 

I remember that there's a ticket open w/ Intel

 for this issue with the Puma 7 , but they haven't acknowledged the issue with Puma 7.

Re: CODA-4582 - Open Issues for Investigation

Jeffj
I plan to stick around

@Telek wrote:

Hmmm. I enabled the DHCP server on the modem and turned off mine so that I could test out IPV6, which seemed to work perfectly. So far so good.  Shortly thereafter the modem management pages locked up though. I could ping the modem, I could use the internet, but both 192.168.1.1 and 192.168.100.1 (yes, I changed it to 1.1) refused to load from any computer. I had to reboot the modem to get it to come back.


What are you using to serve your own dhcp? i would have stuck with it. less grief.

 

 

Re: CODA-4582 - Open Issues for Investigation

@RyzenFX well, its still early days for that firmware as it's only been out for two or three days now.  There is still a lot of testing that should be done, and from NetDog's comments, that version is the first kick at the cat for TCP and UDP.  

 

I think there are a few details that have to be worked out such as testing UDP, primarily, whats the correct packet size and what do you use to test it.  There are discussions on this on the github Iperf page.  From random reading, the actual DNS UDP packet is pretty small, much smaller than the packet sizes that I have been testing with, however, from the github site, you should be able to run tens of Gb/s, using UDP during a data transfer.  I don't know what the packet sizes are for gaming, but I'm interested in what they are.  On my list of things to do is load an older version of Iperf and compare that to the recent version which apparently generates excessive UDP test errors.  This is where the network experts need to have at it to make sure that the testing is reasonable and correct, and that the results are correct. Someone, or a team of people with Dave's knowledge and experience level comes to mind.  So, that's just one detail.  I'm sure that there are others lurking about.  

 

In terms of how this will translate to the Puma 7 is a very good question indeed.  The results with the Puma 7 in terms of latency are much better than the Puma 6/6MG, but, the question that needs to be asked, is, "Is that all there is?"  The high latency spikes are gone, but what is visible now are low level latency spikes, even in quiet hours, but, I don't have a similar view of a Broadcom modem on the Rogers network in order to make a comparison and know if the low level spikes are the result of the Puma 7 modem or of the CMTS loading.  The interesting question at this point is whether or not DOCSIS 3.1 will change the latency behaviour.  That might not happen until 3.1 upstream is enabled, which is several months away.  So, at the moment its a guess.  Porting the changes over to the Puma 7 in theory might make a difference, but, I'm not aware at the moment if the packet processing is done by the CPU or by the Hardware Accelerator in the Puma 7.  Dave would probably be able to tell us that.  If the processing is done by the CPU, then my guess is that the same changes would yield better results.  If the processing in already done by the Hardware Accelerator, then as I said, "Is that all there is?"

 

Here's an examle of what I'm talking about:

 

https://www.dslreports.com/forum/r31232323-

 

Those are TCP/IP to google.ca.  You could just call it ICMP to the CMTS for all that it matters although the base latency to the CMTS is 10 ms throughout the day with much higher latency spikes in the evening.  The issue is still the same, low level latency that continues throughout the day and that becomes much worse during evening hours. Can that be improved with either a firmware update such as what is occurring with the Puma 6/6MG or with DOCSIS 3.1?  Good question.  Wish I had that answer at the moment. 

 

 

Re: CODA-4582 - Open Issues for Investigation

jalalinator
I plan to stick around

Can confirm docsis 3.1 downstream flipped on here in Richmond Hill tonight @ 12:30am! SOOOOO Excited couldn't believe it when I saw it, the light is like a veryyyy light blue(almost purple now), first impressions, I am getting 50+ on the upload for the first time ever, the download speed + ping times is very consistent and even with not alot of variation at all between multiple tests, ping time is around 6ms to telus toronto, going to test out usenet and gaming and I will report back, still seeing around the same max speeds but the signal is MUCH cleaner I can tell already.

Re: CODA-4582 - Open Issues for Investigation

MegnaGaming
I plan to stick around

@Datalink @RyzenFX

 

Regarding the packet loss - during my livestream broadcast tonight I had over 100K dropped frames (the most I've ever experienced) and lots of packet loss (the obvious cause).

 

https://i.gyazo.com/be9c03109d46ab91fa8be71d65d7df06.png

 

I'm not sure what to do about this but .24 definitely seems to hate me streaming content a lot more than .19 did.