cancel
Showing results for 
Search instead for 
Did you mean: 

FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

RogersMargaret
Community Manager (Retired)
Community Manager (Retired)

Hello Community,

 

We are currently offering our users an exclusive opportunity to participate in an upcoming trial of the new firmware for our Rocket Wi-Fi Modem (CGN3ACR, CGN3AMR and CGN3ACSMR) and Rocket Gigabit Wi-Fi Modem (CGN3552 and CODA-4582). For details of this program, please see this thread.

 

This thread will be used for feedback regarding the firmware.  We've invited @RogersSergio@RogersSyd & @RogersBob from our Networking team to participate in this thread.  Your feedback is very valuable and will be used to enhance the firmware before it is released publicly.

 

Thank you for your continued feedback and support.

4,906 REPLIES 4,906

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

fd123
I Plan to Stick Around

Your best bet is to use your own router, and put the cable modem in dumb mode (Turn off the gateway).  That is the only way you will have control, firmware is pushed whenever it is released.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

sgobiraj
I Plan to Stick Around
The sad thing is it appears even the bridge mode with this new firmware has issues. I never signed up for a trial either but got pushed this buggy firmware 2 months ago. Everytime I speak to a tech they say they aren't aware of any issues. They say thousands of people are fine on bridge mode with this firmware as if they know for certain. I keep pointing them to threads about people complaining. They don't bother reading them and tell me they don't support third party routers and to switch to gateway mode. It's quite frustrating.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

jayrabbit
I've Been Here Awhile
I'm fine on Bridge Mode, but then again, I'm not using a VPN (I don't need one for remote access, and I'm not sold on it for reasons of privacy either). In Gateway Mode though, it's a friggin disaster. I work in IT for home and small biz, and am getting calls in my area constantly about this problem where customers are blaming the computer for disconnecting from the network. My advice to customers: buy a third-party router (doesn't fix VPN problems), ask Rogers to downgrade the modem to a CGN3AC, or switch to FttH if it's available in the area. Those are the only solutions, as half-arsed as they are.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

pburton
I've Been Here Awhile

Yes! Yes! and Yes! This is exactly the behaviour I see.

Rogers has sent 4 techs to my house and the line is absolutely clean and I am on my 3rd CGN3 modem.  I am so tired of being pushed off.  I was pushed to the point of having to diagnose this myself.  There is a single cable plugged into my CGN3 and WIFI is disabled.  My current firmware is 4.5.8.40T2 and it runs in gateway mode.  I installed ZABBIX to monitor the rogers connection.  The 1st screenshot shows the typical slow escalation of the RTT to the local IP of the modem.  Latency becomes so bad that the modem becomes completely unresponsive from both in the house and when Rogers tries to login.  The bottom 3 graphs in that same screen show 3 different devices in the house, each with rock solid RTT.

 

 

image.png

 

This next ZABBIX screenshow shows how often I need to reboot my modem.  Every spike results in a reboot.  There was a quiet period from Jul 31 to Aug 7 while I was away on vacation and there was no traffic in the house.  Without any significant traffic, the modem stayed up.  I guess Rogers only supports Internet stability as long as you don't use it.

 

image.png

I have now configured a traffic stress test.  Our typical usage is less than 50Mbps.  I am going to load the link up to >150Mbps (on a 300M services) and see if I can cause the problem to occur more often.

 

I shouldn't have to be the one debugging this.  I'm paying for a service that should work without a need to reboot the modem daily.

 

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

pburton
I've Been Here Awhile

EDIT: images above seem fine now.

---

sorry - 1st time posting. The images are not uploading

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

pburton
I've Been Here Awhile

I'll continue from my post above 

 

The problem recurred last night.  I'm including my information here in the hopes that it helps someone to diagnose the root cause of this issue.  The agents, field techs and senior field techs have not looked at this data.  They just check the lines and replace the modem hardware.  That won't resolve what I believe is a software issue in the modem.

  • I only drove traffic for about 5 hours.  I did not run the traffic over night

20200811-traffic throughput.png

  • You can see that the problem was starting during the 5 hours of receiving traffic by the slight, but gradual increase in RTT.  This RTT is measured from a server in my house to the modem's IP address in my house.
  • The downstream traffic is generated by a simple script that  performs a continual and repeated download of a multi-gigabit file which is sent directly to /dev/null.
  • This is the first time I purposely placed load on my downstream traffic beyond normal usage.  I am not convinced that this triggered the recurrence of the problem at this time, but there is insufficient data to conclude anything.  I stopped the traffic generation script because I read elsewhere in this forum that Rogers will throttle people who use excessive bandwidth. 
  • As you can see in the next graph, the RTT to all other devices in the house remains healthy and constant around 1ms

20200811-ping RTT.png

  • Along with the delay, packet drops gradually increase, until the router hits 100% packet loss.  The bottom graph on the next screen shot illustrates this.  This is a simple ping from a server in my house to the Rogers DNS.  

20200811 - packet loss.png

  • The device on the other end of the ethernet cable from the Rogers modem is a SNMP managed device.  During the time window above:
    • The link stayed operational
    • There were no changes in duplex
    • There were no inbound packet discards or inbound packet errors
    • The interface speed is constant at 1Gbps
    • The modem power is still on
  • If you are an agent reading this post, my answer is “No.  My modem is not plugged into a power bar.”
  • The agent I spoke with today will be sending another field tech to my house, but I honestly don't expect anything to get resolved from that visit.  If they choose to replace the modem, it will be my 4th modem in the last 1-2 months.  This really doesn’t look like a HW issue.
  • To me, this appears to be a software problem within the modem.  Some task (whatever it is), gets progressively slower over time.  It is as though the CPU is processing a list of intense and high priority CPU tasks.  As the list grows, the CPU becomes more and more consumed with processing the list. While the CPU is processing this growing list, it is unable to forward packets.  This would result in long RTT and occasional dropped packets.  Eventually, this “high priority” CPU task consumes the modem completely and all packet forwarding stops.  This includes dropping all packets destined to the GUI of the router (which I run in gateway mode)
  • I don’t have access to any debug logs or stats within the modem to diagnose what the modem itself is doing during this time.

I would encourage Rogers to please have someone beyond agents and field techs take a look at this problem profile.  I would be happy to continue the problem isolation and resolution with them.  I need a stable connection before September when school starts and I will be following this issue carefully.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

sgobiraj
I Plan to Stick Around

This looks identical to the issue I'm experiencing. The tech support people are useless. They can't seem to understand this is a firmware related issue and not the hardware. Like you the tech support individuals are . bent on replacing my modem when I keep stating that this only occurred after the firmware update. I've given up for the time being waiting for others to make noise and see if Rogers will take us seriously.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

RogersIan
Product Manager
Product Manager

We have new code coming this week for our coda users.

Firmware ver 7.1.1.33
The main fix is a Intel patch which improves network stability.

RogersIan

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

Alborze
I Plan to Stick Around
Thanks for the update @RogersIan.

Is there any other update/fix in the new firmware? I'm still facing issues of not being able to switch my modem to bridge mode because it's just riddled with problems and issues. Is there any possible way of switching back to 36T6, that was the most stable firmware and I'm willing to pay extra to switch back. These new firmwares are absolutely garbage, to the point where I'm considering switching to Bell 100mb from Rogers Gigabit since fibe is not available in my area.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

jjdunn10
I Plan to Stick Around
I've had an issue come up while on the new firmware where after I unplugged the modem it wouldn't turn back on. After many attempts the power light was the only thing coming on. The two houses would not even initiate while trying the reset button wouldn't make any difference. Not sure if it's related to the modem or the firmware but something that randomly came up.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

@jjdunn10 the code was pushed this morning so your issue is not related to the .33 firmware.

It does sound like a hardware issue, however, so if you're still stuck at the single (power) LED you'll need to exchange the modem.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

MisterPinst
I Plan to Stick Around

Rogers - do you post patch notes anywhere for the firmware updates? Would be nice to see that kind of transparency and I'm sure it will prevent people from asking if feature 'x' or bug 'y' has been implemented/addressed.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

pburton
I've Been Here Awhile

Here is an update on the issue I reported previously.  I have been working with a Senior service tech for the last 10 days, during which time I have had a CODA test modem in my house rather than the CGN3AC.  We tested the CODA modem on both 2.0.1036T8 (Aug 15-19th) and 7.1.1.32 (Aug 19-24).  Both software versions were reliable over that time period.

RTT-coda.pngping-loss-coda.png

While a root cause was not identified, it is definitively isolated to the CGN3AC or the software that it was running.

 

Thanks for your assistance.  Now I just need Rogers to provide me with a CODA modem that I can keep since I need to return to test modem tomorrow night.

 

 

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

@ pburton here’s some background information that you might find interesting.  Just for the record, the Hitron CGN3xxx and CGNM-3552 modems are all Intel Puma 6 modems.  The Hitron CODA-4582 is an Intel Puma 7 modem.

 

Intel bought the Puma modem line from Texas Instruments in 2010:

 

https://www.cnet.com/news/intel-to-buy-texas-instruments-cable-modem-unit/

 

Intel then put its own spin on the modem, building it with an Atom and Arm processor and Maxlinear cable tuner.  That modem was introduced into service around 2012.  I think Rogers introduced the first CGN3 modems in 2014.  It wasn’t pretty, and that is an understatement.  In short, users experienced latency to and thru the modem for all protocols, and for UDP, which is used for DNS address queries and gaming, there was packet loss as well. 

 

So, it appears that Intel attempted to virtualize the modem, running everything in a software process thru the CPU.  There was a hardware processor / accelerator on the chipset, but, at that time it wasn’t used.  So, the major problem is that there was an unknown task, running approx. every 1.92 seconds.  That task would stop all traffic thru the modem, resulting in latency for all traffic (IPV4 and IPV6).  In the case of UDP traffic, my guess is that it timed out in the modem and those timed out UDP packets were discarded.  That latency and UDP loss resulted in numerous complaints and as result, Rogers engineers started their own investigation into latency and packet loss in mid 2016.  Here’s one of the original Pingplotter plots that I submitted at the prior to the start of that investigation:

 

https://communityforums.rogers.com/t5/media/gallerypage/user-id/829158/image-id/3712iA4BE7806BFF77A9...

 

That’s a new Hitron Puma 6 CGNM-3552 modem running on a new Casa Systems CMTS, with no traffic other than a ping test running to the CMTS, which is the next IP address beyond the modem.  Its unfortunate that at the time I used Pingplotters default 2.5 second ping interval.  If I had lowered that interval down to 20 milli-seconds, that plot would be filled with high latency returns.  Its has a large number of high latency returns, but, there would be a much larger number of those high latency returns.  

 

Ok, with Rogers input in mid 2016, Intel started working on updates to resolve the latencies observed in the Puma 6 modems.  The first release which was designed to resolve IPV4 ICMP (ping) latency was loaded onto Rogers CGN3 modems in Sept 2016.  That did work as expected, but, it left IPV4 (TCP/IP and UDP) and IPV6 (ICMP, TCP/IP, and UDP) for future updates. 

 

Since the introduction of the same Puma 6 modems in the U.S. and Europe, there were similar grumblings of poor latency and packet loss.  That really came into view when DSLReports member xymox1 started a thread in DSLReports outlining his observations.  That first thread is located here:

 

https://www.dslreports.com/forum/r31079834-Modem-SB6190-is-a-terrible-modem-Intel-Puma-6-MaxLinear-m...

 

The next follow-on thread is located here:

 

https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-UDP-Network-Latency-Issue-Discussion

 

 

That rolling stone gathered momentum over time and it was no longer possible to deny that there were problems with the modem’s hardware and firmware design.  During the course of the posts throughout that second thread, it was discovered that the Puma modems were susceptible to low frequency DOS attacks.  Never mentioned through any of this, I wonder if there was a discovery that the DOS attack would or could lead to a takeover of the modem.  Shown enough evidence by posters in that thread, Intel probably had no choice but to recognize the problems, issue a CVE and issue corrective firmware updates to resolve the DOS attack problems for Puma 5, 6, and 7 modems. 

 

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5693

 

When Rogers issued the first update in Sept 2016, that appeared to be the final involvement with Hitron and Intel regarding the latency issue.  It would appear that Rogers looked into a crystal ball to determine which way to go, wait for the Puma 6 updates to appear eventually or, push ahead with the Puma 7 (Hitron CODA-4582) modem.  Rogers appears to have chosen to pursue the Puma 7 deployment and left Arris and Intel to wrestle with the Puma 6 latencies.  The first CODA-4582s were released by Rogers in Dec 2016, most likely long before Intel produced a final resolution to all of the latency and packet loss problems, if ever. 

 

If you look at the following post which shows the history of the modem updates, you will see a series of updates for all modems.  The question is, did Intel every solve the problems?  No one really knows as no company, from Intel to the modem manufacturers to the ISPs have every produced any test latency plots to show that those problems have been resolved.  And, there is no test plot proof that the DOS problems have been fully resolved.

 

https://communityforums.rogers.com/t5/Internet/FEEDBACK-Rogers-Rocket-Wi-Fi-Modem-Firmware-Trial/m-p...

 

Ok, so, that leads to your current situation.  In theory, you shouldn’t be seeing any latency to or thru a Puma 6 modem.  Keep in mind that there has never been any proof provided that shows that the latency issues have been resolved.  I wonder if Intel has done something wrong in the process and you happen to be the first Rogers customer to either determine that the problems still exist, or that Intel has introduced some new problem.  In either case, your next path is to bring this to the attention of @RogersIan for forwarding to Hitron and possibly to Intel, or now Maxlinear.   Working with the techs has led to the determination that there is an issue with the CGN3AC modem, the next step is to address it with the engineering staff, assuming that you return to the CGN3AC modem for everyday use.  If not, this falls to someone else down the road.  

 

One point of interest that arose thru the whole Puma 6 debacle is that there’s no organization that tests modems for performance.  Cablelabs tests the modems for interoperability with CMTS and other cable modems, but they’ve washed their hands of any performance testing.  That performance testing would have shown the problems that existed when the Puma 6 modems were first introduced in 2012.  So, its up to the end users to test modems for latency and packet loss, for any protocol that you might be interested in, within the usual IPV4 and IPV6 (ICMP, TCP/IP and UDP) group and for any other protocol that you might be using beyond that group.  At the point where users are discovering problems what should have been discovered in development testing, its far too late.  The development company, whatever that company happens to be has failed in its development and testing process.  As a result, the end user pays the price.  I wonder if your download testing has stumbled across a particular protocol that results in slowly decreasing modem performance? 

 

Ok, so, where does this leave you and a return to a CGN3 modem.  If you do return to that modem, you may want to start a test program to test all protocols,  To do that you need to run low interval test pings or UDP queries, in the range of 20 milli-seconds, which will show if that unknown task is still an issue, or, if the shift of the protocol processing to the hardware accelerator / processor has in fact resolved the latency and packet loss issues.  That’s what I started to do in 2016, but, after I changed to a Puma 7 CODA-4582 modem in Dec 2016, I never had any reason to return to a Puma 6 modem to retest the firmware updates, so, I don’t know if those updates have really been successful.

 

Just to point out, running a CGN3xxx modem, I was able to crash the modem after about an hour, running a VPN and simultaneous ping test to the CMTS.  Crashing the modem wasn’t a problem.

 

Here’s an example of the modems task processing at work, causing latency thru the modem.  In this case xymox1 is running two Pingplotter plots.  At the time, the modem he was using had the first update to resolve the ICMP latency by shifting the ICMP processing over to the hardware accelerator / processor.  The top plot is the uncorrected TCP/IP processing, the bottom plot is the corrected ICMP processing.  The top plot is what the bottom plot used to look like.  And, its an example of how every protocol latency looked like, as in IPV4 and IPV6 (ICMP, TCP/IP and UDP).  By shifting the ICMP or TCP/IP ping or UDP query interval down to the 20 milli-second range, the 1.92 second interval task shows up in the plot, causing latency spikes, as you can see from the top plot. 

 

https://www.youtube.com/watch?time_continue=9&v=15cJ400yR_E&feature=emb_logo

 

Ok, so that’s a short history of the Puma 6 problems.  Hopefully this provides some useful information for you.  Personal opinion, Puma modems should be consigned to the s . heap (recycle bin).  Now its Maxlinear’s problem as Intel has sold the Home Gateway Platform Division to Maxlinear.  Maxlinear manufactures the cable tuner for the modems, so, now they own the whole can of worms. 

 

 



Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

sgobiraj
I Plan to Stick Around
What I find funny is that given there are modems avaiable with better chips (e.g. Broadcom) compared to Puma they (Rogers) still decide to go with a Puma successor. The Technicolor TC4400 (Broadcom chip) approved by Rogers for use by third party ISP is solid. The SmartRG 808 as well is another example. When will Rogers get their heads out of their ...???

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

I would guess that Rogers has a somewhat long standing relationship with Hitron, and just to point out, I believe that Hitron has been grappling with the Puma 7 issues long before Arris, which produces the Puma 7 XB6 (TG3482ER) used for the Ignite TV (IPTV) system.  

 

Broadcom didn't play in the highspeed modem market until they released the BCM-3390 chipset modem, years after Intel's Puma modems were in the market.  Intel was the only game in town.  Its really unfortunate that the problems with the Puma 6 modem weren't discovered until the modem was in service and ramping up to higher speeds with greater numbers of download channels.  So, Intel had the market cornered for a number of years.



Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

Triple_Helix
I Plan to Stick Around

Confirming that I have the new version installed.

 

 

System Information

This menu displays general information of the device

 

Hardware Version2A
Software Version7.1.1.33

 

 

 

 

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

blues_clues
I Plan to Stick Around

@RogersIan 

Still seeing the 2.4Ghz band wireless frame broadcast bug in 7.1.1.33.  This switches off/on my wireless plugs.  I had to setup a wireless hotspot on one of my RasberryPis as a workaround.  When will this issue be addressed?

 

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

sgobiraj
I Plan to Stick Around
@Rogerslan is there a way I can get this 7.1.1.33 firmware to see if it addresses my modem stability issues in Bridge mode? I'm currently on 7.1.1.32.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

roy86
I Plan to Stick Around

how can i opt out of the beta firmware program?

 

it seems the latest firmware is quite unstable.

Re: FEEDBACK - Rogers Rocket Wi-Fi Modem Firmware Trial

sgobiraj
I Plan to Stick Around

If you're in bridge mode then this new firmware is a piece of junk. Highly unstable. I don't know why Rogers decided to push this firmware during a time when everyone is dependent on stable internet and working from home. Mind boggling.