This looks identical to the issue I'm experiencing. The tech support people are useless. They can't seem to understand this is a firmware related issue and not the hardware. Like you the tech support individuals are . bent on replacing my modem when I keep stating that this only occurred after the firmware update. I've given up for the time being waiting for others to make noise and see if Rogers will take us seriously.
@jjdunn10 the code was pushed this morning so your issue is not related to the .33 firmware.
It does sound like a hardware issue, however, so if you're still stuck at the single (power) LED you'll need to exchange the modem.
Rogers - do you post patch notes anywhere for the firmware updates? Would be nice to see that kind of transparency and I'm sure it will prevent people from asking if feature 'x' or bug 'y' has been implemented/addressed.
Here is an update on the issue I reported previously. I have been working with a Senior service tech for the last 10 days, during which time I have had a CODA test modem in my house rather than the CGN3AC. We tested the CODA modem on both 2.0.1036T8 (Aug 15-19th) and 22.214.171.124 (Aug 19-24). Both software versions were reliable over that time period.
While a root cause was not identified, it is definitively isolated to the CGN3AC or the software that it was running.
Thanks for your assistance. Now I just need Rogers to provide me with a CODA modem that I can keep since I need to return to test modem tomorrow night.
@ pburton here’s some background information that you might find interesting. Just for the record, the Hitron CGN3xxx and CGNM-3552 modems are all Intel Puma 6 modems. The Hitron CODA-4582 is an Intel Puma 7 modem.
Intel bought the Puma modem line from Texas Instruments in 2010:
Intel then put its own spin on the modem, building it with an Atom and Arm processor and Maxlinear cable tuner. That modem was introduced into service around 2012. I think Rogers introduced the first CGN3 modems in 2014. It wasn’t pretty, and that is an understatement. In short, users experienced latency to and thru the modem for all protocols, and for UDP, which is used for DNS address queries and gaming, there was packet loss as well.
So, it appears that Intel attempted to virtualize the modem, running everything in a software process thru the CPU. There was a hardware processor / accelerator on the chipset, but, at that time it wasn’t used. So, the major problem is that there was an unknown task, running approx. every 1.92 seconds. That task would stop all traffic thru the modem, resulting in latency for all traffic (IPV4 and IPV6). In the case of UDP traffic, my guess is that it timed out in the modem and those timed out UDP packets were discarded. That latency and UDP loss resulted in numerous complaints and as result, Rogers engineers started their own investigation into latency and packet loss in mid 2016. Here’s one of the original Pingplotter plots that I submitted at the prior to the start of that investigation:
That’s a new Hitron Puma 6 CGNM-3552 modem running on a new Casa Systems CMTS, with no traffic other than a ping test running to the CMTS, which is the next IP address beyond the modem. Its unfortunate that at the time I used Pingplotters default 2.5 second ping interval. If I had lowered that interval down to 20 milli-seconds, that plot would be filled with high latency returns. Its has a large number of high latency returns, but, there would be a much larger number of those high latency returns.
Ok, with Rogers input in mid 2016, Intel started working on updates to resolve the latencies observed in the Puma 6 modems. The first release which was designed to resolve IPV4 ICMP (ping) latency was loaded onto Rogers CGN3 modems in Sept 2016. That did work as expected, but, it left IPV4 (TCP/IP and UDP) and IPV6 (ICMP, TCP/IP, and UDP) for future updates.
Since the introduction of the same Puma 6 modems in the U.S. and Europe, there were similar grumblings of poor latency and packet loss. That really came into view when DSLReports member xymox1 started a thread in DSLReports outlining his observations. That first thread is located here:
The next follow-on thread is located here:
That rolling stone gathered momentum over time and it was no longer possible to deny that there were problems with the modem’s hardware and firmware design. During the course of the posts throughout that second thread, it was discovered that the Puma modems were susceptible to low frequency DOS attacks. Never mentioned through any of this, I wonder if there was a discovery that the DOS attack would or could lead to a takeover of the modem. Shown enough evidence by posters in that thread, Intel probably had no choice but to recognize the problems, issue a CVE and issue corrective firmware updates to resolve the DOS attack problems for Puma 5, 6, and 7 modems.
When Rogers issued the first update in Sept 2016, that appeared to be the final involvement with Hitron and Intel regarding the latency issue. It would appear that Rogers looked into a crystal ball to determine which way to go, wait for the Puma 6 updates to appear eventually or, push ahead with the Puma 7 (Hitron CODA-4582) modem. Rogers appears to have chosen to pursue the Puma 7 deployment and left Arris and Intel to wrestle with the Puma 6 latencies. The first CODA-4582s were released by Rogers in Dec 2016, most likely long before Intel produced a final resolution to all of the latency and packet loss problems, if ever.
If you look at the following post which shows the history of the modem updates, you will see a series of updates for all modems. The question is, did Intel every solve the problems? No one really knows as no company, from Intel to the modem manufacturers to the ISPs have every produced any test latency plots to show that those problems have been resolved. And, there is no test plot proof that the DOS problems have been fully resolved.
Ok, so, that leads to your current situation. In theory, you shouldn’t be seeing any latency to or thru a Puma 6 modem. Keep in mind that there has never been any proof provided that shows that the latency issues have been resolved. I wonder if Intel has done something wrong in the process and you happen to be the first Rogers customer to either determine that the problems still exist, or that Intel has introduced some new problem. In either case, your next path is to bring this to the attention of @RogersIan for forwarding to Hitron and possibly to Intel, or now Maxlinear. Working with the techs has led to the determination that there is an issue with the CGN3AC modem, the next step is to address it with the engineering staff, assuming that you return to the CGN3AC modem for everyday use. If not, this falls to someone else down the road.
One point of interest that arose thru the whole Puma 6 debacle is that there’s no organization that tests modems for performance. Cablelabs tests the modems for interoperability with CMTS and other cable modems, but they’ve washed their hands of any performance testing. That performance testing would have shown the problems that existed when the Puma 6 modems were first introduced in 2012. So, its up to the end users to test modems for latency and packet loss, for any protocol that you might be interested in, within the usual IPV4 and IPV6 (ICMP, TCP/IP and UDP) group and for any other protocol that you might be using beyond that group. At the point where users are discovering problems what should have been discovered in development testing, its far too late. The development company, whatever that company happens to be has failed in its development and testing process. As a result, the end user pays the price. I wonder if your download testing has stumbled across a particular protocol that results in slowly decreasing modem performance?
Ok, so, where does this leave you and a return to a CGN3 modem. If you do return to that modem, you may want to start a test program to test all protocols, To do that you need to run low interval test pings or UDP queries, in the range of 20 milli-seconds, which will show if that unknown task is still an issue, or, if the shift of the protocol processing to the hardware accelerator / processor has in fact resolved the latency and packet loss issues. That’s what I started to do in 2016, but, after I changed to a Puma 7 CODA-4582 modem in Dec 2016, I never had any reason to return to a Puma 6 modem to retest the firmware updates, so, I don’t know if those updates have really been successful.
Just to point out, running a CGN3xxx modem, I was able to crash the modem after about an hour, running a VPN and simultaneous ping test to the CMTS. Crashing the modem wasn’t a problem.
Here’s an example of the modems task processing at work, causing latency thru the modem. In this case xymox1 is running two Pingplotter plots. At the time, the modem he was using had the first update to resolve the ICMP latency by shifting the ICMP processing over to the hardware accelerator / processor. The top plot is the uncorrected TCP/IP processing, the bottom plot is the corrected ICMP processing. The top plot is what the bottom plot used to look like. And, its an example of how every protocol latency looked like, as in IPV4 and IPV6 (ICMP, TCP/IP and UDP). By shifting the ICMP or TCP/IP ping or UDP query interval down to the 20 milli-second range, the 1.92 second interval task shows up in the plot, causing latency spikes, as you can see from the top plot.
Ok, so that’s a short history of the Puma 6 problems. Hopefully this provides some useful information for you. Personal opinion, Puma modems should be consigned to the s . heap (recycle bin). Now its Maxlinear’s problem as Intel has sold the Home Gateway Platform Division to Maxlinear. Maxlinear manufactures the cable tuner for the modems, so, now they own the whole can of worms.
I would guess that Rogers has a somewhat long standing relationship with Hitron, and just to point out, I believe that Hitron has been grappling with the Puma 7 issues long before Arris, which produces the Puma 7 XB6 (TG3482ER) used for the Ignite TV (IPTV) system.
Broadcom didn't play in the highspeed modem market until they released the BCM-3390 chipset modem, years after Intel's Puma modems were in the market. Intel was the only game in town. Its really unfortunate that the problems with the Puma 6 modem weren't discovered until the modem was in service and ramping up to higher speeds with greater numbers of download channels. So, Intel had the market cornered for a number of years.