Yes! Yes! and Yes! This is exactly the behaviour I see.
Rogers has sent 4 techs to my house and the line is absolutely clean and I am on my 3rd CGN3 modem. I am so tired of being pushed off. I was pushed to the point of having to diagnose this myself. There is a single cable plugged into my CGN3 and WIFI is disabled. My current firmware is 4.5.8.40T2 and it runs in gateway mode. I installed ZABBIX to monitor the rogers connection. The 1st screenshot shows the typical slow escalation of the RTT to the local IP of the modem. Latency becomes so bad that the modem becomes completely unresponsive from both in the house and when Rogers tries to login. The bottom 3 graphs in that same screen show 3 different devices in the house, each with rock solid RTT.
This next ZABBIX screenshow shows how often I need to reboot my modem. Every spike results in a reboot. There was a quiet period from Jul 31 to Aug 7 while I was away on vacation and there was no traffic in the house. Without any significant traffic, the modem stayed up. I guess Rogers only supports Internet stability as long as you don't use it.
I have now configured a traffic stress test. Our typical usage is less than 50Mbps. I am going to load the link up to >150Mbps (on a 300M services) and see if I can cause the problem to occur more often.
I shouldn't have to be the one debugging this. I'm paying for a service that should work without a need to reboot the modem daily.
I'll continue from my post above
The problem recurred last night. I'm including my information here in the hopes that it helps someone to diagnose the root cause of this issue. The agents, field techs and senior field techs have not looked at this data. They just check the lines and replace the modem hardware. That won't resolve what I believe is a software issue in the modem.
- I only drove traffic for about 5 hours. I did not run the traffic over night
- You can see that the problem was starting during the 5 hours of receiving traffic by the slight, but gradual increase in RTT. This RTT is measured from a server in my house to the modem's IP address in my house.
- The downstream traffic is generated by a simple script that performs a continual and repeated download of a multi-gigabit file which is sent directly to /dev/null.
- This is the first time I purposely placed load on my downstream traffic beyond normal usage. I am not convinced that this triggered the recurrence of the problem at this time, but there is insufficient data to conclude anything. I stopped the traffic generation script because I read elsewhere in this forum that Rogers will throttle people who use excessive bandwidth.
- As you can see in the next graph, the RTT to all other devices in the house remains healthy and constant around 1ms
- Along with the delay, packet drops gradually increase, until the router hits 100% packet loss. The bottom graph on the next screen shot illustrates this. This is a simple ping from a server in my house to the Rogers DNS.
- The device on the other end of the ethernet cable from the Rogers modem is a SNMP managed device. During the time window above:
- The link stayed operational
- There were no changes in duplex
- There were no inbound packet discards or inbound packet errors
- The interface speed is constant at 1Gbps
- The modem power is still on
- If you are an agent reading this post, my answer is “No. My modem is not plugged into a power bar.”
- The agent I spoke with today will be sending another field tech to my house, but I honestly don't expect anything to get resolved from that visit. If they choose to replace the modem, it will be my 4th modem in the last 1-2 months. This really doesn’t look like a HW issue.
- To me, this appears to be a software problem within the modem. Some task (whatever it is), gets progressively slower over time. It is as though the CPU is processing a list of intense and high priority CPU tasks. As the list grows, the CPU becomes more and more consumed with processing the list. While the CPU is processing this growing list, it is unable to forward packets. This would result in long RTT and occasional dropped packets. Eventually, this “high priority” CPU task consumes the modem completely and all packet forwarding stops. This includes dropping all packets destined to the GUI of the router (which I run in gateway mode)
- I don’t have access to any debug logs or stats within the modem to diagnose what the modem itself is doing during this time.
I would encourage Rogers to please have someone beyond agents and field techs take a look at this problem profile. I would be happy to continue the problem isolation and resolution with them. I need a stable connection before September when school starts and I will be following this issue carefully.
We have new code coming this week for our coda users.
Firmware ver 7.1.1.33
The main fix is a Intel patch which improves network stability.
RogersIan
Here is an update on the issue I reported previously. I have been working with a Senior service tech for the last 10 days, during which time I have had a CODA test modem in my house rather than the CGN3AC. We tested the CODA modem on both 2.0.1036T8 (Aug 15-19th) and 7.1.1.32 (Aug 19-24). Both software versions were reliable over that time period.
While a root cause was not identified, it is definitively isolated to the CGN3AC or the software that it was running.
Thanks for your assistance. Now I just need Rogers to provide me with a CODA modem that I can keep since I need to return to test modem tomorrow night.
@ pburton here’s some background information that you might find interesting. Just for the record, the Hitron CGN3xxx and CGNM-3552 modems are all Intel Puma 6 modems. The Hitron CODA-4582 is an Intel Puma 7 modem.
Intel bought the Puma modem line from Texas Instruments in 2010:
https://www.cnet.com/news/intel-to-buy-texas-instruments-cable-modem-unit/
Intel then put its own spin on the modem, building it with an Atom and Arm processor and Maxlinear cable tuner. That modem was introduced into service around 2012. I think Rogers introduced the first CGN3 modems in 2014. It wasn’t pretty, and that is an understatement. In short, users experienced latency to and thru the modem for all protocols, and for UDP, which is used for DNS address queries and gaming, there was packet loss as well.
So, it appears that Intel attempted to virtualize the modem, running everything in a software process thru the CPU. There was a hardware processor / accelerator on the chipset, but, at that time it wasn’t used. So, the major problem is that there was an unknown task, running approx. every 1.92 seconds. That task would stop all traffic thru the modem, resulting in latency for all traffic (IPV4 and IPV6). In the case of UDP traffic, my guess is that it timed out in the modem and those timed out UDP packets were discarded. That latency and UDP loss resulted in numerous complaints and as result, Rogers engineers started their own investigation into latency and packet loss in mid 2016. Here’s one of the original Pingplotter plots that I submitted at the prior to the start of that investigation:
That’s a new Hitron Puma 6 CGNM-3552 modem running on a new Casa Systems CMTS, with no traffic other than a ping test running to the CMTS, which is the next IP address beyond the modem. Its unfortunate that at the time I used Pingplotters default 2.5 second ping interval. If I had lowered that interval down to 20 milli-seconds, that plot would be filled with high latency returns. Its has a large number of high latency returns, but, there would be a much larger number of those high latency returns.
Ok, with Rogers input in mid 2016, Intel started working on updates to resolve the latencies observed in the Puma 6 modems. The first release which was designed to resolve IPV4 ICMP (ping) latency was loaded onto Rogers CGN3 modems in Sept 2016. That did work as expected, but, it left IPV4 (TCP/IP and UDP) and IPV6 (ICMP, TCP/IP, and UDP) for future updates.
Since the introduction of the same Puma 6 modems in the U.S. and Europe, there were similar grumblings of poor latency and packet loss. That really came into view when DSLReports member xymox1 started a thread in DSLReports outlining his observations. That first thread is located here:
The next follow-on thread is located here:
https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-UDP-Network-Latency-Issue-Discussion
That rolling stone gathered momentum over time and it was no longer possible to deny that there were problems with the modem’s hardware and firmware design. During the course of the posts throughout that second thread, it was discovered that the Puma modems were susceptible to low frequency DOS attacks. Never mentioned through any of this, I wonder if there was a discovery that the DOS attack would or could lead to a takeover of the modem. Shown enough evidence by posters in that thread, Intel probably had no choice but to recognize the problems, issue a CVE and issue corrective firmware updates to resolve the DOS attack problems for Puma 5, 6, and 7 modems.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5693
When Rogers issued the first update in Sept 2016, that appeared to be the final involvement with Hitron and Intel regarding the latency issue. It would appear that Rogers looked into a crystal ball to determine which way to go, wait for the Puma 6 updates to appear eventually or, push ahead with the Puma 7 (Hitron CODA-4582) modem. Rogers appears to have chosen to pursue the Puma 7 deployment and left Arris and Intel to wrestle with the Puma 6 latencies. The first CODA-4582s were released by Rogers in Dec 2016, most likely long before Intel produced a final resolution to all of the latency and packet loss problems, if ever.
If you look at the following post which shows the history of the modem updates, you will see a series of updates for all modems. The question is, did Intel every solve the problems? No one really knows as no company, from Intel to the modem manufacturers to the ISPs have every produced any test latency plots to show that those problems have been resolved. And, there is no test plot proof that the DOS problems have been fully resolved.
Ok, so, that leads to your current situation. In theory, you shouldn’t be seeing any latency to or thru a Puma 6 modem. Keep in mind that there has never been any proof provided that shows that the latency issues have been resolved. I wonder if Intel has done something wrong in the process and you happen to be the first Rogers customer to either determine that the problems still exist, or that Intel has introduced some new problem. In either case, your next path is to bring this to the attention of @RogersIan for forwarding to Hitron and possibly to Intel, or now Maxlinear. Working with the techs has led to the determination that there is an issue with the CGN3AC modem, the next step is to address it with the engineering staff, assuming that you return to the CGN3AC modem for everyday use. If not, this falls to someone else down the road.
One point of interest that arose thru the whole Puma 6 debacle is that there’s no organization that tests modems for performance. Cablelabs tests the modems for interoperability with CMTS and other cable modems, but they’ve washed their hands of any performance testing. That performance testing would have shown the problems that existed when the Puma 6 modems were first introduced in 2012. So, its up to the end users to test modems for latency and packet loss, for any protocol that you might be interested in, within the usual IPV4 and IPV6 (ICMP, TCP/IP and UDP) group and for any other protocol that you might be using beyond that group. At the point where users are discovering problems what should have been discovered in development testing, its far too late. The development company, whatever that company happens to be has failed in its development and testing process. As a result, the end user pays the price. I wonder if your download testing has stumbled across a particular protocol that results in slowly decreasing modem performance?
Ok, so, where does this leave you and a return to a CGN3 modem. If you do return to that modem, you may want to start a test program to test all protocols, To do that you need to run low interval test pings or UDP queries, in the range of 20 milli-seconds, which will show if that unknown task is still an issue, or, if the shift of the protocol processing to the hardware accelerator / processor has in fact resolved the latency and packet loss issues. That’s what I started to do in 2016, but, after I changed to a Puma 7 CODA-4582 modem in Dec 2016, I never had any reason to return to a Puma 6 modem to retest the firmware updates, so, I don’t know if those updates have really been successful.
Just to point out, running a CGN3xxx modem, I was able to crash the modem after about an hour, running a VPN and simultaneous ping test to the CMTS. Crashing the modem wasn’t a problem.
Here’s an example of the modems task processing at work, causing latency thru the modem. In this case xymox1 is running two Pingplotter plots. At the time, the modem he was using had the first update to resolve the ICMP latency by shifting the ICMP processing over to the hardware accelerator / processor. The top plot is the uncorrected TCP/IP processing, the bottom plot is the corrected ICMP processing. The top plot is what the bottom plot used to look like. And, its an example of how every protocol latency looked like, as in IPV4 and IPV6 (ICMP, TCP/IP and UDP). By shifting the ICMP or TCP/IP ping or UDP query interval down to the 20 milli-second range, the 1.92 second interval task shows up in the plot, causing latency spikes, as you can see from the top plot.
https://www.youtube.com/watch?time_continue=9&v=15cJ400yR_E&feature=emb_logo
Ok, so that’s a short history of the Puma 6 problems. Hopefully this provides some useful information for you. Personal opinion, Puma modems should be consigned to the s . heap (recycle bin). Now its Maxlinear’s problem as Intel has sold the Home Gateway Platform Division to Maxlinear. Maxlinear manufactures the cable tuner for the modems, so, now they own the whole can of worms.
I would guess that Rogers has a somewhat long standing relationship with Hitron, and just to point out, I believe that Hitron has been grappling with the Puma 7 issues long before Arris, which produces the Puma 7 XB6 (TG3482ER) used for the Ignite TV (IPTV) system.
Broadcom didn't play in the highspeed modem market until they released the BCM-3390 chipset modem, years after Intel's Puma modems were in the market. Intel was the only game in town. Its really unfortunate that the problems with the Puma 6 modem weren't discovered until the modem was in service and ramping up to higher speeds with greater numbers of download channels. So, Intel had the market cornered for a number of years.
-
935068 views
-
669 Likes
-
-
