CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[Telek]

---

@Double_K wrote:

---

@Queoder wrote:

Mar 8/17 - Apr 7/17	2351 GB
Feb 8/17 - Mar 7/17	889 GB

---

@QueJoder

The increase in your usage coincides with the dates that your modem's private info was shared publicly in February.

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/421...

Therefore, it appears someone else is spoofing your MAC address to associate usage to your account. 

---

I think it's more likely that someone is simply hitting his IP address. Once you have the MAC address you can use DNS to get the IP address.

I don't think that you can simply "spoof" someone's MAC address to the internet and rack up usage on their account due to the way that the modems work.  If this was possible, you wouldn't need to actually find someone's leaked MAC address as all the MAC addresses are created sequentially and it wouldn't take long to brute force finding another valid MAC address on the Rogers' system.

Keep in mind that the Modem itself authenticates internally to the Rogers' network before providing you with your internet access.

[Jeffj]

---

@Telek wrote:

---

@Double_K wrote:

---

@Queoder wrote:

Mar 8/17 - Apr 7/17	2351 GB
Feb 8/17 - Mar 7/17	889 GB

---

@QueJoder

The increase in your usage coincides with the dates that your modem's private info was shared publicly in February.

http://communityforums.rogers.com/t5/forums/forumtopicpage/board-id/Getting_connected/message-id/421...

Therefore, it appears someone else is spoofing your MAC address to associate usage to your account. 

---

I don't think that you can simply "spoof" someone's MAC address to the internet and rack up usage on their account due to the way that the modems work.  

---

Yes you can, it hapens all the time, and its a major problem for every single cable provider on the planet. 

I'd explain how to but I wont risk my account here. Use google you will see.  It's not even dificult. 

[Telek]

---

@Jeffj wrote:

---

Yes you can, it hapens all the time, and its a major problem for every single cable provider on the planet. 

I'd explain how to but I wont risk my account here. Use google you will see.  It's not even dificult. 

---

If you're referring to MAC address spoofing of your client end, this hasn't been possible since DOCSIS1.0.

Ever since then, the modem itself uses the built-in HFC MAC address along with (possibly) EAE (we don't use  on Rogers) and BPI (which is used by Rogers) to authenticate and encrypt, and this happens at a modem firmware level to the cable modem network, before any consumer devices are attached. Unless you have a hacked cable modem, I don't see how this can be bypassed or taken advantage of, certainly not easily. Furthermore, with BPI enabled, even if you have a hacked or completely dumb modem, you still won't be able to spy on neighbouring traffic to get any useful MAC addresses.

http://www.dslreports.com/faq/3455

[Datalink]

Fwiw, this was more of a problem two to three years ago on Rogers.  It was common to see complaints where the usage had skyrocketed for a day or two or users accounts, more commonly when the user was away.  In any event, it would seem that Rogers has done a lot of work in this area.  The current state obviously isn't perfect as this does come up from time to time now, but, the incident rate is much much lower than it was before. 

[Telek]

Keep in mind as well that another explanation is that some device in your network is infected, and being used externally.

[Double_K]

---

@Telek wrote:

Keep in mind as well that another explanation is that some device in your network is infected, and being used externally.

---

Yes, this is also a possibility, and would be validated if the upload qty was higher/closer to the download qty.  Given the disparity between his/her download & upload, their personal devices don't appear to be used as a proxy.  2TB+ is a lot of movies to store somewhere.  However there could be a rogue device connected to his/her wifi doing the downloading.  

Just way too coincidental that his/her usage has increased around the same time as the MAC address was shared on this forum.

[Telek]

---

@Double_K wrote:
Yes, this is also a possibility, and would be validated if the upload qty was higher/closer to the download qty.  Given the disparity between his/her download & upload, their personal devices don't appear to be used as a proxy.  2TB+ is a lot of movies to store somewhere.  However there could be a rogue device connected to his/her wifi doing the downloading.  

Just way too coincidental that his/her usage has increased around the same time as the MAC address was shared on this forum.

---

I'll agree there -- however I noticed something else interesting in their message:

OTHER than using speedtest.net about 60 measurements every day

If this is true, that's a massive number of bandwidth tests. I assume that this is an exaggeration, however this could be a large contributing factor since this also happened around the same time as their bandwidth usage drastically increased.

On a gigabit connection, according to my modem, 60 tests per day would be about 3TB/month.

[Hwaiting]

Is there anyway to set the time/timezone reported in the firmware?  It's always off.

[JohnBeaudin]

Software Version 2.0.10.26T2 LAN Up Time 011 days 14h:03m:57s One thing for sure.. .26T2 fixed the reboot on it's own issue, I never went past 24h before and now look at it go!

[RyzenFX]

@JohnBeaudin I agree. It seems to have fixed the reboot issue. My modem has been up for 6 days and I haven't seen it suffer from any performance issues even with 15+ devices on my network. 

[Mythen]

It hasnt fixed it for me it seems. Just randomly rebooted this morning for no reason that I can see. I also upgraded to the 250u package. Still getting buffering on twitch streams. It's not the source as I seem to be the only one affected. Even watch a simple YouTube video buffers sometimes

[JohnBeaudin]

@Mythen

Looks like you might be affected by high node congestion or maybe something else, but I doubt it's the modem firmware.

[asns]

with hardwire I can connect with VPN and have access to most of the servers and only few ones the connection gets failed, but with Wifi the VPN connection get stablished but I can not connect to any server on our companies net work.

I didn't have any issues with the previous old modem.

[Telek]

---

@asns wrote:

with hardwire I can connect with VPN and have access to most of the servers and only few ones the connection gets failed, but with Wifi the VPN connection get stablished but I can not connect to any server on our companies net work.

I didn't have any issues with the previous old modem.

---

If you tether your laptop to your cellphone over WiFi, can you connect to your VPN that way?

Keep in mind, VPN software has to install drivers and bypass local adapters. It's possible that it's something on your end that works over wired by has a problem over WiFi. Hypothetically, at least, unless there's some bizarre firewall rules, either wired or wireless connection to the router should provide no difference in connectivity. 

[RogersDave]

Based on the feedback received (and I know there are still corner cases being worked on for some of you) the firmware 2.0.10.26T2 addresses multiple performance and stability related issues.

We have therefore made the decision to make this firmware the new baseline as of today.

What this means is that:

1. A factory reset will upgrade you or keep you on firmware 2.0.10.26T2 if the firmware was manually locked on your modem (trial firmware participants)

2. A modem reboot will upgrade firmware 2.0.10.24 to 2.0.10.26T2 for regular modems (all customers)

We are still working on completing finalizing firmware 2.0.10.27 for trial firmware participants which should be available later this week or next week. I will post release notes as soon as I have them.

Dave

[Triple_Helix]

---

@RogersDave wrote:

Based on the feedback received (and I know there are still corner cases being worked on for some of you) the firmware 2.0.10.26T2 addresses multiple performance and stability related issues.

We have therefore made the decision to make this firmware the new baseline as of today.

What this means is that:

1. A factory reset will upgrade you or keep you on firmware 2.0.10.26T2 if the firmware was manually locked on your modem (trial firmware participants)

2. A modem reboot will upgrade firmware 2.0.10.24 to 2.0.10.26T2 for regular modems (all customers)

We are still working on completing finalizing firmware 2.0.10.27 for trial firmware participants which should be available later this week or next week. I will post release notes as soon as I have them.

Dave

---

So we trial firmware participants need to do a factory reset to get 2.0.10.27 when it comes out? Or do we need to do a factory reset in any case?

TIA,

Daniel 

[JohnBeaudin]

@RogersDave

Thanks For the update.

As a Firmware participants do I need to do anything ( factory reset?) or I just wait until .27 is deployed later this week and no action is required?

[RogersDave]

Nothing to be done.

If you are a trial participants, I will push 2.0.10.27 when it is available. The only difference for now is that if you perform a factory reset, you won't go back to .24, you will stay on 26T2 as this is the new baseline firmware.

Dave

[JohnBeaudin]

Ok perfect Captain! with such amazing results from .26T2, I am really looking forward to .27 I really appreciate the effort from you and the rest of the team, Thanks guys!

[Triple_Helix]

Thanks Dave and I agree things are working well for me with .26T2 but I'm not a hardcore gamer.

[RyzenFX]

Is .27 supposed to be a major release with a lot of improvements, and enhancements? (i'm dying to hear whats new lol) 

If I remember correctly one of the enhancements coming with .27 will deal with the modem LEDs being dimmed (finally).