CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[RyzenFX]

I will make the switch back to the CODA-4582 sometime this week. I will be playing a couple games of LOL and will use the raw network logs and translate it into a graph which shows the packets lost/min.

I will also be running the test with less than 10 devices connected to the modem, and 10+ devices connected to it. This test should be run on firmware .26 in order to verify if the reboot fix actually worked. There's limited feedback on .26, so hopefully there's more feedback in the next few days.

[JohnBeaudin]

@RyzenFX

Thanks for your contribution to the community.

[Jeffj]

---

@Telek wrote:
Alternately, just try nping (part of nmap) in echo mode. Setup a server somewhere and flood UDP pings and see how much loss you get.

https://nmap.org/book/nping-man-echo-mode.html

---

 its not just lost packets its out of order packets.  iperf might be more suitable as it will verify if the data recieved is the same as sent and if the order is correct. 

[tonytoronto]

---

@Telek wrote:

---

@Datalink wrote:

After looking at a League of Legends data capture and from running UDP ping tests to pingplotter.com, I'm not convinced that the UDP problems are all attributable to the modem.  I think the data path from the start point to the end point might be playing a big part in the UDP problems, but, that's for Dave to confirm or deny. 

---

All attributable? No. But as others have mentioned the AC modem didn't have nearly the same level of the problem as the CODA one does.

---

  That. Since switching to 3552 kids have not any packet loss or ping problems with any of the games. Only  thing that  changed was the modem. 

[J04DAN1]

I need to disable IPv6 on my CODA-4582 Modem. Unforutnately Rogers IPv6 causes connection issues with some EA games on the Xbox One. The most recent is Mass Effect Andromeda that makes it impossible to play multiplayer with Rogers Internet. NHL 17 also has this issue but only in head-to-head games. 

I have tested this enought to confirm 100% it is a Rogers Internet issue. The issue is gone when using an IPv4 only connection. Is there anyway to disable IPv6 on my modem?

[toolcubed]

Noticed some slowness in recent days and also some random time-outs when attempting to load certain sites. Started running ping plotter over the last few days and notice a lot of packet loss at multiple hops when testing IPv6 (using Google). The problem is consistent, meaning the packet loss is always there every time I test. There are zero issues with IPv4 (no packet loss and consistently low pings). Not sure if this problem manifested itself in firmware .26. I'm running CODA in bridge mode with Asus RT-AC68U. IPv6 is configured correctly on the Asus. Anyone else having issues with packet loss using IPv6?

[Jeffj]

---

@J04DAN1 wrote:

I need to disable IPv6 on my CODA-4582 Modem. Unforutnately Rogers IPv6 causes connection issues with some EA games on the Xbox One. The most recent is Mass Effect Andromeda that makes it impossible to play multiplayer with Rogers Internet. NHL 17 also has this issue but only in head-to-head games. 

I have tested this enought to confirm 100% it is a Rogers Internet issue. The issue is gone when using an IPv4 only connection. Is there anyway to disable IPv6 on my modem?

---

I'm over 24h of game play into the new Mass ffect on Xbox One with IPV6 enabed and have not experienced the same issues.  Something else must be afoot for you. 

[KyleW1988]

I just bit the bullet and bought a router from Best Buy. Now I'm solely on IPv4 I couldn't play P2P games. All server based games worked fine though before.

[J04DAN1]

Unfortunately no it is a Rogers issue. I have tested this many ways. It also only impacts Mass Effects MULTIPLAYER. Steps I have done to confirm the issue is with Rogers. 

1. NAT is Open on Xbox One with a static IP address assigned and all the ports suggested by Microsoft and EA for Andromeda correctly routed to Xbox One 

2. Every other multiplayer game works perfectly including recent releases like For Honor, Battlefied 1, Madden 17, Destiny, Ghost Recon Wildlands, Rainbow Six Seige. 

3. The issue is not present if I use my cell phone as a mobile hotspot and connect to Xbox Live. The phone is giving an IPv6 address as well but whatever routing issue is present with Rogers Cable Internet is not present on their mobile network. 

4. I have used two separate modems the older CGN and now the new CODA modem and both had the same issue. 

5. The issue is not present when I took my Xbox to a family members house whose Rogers modem was running an older firmware that was only providing IPv4 addresses. When only connected to IPv4 the issue again was fixed. 

6. I have factory reset both modems I tried, tested with UPNP on and off, tested with DMZ on and off. Factory reset Xbox One, and hard booted Xbox One multiple times. 

No matter what troubleshooting step I took Mass Effect Andromeda multiplayer and NHL 17 head to head both are completely unplayable as neither will connect to the opponent/host while using Rogers Cable Internet with an IPv6 address. 

[JohnBeaudin]

@J04DAN1

This is going.... to be very hard to re-create for Hiltron.. Might take a while if they ever find a fix for that one.

[Jeffj]

---

@J04DAN1 wrote:

Unfortunately no it is a Rogers issue. I have tested this many ways. It also only impacts Mass Effects MULTIPLAYER. Steps I have done to confirm the issue is with Rogers. 

1. NAT is Open on Xbox One with a static IP address assigned and all the ports suggested by Microsoft and EA for Andromeda correctly routed to Xbox One 

2. Every other multiplayer game works perfectly including recent releases like For Honor, Battlefied 1, Madden 17, Destiny, Ghost Recon Wildlands, Rainbow Six Seige. 

3. The issue is not present if I use my cell phone as a mobile hotspot and connect to Xbox Live. The phone is giving an IPv6 address as well but whatever routing issue is present with Rogers Cable Internet is not present on their mobile network. 

4. I have used two separate modems the older CGN and now the new CODA modem and both had the same issue. 

5. The issue is not present when I took my Xbox to a family members house whose Rogers modem was running an older firmware that was only providing IPv4 addresses. When only connected to IPv4 the issue again was fixed. 

6. I have factory reset both modems I tried, tested with UPNP on and off, tested with DMZ on and off. Factory reset Xbox One, and hard booted Xbox One multiple times. 

No matter what troubleshooting step I took Mass Effect Andromeda multiplayer and NHL 17 head to head both are completely unplayable as neither will connect to the opponent/host while using Rogers Cable Internet with an IPv6 address. 

---

I'm not saying you dont have an issue, I'm saing I have the same game and dont have these issues with IPV6 enabled and multiplayer, which is why i stated something else is afoot. 

Aside when xbox live was having issues for the past few days i have not had any issues connectivity wise on Mass Effect: Andromeda.

[J04DAN1]

@JohnBeaudin

The fix actually quite easy. Just give us an option in the firmware of CODA modems to disable IPv6

[KyleW1988]

I couldn't play NHL 17. This is what EA has posted. The only fix for me was buying a 3rd party router. For $60 everything is working again.

If you're having frequent disconnections on a peer-to-peer connection, particularly in games like FIFA, it's possible that your network is using newer IPv6 IP addresses rather than the more common (and previously standard) IPv4 type.

IPv4 IP addresses are 32-bit numerical strings separated by periods. For example, 192.168.1.1
IPv6 IP addresses are 128-bit hexadecimal strings separated by colons. For example, 2001:0db8:85a3:0000:0000:8a2e:0370:7334
Unfortunately, IPv6 and IPv4 are not yet at the point where they work well together on peer-to-peer connections. If you find that you lose your connection every time you play against a particular friend or whenever you join a match, chances are that you or your opponent are using IPv6.

You may need to speak to your ISP if you suspect this is the case, as they can advise you on any hardware or router firmware changes you might need to make to switch to IPv4 for gaming.

[Jeffj]

---

@J04DAN1 wrote:

@JohnBeaudin

The fix actually quite easy. Just give us an option in the firmware of CODA modems to disable IPv6

---

If IPV6 is causing you issues with your peers, Set a static IPV4 address on the xbox one this should disable IPV6 dhcp leasing on the console. 

[J04DAN1]

@KyleW1988

The problem with that stock reply from EA support is it isn't true. Multiplayer games work fine with IPv6. As I mentioned I can play every other multiplayer game with no issues on Xbox Live  even some EA titles like Madden and Battlefield 1. It's these two games and whatever routing technique they use to connect peers. The only fix is for them to patch the game which they won't since this issue impacts probably less then 1% of the player base or for Rogers to fix whatever is clogging the routing or give us the option to disable IPv6

[J04DAN1]

@Jeffj

Any idea on how to do that?

[Jeffj]

It should be under network settings on the console, then choose advanced settings.  

You can also do this on the modems end by address reservation via mac address. But setting it on the console will ensure that its forced to IPV4.

[KyleW1988]

Every friend I have that has Rogers has this problem on NHL. Server based games work fine but for example HUT in NHL won't find an opponent but EASHL in NHL works fine. So it's totally dependent whether that game mode works on a peer to peer connection or server based connection. I also couldn't play rocket league. Any friend I have that has bell which hasn't upgraded to IPv6 yet has no problem. I only started having problems when Rogers enabled IPv6. So it would be a weird coincidence if all these games started giving me the finger at the same time. It's something to do with Rogers enabling IPv6 they should of put in a way to disable it. Like I said once I got my own router and made it IPv4 only there has been no issues.

[J04DAN1]

@Jeffj

Unfortunately all this does is assign a static local IP address the same as DHCP reservation. This doesn't prevent the modem from issuing an IPv6 address as well. 

[Telek]

---

@Jeffj wrote:

---

@Telek wrote:
Alternately, just try nping (part of nmap) in echo mode. Setup a server somewhere and flood UDP pings and see how much loss you get.

https://nmap.org/book/nping-man-echo-mode.html

---

 its not just lost packets its out of order packets.  iperf might be more suitable as it will verify if the data recieved is the same as sent and if the order is correct. 

---

Except how can this be a probem? UDP is specifically designed that there is no packet order.  By definition any system that uses UDP must be able to accept missing and out of order data.

[Telek]

---

@J04DAN1 wrote:

@JohnBeaudin

The fix actually quite easy. Just give us an option in the firmware of CODA modems to disable IPv6

---

This.

I'm actually very upset that Rogers is forcing IPV6 on us, especially considering the massive security implications.