CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[MegnaGaming]

---

@Telek wrote:

Have you tried on multiple devices? Are you plugged directly into the modem in gateway mode? Have you swapped out the modem before?

---

I have! Most of these things definitely would have been checked by the senior tech that was out at the house many times as well. Thanks for the suggestion though!

@Jeffj I'm worried you're right about the infastructure but I find it odd that even the senior techs haven't been able to track down what's up. Also seems odd that it's fixed with a high enough upload rate via ethernet but still presents itself via wifi.

[RyzenFX]

@Datalink

I'm highly confident that the Puma 7 is affected by UDP latency from what you said based in your last paragraph.

That is exactly what I've observed with League OF Legends on BOTH the Puma 6 and PUMA 7.

I've been monitoring this closely with the Puma 7, and the UDP latency is fine during the day, however during peak hours, the UDP latency becomes much worse, and during midnight, the packet loss goes down to its normal levels.

From the results from the first engineering code with the Puma 6, there's a lot of room for improvement with the Puma 7.

So basically, UDP latency exists on the Puma 7. This needs to be opened w/ Intel ASAP.


I'll be able to post examples of UDP latency thru league if legends this afternoon.

Datalink, do you think I have a credible lead here?

[Jeffj]

---

@MegnaGaming wrote:

---

@Telek wrote:

Have you tried on multiple devices? Are you plugged directly into the modem in gateway mode? Have you swapped out the modem before?

---

I have! Most of these things definitely would have been checked by the senior tech that was out at the house many times as well. Thanks for the suggestion though!

@Jeffj I'm worried you're right about the infastructure but I find it odd that even the senior techs haven't been able to track down what's up. Also seems odd that it's fixed with a high enough upload rate via ethernet but still presents itself via wifi.

---

the 2.4GHz band is so over saturated, I can say with confidence you will always see issues streaming on it, let alone wifi. I'm inclined to say that ur issue is localized to your hardware.  Wireless works off first in first out concept. So if you're streamign on it and another device makes a data transaction your packets will have to wait while the transaction that occoured before some of your packets gets handled. 

If streaming on wired yeilds proper results then you have answered your own question right there. 

Wireless iss extreemly susceptible to interference, and packet loss, causing retransmits. Also one thing people dont know is that a wireless access point is a dumb hub One talks and everyone has to stop, listen, waits for their turn to talk, so any transactions on your network your streaming device has to go, "wait, is this for me, no ok. wait i need to talk, ok its my turn? go?, ok is this my response? ok good."

If you have a monitored switch (most dont) or you use wireshark to packet sniff your wireless network, i think you will find something on your network is doign something during those lag spikes, or that the wireless is retransmitting the same packets due to loss on those lag spikes.

[Telek]

---

@Jeffj wrote:

---

Wireless iss extreemly susceptible to interference, and packet loss, causing retransmits. Also one thing people dont know is that a wireless access point is a dumb hub One talks and everyone has to stop, listen, waits for their turn to talk, so any transactions on your network your streaming device has to go, "wait, is this for me, no ok. wait i need to talk, ok its my turn? go?, ok is this my response? ok good."

Also don't forget that wifi also has to operate at the lowest level based on all devices connected.

Yeah, 2.4GHz (unless you're in an isolated area) should really only be used for low-bandwidth things. Even the stupid 5GHz networks are saturated now in busy areas because of our stupid limitations on channels that we can use.

[MegnaGaming]

@Telek @Jeffj

---

@Telek wrote:

---

@Jeffj wrote:

---

Wireless iss extreemly susceptible to interference, and packet loss, causing retransmits. Also one thing people dont know is that a wireless access point is a dumb hub One talks and everyone has to stop, listen, waits for their turn to talk, so any transactions on your network your streaming device has to go, "wait, is this for me, no ok. wait i need to talk, ok its my turn? go?, ok is this my response? ok good."

Also don't forget that wifi also has to operate at the lowest level based on all devices connected.

Yeah, 2.4GHz (unless you're in an isolated area) should really only be used for low-bandwidth things. Even the stupid 5GHz networks are saturated now in busy areas because of our stupid limitations on channels that we can use.

---

You guys misunderstood - my fault for lumping two issues together!

I would never stream wirelessly as that would be insane. The ridiculous packet loss is happening on a wired connection and was not anywhere near this bad when on .19

You can see the packet loss as an example here: https://i.gyazo.com/be9c03109d46ab91fa8be71d65d7df06.png

The issue I'm seeing wirelessly is the following latency spikes while pinging google: 
https://i.gyazo.com/9932f1e48b0e2fc51a37cf5e0c8ae736.png

The latency spikes was also happening while wired when on the 100/10 plan. I had thought the problem was fixed by the plan change but it seems WiFi is reproducing that problem. This shows up on 2.4, 5 and from multiple devices.

[JohnyR]

Yup, 3.1 activated in Thornhill at around the same time. 

I'm interested in seeing how this holds up in peak hours. I'm hoping this will fix the crazy congestion I've been experiencing over the last 2 years.

[QueJoder]

Last night came and went,. my white CODA 4582 one black dot is now supportiung the .24 production version.. Additionally i did a reboot for good measure

i am told by Rogers Tech Support that i am on Docsis 3.1 in Toronto West End> My speeds today have not been upgraded. Running between 500/600 download and have degraded to less than 15 Mbts upload.

Nothing in here has changed for the better. What am i to expect now? My system structure has seen before and with previous modems a couple of 930ish download and high 30's upload speedtest .net readings. therefore my hardware here is capable of those speeds.

What would RogersDave do if he was in my shoes knowing what he knows abnout the status of this issue?

[mikeepolmic]

Just wanted to update - last evening I got the .24 firmware upgrade - it's now been 25+ hours and haven't had to reboot the modem.  Internet connectivity has been solid, speed is very good.  On a direct ethernet connection to modem it's on average 905/43, download/upload.

(Knocking on wood while trying to type, so excuse any typos) ...

🙂

[gduesbur1]

Packet Loss and excessive Ping times

-----------------------------------------------

CODA-4582 Modem in Bridge Mode. -  Commercial Router and computers hardwired

Hardware Version            1A

Software Version             2.0.10.24

Gateway Serial Number 25116A0xxxxx

HFC MAC Address            xx:xx:xx:xx:xx:70xx

System Time                      Sat, 25 Feb 2017 13:35:41

Private LAN IP Address  192.168.100.1/24

LAN Receiving                    159.88M Bytes

LAN Sending                       438 Bytes

LAN Up Time                      000 days 11h:11m:14s

I have ran PingPlotter over night and I can see a consistent / repeatable error

Click on Link to right  -   https://share.pingplotter.com/cxXcWBwez9k

Its during this 8 to 10 sec time frame when Pings go to over 2400ms  - See Link -> https://share.pingplotter.com/JKMsA4AR3ej

This happends 7 x 24 since I replaced my modem.


 

[Telek]

---

@gduesbur1 wrote:

Packet Loss and excessive Ping times

CODA-4582 Modem in Bridge Mode. -  Commercial Router and computers hardwired
 

---

I know it sucks, but can you try in gateway mode without your router in the picture?  Also try with a different computer, just to ensure that it's nothing with your setup.

[IbeforeE]

I've recently switched to the gigabit plan with the CODA-4582, and I'm having serious issues with my download speed.

Starting with firmware .13, I did a speed test, and the results were excellent.  Signed up for the beta firmware to receive .24, and there was a definite speed loss on the test, but I chalked that up to network traffic.

From there I switched to bridge mode and my router, only to find a slightly higher ping than I was expecting, but then also lag swings that made games completely unplayable.  (Since then, I've disconnected the external router entirely, to take it out of the troubleshooting equation.)

Having done a number of factory resets and power cycles, this is the graph of all the speed tests I've done, mostly using a single machine over ethernet directly to the modem.

I'm open to any ideas, here.

This is the DOCSIS WAN info (I have an IPv6 address, but not 3.1 enabled, as far as I can tell)

Downstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Signal noise ratio (dB)
1	591000000	256QAM	3.600	31	37.356
2	363000000	256QAM	0.200	10	37.636
3	369000000	256QAM	0.300	11	37.636
4	375000000	256QAM	0.700	12	37.636
5	381000000	256QAM	0.700	13	38.605
6	387000000	256QAM	0.700	14	37.636
7	393000000	256QAM	0.800	15	38.605
8	399000000	256QAM	1.200	16	38.605
9	405000000	256QAM	1.300	17	38.605
10	411000000	256QAM	1.100	18	38.605
11	417000000	256QAM	1.400	19	37.636
12	423000000	256QAM	1.700	20	38.605
13	429000000	256QAM	2.000	21	38.605
14	435000000	256QAM	1.800	22	38.605
15	441000000	256QAM	1.500	23	37.636
16	447000000	256QAM	1.700	24	38.605
17	555000000	256QAM	2.900	25	37.636
18	561000000	256QAM	3.200	26	37.636
19	567000000	256QAM	3.300	27	37.636
20	573000000	256QAM	3.300	28	37.356
21	579000000	256QAM	3.500	29	37.356
22	585000000	256QAM	3.400	30	37.636
23	357000000	256QAM	0.400	9	38.605
24	597000000	256QAM	3.700	32	37.636
25	603000000	256QAM	4.300	33	38.605
26	609000000	256QAM	4.700	34	38.605
27	615000000	256QAM	4.900	35	38.605
28	621000000	256QAM	4.900	36	37.636
29	633000000	256QAM	5.500	37	38.605
30	639000000	256QAM	5.800	38	38.605
31	645000000	256QAM	5.600	39	38.605
32	651000000	256QAM	5.600	40	38.605

OFDM Downstream Overview

Receiver	FFT type	Subcarr 0 Frequency(MHz)	PLC locked	NCP locked	MDC1 locked	PLC power(dBmv)
0	NA	NA	NO	NO	NO	NA
1	NA	NA	NO	NO	NO	NA

Upstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	30596000	ATDMA - 64QAM	39.750	1	6400000
2	38596000	ATDMA - 64QAM	43.500	3	3200000
3	23700000	ATDMA - 64QAM	38.250	2	6400000

OFDM/OFDMA Overview

Channel Index	State	lin Digital Att	Digital Att	BW (sc's*fft)	Report Power	Report Power1_6	FFT Size
0	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K
1	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K

[dallasdinatake]

Anyone see any issue with the Xbox one speed test ?

[MegnaGaming]

@gduesbur1 wrote:

Packet Loss and excessive Ping times

@gduesbur1,

Might I suggest editing your post ASAP to remove those links as you're leaving your IP address exposed for all the internet to see. 

Are you very familiar with PingPlotter? There are a lot of great articles that provide some detailed explainations of results you're seeing. Unfortunately when I attempted to download the data file it only gives about 1 minute of the original data so unfortunately I can't troubleshoot too much. 

Just going off the images attached,there appears to be major latency at hop 9. I would suggest trying to ping Google or Rogers. Secondly you seem to have major packet loss happening at your router/modem (50%) but upon closer inspection it seems you may be pinging REALLY frequently. Do you have your interval set to 0.5 or 1 second by chance? I suggest keeping it on 2.5 seconds.

If you want more help with PingPlotter, send me a DM and I can take another look!

[Alex4161]

@IbeforeE

I would suggest the following:

- factory reset the modem

- turn off the 2.4 Ghz band

- Connect a CAT 6 or CAT 7 cable in port 2 into your PC and run some speed tests from these sites:

speedtest-tor1.digitalocean.com/

http://speedcheck.rogers.com/en.html

You should be getting good speeds and if not, it could be congestion on your node.  Contact Rogers Support and they can troubleshoot things for you.

[Kbursey]

i have, have yet to get proper speeds on pc get 900+ on xbox i get around 100

[gp-se]

---

@Kbursey wrote:
i have, have yet to get proper speeds on pc get 900+ on xbox i get around 100

---

@Kbursey Slow speeds on xbox is being looked into by @RogersDave. I believe he is working with Microsoft on the issue.

[lethalsniper]

Yea and I think thats what's causing games to play bad online when it comes to the Xbox due to the slow speeds. Hope @RogersDave does get this fixed with Microsoft , when can we expect a fix for the udp ?

[Gdkitty]

Havent been fully watching this thread, since I didnt have this modem, till this weekend.

My original CGN3.. well started to go wonky.  Where it was still conntected from rogers side (uplink/downlink, etc still there, rogers could see it) but no LAN side.. no connectivity at all from wired/wireless/static/dynamic.  Cant get to the modem interface.   Swapped it out and was given a new CGN3, and same thing, something wonky with the current firmware?

So was swapped out for the CODA.  

Generally that one was working fine for about 1/2 the day.   Then thats when stuff started getting wierd.

Not in bridge mode.
Randomly, a good chunck of the devices.. would not be connected.. well they would connect to the wifi.. but not getting an IP address properly..
Any of my devices which were IPv4 only.  Were connecting fine.

Anything which had a STATIC IPv4 addres, was fine.
The issue seemed to be with devices which were on dynamic.. and getting both IPv4 and IPv6 addresses?
My iphone/ipad were ones which were effected.  They are not able to turn on/off IPv6..  it appeared it got a v6 address.. but wouldnt keep properly the v4 address.

And from what i could find.. very little control over the v6 DHCP in the modem itself?

Since then.. swapped over to bridge mode.  Using an Asus RT-N66U.
Currently, have v6 OFF with it.. just v4 only.. and appears to be working fine.

[Alex4161]

@Gdkitty

That would explain why my iPad Air 2, Pixel was offline.  It was the IP6 Address. 

Is there a way we can turn this thing off?

[Mythen]

I have the same issue with my pixel XL. Frequent disconnects since IP v6 was enabled. Anyone have any suggestions to fix this

[Triple_Helix]

---

@Mythen wrote:
I have the same issue with my pixel XL. Frequent disconnects since IP v6 was enabled. Anyone have any suggestions to fix this

---

@RogersDave I'm having the same disconnects with my Moto Z Phone as well since IPv6 so I don't think it's only the Pixel XL.

Also my Upload has not been great as well since IPv6 has been enabled. I was getting over 40Mbps before.