CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[Jeffj]

---

@JohnBeaudin wrote:

For some reason ARK is the game where I get the lowest ping and by far out of any games I play.. I can get as low as 29 ping.

Dota2: 65 ms

CSGO: 70-75 MS

Overwatch: 45-50 ms

ARK: 29-31 ms

all on US-East servers..

---

I'm about to see if my ping is better i typically get 55-75 to my Private server.  Its the being booted from party chat that was killing me. it would happen all night multiple times, and that was due to latency, parties drop like flies when they get a few seconds of latency/dropped packets, and (possibly UDP?) 

[RyzenFX]

For anybody who is able to do this on their CODA-4582 and has 10 or more devices connected to the modem and is affected by the UDP packet loss issue:

1. Try testing for the issue when you have 10 or more devices connected to it. 

2. Try testing for the issue when you have only 1 device connected to it.

If you can, please post the results of both tests including ping, packet loss, and state responsiveness of the game. I'm on to something and this should give me a good idea what's happening.

[Datalink]

Here is some food for thought on this.  For anyone running a pc based game, download and install Wireshark. Then, when you are in a game fire up Wireshark and run a short data capture while you're in game, actually doing some activity within the game.  The more activity on your part while engaging another external party, the better.  The goal is to cause/generate UDP packet transfer in both directions.  You probably only need somewhere between 2 to 5 min to capture enough data. Then stop the capture and save the file.  You don't want to run long captures just yet as the file size can grow pretty fast.  I'm sure there is a way to filter data and save that to a file but I havent' looked at Wireshark closely enough to determine exactly how to do that.  I would suggest starting with small files for now and work our way up if necessary.  

You don't necessarily have to be a Wireshark guru to read the data, although, it does help, and I'm not a Wireshark guru by any means.  But, you should be able to identify inbound and outbound UDP packets from the display and determine if any of the inbound packets indicate packet corruption, fragmentation or anything else.  

You could do a screen capture and send that to Dave via pm as it will contain your IP address and various IP addresses that your pc is communicating with.  Possibly Dave will request the file via pm, don't know at this point.

It would be interesting to see what turns up on the data capture.   

https://www.wireshark.org/

Edit:  Don't post a Wireshark screen capture to this or any other public forum due to the type and amount of IP addresses contained in the capture.  Or, in other words, for your own online security, keep your own data secure. 

[tonytoronto]

  You welcome.

   Kids are finally happy and wife too, no more complaning about internet not working.

   I like using my ipad while in the threadmill, usually watching Netflix or from somethig on the server, with CODA was getting disconnects 2-3 times every hour, so freaking annoying, plus trying to "fix" while walking was pain in the b***.  no more disconnects with 3552.

   Spent little time last night checking pings and such, to some servers might have gone up couple ms, but nothing that will make diference surfing or playing games. Theres couple little things with this modem too, but no biggie, all things i can leave with for now.

---

@lethalsniper wrote:

@tonytoronto@ so i picked up the  CGNM-3552 modem and me and brother both played online using the xbox one, and so far so good game feels more responsive and alot less laggy now dont know why but could be the UDP like you said Thanks @tonytoronto was having bad experience and very frustrated when on the coda modem.

@hopefully @RogersDave can rollout a udp fix for the coda modem because its  useless for gamers and there are a lot of us out there.

---

[Jeffj]

---

@Datalink wrote:

Here is some food for thought on this.  For anyone running a pc based game, download and install Wireshark. Then, when you are in a game fire up Wireshark and run a short data capture while you're in game, actually doing some activity within the game.  The more activity on your part while engaging another external party, the better.  The goal is to cause/generate UDP packet transfer in both directions.  You probably only need somewhere between 2 to 5 min to capture enough data. Then stop the capture and save the file.  You don't want to run long captures just yet as the file size can grow pretty fast.  I'm sure there is a way to filter data and save that to a file but I havent' looked at Wireshark closely enough to determine exactly how to do that.  I would suggest starting with small files for now and work our way up if necessary.  

You don't necessarily have to be a Wireshark guru to read the data, although, it does help, and I'm not a Wireshark guru by any means.  But, you should be able to identify inbound and outbound UDP packets from the display and determine if any of the inbound packets indicate packet corruption, fragmentation or anything else.  

You could do a screen capture and send that to Dave via pm as it will contain your IP address and various IP addresses that your pc is communicating with.  Possibly Dave will request the file via pm, don't know at this point.

It would be interesting to see what turns up on the data capture.   

https://www.wireshark.org/

Edit:  Don't post a Wireshark screen capture to this or any other public forum due to the type and amount of IP addresses contained in the capture.  Or, in other words, for your own online security, keep your own data secure. 

---

The only problem with this is, unles you have a managed switch with a monitoring port you're only going to see one side of the traffic, even in permiscuous mode. An alternitive is to man in the middle the connection but then you have to log the connection one way at a time so it isnt exactly matching in and out transactions, and it also causes you to only duplex at 10 base t when doing so.

So unless an end user has the equipment to do it, the data collected might not be very useful to @RogersDave

[Datalink]

Yup, absolutely agree, passive tap to log both inbound and outbound would be ideal.  Too bad this never occurs under ideal conditions 😞  You're correct in that it would only show the effects on one side of the traffic which is the inbound traffic.  But, its better than nothing and it would serve as a check to determine if there are issues with inbound UDP traffic for gaming or any other application.  Ideally, you would monitor both sides of the modem for both inbound and outbound traffic to see the before and after effects that are imposed by the modem.  But, that requires a much higher level of equipment and ability to match the packet frames.  Seems to me that the chipset manufacturers should be the ones doing this on a regular basis in order to ensure that the modem operates as it should.  It shouldn't be left to the tail end of the manufacturing and distribution chain to come up with methods to accomplish this. Personal opinion of course 😞

[Jeffj]

---

@Datalink wrote:

Yup, absolutely agree, passive tap to log both inbound and outbound would be ideal.  Too bad this never occurs under ideal conditions 😞  You're correct in that it would only show the effects on one side of the traffic which is the inbound traffic.  But, its better than nothing and it would serve as a check to determine if there are issues with inbound UDP traffic for gaming or any other application.  Ideally, you would monitor both sides of the modem for both inbound and outbound traffic to see the before and after effects that are imposed by the modem.  But, that requires a much higher level of equipment and ability to match the packet frames.  Seems to me that the chipset manufactures should be the ones doing this on a regular basis in order to ensure that the modem operates as it should.  It shouldn't be left to the tail end of the manufacturing and distribution chain to come up with methods to accomplish this. Personal opinion of course 😞

---

I agree with you, this should be Hitron soing tests on Reported gaming infrastructure and other reported service issues with UDP.

I have no problems letting Dave set up some sort of monitor on my Wan connection on his end but i dont even know if thats possible for him to track my I/O in that aspect.

One thing ive noticed pinging download.xbox.com which hits their CDN, is the thing flip flops back and forth between IPV6 and IPV4 and whn it does i get a latency spike.  I wonder if this is also salting the wound so to speak on the issues with gaming CDN's and UDP.

[Datalink]

Wonder why it doesn't operate solely on IPV4 or IPV6 instead of switching back and forth?  You should pm Dave with the details and logs if you have any that show that.

[Jeffj]

Trying to ponder on a way to send this data to @RogersDave in some sort of meaningful format as my trial of PinPlotter has expired so it only shows real time data and ive just happened to catch it in the ipv4 ipv6 flip flop act a few times today. 

[Datalink]

You can see red arrows on the bottom? And if you hover the mouse over the arrow it displays the IP address change?

I've seen that over the last couple of days, running a ping test to google.ca where the IP address for google changes, and then changes back.  The first change results in a slightly higher ping time and remains that way until it changes back.  This is all with IPV4 as I have IPV6 turned off for now.   The flip flop from IPV4 to IPV6 and back is a different twist on this.  

With the freebie pingplotter, the data will still be there, but you won't be able to access it.  Anyone with the Standard or Pro version will be able to access the file if you export the sample set.  I run the standard version.

Edit:  If you go Edit .... Copy as Image and then paste that into something like MS Paint, you will see the notes at the bottom of the image which includes the details of the events that are marked by the markers or arrows.  At least that's what the Standard version does.  I would expect the freebie version to do the same, unless they cut that off as well.   So, it should be a matter of waiting until one of these appears on the display, then run the Copy as Image function and dump it into something to see if the notes show up at the bottom left of the image. 

[Jeffj]

---

@Datalink wrote:

You can see red arrows on the bottom? And if you hover the mouse over the arrow it displays the IP address change?

I've seen that over the last couple of days, running a ping test to google.ca where the IP address for google changes, and then changes back.  The first change results in a slightly higher ping time and remains that way until it changes back.  This is all with IPV4 as I have IPV6 turned off for now.   The flip flop from IPV4 to IPV6 and back is a different twist on this.  

With the freebie pingplotter, the data will still be there, but you won't be able to access it.  Anyone with the Standard or Pro version will be able to access the file if you export the sample set.  I run the standard version.

What are you pinging that you see the shift from IPV4 to IPV6?

Edit:  If you go Edit .... Copy as Image and then paste that into something like MS Paint, you will see the notes at the bottom of the image which includes the details of the events that are marked by the markers or arrows.  At least that's what the Standard version does.  I would expect the freebie version to do the same, unless they cut that off as well. 

---

Yep seeing the red arrow and seeing the change from ipv4 to 6.  when it happens it flips and then 5 min later flops. I'll start up the test again and get some data and take a image.

download.xbox.com

Further more i am seeing it switch me from local CDN servers and then to the netherlands as well some times, and some tiems seeing my route go through Verizons layer in the states and soem times it doesnt. its realy unstable. 

[Datalink]

More than a little strange considering that Pingplotter should be using an IP address.  But, if you are using download.xbox.com as the target, I wonder if pingplotter goes out with a DNS request and gets a different response at some point?  That would show up on a Wireshark log perhaps?  What do you use for DNS servers?  I'm using OpenDNS and see and address changes which are all within the Rogers IP address ranges.  If you're using the Rogers DNS and seeing an address shift from IPV6 to IPV4 and back, thats a little interesting as well.  

Just thinking about this I'm using Google.ca as a target.  I'll have to give this a try with an IP address to see what happens.  Maybe this is nothing more than a DNS problem at the DNS servers?  

[Jeffj]

---

@Datalink wrote:

More than a little strange considering that Pingplotter should be using an IP address.  But, if you are using download.xbox.com as the target, I wonder if pingplotter goes out with a DNS request and gets a different response at some point?  That would show up on a Wireshark log perhaps?  What do you use for DNS servers?  I'm using OpenDNS and see and address changes which are all within the Rogers IP address ranges.  If you're using the Rogers DNS and seeing an address shift from IPV6 to IPV4 and back, thats a little interesting as well.  

Just thinking about this I'm using Google.ca as a target.  I'll have to give this a try with an IP address to see what happens.  Maybe this is nothing more than a DNS problem at the DNS servers?  

---

I am using Rogers DNS. I decided to stick with it for testing, i typically use googles DNS. 

THis could very well be a dns issue I agree, I just find it curious as once pingplotter resolves a name, and gets its target it should stick to it unless its forced to relookup?

[Pejmank]

Hi, 

When I found this forum I was happy that I'm not the only one having issues with Gig Internet.

I followed all the steps here:

-Changed modem to one with 2 black dots.

-reboot and tested with and without bridge mode

-asked tech to come and check my signal 

-tested with one device, multi devide, waited to have the latest firmware and anything else that I know of

but still having issues with download speed.

my Modem firmware

Hardware Version	1A
Software Version	2.0.10.24

also this is what my modem shows in the Docsis wan

Downstream Overview
Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Signal noise ratio (dB)
1	615000000	256QAM	5.500	11	38.983
2	561000000	256QAM	5.800	2	38.605
3	567000000	256QAM	5.900	3	37.636
4	573000000	256QAM	6.700	4	37.356
5	579000000	256QAM	6.400	5	38.605
6	585000000	256QAM	6.400	6	38.605
7	591000000	256QAM	6.000	7	38.605
8	597000000	256QAM	6.100	8	38.605
9	603000000	256QAM	5.700	9	38.605
10	609000000	256QAM	5.700	10	38.983
11	555000000	256QAM	6.300	1	38.605
12	621000000	256QAM	5.200	12	38.605
13	633000000	256QAM	4.800	13	38.605
14	639000000	256QAM	5.800	14	38.983
15	645000000	256QAM	4.800	15	38.605
16	651000000	256QAM	4.400	16	38.605
17	657000000	256QAM	5.700	17	38.605
18	663000000	256QAM	6.500	18	38.983
19	669000000	256QAM	5.200	19	38.605
20	675000000	256QAM	4.800	20	38.605
21	681000000	256QAM	5.800	21	38.605
22	687000000	256QAM	6.400	22	38.605
23	693000000	256QAM	5.100	23	37.356
24	699000000	256QAM	5.500	24	37.636
25	705000000	256QAM	5.700	25	37.636
26	711000000	256QAM	5.700	26	37.356
27	717000000	256QAM	5.600	27	37.356
28	723000000	256QAM	5.200	28	37.356
29	825000000	256QAM	2.500	29	36.387
30	831000000	256QAM	2.500	30	36.610
31	837000000	256QAM	2.200	31	36.610
32	843000000	256QAM	1.600	32	35.780
OFDM Downstream Overview
Receiver	FFT type	Subcarr 0 Frequency(MHz)	PLC locked	NCP locked	MDC1 locked	PLC power(dBmv)
0	NA	NA	NO	NO	NO	NA
1	4K	275600000	YES	YES	YES	6.000000

Upstream Overview
Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	38595609	ATDMA - 64QAM	45.750	3	3200000
2	30596000	ATDMA - 64QAM	41.500	1	6400000
3	23700000	ATDMA - 64QAM	40.750	2	6400000
OFDM/OFDMA Overview
Channel Index	State	lin Digital Att	Digital Att	BW (sc's*fft)	Report Power	Report Power1_6	FFT Size
0	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K
1	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K

 anyone have any idea what to try next?

@RogersDave I really appreciate if you can take a look and help me to figure this out. I'm dealing with this since Oct last year!

[JohnBeaudin]

@RogersDave

Will .26 be ready by the end of the week? I can't wai to give you feedback on modem reboots, mine reboot average twice a day 🙂

[RyzenFX]

@JohnBeaudin Hopefully it releases soon. I'm really curious to see how it performs in gaming with say 5-8 devices connected to it.

EDIT: Firmware .26 will be released today. Please let me know if the UDP issue is still present.

[JohnBeaudin]

@RyzenFX

Fingers crossed!

[JohnBeaudin]

@RyzenFX

Out of curiosity how do you know .26 will be released today?

[RogersDave]

Good morning Community,

I just updated the release notes for firmware 2.0.10.26 in this post.

I will be starting the upgrades of non CODA modems today. For CODA-4582 I don't know yet if I'll be able to push them today or only start tomorrow.

For technical background, the CODA-4582 are currently configured to automatically upgrade on boot to version 2.0.10.24. If I upgrade any modem at this point, it will automatically downgrade back at reboot. We are making a change in our configuration file and I'll start the deployment as soon as the configuration is in place.

Now there is an important change in process for trial participants on the CODA modem. When I'll push the trial firmware, your modem will be a in state where it ignores upgrade commands from the network (so that it doesn't downgrade back to 2.0.10.24). However, if you perform a factory reset, the modem will automatically be set back in network upgrade mode and go back to 2.0.10.24.

The new configuration file will also include the "Pixel/Android fix" that greatly improves the stability of WiFi connections for Android devices. Because this will be included in the configuration file, all modems will receive this fix with a reboot (no need to be part of the trial or on firmware 2.0.10.26). 

As a side note, the performance/range of 5 GHz channels 36-40-44-48 on the CODA modem seems to be really poor. I recommend to everybody using WiFi functionality on the CODA-4582 to manually set their 5 GHz channel to 149-153-157-161.

Dave

[bahha]

Missing downsteam problem is back on a black sticker CODA-4582. Channels 2 to 10 are gone,  was connecting to all channels previously.  Not sure if it's realated to the main line break in my neighbourhood, a temp line was installed as a thaw is needed to bury it.  First RCS partial service error happened on March 14. 

[Triple_Helix]

---

@bahha wrote:

Missing downsteam problem is back on a black sticker CODA-4582. Channels 2 to 10 are gone,  was connecting to all channels previously.  Not sure if it's realated to the main line break in my neighbourhood, a temp line was installed as a thaw is needed to bury it.  First RCS partial service error happened on March 14. 

---

Unplug the Modem from the wall for about 10 to 15 seconds then plug it back in and check again when it's done.