CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[Telek]

Thanks again, @RogersDave!

[Alex4161]

@RogersDave

Thanks for pushing the update.  Could you or someone please remind me what benefit would I see if I enable DFS?

Thanks

[Telek]

DFS opens up more channels which would otherwise be inaccessible in Canada, @Alex4161

If you live in an area where there are a lot of 5G network users (like, say, a cluster of 5 condos), then those networks will typically be a lot less congested.

[RyzenFX]

---

@RogersDave wrote:

Community,

Quick update for everybody. I just pushed code 2.0.10.31 to some of you. Based on feedback, I will continue deployment in the coming days (and will resume firmware upgrades for new participants as well).

Code 2.0.10.31 on CODA is the same as 2.0.10.29 with the following changes:

• Band steering is visible in GUI (advanced WiFi 2.4 GHz and 5 GHz)
• DFS is visible in GUI (advanced WiFi 5GHz)
• Available DFS channel list updated
• USB function control displayed in gateway function GUI

Dave

---

@RogersDave I can confirm that all those features are present on 2.0.10.31. I have done some initial testing with this firmware and all the WiFi speeds and Wired speeds are as normal. In terms of WiFi speed complaints on the DFS channels, they have now been resolved with this firmware. 

Be advised that if you do switch to one of the DFS channels, the SSID will take 2-3 minutes longer to appear on your device than if you were on the non-DFS channels. 

The latency problem reported on 2.0.10.30T1 appears to be gone from my initial testing with League of Legends. My ping is averages 24-27ms on this firmware compared to 30-35ms on .30T1. I will see what others have to say about the latency in the firmware, but from what I have tested everything seems to be working nicely so far. 

[nobs2016]

Two disruptions within the last 2 hours?

WAN up made it to 17 hours this morning then THREE disruptions as I write this.

And there goes the VOIP ATA.

Will .31 resolve?

Bell just installed their FTTH access right across the street. Wonder if they're any better?

[Alex4161]

Hello @RogersDave

I did some speed tests on the 5 Ghz with DFS on and DFS off and here are my results:

Connected to Modem via CAT 6:

5 Ghz DFS On:

5 Ghz DFS Off:

I tried multiple speed tests around the same time to rule out sampling differences.  It appears that the extra channels with DFS on are causing some performance issues.  My WiFi card on my laptop is an Intel AC-7260 running the latest drivers (18.00.7.2)

[Telek]

@Alex4161 do you know what channels you were using for your DFS on and off tests?  Keep in mind since you're (probably) using different channels, the results can be different.

Also, clearly, the DigitalOcean speed test isn't reliable based on the wired results being way too high.  Suggest running on speedtest or DSL reports.

Keep in mind that when DFS is enabled, the AP:

• Looks for radar detection before securing a frequency channel.
• Scans continuously for radar signal patterns during normal operation.

If you're using a DFS enabled channel, there may be some loss in order to accomplish the second.  However, in the event that you are in an area that has congested 5GHz bands, that loss may be less than having to share the non-DFS channel with other users.

[Alex4161]

@Telek

Thanks for letting me know.  

Using http://www.nirsoft.net/utils/wifi_information_view.html, I scanned 26 WiFi networks in my area and only 1 additional 5 Ghz network in my area on channel 36.  I just turned DFS off as I clearly don't have any issues with congestion on the 5 Ghz band.  🙂

[gp-se]

---

@Telek wrote:

@Alex4161 do you know what channels you were using for your DFS on and off tests?  Keep in mind since you're (probably) using different channels, the results can be different.

Also, clearly, the DigitalOcean speed test isn't reliable based on the wired results being way too high.  Suggest running on speedtest or DSL reports.

Keep in mind that when DFS is enabled, the AP:

• Looks for radar detection before securing a frequency channel.
• Scans continuously for radar signal patterns during normal operation.

If you're using a DFS enabled channel, there may be some loss in order to accomplish the second.  However, in the event that you are in an area that has congested 5GHz bands, that loss may be less than having to share the non-DFS channel with other users.

---

I Agree complexity with this, different channels can make a huge difference in speed. The best thing is to run a WiFi analyzer and see what channels are in use, then choose a channel that isn't in use/low number of users. A good example is when I was using channel 149 and over WiFi it would max out around 600mb/s. Changing to channel 40 and I can speed test at 750-800mb/s over WiFi. I am using a TP-Link Archer C3150 set as an access point going though a PF Sense box with the CODA in bridge mode. 

One more thing about DFS, they are dynamic channels so performance can be all over the place, if you aren't in a congested area I would say keep it turned off. 

[ishine80]

@Datalink I can confirm I only have IPV4 enabled. So if the issue is modem to CMTS issue how do I fix it? Is really love to have no more issues with the Internet. It's been over a month of us dealing with this. @RogersDave if as @Datalink suggests and the issue is modem to CMTS what do we do next?

[mednorth]

@RogersDave

I've been monitoring my latency issues, as I'm coming up on 5 days left on my PingPlotter trial.

No change in the issue at all, here are a couple quick graphs from this morning.

A 5min run of the regular (and horribly annoying) ping spikes:

And a 10min run that captured a very fun looking mountain solid  800ms+ spike:

Both of these were run hardwired to the CODA-4582, with that computer being the only connection to the modem and with wifi disabled. I'm still on .27, and being that .27 was supposed to fix latency issues would this be a larger network issue? I've discussed this with @datalink earlier and would love to have some network input on it. My end-to-end momentary lag is causing a lot of gaming issues, of which I didn't have until the last few months.

Tech support has repeatedly run 'tests' and all is okay. Help, please!

[tmooney]

CODA-4582 VPN disconnects

Hello fellow Tech-enthuasists , I have started a new Work-From-Home job and have to connect to a VPN to do business, this is a Paid VPN service by the way.

I have talked to at least 3 tech's and have spoken to the online tech support chat countless of times, there seems to be an ongoing problem with Rogers ISP and VPN. 

I have read through the similar threads about other routers problems and VPN and have tried many of the tricks (disable usb storage,ect).

Im almost at my wits end as I'm down 2 days pay now because of this, I have done many networking courses and I work in I.T myself, so you can understand how much its driving me nuts not being able to get it working.

Can anyone shed some light on what else I can try if anything?

[Datalink]

@tmooney are you running the modem in Gateway or Bridge mode?  If its in Gateway mode try disabling IPV6 to see if that improves the VPN performance.  

Log into the modem, navigate to the BASIC .... GATEWAY FUNCTION tab and select IPV4 only for the Router mode.  Save the data.  It will take the modem about three to four minutes to switch to IPV4 only mode.  Personal opinion, reboot the modem, ADMIN .... DEVICE RESET and reboot the pc to switch both to IPV4 only.  To return to Dual stack (IPV4 and IPV6) mode, log back into the modem, navigate to that tab, select DUAL (Stack) and save the data.  Once again, I would reboot both the modem and pc to ensure that they pick up the IPV4 and IPV6 addresses. 

If you have the modem in Bridge mode with a follow-on router, I would disable IPV6 in the router as well as an experiment and reboot both the router and pc.  

Have you checked the VPN protocol type to determine if any of the SECURITY .... PASS-THROUGH VPN types require enabling?

One last item comes to mind and that is to kick the modem into Bridge mode, connect the pc or laptop and try the VPN.  In Bridge mode the modem is just that, a modem only, so, no services like the firewall, or other security features.  I would connect the pc or laptop, fire up the VPN and see if it functions.  Run that for a minute or two and then disconnect the VPN and then log back into the modem to kick it back into Gateway mode. 

To switch the modem from Gateway to Bridge mode, log into the modem using 192.168.0.1, or 192.168.100.1.  Either one will work in Gateway mode.  Navigate to the same BASIC .... GATEWAY FUNCTION page and disable the Residential Gateway Function.  Save the change and the modem will reboot into Bridge mode.  

Can you give that a go and let us know what you find?

to switch the modem from Bridge mode to Gateway mode, log into the modem using 192.168.100.1.  That is the only address that will work in Bridge mode.   Navigate to the same BASIC .... GATEWAY FUNCTION page and enable the Residential Gateway Function.  Save the change and the modem will reboot into Gateway mode with it previous settings intact. 

[Datalink]

@ishine80 the next step is to monitor the modem to Cable Modem Termination System (CMTS) connection.  In simple terms, ping the CMTS on a continual basis, looking for packet loss, disconnects and latency.  The CMTS is essentially the neighborhood fibre to copper converter and it supports all of the neighborhood Rogers modems, either internet, cable tv or home phone, as well as any TPIA provider modems that run off of the Rogers network.  

There are a couple of ways to do this.  The first is to download and install Pingplotter.  It will run in a Pro mode for 14 days before locking down into a basic freebie mode if you don’t buy a Pro or Standard license.  The pro mode allows you to view 7 days of data compressed into one screen, the standard allows you to view a max of two days of data in one screen, even though its from a larger data file, and the freebie mode allows you to only view the last 10 minutes of data, and you can’t scroll back in time as you can with the pro or standard licenses.  In the case of the Pro or Standard modes, you can scroll back and forth along the data to determine what has occurred in the past.  

If you have an unlimited internet plan, I would set Pingplotter to ping the CMTS at 1 second intervals.  Pingplotter will display any occasional packet loss and keep running.  This might answer one question of whether or not the problem here is with an occasional packet loss, or if you’re seeing complete disconnects from the CMTS.

To determine how to set up Pingplotter, have a look at the following post:

http://communityforums.rogers.com/t5/Internet/Intermittent-disconnects/m-p/364602#M35528

The next method of doing this is to run HRPing, which is a freebie command line application and set it to post the ping results to a log file.  Then you can scroll thru the log file to determine if there have been any packet loss or disconnect instances.  In this case, numerous successive packet loss reports would be an indication of a disconnect.  HRPing will keep pinging, so, if the modem recovers at some point you should see that reflected in the log, unless of course the pc or laptop does not recover from the loss of internet connectivity.  

HRPing can be download from here:

https://www.cfos.de/en/ping/ping.htm

Download HRPing and unzip the files to a folder that is easy to navigate to with a command prompt.  Start a Command prompt with Administrator rights and navigate to the folder where you placed the files.  Enter the following command:  

Tracert www.google.ca

That will produce a trace to google.  All you want is the second address shown in the list, assuming that the first address is your modem’s address.  

The command for running HRPing is as follows:

hrping 99.239.48.x -t -F c:\temp\24Juntest.txt -T

In my case the second address is the CMTS address, 99.239.48.x, as determined from the trace to google.  Yours will be different.  

The -t command runs the ping application until you use Control C to bail out of it.

The -F command writes the result to the file of your choosing.  In my case I dumped the results to a file in the C:\temp directory.  You can choose your own folder destination and file name.  Include the .txt file type.  HRPing will create the file and keep updating it with the results.  

The -T command writes a time stamp on each line of the results file.  The windows ping command does not time stamp the data, so if there is any disconnect or packet loss, you don't know when that occurred.  HRPing, with its time stamp capability shows when the disconnect starts and stops, same with any packet loss.

The -s 1000 command results in 1 second, or 1000 ms between each successive ping.

On the first run, you will have to step thru the licence details and agree to the licence and warranty conditions.  After you accept the conditions, HRPing will start and keep running until you stop it.  

Ok, here’s an example of the results:

hrping 99.239.48.x -t -F c:\temp\24Juntest.txt -T -s 1000

This is hrPING v5.07.1148 by cFos Software GmbH -- http://www.cfos.de

2017-07-24 01:06:39.534: 99.239.48.x -t -F c:\temp\24Juntest2.txt -T -s 1000

Source address is 10.0.0.8; using ICMP echo-request, ID=5000

Pinging 99.239.48.x [99.239.48.x]

with 32 bytes data (60 bytes IP):

2017-07-24 01:06:46.559: From 99.239.48.x: bytes=60 seq=0008 TTL=63 ID=5783 time=10.289ms

2017-07-24 01:06:47.557: From 99.239.48.x: bytes=60 seq=0009 TTL=63 ID=5784 time=8.680ms

2017-07-24 01:06:48.560: From 99.239.48.x: bytes=60 seq=000a TTL=63 ID=5785 time=11.295ms

2017-07-24 01:06:49.557: From 99.239.48.x: bytes=60 seq=000b TTL=63 ID=5786 time=8.112ms

2017-07-24 01:06:53.319: From 10.0.0.8: host unreachable; bytes=88 SEQ=000e TTL=64 ID=0029 time=694.759ms

2017-07-24 01:06:54.550: Timeout waiting for seq=000c

2017-07-24 01:06:54.550: Timeout waiting for seq=000d

2017-07-24 01:06:56.550: Timeout waiting for seq=000f

2017-07-24 01:06:56.550: Timeout waiting for seq=0010

2017-07-24 01:06:58.550: Timeout waiting for seq=0011

2017-07-24 01:06:58.550: Timeout waiting for seq=0012

2017-07-24 01:06:59.319: From 10.0.0.8: host unreachable; bytes=88 SEQ=0014 TTL=64 ID=002b time=769.152ms

2017-07-24 01:07:00.549: Timeout waiting for seq=0013

2017-07-24 01:07:01.771: From 99.239.48.x: bytes=60 SEQ=0017 TTL=63 ID=5787 time=13.746ms

2017-07-24 01:07:02.561: Timeout waiting for seq=0015

2017-07-24 01:07:02.561: Timeout waiting for seq=0016

2017-07-24 01:07:02.568: From 99.239.48.x: bytes=60 seq=0018 TTL=63 ID=5788 time=7.315ms

2017-07-24 01:07:03.572: From 99.239.48.x: bytes=60 seq=0019 TTL=63 ID=5789 time=14.400ms

2017-07-24 01:07:04.557: From 99.239.48.x: bytes=60 seq=001a TTL=63 ID=578a time=8.117ms

2017-07-24 01:07:05.572: From 99.239.48.x: bytes=60 seq=001b TTL=63 ID=578b time=16.293ms

2017-07-24 01:07:06.558: From 99.239.48.x: bytes=60 seq=001c TTL=63 ID=578c time=8.953ms

2017-07-24 01:07:07.597: From 99.239.48.x: bytes=60 seq=001d TTL=63 ID=578d time=48.280ms

2017-07-24 01:07:08.559: From 99.239.48.x: bytes=60 seq=001e TTL=63 ID=578e time=9.171ms

[Aborting...]

Packets: sent=36, rcvd=25, error=2, lost=9 (25.0% loss) in 35.009182 sec  

RTTs in ms: min/avg/max/dev: 6.758 / 67.670 / 769.152 / 188.786     

Bandwidth in kbytes/sec: sent=0.061, rcvd=0.047   

I deliberately disconnected the pc to simulate an internet connection loss and reconnected it. The results show the loss of connection.  You can see the timeout indications as well as a calculated time from the last time that the target was unreachable.  So, this makes it easy to scroll thru the file looking for sequential time outs which indicate that a disconnect has occurred.  

If your on a limited internet plan, you can back off of the interval time, maybe to something like 5 or 10 seconds, 5000 ms or 10000 ms or greater.  

That test needs to run via ethernet, so that you have connected laptop, modem and then CMTS, without any wifi issues mixed with this.  The purpose of this is to determine if there are internet disconnects, how many and how long.  If you get into a time period where you see continuous disconnects thats the time to call tech support and ask the CSR to ping the modem from the CMTS, or, ping the CMTS from the modem.  Either one will work.  When you see a disconnect, the CSR should see that as well.  The next question is, is it just your modem that is affected, or your neighbors as well.  That's up to the CSR to determine as it will also determine what tech is dispatched to look at the issue.   

This can also be used for wifi investigation purposes, pinging the modem to determine how badly any particular wifi channel might be affected by interference on the channel or affected by interference from adjacent channels in the 2.4 Ghz band.  Although there are applications that will indicate how many modems or routers might be on a particular channel, looking at the pingplotter or HRPing result will give a user a good idea of the actual amount of packet loss that might be incurred and if compared to another channel, help determine if any other channel is any better in terms of remaining connected to the modem or router.

Just to note, the above example is run with the CODA-4582 with V2.0.10.31 loaded.  The ping times to the CMTS and back are in the range of 7 to 14 ms, which is pretty standard.  Then towards the bottom of the results is single ping at 48 ms, which isn’t standard.  There is something amiss with the latest trial firmware (personal opinion at this point).  Those times should all stay within a reasonable low timeframe and shouldn’t be bouncing up into the 100 ms range as I’ve seen in ping tests so far.   

Is your exterior cable underground from the local tap (near your home) or overhead from a utility pole?

[Telek]

---

@Datalink wrote:

If you have an unlimited internet plan, I would set Pingplotter to ping the CMTS at 1 second intervals.

---

Great suggestions, @Datalink! You don't need to worry about unlimited plans for doing ping tests though.

A ping total payload size is about 74 bytes IIRC (with 32 bytes payload).  That's 6.4MB/day and 192MB/month. Not something to worry about regardless of your network cap.

[tmooney]

Hey I have tried all of the above suggestions and still getting the same issue. I'm running out of options here.

The disconnects seem to happen when a program using voip gets logged into, Ive checked the manufactures site for the ports they need so I done port triggering for those ports, but still the same issue. Anyone think of something else?

This is very time sensitive.

[Datalink]

@tmooney, if you're running port triggering I'd do the following:

1.  Assign a static LAN address to the pc/device that uses VOIP.  Navigate to BASIC .... LAN SETUP and use the DHCP Reservation function to assign the static address.

2.  Navigate to BASIC .... GATEWAY FUNCTION and disable the SIP ALG.

3.  Delete the port triggering

4.  Reboot the modem.

5. After the reboot, assign port forwarding rules using the same ports that were used for Port Triggering.  Assign the port forwarding rules to the static LAN IP address.

6.  Reboot the modem. 

Give that a go and see what happens.  I suspect that the SIP ALG is already disabled.  If not, please disable it as it can interfere with the server to VOIP signalling.  There are multiple signalling protocols for VOIP apparently, so its hit and miss as to whether any modem manufacturer gets it right for your particular VOIP device or application.  Usually, disabling the modems SIP ALG function allows the VOIP server to VOIP device signalling to run properly. 

So, just to confirm, you ran the modem in Bridge mode with a direct connection to the pc/laptop and ended up with the same problems?  Thats very unusual as the modem, in Bridge mode shouldn't interfere with end device operation unless perhaps there are still ports that are used by Rogers that can't be used by the end device.  @RogersDave would have to confirm that one. 

[tmooney]

RogersDave is aware of the issue, seems to be an issue with company+vpn+rogers. Looks like I might have to get different internet to fix the issue, its very unfortunate but it seems to be my only option as of right now.

[Datalink]

@tmooney if Dave is already aware of the issue and doesn't have a solution in mind, there isn't much that I or anyone else can help with, unfortunately.  Do you have the ability to select the protocol type for the VPN?  If so, that might be worth looking at.

[Alex4161]

Here is a good article to read for those who are interested in what DFS (Dynamic Frequency Selection) does:

https://www.smallnetbuilder.com/basics/wireless-basics/33082-smallnetbuilder-s-wi-fi-dynamic-frequen...

[Triple_Helix]

Working well with the new Firmware .31!

Daniel