CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[4kgigabitguy]

Thanks @Mythen

On the computer that's directly connected to the modem, can you run a speed test at speed check rogers & dsl reports & post them here?? Are you getting the issue with both the devices using the powerline adapter & the computer that's directly connected? Performance with powerlines has been hit or miss for me (would love to hear about your experience in separate note) 

What would also be important is, whether or not the issue is happening all throughout the day or only at specific times...If it's only at specific times then I would also contact rogers during the times you are experiencing the problem & ask them to have a look at the area/node for any known issues as it may require further investigation. 

[Mythen]

---

@4kgigabitguy wrote:

Thanks @Mythen

On the computer that's directly connected to the modem, can you run a speed test at speed check rogers & dsl reports & post them here?? Are you getting the issue with both the devices using the powerline adapter & the computer that's directly connected? Performance with powerlines has been hit or miss for me (would love to hear about your experience in separate note) 

What would also be important is, whether or not the issue is happening all throughout the day or only at specific times...If it's only at specific times then I would also contact rogers during the times you are experiencing the problem & ask them to have a look at the area/node for any known issues as it may require further investigation. 

---

http://www.dslreports.com/speedtest/13472253

[StephKitty]

---

@Mythen wrote:

last few days I am getting crazy lag spikes.  to the point where anything I watch online buffers for a moment or 2.  What would cause that?  Anyone else seeing this?

---

I'm getting major lag spikes while playing online, as well as stuttering when streaming Netflix. Some games are downright unplayable right now, it's pretty horrendous.  Could it be IPV6 messing things up?  Prior to the .24 firmware update, I had no such issues.  I'm not sure what it is, but my connection also slows to a about 300mb down max on a gigabit connection.  

This is unacceptable, such ridiculous issues with this gigabit service from day one. 

[Alex4161]

Hello @RogersDave

I just got back from holidays and it looks like the Downstream OFDM channel drops are still happening for me and my modem speed drops to under 1 mb/s on the downstream.  I am only able to get connectivity after rebooting (i.e. power cycling the modem).  Please note that this is my 3rd CODA modem (2 black dot version).  My modem info is as follows: Hardware Version
1A
Software Version
2.0.10.26T2

Downstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Signal noise ratio (dB)
1	615000000	256QAM	-4.400	11	35.595
2	561000000	256QAM	-5.600	2	33.834
3	567000000	256QAM	-6.100	3	33.834
4	573000000	256QAM	-4.800	4	33.957
5	579000000	256QAM	-3.600	5	34.346
6	585000000	256QAM	-2.800	6	35.084
7	591000000	256QAM	-1.500	7	36.610
8	597000000	256QAM	-1.000	8	36.387
9	603000000	256QAM	-1.500	9	36.387
10	609000000	256QAM	-2.900	10	35.780
11	555000000	256QAM	-4.800	1	34.484
12	621000000	256QAM	-5.600	12	34.484
13	633000000	256QAM	-5.300	13	34.926
14	639000000	256QAM	-4.700	14	35.084
15	645000000	256QAM	-4.000	15	35.595
16	651000000	256QAM	-4.700	16	35.595
17	657000000	256QAM	-5.300	17	35.084
18	663000000	256QAM	-4.500	18	35.084
19	669000000	256QAM	-4.500	19	34.926
20	675000000	256QAM	-4.600	20	35.595
21	681000000	256QAM	-4.400	21	35.595
22	687000000	256QAM	-4.400	22	35.595
23	693000000	256QAM	-5.300	23	35.595
24	699000000	256QAM	-5.500	24	34.926
25	705000000	256QAM	-6.600	25	34.926
26	711000000	256QAM	-8.000	26	34.346
27	717000000	256QAM	-8.900	27	33.487
28	723000000	256QAM	-9.900	28	32.676
29	825000000	256QAM	-7.100	29	34.926
30	831000000	256QAM	-7.300	30	34.926
31	837000000	256QAM	-7.800	31	34.484
32	843000000	256QAM	-7.800	32	34.926

OFDM Downstream Overview

Receiver	FFT type	Subcarr 0 Frequency(MHz)	PLC locked	NCP locked	MDC1 locked	PLC power(dBmv)
0	NA	NA	NO	NO	NO	NA
1	4K	275600000	YES	YES	YES	0.599998

Upstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	23700000	ATDMA - 64QAM	30.000	2	6400000
2	38595766	ATDMA - 64QAM	35.250	3	3200000
3	30596000	ATDMA - 64QAM	31.500	1	6400000

OFDM/OFDMA Overview

Channel Index	State	lin Digital Att	Digital Att	BW (sc's*fft)	Report Power	Report Power1_6	FFT Size
0	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K
1	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K

[RyzenFX]

@Alex4161 

From what I'm seeing there could be something that could be interfering with your signal (such as noise). As you can see in your downstream overview, the signal strength ranges from -1 all the way to -9.9 and the signal to noise ratio looks lower than usual. 

I reccomend you to call the Rogers Internet support line an have them check if your node is being affected by some sort of noise or interference. Also, have them run a line test to your modem and ask them if they can send a technician to your place so the technician can do some work and try to increase your signal strengths. 

[Alex4161]

@RyzenFX

Thanks for your reply.

I had a Rogers tech come to my home a few weeks ago and he tested the area and found no issues.  He also swapped the modem to the 2 black dot version. 

I replaced the unity gain amp on the line with another amp and my signal levels look better now:

Downstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Signal noise ratio (dB)
1	615000000	256QAM	-1.100	11	35.780
2	561000000	256QAM	-2.100	2	34.926
3	567000000	256QAM	-2.800	3	34.926
4	573000000	256QAM	-1.300	4	35.084
5	579000000	256QAM	-0.100	5	35.084
6	585000000	256QAM	0.600	6	35.780
7	591000000	256QAM	2.000	7	36.610
8	597000000	256QAM	2.400	8	36.610
9	603000000	256QAM	2.000	9	36.610
10	609000000	256QAM	0.400	10	36.387
11	555000000	256QAM	-1.300	1	35.595
12	621000000	256QAM	-2.100	12	35.595
13	633000000	256QAM	-1.900	13	35.780
14	639000000	256QAM	-1.300	14	35.595
15	645000000	256QAM	-0.600	15	35.595
16	651000000	256QAM	-1.400	16	35.780
17	657000000	256QAM	-1.900	17	35.780
18	663000000	256QAM	-1.100	18	35.780
19	669000000	256QAM	-1.000	19	35.595
20	675000000	256QAM	-1.100	20	35.780
21	681000000	256QAM	-0.800	21	35.780
22	687000000	256QAM	-0.800	22	36.387
23	693000000	256QAM	-1.600	23	36.387
24	699000000	256QAM	-1.900	24	35.595
25	705000000	256QAM	-3.000	25	35.780
26	711000000	256QAM	-4.500	26	35.084
27	717000000	256QAM	-5.500	27	34.346
28	723000000	256QAM	-6.600	28	33.377
29	825000000	256QAM	-3.400	29	35.084
30	831000000	256QAM	-3.600	30	35.595
31	837000000	256QAM	-3.900	31	35.084
32	843000000	256QAM	-4.000	32	35.084

OFDM Downstream Overview

Receiver	FFT type	Subcarr 0 Frequency(MHz)	PLC locked	NCP locked	MDC1 locked	PLC power(dBmv)
0	NA	NA	NO	NO	NO	NA
1	4K	275600000	YES	YES	YES	4.800003

Upstream Overview

Port ID	Frequency (MHz)	Modulation	Signal strength (dBmV)	Channel ID	Bandwidth
1	23700000	ATDMA - 64QAM	38.250	2	6400000
2	38595707	ATDMA - 64QAM	44.250	3	3200000
3	30596000	ATDMA - 64QAM	39.500	1	6400000

OFDM/OFDMA Overview

Channel Index	State	lin Digital Att	Digital Att	BW (sc's*fft)	Report Power	Report Power1_6	FFT Size
0	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K
1	DISABLED	0.5000	0.0000	0.0000	-inf	-1.0000	4K

[JohnBeaudin]

LAN Up Time 004 days 19h:18m:06s Beautiful!, since D3.1 and .26T2 never been that stable!

[gp-se]

---

@Alex4161 wrote:

@RyzenFX

Thanks for your reply.

I had a Rogers tech come to my home a few weeks ago and he tested the area and found no issues.  He also swapped the modem to the 2 black dot version. 

I replaced the unity gain amp on the line with another amp and my signal levels look better now

---

Be careful when adding amps, they amplify the entire signal, including the noise on the line and can make things worse. It's best to get Rogers to come and balance the lines using their equipment/amps if needed.

[asns]

Recently we have upgraded our Rogers modem to Hitron CODA-4582 and since then I am not able to access to our work place servers through WIFI, I use a VPN to connect to our company network but no website is accessible. Please advise

[gp-se]

---

@asns wrote:

Recently we have upgraded our Rogers modem to Hitron CODA-4582 and since then I am not able to access to our work place servers through WIFI, I use a VPN to connect to our company network but no website is accessible. Please advise

---

does hard wire ethernet work? or only WiFi is giving you troubles with VPN? 

[Windwalker]

---

@Windwalker wrote:

So I signed up for 1000/50 in June 2016.   Download speeds have been all over the place, but my uploads were always fairly consistent at around 50.

Lo and behold, as of yesterday I can't get over 30 UP.  Was my profile changed on the system?  WTH!

Also, I have noticed that speeds for Gigabit are infuenced by how current a computer and processor I am using.  I just set up an i5 third gen computer and ran the speed test without antivirus installed.  The rogers speedtest resulted in 945/50 prior to yesterdays upload change.  As well the download graph was flat with no spikes or dropouts.

DSL reports yelded similar results.

@RogersDaveare you looking at the impact of antivirus software on speeds in your testing?

---

Update:  Today the upload is back to 50 up. 

DSL Speed test:  940/50.6

Quality = A+

Bufferbloat = B

This is on the clean PC.  No antivirus.  Will follow up once I finish installing all my programs.

May have been congestion as @RyzenFX mentioned.  I did not get around to calling in.

[Mythen]

speed test this evening

http://www.dslreports.com/speedtest/13579628

[RyzenFX]

@Mythen From what I'm seeing the upload bufferbloat is really high. I get similar results as well and see that the bufferbloat peaks when the the upload speed peaks, and then the speed goes down. In your situation, the same thing can be seen b/c the upload speed hits a peak, but then drops down. 

Could there be some sort of accumulation of that causes this high bufferbloat? This is just one of the questions I'm thinking about which could be causing the problem.

[Mythen]

I don't get it honestly.  I am not running any programs that use much of any upload. 

[toolcubed]

Hi, I was wondering if a Rogers engineer (Dave?) could help me out with something.  Over the past week, I've been running some speed tests after noticing sluggish performance with my Internet practically every night after 8pm.  I had this problem a while ago and after reporting it to Rogers back then, they said the problem was with congestion/noise on the neighbourhood node, which was fixed at the time.  The problem went away but now it appears to be back.  I don't have any issues achieving a peak 200Mbps (and then sustained 130Mbps) on my 100/10 plan, but jitter sky rockets and upload speeds tank (usually around 5Mbps but sometimes even as low as 2Mbps) after 8pm.  Everything is perfectly fine throughout the day (i.e. non-peak hours) where jitter is low and upload is 11Mbps.  Again, the problems only seem to happen at night after peak hours.  Do I have to call in to report the problem?  I just really want to avoid going thru the same song and dance that I went thru last time where they sent 3 technicians to the house when it ended up being a problem with the node.  I know for certain it's not a problem inside my house...my cables are in top notch shape, my connectors were re-done by the last tech that came, and my signal levels are great.  It sounds like there's noise or congestion on the network somewhere (most likely the node again).  Thanks.

[RyzenFX]

@toolcubed

It definitley sounds like upload congestion, my node is running high on the upload side and I'm seeing high jitter as well. You can definitley tell that it is upload congestion when your consistently hitting 130Mbps download, but your upload speed is low and your jitter is higher than your ping.

Maybe you can call Rogers and escalate to a Tier 2 support agent and ask if you can submit a ticket to clear congestion?

[toolcubed]

Thanks, I'll try that. Hopefully, they can escalate right away without wanting to send a tech to the house.

[asns]

When I am connected wired I can access most of the servers of our company through VPN, but still few of them are not accessible. but , with WIFI my VPN connection is established but I can't access any of the servers or even my work email.

[Datalink]

@asns have you enabled any of the VPN passthrough settings at all?  This might be an issue for @RogersDave to have a look at.  Don't know why it would be different for wired vs wifi.  Just wondering if by any chance the VPN passthrough settings only apply to the wired, and not to the wifi path.  That would be rather strange, but, its a new modem, so, food for thought.....

[Datalink]

@toolcubed is your modem running DOCSIS 3.1 at the present time?  Just thinking, unless this is a noise issue for your area, the only way out would be switching to DOCSIS 3.1 or a node split which would take weeks if not months to accomplish. 

[toolcubed]

Yup, been running on DOCSIS 3.1 for a while. Is there any way to find out what work Rogers already performed on the node without calling in? I tried asking tech support a few times and they won't give me any specifics. I'm wondering if the node was already split because there were a few occasions in the past few months where several Rogers trucks were in the area pulling fibre and it appeared they even moved my coax line to a different tap (physically different green box) on the street. Would be good if someone here could provide some detail.