cancel
Showing results for 
Search instead for 
Did you mean: 

CODA-4582 - Open Issues for Investigation

RogersDave
Retired Support
Retired Support

*** This post was last edited May 2, 2017 ***

 

Good morning Community,

 

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

 

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

 

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

 

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

 

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

 

 

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

 

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

 

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

 

Update April 22: This problem seems resolved in firmware 2.0.10.27

 

 

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

 

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

 

Workaround: manually select higher channels (149-153-157-161)

 

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

 

 

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

 

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

 

 

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

 

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

 

 

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

 

This problem has been reported to Hitron for investigation.

 

 

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

 

Update April 12: This problem is resolved in 2.0.10.26T2

 

 

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

 

Investigation continues with Hitron.

 

 

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

 

 

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

 

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users


 

Future Planned Improvements

The following are items that we are working on in parallel of the above.

  • Improvement in WiFi speeds
  • Improvement in latency / bufferbloat

 

 

Dave

 

*Edited Labels*

2,620 REPLIES 2,620

Re: CODA-4582 - Open Issues for Investigation

??

Re: CODA-4582 - Open Issues for Investigation

lupinglade
I plan to stick around
https://github.com/esnet/iperf/releases - its the latest release

Re: CODA-4582 - Open Issues for Investigation

Yes, but, as found on the github site, that version is for CentOS Linux, FreeBSD, and MacOS X. 

 

Iperf 3.1.7 is functionally identical to iperf 3.1.6. Its only changes consist of updated documentation files and text in the RPM spec file.

 

Iperf 3.1.6 2017-02-02 == The release notes for iperf 3.1.6 describe changes, including bug fixes and new functionality, made since iperf 3.1.5.  

 

* User-visible changes

* Specifying --fq-rate or --no-fq-socket-pacing on a system where these options are not supported now generate an error instead of a warning. This change makes diagnosing issues related to pacing more apparent.

* Fixed a bug where two recently-added diagnostic messages spammed the JSON output on UDP tests.

 

3.1.6 Windows version can be downloaded here:

 

https://www.neowin.net/forum/topic/1596631033234695-iperf-316-windows-build/#comment-

 

Re: CODA-4582 - Open Issues for Investigation

hoob
I plan to stick around

Anecdotal observation, up until and including .26T2 I had no problems with my Honeywell Wifi thermostat. Since .27 was loaded onto the modem, the Honeywell goes for long stretches of disconnection.

 

Since Honeywell emails me every time the thing goes offline for more than a minute, it's pretty obvious when it happens.

 

Not critical but just thought I'd mention it.

 

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

So we're now in June!

 

Anyone know if Rogers started to roll-out DOCSIS 3.1 upstream in Ontario yet?

Re: CODA-4582 - Open Issues for Investigation


@JohnBeaudin wrote:

So we're now in June!

 

Anyone know if Rogers started to roll-out DOCSIS 3.1 upstream in Ontario yet?


@JohnBeaudin

I don't think they've rolled it out yet, I think their resources are going towards IPTV. Also they're releasing another Gateway/Modem so resources are also on tuning/refining that modem experience.

Re: CODA-4582 - Open Issues for Investigation

WestPoint
I plan to stick around
@hoob MINE TOO, mines been doing it since these new coda modems came out the, so like 6 months or so now

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

@gp-se

 

About that new gateway, the release should be soon according to the news ( they said mid 2017). I am interested even thought I just got the CODA in December.

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around

@JohnBeaudin wrote:

@gp-se

 

About that new gateway, the release should be soon according to the news ( they said mid 2017). I am interested even thought I just got the CODA in December.


It also took 4 months for the gateway to be stable enough to use. Sure you want to be bleeding edge on the next one? Smiley Very Happy

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

One can always dream that it won't be a puma chipset haha

Re: CODA-4582 - Open Issues for Investigation

huh666
I plan to stick around

I jumped on the CODA train immediately too, but I am anxiously awaiting the new gateway as well.  Its kind of fun testing this stuff out.

Re: CODA-4582 - Open Issues for Investigation

dallasdinatake
I plan to stick around
Will they be a new update soon for the firmware

Re: CODA-4582 - Open Issues for Investigation

Working very well here:

 

 

Daniel Smiley Very Happy

Re: CODA-4582 - Open Issues for Investigation

I signed up for the 1000/50 But only got 30upload this was last year to

Re: CODA-4582 - Open Issues for Investigation

AnnoyingRegistr
I've been here awhile

Any update on this?


UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

 

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).


I'm having packet loss while playing league of legends at night. I'm on the CODA-4582 and .27 firmware on a wired connection.

Re: CODA-4582 - Open Issues for Investigation

RogersDave
Retired Support
Retired Support

Hello community,

 

Just finished reading the last 4 pages of this thread (sorry, I was pulled in other situations and couldn’t spend as much time as I wanted on the forums).

 

I’ll try to address the concerns raised (or at least explain why).

 

Automatic downgrade on factory reset

I understand that this is inconvenient for beta tester and wish I could move out of there. It will happen but we need more time. A bit of history, at Rogers we never really used the automatic update on boot feature. We always manually controlled firmware versions on modem which allowed me for example to push a beta firmware and it would stick.

 

On the CODA, the initial firmware that we received on brand new modems from the factory was far from perfect. To provide an acceptable customer experience, we had to enable automatic update on boot because otherwise a customer would stay on a very old firmware for up to 72 hours before our systems would pick it up. This also helped customers with specific issues on the more recent firmwares as a reboot would get them to a newer firmware addressing their problem immediately. It was also necessary so that IPv6 could be enabled on that modem (IPv6 is a global setting per modem type but old firmware didn’t support it well).

 

This is great for all CODA customers but affects the trial users (on factory reset, the modem picks up the configuration that says for a new modem, update to version X immediately). Unfortunately, we have 0.2% of the CODA-4582 users who are beta firmware participants so that decision had to be made for the greater population.

 

We are currently working on replacing our firmware management system (discussing with potential vendors) and once we have the new system in place, we will be able to manage this better.

 

Packet loss on / around CMTS

I will need to investigate further on this one. However, firmware 2.0.10.27 is an Intel SDK change so it might have to go back to Intel. I’ve seen some private messages on the topic and will look into them when I have a little free time. It is however a cosmetic issue (the end destination doesn’t report packet loss) so although annoying, it is lower priority for me at the moment. I will however focus on the IPv6 packet loss issue more carefully to understand if there is any impact.

 

Packet Loss on UDP and impact of VPN

Although I haven’t looked in detail at the neighborhood where this has been reported recently, it is usually tied with very high usage in the uplink direction. We have seen a dramatic increase of usage in the uplink in multiple areas and we need to increase the capacity. This can be in some cases a quick process (cabling reconfiguration) or a very lengthy process (construction permit, digging, new equipment installation).

 

Nonetheless, there is a lot of effort put into this right now where we are increase the capacity in multiple nodes of our network on a daily basis. We are also building a better mathematical model to predict areas where customers are impacted or are likely to be impacted based on live data from the network. This will help us schedule capacity augmentations sooner.

 

The problem with UDP is that there is no protection at all for these packets. In case of congestions, if the packets are dropped they won’t be retransmitted. When using a VPN from your home network to an endpoint on the Internet, all UDP traffic gets encapsulated in TCP traffic (with retransmission capabilities). This means that if the VPN packet will get retransmitted (along with the encapsulated UDP packet) whereas the UDP packet would simply be dropped.

 

Firmware 2.0.10.29

This is still work in progress and scope is not fully locked. I am hoping that the following will be included (some of these may come in additional builds T2, T3, etc.)

  • More flexible IPv6 firewall
  • IPv6 DNS override
  • USB port fix
  • WiFi performance improvement
  • DDNS freedns.afraid.org fix
  • Guest network fix

 

We will have the first builds sometime in June (expected)

 

DOCSIS 3.1 Upstream

Not yet and I don’t have a date. I’m a little disconnected from this project unfortunately.

 

New gateway

I can’t talk about it but there is a lot of work on it. I’m busy testing a lot of the issues reported on the CODA since the launch to make sure we don’t have similar problems.

 

Other items

Waiting for the next firmware to be available, I have a few more things to test with volunteers for those of you interested. The firmware 2.0.10.27 supports 2 features on WiFi that are currently not enabled by default, DFS and band steering. I need some people to test them so if you are interested, send me a PM and I’ll enable both features on your modem.

 

DFS will enable the UNII-2 and UNII-2 Extended WiFi bands (channels 40 to 140). This can help in congested WiFi areas.

 

Band steering allows to set the same SSID on both the 2.4 GHz and the 5 GHz band so that devices can dynamically pick the best channel available based on the radio conditions. Once I enable band steering, you won’t be able to have different SSID for both radio bands.

 

You can always bail-out of these two settings with a factory reset.

 

 

Happy testing!

 

Dave

Re: CODA-4582 - Open Issues for Investigation

Just to add some info for Dynamic Frequency Selection (DFS), for those who might be interested in trying it out:

 

The DFS frequency space is protected for use by Airport Weather Radar, and possibly by extension, airborne radar as well. Wifi modems and routers are allowed to operate within that frequency band with the following Industry Canada regulation. These are the main portions of the regulations that will have the most impact on DFS users:

 

http://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf10971.html#s6.3

 

6.3.2 Operational requirements

 

Devices shall comply with the following requirements, however, the requirement for in-service monitoring does not apply to slave devices without radar detection.

 

a. In-service monitoring: an LE-LAN device shall be able to monitor the operating channel to check that a co-channel radar has not moved or started operation within range of the LE-LAN device. During in-service monitoring, the LE-LAN radar detection function continuously searches for radar signals between normal LE-LAN transmissions.

 

b. Channel availability check time: the device shall check whether there is a radar system already operating on the channel before it initiates a transmission on a channel and when it moves to a channel. The device may start using the channel if no radar signal with a power level greater than the interference threshold value specified in Section 6.3.1 above is detected within 60 seconds. This requirement only applies in the master operational mode.

 

c. Channel move time: after a radar signal is detected, the device shall cease all transmissions on the operating channel within 10 seconds.

 

d. Channel closing transmission time: is comprised of 200 ms starting at the beginning of the channel move time plus any additional intermittent control signals required to facilitate a channel move (an aggregate of 60 ms) over the remaining 10‑second period of the channel move time.

 

e. Non-occupancy period: a channel that has been flagged as containing a radar signal, either by a channel availability check or in-service monitoring, is subject to a 30-minute non-occupancy period where the channel cannot be used by the LE-LAN device. The non-occupancy period starts from the time that the radar signal is detected.


The following applies across the entire DFS channel range:

 

6.2.3.1 Power limits

 

The maximum conducted output power shall not exceed 250 mW or 11 + 10 log10B, dBm, whichever is less. The power spectral density shall not exceed 11 dBm in any 1.0 MHz band.

 

The maximum e.i.r.p. shall not exceed 1.0 W or 17 + 10 log10B, dBm, whichever is less. B is the 99% emission bandwidth in megahertz. Note that devices with a maximum e.i.r.p. greater than 500 mW shall implement TPC in order to have the capability to operate at least 6 dB below the maximum permitted e.i.r.p. of 1 W.

 

 

In a nutshell, any modem or router operating within DFS frequencies shall

 

a. monitor for the presence of a radar signature on the selected wifi channel


b. if a DFS wifi channel is selected for use, the modem/router has to listen for a radar signature for 60 seconds prior to commencing operation on that channel, if in fact that channel turns out to be clear of any radar systems.


c. if a radar is detected, cease operations on that channel within 10 seconds.


d. close the channel, conducting channel change operations and broadcast instructions to remote devices within a 200ms window


e. lock out that affected channel for 30 minutes before it can be considered for any use as determined by step b.


f. limit the output power to 250 mW or operate at least 6 dB below the maximum permitted e.i.r.p. of 1 W

 

Note that the power limits of the 5 Ghz lower channels (36 to 48) is either 50 or 200 mW depending on the date of approval of the device by Industry Canada and the limit of the upper channel range (149 to 161) is 1 watt.


Operational impact:

 

If you happen to live near an airport, or near a weather radar, you will probably find that the DFS channels are of no use as they will shut down unexpectedly and shift or try to shift to another DFS channel due to the weather radar detection.

 

With DFS channels enabled, if the modem or router attempts to use a DFS channel and determines that its occupied by a radar system, it might then pick yet another DFS channel, wait another 60 seconds to determine if its occupied by a radar, and if so, move to yet another DFS channel.

 

If the logic in the channel selection is to use DFS channels first, then its possible that the modem/router might cycle thru all of the DFS channels before selecting a non DFS channel. So, for perhaps 14 minutes, there might not be any wifi network available from the modem or router as it cycles thru all of the DFS channels before arriving on a workable non-DFS channel.  I suspect that the first time that someone goes through this, it will be rather infuriating to say the least.

 

Due to the reduced output power in the DFS channels (52 to 144), the max range or operating distance from the modem or router will be better than the lower channels but not as good as operating on the higher channels.

 

To see this visually, have a look at the following page:

 

http://www.semfionetworks.com/blog/5ghz-band-channel-availability-in-canada

 

For those who are able to make use of DFS channels, you might be able to finally run your wifi network without fighting with the neighbor for clear channels, or, perhaps the neighbor will move to a DFS channel leaving the upper channel clear for your use.  That would be the better solution, a clear upper channel with its higher power output level.  Perhaps a bribe for the neighbor with a hint to use DFS channels might be in order someday 🙂

Re: CODA-4582 - Open Issues for Investigation

Telek
I plan to stick around
Thank you Dave, once again, for your assistance to this community! Also, congratulations on your 500th post 😄

Re: CODA-4582 - Open Issues for Investigation


@Telek wrote:
Thank you Dave, once again, for your assistance to this community! Also, congratulations on your 500th post 😄

I'm glad this 500th post had some substance and wasn't a random comment (like this one) 🙂

Re: CODA-4582 - Open Issues for Investigation

JohnBeaudin
I'm a senior contributor

Thanks for the update @RogersDave

 

Feels good to see you back 😉

Re: CODA-4582 - Open Issues for Investigation

dallasdinatake
I plan to stick around
I would test this out but I'm using a Netgear R7000