CODA-4582 - Open Issues for Investigation

[RogersDave]

*** This post was last edited May 2, 2017 ***

Good morning Community,

As I mentioned in a post two days ago, we have received the next firmware 2.0.10.20 from Hitron. We are currently running initial testing on this version and will push it out to participants in the firmware trial program as soon as it passes initial testing.

However, while running these tests, we discovered abnormal behavior with ICMP and are awaiting feedback from Hitron today to asses how this will be addressed. As soon as I this is confirmed, I’ll update the change log with the correct version information and start pushing it out.

In parallel, we are still working on the following high priority items. In some cases below, I requested affected customers to reach out to me via private message. If you do so, please include your modem MAC address in the subject line (even if we exchange messages daily) as there are a lot of you reaching out to me daily 🙂

UDP Packet Loss

The investigation for what has been reported as UDP packet loss is still ongoing. We have deployed a probe at one fellow forum member on both a CODA-4582 and a CGNM-3552 to collect additional data. We are actively working with Hitron and Intel on the results observed.

Based on what we know so far, in most instances UDP packet loss is coupled with higher uplink usage in the area. Although the impact is noticeable in specific logs (League of Legends), the root cause for the perceivable impact (while playing) is likely related to bufferbloat (see next issue).

Bufferbloat

When comparing the performance of a CODA-4582 to a CGNM-3552 in the same network conditions, the CODA-4582 consistently reports higher bufferbloat when tested on DSLReports.

Update April 12: The solution for this problem will come in two folds. It will require a change in software which will possibly be included in 2.0.10.27 but more likely in 2.0.10.28 and a change in network configuration.

The network configuration change is not compatible with the current firmware so this change will only come after a vast majority of the modems are running the new code. We are however looking at a way to make the change only for specific modems to support testing in the community.

Update April 22: This problem seems resolved in firmware 2.0.10.27

5 GHz WiFi Low range for channels 36 to 48

Lower WiFi channels on the modem have a much smaller range. This is due in part to the limit imposed by Industry Canada to maximum transmit power.

Furthermore, the current automatic channel selection (auto mode) tends to select the lower channels when in similar load conditions.

Workaround: manually select higher channels (149-153-157-161)

Update April 22: The channel selection algorithm has been improved in firmware 2.0.10.27

Loss of OFDM Channel Lock

Under some RF conditions, the modem fails to lock properly on the OFDM channel. This typically result in variable performance.

Update April 12: This problem is resolved in 2.0.10.26T2

List of connected device does not get fully populated

This is a known issue that has been tracked since firmware 2.0.10.13. We are making improvements at every firmware but it is not a perfect system.

The situation is worst after a reboot or firmware upgrade as the list gets reset and must be repopulated as devices renew their DHCP lease.

NAT Loopback not working for wired clients

When setting up port forwarding to an internal server, it is possible for a client on WiFi to reach the server using the external IP/port. If the client is on a wired interface, it doesn't work.

Update April 12: This problem is resolved in 2.0.10.26T2 (not confirmed)

LAN Counters not working

Some customers reported that LAN counters (especially in bridge mode) are reporting inaccurate values.

This problem has been reported to Hitron for investigation.

Unexpected modem reboot

Some customers reported their modem reboots unexpectedly. We have also seen this behavior in our lab.

Update April 12: This problem is resolved in 2.0.10.26T2

Missing SC-QAM Channels

After a reboot, some modems are missing SC-QAM channels. A fix has been implemented in 2.0.10.26T2 to address this behavior but it has not corrected all scenarios.

Investigation continues with Hitron.

WiFi Survey

The WiFi Survey functionality in firmware 2.0.10.26T2 (and possibly before) reports incorrect SSID names.

Guest Network

When connecting to the Guest Network, an error message is displayed "only allow DHCP client to use this wireless".  This has been reported in firmware 2.0.10.26T2.

Update April 22: This issue has been resolved in firmware 2.0.10.27

Update May 2: It seems this issue is not fully resolved and still experienced by some users

---

Future Planned Improvements

The following are items that we are working on in parallel of the above.

• Improvement in WiFi speeds
• Improvement in latency / bufferbloat

Dave

*Edited Labels*

[RogersDave]

*** This message is a placeholder for updates and version log ***

[JohnBeaudin]

That's a step in the right direction,I will try to recreate the lag buildup over time if I can.

[RyzenFX]

UDP Packet Loss

There has been a few report of this lately. In order to report this problem and have it investigated, I will need an easy way to reproduce it and measure the impact (percentage of packet loss). If you have any idea, send me a private message.

---

---

I am easily able to replicate this in League of Legends. League provides a network diagnostic of each game, which can be convert to a graph which meaures the amount of packet loss per minute. I seem to be averaging a packet loss of 10-11 packets/minute. 

In Discord, a VOIP program this is also noticable but I still have yet to find a way to measure the packet loss.

[prateeck7]

@RogersDaveJust to confirm if anybody is facing the same issue. It takes me 2 mins to download 1GB files (since CODA); same as with Bell 150 (actually Bell 150 is faster). I don't even know why I upgraded to Rogers Gigabit then.

Yesterday, I was downloading 1.2 GB from Microsoft store. It took me approximately 2 mins. I checked internet speed just then. I am hitting 900+mbps at different speed test sites (bar testmy.net which I feel is the real speedtest as it measures downloaded size over time).

I thought it was traffic. I try downloading the app again after couple of hours and I have been downloading apps from microsoft store and 'thinkbroadband speed test' for the past 12 hours in intervals. I still take approximately 2mins to download 1GB files. Speedtest sites meanwhile has been showing 900+mbps speed for the same past 12 hours consistently.

I noticed that while downloading (and I have been mentioning this issue since the first day of this modem) the file is being downloading really fast, then get stucks, really fast and stuck and cycle repeats.

Anybody know what is going on here?

[Jeffj]

---

@prateeck7 wrote:

@RogersDaveJust to confirm if anybody is facing the same issue. It takes me 2 mins to download 1GB files (since CODA); same as with Bell 150 (actually Bell 150 is faster). I don't even know why I upgraded to Rogers Gigabit then.

Yesterday, I was downloading 1.2 GB from Microsoft store. It took me approximately 2 mins. I checked internet speed just then. I am hitting 900+mbps at different speed test sites (bar testmy.net which I feel is the real speedtest as it measures downloaded size over time).

I thought it was traffic. I try downloading the app again after couple of hours and I have been downloading apps from microsoft store and 'thinkbroadband speed test' for the past 12 hours in intervals. I still take approximately 2mins to download 1GB files. Speedtest sites meanwhile has been showing 900+mbps speed for the same past 12 hours consistently.

I noticed that while downloading (and I have been mentioning this issue since the first day of this modem) the file is being downloading really fast, then get stucks, really fast and stuck and cycle repeats.

Anybody know what is going on here?

---

Speeds at which you receive a file are not just determined by your Download speed, they are also determined by how fast the content distribution server is willing to serv them to you. so if they decide you're going to get xx  amount of bandwidth to be fair to other content users using their distribution server at the time. thats what you get. And this explains exactly why your speed tests are 900 MBPS and your downloads are not that fast. 

[prateeck7]

I am downloading files from Microsoft store (not torrent). And to verify that the content provider was not limiting it, I went to 'thinkbroadband' and 'testmynet' which is legit. Same issue.

In fact, I have never crossed 70mbps at 'testmynet' while with CGN I would easily get past 400mbps.

[Jeffj]

---

@prateeck7 wrote:

I am downloading files from Microsoft store (not torrent). And to verify that the content provider was not limiting it, I went to 'thinkbroadband' and 'testmynet' which is legit. Same issue.

In fact, I have never crossed 70mbps at 'testmynet' while with CGN I would easily get past 400mbps.

---

You're not understanding what im saying. The microsoft tore uses COntent distribution networks like Akamai, which is the one Microsoft uses and partiners with and uses. Akamai will limit your download speed to make it fair for everone who is pulling off the same server. Its not just you on your loansom that is pulling content from them at any given time, its thousands of people at once. Sure if you were the lone ranger downloading from microsoft at that time. You would have been downloasing at your full 900Mbps speeds. But i can assure yiu, you were not the only user downloading from their content network at that time.  This is called traffic shaping that is deployed for QoS, which im not sure if you are farmiliar with but it means Quality of service. which tries to make it fair for every one.

You stated your self you were reaching 900 Mbps speeds from speed tests. which are designed not to traffic shape in any way to provide the best and most accurate result possible.

So like i stated before its not just your download speed that determines your download rate, but also the rate the content server is willing to provide it, which is not a rogers issue.  

As a side note, a torrent will be more likely to achieve 900Mpbs speeds then a file downloaded from a content distribution network as torrents being P2P deploy not traffic shaping. GO try downloading the latest torrnt of Ubuntu and see, as long as your hard disk can write fast enough it should max out your connection. on a properly configured torrent client. 

[prateeck7]

@JeffjWhat about 'testmynet' and 'speedof.me'. My brother in law in another neiborhood can get above 300mbps at that site right now. I myself used to get 400mbps before a month with CGN modem consistently.

So you are telling me that 'testmynet' all of sudden decided to limit my download since I got white CODA modem.

May be your claim on Microsoft store is right. I am in no way tech savvy when it comes to these things. But, I used to download apps from the microsoft store very fast with CGN modem. May be it was a conincidence that there were no traffic when I downloaded stuffs over intervals of time.

Also 'testmynet' and there are many other sites that let you download files without limit. They are legit. Infact, i have always found 'testmynet' real as it always resembles the speed I am getting during the download. Same with speedof.me. I rarely get above 90mbps at speedof.me since this modem. I am mentioning this because I have experienced this. And I am checking the internet speed at testmynet every time I test the internet speed in the hope that I am getting the speed at this site as before.

[Jeffj]

I cannot put any merrit to test my net as its  servers are quite far from my geografical location, and with all the hops it takes me to get there i cant trust their results for my tests, however my tests typically look fine with them i still see them odly fall and i think distance and hops is a factor in the tests.  Same with speedofme, always says test server "unknown".  I typically use DSL reports speed tests as they have been and always will be a trusted site, they also keep a graph of your results for you. They have been there for decades for the users.  Contorary to belif, Speedtest.net isnt faking results. Some people on occasion decide for some reason they are going to say they are lieing, mostly to try an validate their own point. but MANY MANY reputable companies, ISPs, and others  provide servers for speedtest.net, to broaden the network for users to test. Digital Ocean, a very reputable Hosting company that i personally use because of their stellar reputation amongst  the dev community for being an honest company, even host multiple servers around the world for users to have a transparent speed test. http://speedtest-sfo1.digitalocean.com/

[RogersDave]

The microsoft store does indeed uses Akamai and Azure amongst other content distribution server. You are likely hitting Akamai and since just before the holidays, we have seen a slow down of all Microsoft content from Akamai, regardless of the modem used.

I currently have a ticket open with both Microsoft and Akamai on this specific issue.

For thinkbroadband, this is UK based if I recall and is largely affected by latency. I wouldn't use this as a reference.

Speedof.me is HTML5 based. I've had mixed experience with it and it highly depends on the browser you use. What browser were you using for your tests and can you try another one to compare?

Dave

[prateeck7]

@JeffjOut of curiosity, how much are you getting at testmynet?

@RogersDaveI am testing with Mozilla and Chrome. Testmynet is very crucial to me as from my expereience I find it quite reliable than others. That site is my benchmark.

When I run testmynet for the first time after rebooting the modem, it is all fine. 100MB file gets transfer in a second or so. It is BLAZING fast.

Its after the first time. Mostly when they verify the test (to be consistent over 7 secs) to download 200MB files. It takes me around 40 secs to download it now. Before it would take me 8 secs at max with the CGN modem frequently (didn't matter 1st try or zillion tries). As I mentioned it before many time, i can actually notice it. Its blazing fast, it gets stuck (for a couple of seconds), fast (not blazing fast), stuck (for a couple of secs) and the so on until the download is completed. I barely hit above 70mbps with testmynet which is pathetic for Gigabit plan. People on 250u are getting more than double at that site than me. I called Rogers, they said they can't help me, outright, as expected.

[Jeffj]

@prateeck7

---

@prateeck7 wrote:

@JeffjOut of curiosity, how much are you getting at testmynet?

---

I'm on the 250u Package, my speeds have been fine since we got pushed .19

250 down and 17 up. before 178 down and 12 up testmynet NY server that has to hop through the most congested part of the Untied Staes/North America internet wise (the east coast) but its the closest to me.

I run a test on the Digital Ocean Toronto server and i get 340 down and 25 up.

If i run the DSL reports which uses a server in Toronto i get 320 down and 21 up. 

Speed of me 270 down 12 up.

these tests were done one right after the other. 

@as for the Microsoft downloads running at around 150mbps speeds thats about normal, the way it always has been for me on Xbox.  However I did have issues before Xmas wich i believe is the ticket @RogersDave is referring to were i was only getting 1.5megabit to 800 kilobits down on the Microsoft content servers. Now that was an issue. 

[Triple_Helix]

Thanks Dave! Just waiting to test new Firmware for ya!

Cheers,

Daniel 

[Hybrid_Noodle]

@RogersDave There may be an issue with the Rogers Home Monitoring router (Netgear WNR2000v5) getting an IP address while the CODA is in bridge mode as after the .19 firmware push it appears as though the router has lost connectivity to the outside world (Internet LED is solid amber which states "The router detected an Ethernet cable connection to the modem" and will not go to the standard solid green).

I have moved the Netgear behind my personal router for the time being to give it access.

[gp-se]

In terms of the exchange program for users with the interference issue, how will those users know they are affected?
Is there a certain batch of serial numbers that are known to have the issue? I ask because I'm curious if my modem is affected and would need to be exchanged once the program launches.

[reefermajic69]

Here is my issue.  I have signed up for the the GIG internet.  Since Oct 31/16 when I signed up, I have not got even close to GIG speeds, more like an average of 300-500MBPS.

I have the new CODA modem.

Now when I run it like just now under Rogers Speed Test I am getting with CODA as default Gateway directly wired into my PC.  I am getting 763D and 30U.

When I run the TestMySPeed I am getting 257D and 32U.  Why such an extreme variance?  I do not trust the speed output from the Rogers Test because the Variance to every other testing online is so extreme.  Okla etc.

I have attached screen prints.

Also I would rather use my ASUS2400AC as my gateway and CODA as bridge but for whatever reason the speed through ASUS drops by 50% in the 300-450mbps range.  I made sure no QOS is on etc.  This is very frustrating and I would like some help.

Setting expectations.  What should my average be?  800? 850? 900? 950?  1,000?

[Makaveli99]

What is a Asus2400AC?

Do you mean The Asus AC2400 RT-AC87U ?

If is it the AC87 is Nat acceleration on?

There are other settings besides QOS that will cause slow downs like traffic moniter etc.

I would recommend you resetting the Asus device back to default then testing it.

[reefermajic69]

Thanks.  Yes its the ASUS as you mentioned.  NAT is off.  The screen prints I just did though are with the CODA as default gateway and not even using the ASUS.

My question is why the extreme variance in speed test from then Rogers site to the Testmy.net?

I also find the WIFI for the 5G and I am in a condo 1000sqft.  The CODA 5G signal on my Iphone 7 turns into like 2 bars whereas with the ASUS its full 5 bars wifi signal.

[Datalink]

The LAN .... SWITCH CONTROL .... NAT Acceleration must be on for Asus routers if you want to run anything over 100 Mb/s.  Now, the one thing that you have to keep in mind is that there are functions that will kick off the NAT Acceleration, such as Traffic Monitoring, Port Forwarding and others.  The suggestion to reset the router back to default settings as indicated by @Makaveli99 is spot on as that will kill any functions that disable the NAT Acceleration.

When that is done, run a speed test using the www.speedtest.net Toronto Rogers server, or Montreal Rogers Server.  If and when that is complete, go thru the entire menu list on the left hand side, look at every sub-menu item and ask yourself if you're using it or not.  If not, disable it.  If you decide to enable something, follow that with an immediate speedtest so that you know whether or not you're crippling the throughput by enabling that function.  

Just to note, the RT-AC87U has been nothing but trouble for a good number of its users.  If you have avoided all of its problems, consider yourself very lucky.  If you're having issues with it, as many others have had, strongly consider getting rid of it and moving on to something else with better stability.  

As for the speedtests, and I say this to everyone, you need to do some research to determine what the speedtest is doing:

1.  single thread or multi-thread; and

2.  what protocol is used; and

3.  what browswer is best suited for that particular site; and

4.  whether or not its so far away that any results you see are questionable at best.  

Add to that discussion any device that you might decide to use to test the top speed of the gigabit service.  Can the device actually support gigabit rates?

That all sounds rather harsh, but, we've had many people come into the forum in the past complaining about data rates and when we look very closely at routers, cabling, end devices, etc, we find that they are old, not configured properly or that they won't support gigabit rates.   So, moving users up above 100 Mb/s was the first challenge, the challenge now is to get them to gigabit rates, while at the same time dealing with new modem issues.  

So, best advice, bite the bullet and reset the modem.  Watch the NAT Acceleration very closely to ensure that it doesn't get kicked off.  Run the speedtests at www.speedtest.net Toronto Rogers, Toronto Beanfield, or Montreal Rogers or Fibrenoire if you're closer to Montreal than Toronto.  Those servers have enough horsepower to test the last mile to the modem and drive higher data rates. 

[Wizerobe]

Just chiming in with my current results/issues

I had the 250Mbps package hooked up yesterday (wednesday Jan 4th) and even up until this morning I had an average speedtest of 330Mbps, though as of tonight, I've been stuck at 5-10Mbps Down.

Not sure what I should provide here, but I'm currently running firmware 2.0.10.19

Hopefully this is resolved soon, as it leaves a really bad impression to be having issues like this on the second day of service.

Good luck Dave with finding the solution, if possible, I would like to participate in any advanced/beta programs for firmware, if that is related to the issue.

Thank you!