horrible speeds over VPN

[hoek]

Hi,

I recently switched to Rogers gigabit from Bell. I have a CODA-4582 modem, firmware 2.0.10.34T6.

I frequently work from home connecting to work over a Cisco VPN.

Everything is much slower compared to when I was on Bell and compared to a friend using a TP-Link modem on Teksavvy.

This post seems to indicate a problem with processing VPN traffic:
http://communityforums.rogers.com/t5/Internet/Rogers-Hitron-CODA-4582-Hardware/td-p/378263/page/150

I won't pretend to understand it all but I can definitely see there is a problem with this modem related to VPN traffic.

Copying down large files is out of the question now and RDP connections are much less responsive.
During a file copy I see speeds around 1-2 Mbps and sometimes only a few 100 KBps. On Bell it was a steady 4-5 MBps.

I've tried bridged and gateway mode doesn't matter.
It seems there has been some issue about this since 2016.  Is there any resolution?

I'd be willing to try beta firmware, different modem (though I imagine nothing else is available).

Its great and all that 5 people in the house can stream 1080p youtube but it kinda kills things for me if simple telework activities are crippled by modem firmware.

*** Edited Labels ***

[stepy2015]

Reset your modem it will roll back to the old firmware. The current firmware is very unstable. but it may not stay for long as your modem may try to update

[hoek]

Tried factory reset comes back with same firmware.

I noticed speeds are better for a short time after rebooting slowly getting back to the crippled state.

Almost like a buffer or something on the modem overflows and it can't keep up.

[RogersTony]

Hello @hoek,

Thank you for your post and welcome to the Rogers Community Forums!

Having your ability to Work from Home become impaired due to slow speed issues can become super disruptive! We definitely want to take a closer look into this to see what is going on. We can start by answering the questions below:

• Have you tried connecting with an alternative VPN?
• Have you confirmed no issues with your VPN settings/routing?
• Have you reached out to your VPN provider?

We are not aware at this time of any VPN connection issues with our production firmware. Our testers have confirmed no known concerns at this time.

We look forward to hearing back from you!

RogersTony

[hoek]

• Have you tried connecting with an alternative VPN?
• Have you confirmed no issues with your VPN settings/routing?
• Have you reached out to your VPN provider?

---

I've tried both my workplace's VPN and my wife's.  Both the same behaviour.  Laggy RDP connection, slow file copy. I use a Cisco client she uses an Avaya client.  Both are IPSec.

Nothing has changed with the VPN settings compared to when the connections worked fine while on Bell fibe.

My VPN provider is my employer and they aren't about to change anything for 1 employee when everything was fine before and hundreds of remote users have no issues.

I should have taken a video when I still had Bell.  The difference is very obvious.  Also the fact that a colleague who switched from Bell to Teksavvy sees more stable and faster speeds using the same VPN compared to me adds more weight to it being the Rogers modem.

Check the link I mentioned... "slow IPSEC throughput has already been noted on the Hitron CGN3xxxx and CODA-4582 modems".  

[stepy2015]

Test the speed on a different connection. I think it may be your VPN not the Rogers connection. What are your speeds when you are disconnected from the VPN? 

[hoek]

There is no issue with the Rogers connection.  Speeds are well within spec with typical http/https activity.

The issue is with IPSec VPN (specifically) traffic  through this modem.  Is there no one that can validate whether IPSec is indeed hampered by this firmware as the various posts seem to indicate?

I'll try to get some screen caps of what I'm seeing.  I can probably even get a VM on Azure going to test the VPN throughput from there.

[Datalink]

Low IPSEC rates through the CGN3 series (Intel Puma 6 modem) and the newer CODA-4582 (Intel Puma 7 modem) are a known problem.  Having said that, higher IPSEC rates have been observed, but, that might depend on the IPSEC configuration settings and the capability of your pc to support client IPSEC/VPN use.

Personal opinion, if you have access to Bell fibre, your best bet is to:

1.  sign up for Bell Fibre

2.  Replace Bell's Home Hub 3000 as outlined in the following link with a gigabit fibre to ethernet converter:

http://www.dslreports.com/forum/r31118482-Yes-you-CAN-bypass-the-HomeHub-3000

3.  Connect the fibre to ethernet converter to:

     a.  An Asus RT-AC86U with Merlin's Asuswrt loaded (IPSEC support is enabled);

     b.  A pfSense router built with a CPU containing AES-NI capability which will support VPN

          & IPSEC requirements.  Here's a thread that discusses this:

https://news.ycombinator.com/item?id=14239580

     c.  A business class router which will support VPNs.

Doing some background reading on this yesterday I came across a note indicating that pfSense 2.5, which is not out, will require a cpu containing AES-NI capability in order to support route-based IPSEC operation.  

https://www.netgate.com/blog/pfsense-2-5-and-aes-ni.html

As far as the Puma 6 & 7 situation with IPSEC and VPNs, this has been a long standing problem and there hasn't been any comment from Rogers or Intel regarding any plans to resolve the low throughput problem.  I suspect that this actually dates back to the Puma 5 which was developed in the 2008 timeframe by Texas Instruments.  Whatever limits or design issues that exist in the Puma 5 firmware may have been carried forward into Intel's Puma 6 and 7 firmware development without anyone taking a close look at those limits or design issues.  This is an Intel issue to resolve, Rogers or Hitron won't be able to do anything until Intel expends some effort on this.  At the present time Intel has its hands full resolving other latency issues with the Puma 6 modems, DOS issues with the Puma 5, 6 and 7 modems, and a fighting class action lawsuit or two over the latency issues in the Puma 6 modems.

Plan B would be to move to a TPIA such as Start, TekSavvy or Ebox and use a Technicolour TC4400 modem which is a Broadcom modem.  That isn't cleared for use on Start or Techsavvy at the present time, but is available with EBox.  So, your options would depend on where you're located.  Ideally the TC4400 would be available for all TPIAs.  The TC4400 is DOCSIS 3.1 capable, which is currently in use with Rogers.  In terms of IPSEC/VPN performance, you would have to do some research, looking for comments from other users to determine if this would really be a viable option for your VPN use.   You would want to avoid an Intel Puma 5/6/7 modem and look for a Broadcom modem, but, you would definitely have to do some homework to determine the suitability of any of the TPIA offered Broadcom modems, specifically for VPN use.  

[hoek]

Thanks for confirming it.

Bell fibre isn't an option in my area.

I guess I should have stuck with Teksavvy 250 so I could pick a different modem and avoid this Puma nonsense.

I had my hands on a TP-Link docsis 3.0, broadcom based modem. Then stupidly switched to Rogers at the last minute due to, of course, a deal that undercut TS and offered gigabit for less slightly less than TS 250.

The truth is gigabit is great if everyone in your house is power downloading.  Otherwise it's really nothing more than bragging rights.

I've confirmed with a colleague that the TP-link (docsis 3.0) modem on Teksavvy 75 Mb provides a more stable connection with higher throughput to our workplace VPN than the Hitron.

I guess I have no choice but to switch to another provider so I can use another modem.

[Datalink]

@hoek pm sent.  Check your message inbox.  Follow the avatar (link) at the upper right hand corner when you're logged into the forum.   That will take you to your profile and message inbox/outbox.

[BS]

@Datalinkand @hoek  It is intriguing watching this discussion and I am glad that datalink has popped in to support hoek in this.  

It will be nice to hear if there are actually inherent issues with IPSEC over VPN.

This clearly is an important discussion for anyone who is using VPN clients to access corporate defined VPN services.

Fortunately, you had your wife using a different client, to evaluate, as you have no capacity to use different service, or VPN client - unless I am wrong you are on a defined corporately managed VPN connection, with a configured client, defined again by your internal or IT consultant team.

If your support is like the team I managed for corporate VPN access to my internal customers (staff and clients of the services) in a previous job, and having been on the other side too, the team is limited by what they can do when they are confronted by an issue that may be modem/service/router based on the client side of things.

They have usually tested extensively and researched and have experience on "fine tuning" the client side, but there is only so much they can do.

I know I personally had to advise staff that we couldn't support certain residential network, and eventually moved to us providing the business access to residential setting in order that we had more control over the client side.

I hope that you can over come it on Rogers, since it sounds like you pay for your own services, and it also impacts your wife's work too, so you need clarity on whether you have to move to another service - maybe back to TS, or maybe there is an alternative modem or firmware build that is available for your situation.

For my own curiousity, and the benefit of others, I would love to hear the outcome.  It is just past professional career curiousity.  Certainly been there.  My wife had her own challenges with a Cisco connection with her past company over Rogers a few years ago.  We ended up in her case that we had to go to a wired connection, that for some reason the WIFI wouldn't connect with the client properly.

Good luck in finding a solution.

Bruce

[hoek]

@BS you are correct about the corporately managed VPN connection.  I have no experience setting up Cisco VPNs but I suspect asking IT to mess with configuration for the sake of testing that may go nowhere is a non starter.

@Datalink mentioned AES-NI CPU extensions which I have.  This is supposed to accelerate AES encryption/decryption.  (Though I don't quite understand  how that helps if the problem happens before the client.) No clue if the Linux vpnc client automatically makes use of this.  I see the AES module loaded and openssl appears to know about it.

I'll remain in a holding pattern at least for a while (or until my "other boss" has had enough) in the hopes that maybe a new firmware down the road will help.  Though it sounds like a hardware problem / design choice that can't be fixed with firmware.

[BS]

@hoek

Are you possibly able to take your device and try to connect to someone you know on a Rogers connection, possibly same modem or another one, or go to a open WIFI.

Does someone you know have a router you could set in and run bridge (you may have mentioned that you have).

Just to isolate modem, site specific to your home.  Obviously there must be others using corporate VPN's and clients (either software or hardware), to access work securely - and definitely there have been reported issues with this modem across all ISP's, but I suspect it must be working for some.

Difficult one to diagnose as I would agree, your IT will probably only go so far, but I would suggest asking, can't hurt, or your wife asks hers.  Very little Rogers can do because they can't touch your VPN client and I suspect they would be very reluctant to get into diagnosing or playing with your VPN client and protocols, although they may have some knowledge of a few things depending upon who you get.

Only thoughts that come to mind for testing/troubleshooting that I can think of.

Have a look at these - I am  wondering if your VPN is using port and protocol combinations that the Hitron does not support well.  Doesn't change the reality that your employer boss, and your real boss will probably want to see this fixed soon.

Good luck with it.

Bruce

[TECHHEAD]

@hoek,

I recently switched to Ignite 500U/20D and use Global Protect VPN client for work and have no issues with connection speeds.

The company I work for used to use Cisco VPN Client and changed over to Global Protect because of some issues.

I use RDP regularly to connect to my desktop or servers at work and have never encountered any issues.

My Hitron is setup in Bridge mode with an ASUS modem. I believe my Hitron has the same firmware as yours.

[hoek]

@BS Good suggestion.  I have a friend using Rogers 100 who signed up over a year ago so 99% sure it's a different modem and in bridged mode.  I can take a laptop over and do some a/b testing.

I'll try and corner our IT guys to see if there's some alternate VPN configuration available.

I don't expect Rogers to be able to do anything at all unless they have some alternate modem trial they'd allow me to enrol into.

@TECHHEAD thanks, I googled Global Protect.  Correct me if I'm wrong but it looks like you have to use their service.  It isn't just a client like say Shrew Soft (which I tried) that you can use with another VPN provider.  Anyway I'll poke around. 

edit: Ok I see.  It's a whole other platform so this is not an option

I doubt a client can help as I've tried the Linux vpnc client which is very lightweight and the Windows Cisco client (older one not that anyconnect stuff) both with the same results.

[BS]

Cisco client once configured and tuned for the corporate VPN server is a pretty stable product. Hope some a/b tests will identify. I know what it means to corner a tech. Even as a manager I let them run the show. I just kept the squeaky wheels at bay. They ran the day to day stuff and that keeps them pretty busy most of the time. I hope even if slow you are getting work done. I also know the concept of the "real boss". Married 38 years I am retired from the field and I am still IT. But no team behind me of experts.

Bruce

[hoek]

🙂 yes that boss tends to have veto power when it affects productivity.

Agreed about the Cisco client.

I think if I had just started VPN'ing to work I would probably just accept this is the way it's supposed to be.  

I can work, little choppy sometimes, and moving data back and forth (git push/pull) is super slow so I'll avoid it when possible.

Others might say well if you can work what are you complaining about.  Yes, I can work.

I've seen the difference though.  I just dropped a solid service that was more stable with better throughput had zero modem issues for a service with a fatter pipe (which truly is a miracle after being on 5-50 Mb) that is hampered by hardware.  ~100$ a month is nothing to sneeze at and this is the discounted rate!  I don't think I'm asking for too much...

[BS]

I was doing some further searching on hitron users and related forms from ISP's and it has been suggested that there may be issues with the port being blocked for RDP.  There is port forwarding capabilities in the interface, but I haven't had to deal with port forwarding in a long time, and it was never one of my skill sets - others wold advise me of the actions, and I would assist in role out where needed.

Again, good luck - any luck with pushing this issue up to a higher level tech support in Rogers, just wondering.  Sometimes you get a person who likes a challenge - too bad RogersDave is no longer around.

Bruce

[hoek]

So I brought a laptop with the same VPN clients I've tested with on my connection to a friend who has 100Mb service.

It  was even worse on his end.  Speeds topped out around 2 Mbit/s.  I attribute it to the fact that he also has a Hitron modem except it's an older CGN3.

I think there's enough evidence that certain VPN traffic is hobbled by Hitron modems.  Presumably anything equipped with this Puma garbage.

Just google it.  These chips seem plagued with issues going back several generations.  Very strong "do not buy" vibe out there.

As I mentioned a colleague using a TP Link 7650 equipped with broadcom easily gets a stable 5-6 Mbit/s. 

Mine is all over the place 600KB/s - 6Mbit/s frequently only 1 or 2 Mbit.

One day soon I hope to work up the stamina to call tech support and see if I can start an actual case about this but I frankly don't see where this can go.  

Rogers please ditch these modems for Broadcom based ones or, perish the thought, let us use our own modems.  Just bring the service to my house and I will worry about the rest PLEASE!

[Datalink]

@hoek this isn't a Hitron issue, its an Intel Puma chipset or firmware issue.  Ideally this would be nothing more than a QOS selection made by the modem (which could be easily changed), but, there has never been any feedback on the real cause of the poor IPSEC and VPN performance, so anything is possible.   Any comments regarding the poor IPSEC and VPN performance should be directed to @RogersSergio so that he can direct/request the engineers to move this issue higher in the "to do" list. 

[hoek]

Yes I understand.  Intel chip/firmware issue.  I can't use a different modem on Rogers' service though so ultimately it becomes a modem issue.

I appreciate that code takes time to go from lab to production factoring in all the stakeholders, service provider priorities, shareholders etc to maintain a service presumably to hundreds of thousands of customers.  I also appreciate that some problems are lower priority than others.  If 500 customers are threatening to leave because their favorite MMO lags I'm sure that will receive more attention that the 12 people complaining about specific VPN issues.

@RogersSergio any mention of where IPSec/VPN throughput issues is on the list of issues being looked at would be nice.  I'm willing to provide any data, testing, try beta firmware, whatever might help.  But it would be nice to know if it's even on the radar and for when.  Weeks, months, years.