I'm curious if any other Twitch broadcasters are experiencing this issue with Rogers. I get 250 down / 20 up consistently and my upload to Twitch will start to drop frames/packets if I exceed around 3000 bps, which is not even a 1/4 of my upload.
I've been troubleshooting the connection issues streaming to twitch, for almost a year now with support. Modems have been replaced. Techs have come and replaced wire and tested to the curb etc. Speedtests are fine. The issue persists across multiple machines/cables and every Twitch ingest server that I can reasonably test location wise.
At some point, someone suggested I encapsulate my data by testing with a VPN to the local cities of the ingest servers I was testing. This is when I realized, that Rogers may in fact be throttling my RTMP uploads by payload. When I use a VPN, and test to any of the ingest servers I had issues with, I get no packet loss or dropped frames to Twitch. I can push the Twitch cap of 6000 bps without issue with the VPN enabled. ( testing to Toronto, NY, Chicago ingests primarily)
I did another test. I streamed my feed to restream.io ( which also uses RTMP) to rule out a rule specific to twitch completely. I saw the exact same issue. Without the VPN, I would get frame/packet loss, but with the VPN on, I would have no issue.
I've been running with the VPN for some time now as a workaround, but it's an additional cost of course, and one I shouldn't need to incur. Ater a few months of solid streaming through the VPN, I tested again without it. As expected I immediately started to lose frames/packets intermittently through the broadcast, due to insufficient network when I disabled the VPN.
I'm at the point where I think I'm going to need to change providers. Calling front line support doesn't seem to get me anywhere. They just want to replace a modem or close my ticket the next day without any feedback/consent or resolution.
Is anyone else out there experiencing these types of issues uploading to twitch via rogers? Advice on how I can escalate this to an appropriate resource within the support team who will look at it seriously?
There are a few of us in the Rogers Online Gaming thread that are experiencing this very issue. I only started experiencing it myself in mid-August after upgrading to the Ignite 500u plan and being given a CODA-4582 modem. At first I thought the issue was related to that, but this week I changed my plan to the Ignite 150u just so i could get the modem model that wasn't giving me issues back (the CGN3ACSMR) but I'm still getting the issues.
That's interesting about the VPN - which one are you using? I hadn't though to test it that way.
I've had several models of modem along the way. Now it's the new white one they give for the gigabit plan. They replaced it a second time because apparently the first was faulty and I needed one with 2 or 3 dots, not 1... I don't have the model numbers handy but, it's definitely not a modem/model issue in my case. Unless of course they have hidden features in it that throttle at that level...
I tried ExpressVPN trial just to test it initially. The upload speeds aren't massive through the VPN, but it works fine enough for twitch uploading. I suppose a CRTC complaint is always an option, but I'm not sure it's worth the headache vs just changing providers.
Food for thought ..... this might not be a matter of Rogers throttling RTMP, or intentionally throttling RTMP. There are a number of factors that are thrown into the mix for any protocol and destination. @AhmedAskar's comments earlier today that he had "0 frame drops on 8-10 hour streams on Ashburn server." is rather interesting. So, that shows that it can be done over the Rogers network, the question is, what's different between your pc, video resolution, stream settings, modem and path to the same server versus those for @AhmedAskar. Somewhere in those details will be the answer, or one of many answers.
Just to point out, a similar situation exists for ESP/IPSEC traffic. From the feedback thread and @RogersDave's continuing post for the version updates:
So, thats a firmware issue that goes way back and is something that Intel will have to address. Are there similar issues with any other protocols, including RTMP? Thats a question that needs to be addressed.
I think the first thing I would look at are the differences in the pc, video resolution and streaming settings with OBS, just to see if anything stands out as an item that could be tested by using different settings. After that its on to the modem model and firmware and route to the server.
Edit: You indicated that you saw a couple of Rogers techs earlier today. Are you in a condo/townhouse/highrise type of building? If so, then that building could have a Multiple-Dwelling Unit (MDU) that provides data services to the building. If the building is big enough, there could be more than one MDU. These are similar to the Cable Modem Termination Systems that service neighborhoods. There is always the possibility that you could be connected to an MDU that has a technical problem.
Now that you have switched modems, can you try pinging the MDU/CMTS, 18.104.22.168 again. Let that run for 10 or 30 minutes and ensure that the focus time in pingplotter is set for Auto. I'd like to see the average time to the MDU/CMTS, which was variable in the previous slides.
@Datalink @RedLir So earlier this morning I had two Rogers techs at my building working on the box after my request on Monday. I went home on my break and fired up a test stream at my test rate of 720p 60FPS 4000 kb/s. After 90 seconds there weren't any issues (which is the longest I've seen in a week) so, good sign. I left it running for ten minutes and when I checked again there were no issues. No dropped frames and no drops in upload speed.
I stopped that stream and changed to my normal stream settings - 720p 60FPS 5500 kb/s and have since left that running while I came back to work. As of right now it's still working fine. I'm monitoring the broadcast with the Twitch Inspector tool and there hasn't been any hits or unstable events in the broadcast. When I get home, it will have been running for 2 or 3 hours, and I will post an update to this post then. So looks like my issue might have been something to do with the box at my apartment.
@Datalink I appreciate your response,
This should not be anything to do with the settings in OBS. The same settings work fine over the VPN. We even went so far to test it with other broadcasting software like xplit just to be sure ( before the VPN idea) For what it's worth, we stream at 720/60FPS using a medium encoding at 5mb/s
It's also been tested on multiple dedicated machines ( I7 6700k, I7 5820, I7 7700k) with multiple fresh OS installs and even different OS installs Win7 vs Win10. Modems, Cat cable, and coax to the box has been replaced. At one point we even installed dedicated intel server grade network cards. The only thing that works is encapsulating the data so it's not known to Rogers what is going out the pipe.
To your second point. If at a firmware level rogers is throttling certain protocols, this is still a (illegal?) problem. At the max high end, the amount of data being pushed is 6mb/s. A limit of 25mb/s would seem trivial, given my max upload is only 20mb/s anyway. We've had at least 3 models of routers, with multiple in those models being replaced through out the life of the issue.
Multiple ingest servers have been tried around the world. They all exhibit the same issue. As soon as we enable the VPN, even if it's not in the same city as the ingest we pick, the problem goes away.
To reiterate, in our case, it's not intermittent on a daily basis. It's intermittent during the stream, almost on a cycle. It will drop 400+ frames and recover. If you check the OBS logs, the issue is not because the encoder couldn't catch up etc, it's because there is no network bandwidth available. This is reproducible every day, 100% of the time. If you drop the payload to somewhere around 2mbps or switch the VPN, the throttling stops.
@MallardQuack I just saw your response. I hope for your case it is fixed, but as for our issue here, I don't see how a VPN enabled vs disabled would fix a hardware issue. Look forward to your update!
@RedLir the point that I would really like to stress is that this isn't necessarily a case of "Rogers is throttling" any given data type. Any statement of that type will cause a never ending stream of comments and criticism from a number of people who rightfully oppose that idea in any form, but who are more than willing to add their thoughts on the situation without looking at any of the details in this discussion. Simply put, I'm not interested in getting into that type of argument.
What I am trying to point out is the technical details here. You indicated that encapsulating the data appears to solve the issue and the suggestion from that is the fact that users have to hide the data from Rogers so that it goes out at an acceptable rate. I would suggest that this is not the case at all. What it points to is yet another possible processing issue with the Puma 6 and 7 chipsets, designed by Intel and running on Intel provided firmware. Somewhere in the past, in the 2010 to 2012 timeframe, some engineer at Intel, or maybe a group of engineers at Intel determined how the Puma 6 (CGN3xxxx) would process various types of protocols and how fast those protocols would run thru the modem. That information doesn't appear to have trickled down thru the manufacture, to the ISP, to the end user. End result, you have users trying to determine what the "problem" is for any given protocol, pointing fingers at the ISP, and by ISP I mean more than just Rogers. My guess would be that at some point in the Puma 7 (CODA-4582) development, the same protocol priority and data rate constraints used in the Puma 6 chipset were possibly ported over to the Puma 7 chipset. Why screw with success, right? And here we are, arguing over the possibility of intentional throttling by Rogers, "this is still a (illegal?) problem" versus recognizing the limits embedded in the hardware and firmware which is designed by Intel and provided to numerous manufacturers.
The one item that has become evident over the last two to three years is that there isn't sufficient testing at any level, from Intel, to the manufacturers, to the ISPs that would result in exact details outlining the protocol capabilities. That should start with Intel, not the end user, personal opinion. If this had been done, my guess is that we wouldn't be having this conversation. At some point in time, the manufacturers and ISPs would have looked at the results for all of the protocols, determined that there was a problem due to the throughput rates and kicked the issues back to Intel long before the users had caught them. This type of conversation shouldn't be happening.
Having said that, testing takes resources, in specialized equipment in some cases, and more importantly in manpower, designing the tests and reviewing the results to confirm that they are correct and that the final results are reasonable. That takes network engineers with very specific knowledge of the protocols who can instantly recognize a problem when they see it. That takes a lot of experience and at the end of the day, it comes down to company budgets. Personal opinion, the engineers who can run those tests and analyze the results are probably worth their weight in gold as they can save any company huge amounts of manpower in terms of tech support that doesn't have to be provided if the equipment works as it should. Once again, that should start with Intel, during the development cycle. If that testing was done correctly by Intel, we shouldn't be seeing any of these issues.
All personal opinion of course ....... fwiw.
Edit: Just to point out, during the Puma 6 latency debacle which is still ongoing, it was pointed out that Cablelabs should be doing performance testing on modems. Cablelabs runs interoperability testing to certify modems for DOCSIS compliance on the cable modem side. Cablelabs response to the Puma 6 situation was "we don't do performance testing." So, there is no central body that looks at the LAN side of the modem and any throughput issues , WAN to LAN or vice versa, that might exist. Too bad because they are probably ideally placed to run performance testing. They have CMTS equipment, the modems and the engineers who understand how the modems operate and probably the intricacies of the protocols.
Edit 2: You were wondering why a VPN makes a difference. Let me try to answer that one although @RogersDave could provide the full explanation. The modem examines every packet or group of packets to determine the applicable transmit/receive/processing rules and to determine if the packet(s) should be processed by the CPU or by the Hardware Accelerator/Processor. As is seen in the case of ESP/IPSEC, the packets are handled by the CPU if I remember this correctly and are only processed at a 25 Mb/s rate or less. Is there a CPU limitation that restricts that rate, or was it selected by a the flip of a coin many years ago? Don’t know, but, the situation exists.
In the case of the VPN, there is an initial inspection, the VPN is set up and handed over to the Hardware Accelerator/Processor. The tunnel keeps running but there are no more inspections of the data. The data transits through the modem at the VPN rate instead of the rate predetermined for a given protocol. There is of course the Host VPN provider limits and the limit of the pc's encryption rate that would affect the final data rate.
So, that is the way that Intel designed the firmware many years ago, and now, the limitations, if they exist, are bubbling to the surface so to speak as the overlooked protocols are now being put to use.
I appreciate your replies!
I understand where you're coming from, I don't care to go on some witch hunt. Like any customer, I want what I'm paying for. I'm here to find some other way to get this on the radar of Rogers as one last attempt before I move on to something else.
Even if by some miracle this is a bug in every modem version and (intel) firmware, they've ever released, it's rogers in the end who is accountable for that, to their customers. I'm presenting the facts and what I see through exhaustive testing, while being met with countless tickets closed with "No problem found". That's not good business, but I digress.
At this point, the issue is not with my setup or hardware.
So it's either:
A) At the router
B) The rogers network
C) Somewhere just outside the rogers network that would still lead to every Twitch ingest and restream.io ingest.
For A), I'm all for hearing form someone who has real information on this. I have my doubts about the connection is being throttled/dropped only when using a very specific application layer protocol. Even if this is the case, my next question would be, how do I get someone at rogers to take this seriously and look at it?
Alternatively, is there a router I can buy and use on the rogers network that doesn't use this chipset. That would at least take it out of the equation.
For B)/C) I think someone at rogers would have to be able to monitor the stream somehow and see why these frames get lost. Again, the question is how to get someone at support to look at this seriously?
@RedLir @Datalink Okay, so I just got home from work and had left my stream running the whole time I was gone. There were only three short spikes where there was some dropped frames experienced, but after 3 and a half hours, the total was 0.2%, so a negligible amount of frames in my opinion, especially following the 50%+ I've been dealing with over the last little while. I'm not sure what the actual cause was, but I'm assuming there was some issue between my computer/modem, the box of my building and wherever that signal was being sent to. Fixing the box seems to have helped.
I'm not saying I'm out of the woods yet, I'll continue to monitor it and let you all know if the issue rears it's head again.
EDIT: The 3.5 hour stream I left running was configured as such in OBS:
720p / 60FPS
5500 kb/s Bitrate
Server selected was "Auto" but that just connected me to the Toronto Ingest server.