09-15-2017 12:25 PM - last edited on 09-15-2017 01:00 PM by RogersZia
Hello,
I'm curious if any other Twitch broadcasters are experiencing this issue with Rogers. I get 250 down / 20 up consistently and my upload to Twitch will start to drop frames/packets if I exceed around 3000 bps, which is not even a 1/4 of my upload.
I've been troubleshooting the connection issues streaming to twitch, for almost a year now with support. Modems have been replaced. Techs have come and replaced wire and tested to the curb etc. Speedtests are fine. The issue persists across multiple machines/cables and every Twitch ingest server that I can reasonably test location wise.
At some point, someone suggested I encapsulate my data by testing with a VPN to the local cities of the ingest servers I was testing. This is when I realized, that Rogers may in fact be throttling my RTMP uploads by payload. When I use a VPN, and test to any of the ingest servers I had issues with, I get no packet loss or dropped frames to Twitch. I can push the Twitch cap of 6000 bps without issue with the VPN enabled. ( testing to Toronto, NY, Chicago ingests primarily)
I did another test. I streamed my feed to restream.io ( which also uses RTMP) to rule out a rule specific to twitch completely. I saw the exact same issue. Without the VPN, I would get frame/packet loss, but with the VPN on, I would have no issue.
I've been running with the VPN for some time now as a workaround, but it's an additional cost of course, and one I shouldn't need to incur. Ater a few months of solid streaming through the VPN, I tested again without it. As expected I immediately started to lose frames/packets intermittently through the broadcast, due to insufficient network when I disabled the VPN.
I'm at the point where I think I'm going to need to change providers. Calling front line support doesn't seem to get me anywhere. They just want to replace a modem or close my ticket the next day without any feedback/consent or resolution.
Is anyone else out there experiencing these types of issues uploading to twitch via rogers? Advice on how I can escalate this to an appropriate resource within the support team who will look at it seriously?
Thanks!
***Edited Labels***
09-15-2017 12:49 PM
There are a few of us in the Rogers Online Gaming thread that are experiencing this very issue. I only started experiencing it myself in mid-August after upgrading to the Ignite 500u plan and being given a CODA-4582 modem. At first I thought the issue was related to that, but this week I changed my plan to the Ignite 150u just so i could get the modem model that wasn't giving me issues back (the CGN3ACSMR) but I'm still getting the issues.
That's interesting about the VPN - which one are you using? I hadn't though to test it that way.
09-15-2017 01:05 PM
I've had several models of modem along the way. Now it's the new white one they give for the gigabit plan. They replaced it a second time because apparently the first was faulty and I needed one with 2 or 3 dots, not 1... I don't have the model numbers handy but, it's definitely not a modem/model issue in my case. Unless of course they have hidden features in it that throttle at that level...
I tried ExpressVPN trial just to test it initially. The upload speeds aren't massive through the VPN, but it works fine enough for twitch uploading. I suppose a CRTC complaint is always an option, but I'm not sure it's worth the headache vs just changing providers.
09-15-2017 01:30 PM
09-15-2017 02:21 PM - edited 09-15-2017 02:35 PM
Food for thought ..... this might not be a matter of Rogers throttling RTMP, or intentionally throttling RTMP. There are a number of factors that are thrown into the mix for any protocol and destination. @AhmedAskar's comments earlier today that he had "0 frame drops on 8-10 hour streams on Ashburn server." is rather interesting. So, that shows that it can be done over the Rogers network, the question is, what's different between your pc, video resolution, stream settings, modem and path to the same server versus those for @AhmedAskar. Somewhere in those details will be the answer, or one of many answers.
Just to point out, a similar situation exists for ESP/IPSEC traffic. From the feedback thread and @RogersDave's continuing post for the version updates:
So, thats a firmware issue that goes way back and is something that Intel will have to address. Are there similar issues with any other protocols, including RTMP? Thats a question that needs to be addressed.
I think the first thing I would look at are the differences in the pc, video resolution and streaming settings with OBS, just to see if anything stands out as an item that could be tested by using different settings. After that its on to the modem model and firmware and route to the server.
Edit: You indicated that you saw a couple of Rogers techs earlier today. Are you in a condo/townhouse/highrise type of building? If so, then that building could have a Multiple-Dwelling Unit (MDU) that provides data services to the building. If the building is big enough, there could be more than one MDU. These are similar to the Cable Modem Termination Systems that service neighborhoods. There is always the possibility that you could be connected to an MDU that has a technical problem.
Now that you have switched modems, can you try pinging the MDU/CMTS, 99.230.32.1 again. Let that run for 10 or 30 minutes and ensure that the focus time in pingplotter is set for Auto. I'd like to see the average time to the MDU/CMTS, which was variable in the previous slides.
09-15-2017 02:47 PM
@Datalink @RedLir So earlier this morning I had two Rogers techs at my building working on the box after my request on Monday. I went home on my break and fired up a test stream at my test rate of 720p 60FPS 4000 kb/s. After 90 seconds there weren't any issues (which is the longest I've seen in a week) so, good sign. I left it running for ten minutes and when I checked again there were no issues. No dropped frames and no drops in upload speed.
I stopped that stream and changed to my normal stream settings - 720p 60FPS 5500 kb/s and have since left that running while I came back to work. As of right now it's still working fine. I'm monitoring the broadcast with the Twitch Inspector tool and there hasn't been any hits or unstable events in the broadcast. When I get home, it will have been running for 2 or 3 hours, and I will post an update to this post then. So looks like my issue might have been something to do with the box at my apartment.
09-15-2017 02:49 PM - edited 09-15-2017 02:52 PM
@Datalink I appreciate your response,
This should not be anything to do with the settings in OBS. The same settings work fine over the VPN. We even went so far to test it with other broadcasting software like xplit just to be sure ( before the VPN idea) For what it's worth, we stream at 720/60FPS using a medium encoding at 5mb/s
It's also been tested on multiple dedicated machines ( I7 6700k, I7 5820, I7 7700k) with multiple fresh OS installs and even different OS installs Win7 vs Win10. Modems, Cat cable, and coax to the box has been replaced. At one point we even installed dedicated intel server grade network cards. The only thing that works is encapsulating the data so it's not known to Rogers what is going out the pipe.
To your second point. If at a firmware level rogers is throttling certain protocols, this is still a (illegal?) problem. At the max high end, the amount of data being pushed is 6mb/s. A limit of 25mb/s would seem trivial, given my max upload is only 20mb/s anyway. We've had at least 3 models of routers, with multiple in those models being replaced through out the life of the issue.
Multiple ingest servers have been tried around the world. They all exhibit the same issue. As soon as we enable the VPN, even if it's not in the same city as the ingest we pick, the problem goes away.
To reiterate, in our case, it's not intermittent on a daily basis. It's intermittent during the stream, almost on a cycle. It will drop 400+ frames and recover. If you check the OBS logs, the issue is not because the encoder couldn't catch up etc, it's because there is no network bandwidth available. This is reproducible every day, 100% of the time. If you drop the payload to somewhere around 2mbps or switch the VPN, the throttling stops.
@MallardQuack I just saw your response. I hope for your case it is fixed, but as for our issue here, I don't see how a VPN enabled vs disabled would fix a hardware issue. Look forward to your update!
09-15-2017 03:34 PM - edited 09-15-2017 04:47 PM
@RedLir the point that I would really like to stress is that this isn't necessarily a case of "Rogers is throttling" any given data type. Any statement of that type will cause a never ending stream of comments and criticism from a number of people who rightfully oppose that idea in any form, but who are more than willing to add their thoughts on the situation without looking at any of the details in this discussion. Simply put, I'm not interested in getting into that type of argument.
What I am trying to point out is the technical details here. You indicated that encapsulating the data appears to solve the issue and the suggestion from that is the fact that users have to hide the data from Rogers so that it goes out at an acceptable rate. I would suggest that this is not the case at all. What it points to is yet another possible processing issue with the Puma 6 and 7 chipsets, designed by Intel and running on Intel provided firmware. Somewhere in the past, in the 2010 to 2012 timeframe, some engineer at Intel, or maybe a group of engineers at Intel determined how the Puma 6 (CGN3xxxx) would process various types of protocols and how fast those protocols would run thru the modem. That information doesn't appear to have trickled down thru the manufacture, to the ISP, to the end user. End result, you have users trying to determine what the "problem" is for any given protocol, pointing fingers at the ISP, and by ISP I mean more than just Rogers. My guess would be that at some point in the Puma 7 (CODA-4582) development, the same protocol priority and data rate constraints used in the Puma 6 chipset were possibly ported over to the Puma 7 chipset. Why screw with success, right? And here we are, arguing over the possibility of intentional throttling by Rogers, "this is still a (illegal?) problem" versus recognizing the limits embedded in the hardware and firmware which is designed by Intel and provided to numerous manufacturers.
The one item that has become evident over the last two to three years is that there isn't sufficient testing at any level, from Intel, to the manufacturers, to the ISPs that would result in exact details outlining the protocol capabilities. That should start with Intel, not the end user, personal opinion. If this had been done, my guess is that we wouldn't be having this conversation. At some point in time, the manufacturers and ISPs would have looked at the results for all of the protocols, determined that there was a problem due to the throughput rates and kicked the issues back to Intel long before the users had caught them. This type of conversation shouldn't be happening.
Having said that, testing takes resources, in specialized equipment in some cases, and more importantly in manpower, designing the tests and reviewing the results to confirm that they are correct and that the final results are reasonable. That takes network engineers with very specific knowledge of the protocols who can instantly recognize a problem when they see it. That takes a lot of experience and at the end of the day, it comes down to company budgets. Personal opinion, the engineers who can run those tests and analyze the results are probably worth their weight in gold as they can save any company huge amounts of manpower in terms of tech support that doesn't have to be provided if the equipment works as it should. Once again, that should start with Intel, during the development cycle. If that testing was done correctly by Intel, we shouldn't be seeing any of these issues.
All personal opinion of course ....... fwiw.
Edit: Just to point out, during the Puma 6 latency debacle which is still ongoing, it was pointed out that Cablelabs should be doing performance testing on modems. Cablelabs runs interoperability testing to certify modems for DOCSIS compliance on the cable modem side. Cablelabs response to the Puma 6 situation was "we don't do performance testing." So, there is no central body that looks at the LAN side of the modem and any throughput issues , WAN to LAN or vice versa, that might exist. Too bad because they are probably ideally placed to run performance testing. They have CMTS equipment, the modems and the engineers who understand how the modems operate and probably the intricacies of the protocols.
Edit 2: You were wondering why a VPN makes a difference. Let me try to answer that one although @RogersDave could provide the full explanation. The modem examines every packet or group of packets to determine the applicable transmit/receive/processing rules and to determine if the packet(s) should be processed by the CPU or by the Hardware Accelerator/Processor. As is seen in the case of ESP/IPSEC, the packets are handled by the CPU if I remember this correctly and are only processed at a 25 Mb/s rate or less. Is there a CPU limitation that restricts that rate, or was it selected by a the flip of a coin many years ago? Don’t know, but, the situation exists.
In the case of the VPN, there is an initial inspection, the VPN is set up and handed over to the Hardware Accelerator/Processor. The tunnel keeps running but there are no more inspections of the data. The data transits through the modem at the VPN rate instead of the rate predetermined for a given protocol. There is of course the Host VPN provider limits and the limit of the pc's encryption rate that would affect the final data rate.
So, that is the way that Intel designed the firmware many years ago, and now, the limitations, if they exist, are bubbling to the surface so to speak as the overlooked protocols are now being put to use.
09-15-2017 05:15 PM
Hi @Datalink
I appreciate your replies!
I understand where you're coming from, I don't care to go on some witch hunt. Like any customer, I want what I'm paying for. I'm here to find some other way to get this on the radar of Rogers as one last attempt before I move on to something else.
Even if by some miracle this is a bug in every modem version and (intel) firmware, they've ever released, it's rogers in the end who is accountable for that, to their customers. I'm presenting the facts and what I see through exhaustive testing, while being met with countless tickets closed with "No problem found". That's not good business, but I digress.
At this point, the issue is not with my setup or hardware.
So it's either:
A) At the router
B) The rogers network
C) Somewhere just outside the rogers network that would still lead to every Twitch ingest and restream.io ingest.
For A), I'm all for hearing form someone who has real information on this. I have my doubts about the connection is being throttled/dropped only when using a very specific application layer protocol. Even if this is the case, my next question would be, how do I get someone at rogers to take this seriously and look at it?
Alternatively, is there a router I can buy and use on the rogers network that doesn't use this chipset. That would at least take it out of the equation.
For B)/C) I think someone at rogers would have to be able to monitor the stream somehow and see why these frames get lost. Again, the question is how to get someone at support to look at this seriously?
09-15-2017 06:02 PM - edited 09-15-2017 06:04 PM
@RedLir @Datalink Okay, so I just got home from work and had left my stream running the whole time I was gone. There were only three short spikes where there was some dropped frames experienced, but after 3 and a half hours, the total was 0.2%, so a negligible amount of frames in my opinion, especially following the 50%+ I've been dealing with over the last little while. I'm not sure what the actual cause was, but I'm assuming there was some issue between my computer/modem, the box of my building and wherever that signal was being sent to. Fixing the box seems to have helped.
I'm not saying I'm out of the woods yet, I'll continue to monitor it and let you all know if the issue rears it's head again.
EDIT: The 3.5 hour stream I left running was configured as such in OBS:
720p / 60FPS
5500 kb/s Bitrate
Server selected was "Auto" but that just connected me to the Toronto Ingest server.
09-15-2017 06:14 PM
@MallardQuack It will be interesting to see what your results are during "Prime Time". Please let me know what you find.
09-16-2017 02:23 PM
In regards to the modem and version, these are the specs I pulled off this morning.
CODA-4582
Hardware Version | 1A |
Software Version | 2.0.10.28T2 |
09-16-2017 07:09 PM
@RedLir Ran a second test stream last night from around 8 until nearly midnight and had no dropped frames that time. Just did a legit stream on my channel, playing a game and with all the add-ons running and went for four hours with no problems. Seems like my issue was tied to the box on my building.
09-18-2017 11:34 AM
@MallardQuack That's good news for you!
I ran several tests this weekend without the VPN, they were all experiencing throttled/dropped frames every 30 seconds or so. As soon as I enable the VPN, solid for hours.
I'm not really sure how to get Rogers to look at this issue or take is seriously. Did you need to escalate your ticket to a specific resource in the support structure, or was it just coincidence that Rogers techs were at your box that day?
Cheers!
09-18-2017 01:52 PM
@RedLir Basically when I had the tech come in on Monday, he checked the signals and said everything was fine but I was showing him the issues I was having with OBS. (I made a second Twitch account that I could stream to freely without sending out pointless notifications to my followers) It seems after I persisted enough he said he would create a ticket to have someone check the main lines coming into the building and that they would be there between 48 hours to a week. I thought it was just something he was telling me to be honest because it seemed rather vague, but on Friday there were two techs outside working and whatever they did rectified the issue.
09-18-2017 02:13 PM - edited 09-18-2017 02:23 PM
The Rogers engineer that can do something about these issues is @RogersDave. You may have noticed his posts in the forum already and I suspect that he's already seen your posts on this issue. Responding to user posts is not his primary job but he takes time, when he can, to address specific issues such as this.
From your previous post:
A. Its very possible that there is a limit on the data rate through the modem for RTMP. Dave would be the engineer on staff who can send this issue to Hitron for further investigation. ESP/IPSEC is one example where there is a definite limit on the data rate through modem. The question is, are there other protocols that are limited for any reason, technical or otherwise? That is something that Hitron and Intel would have to determine.
B. There is no modem that you can buy to use on the network. Rogers supplies the modems at the present time and will only allow approved modems onto the network. At the present time, there is no other non-Intel modem available. Now having said that, are you running a business? If so, I wonder if the business division might have access to other modems? I've never had any interactions with that group so I don't know what access to enterprise equipment they have that would be more suitable for businesses and large organizations. I would think they would.
C. There is always a possibility of congestion issues with the Cable Modem Termination System, which the modem is connected to, and, there is an issue with the CODA-4582 (Intel Puma 7) and large TCP retransmissions. Are the problems you are experiencing a side-effect of congestion or the TCP issue, or is this simply a matter of a capped data rate for RTMP? That is something that Dave, Hitron and Intel will have to sort out.
On another note, have a look at the following post by @J04DAN1 which will be of interest as it details his experience of successfully streaming via wifi instead of ethernet to get around the dropped frame problems that he had while connected thru ethernet to the modem:
http://communityforums.rogers.com/t5/Internet/Rogers-Online-Gaming-Thread/m-p/403570#M48120
09-28-2017 02:41 AM - last edited on 09-28-2017 08:51 AM by RogersMoin
Uploading to Twitch
I'm not sure exactly what information I need to provide, but I've been having an issue with live streaming to twitch. I'm on the Gigabit plan, and while uploading at 4500 bitrate I'll occasionally have my internet speed drop to less than 1mb/s and drop a metric ton of frames.
Examples of this happening:
After countless times of being told that everything seemed fine on Rogers end, I had them send a tech who found that there was indeed an issue with my upload, and he told me that he had fixed it.
Later that night, I did a test and the issue persisted. Here is a picture of about a 2 hours stream I did showing it dropping still.
Here is a Twitch Bandwidth test I did. I typically stream to either the Ashburn, VA server, or the New York, NY server as I've found the Chicago server is always either really good, or really bad and I find the Toronto server is very random as well.
Speedtest.net also shows me about what I'd expect I should be getting, which should be way more than enough for me to be streaming at the 4.5Mb/s that I'm typically at.
If anyone has any insight in what could be wrong, what I can do, or anything like that, please, please help me. I've been struggling with this issue pretty much since the day I upgraded to gigabit, and I've been on the phone with rogers for what feels like 3 weeks now trying to get it fixed just being told that everything seems fine from what they can see.
But if I'm being told everything seems good on their side, and the Tech told me that everything should now be good on my end, but the issue is still happening, obviously something is wrong still and I'm just completely lost at this point.
09-28-2017 11:22 AM
@Token1 I've not found a permanent solution to my issue either. Unfortunately I have the same experience with Rogers support as you have, it just seems to go nowhere.
I'm not sure if you saw my workaround, but for me, the only thing that causes things to be stable, is to use a VPN. You may want to try a free trial ( I'm using Express right now) and connect to the fastest node and leave OBS on auto for ingest selection. For me it's New York or Jersey most of the time. I have 0 dropped frames when using the VPN.
09-28-2017 08:30 PM
Trying it now, running a twitch bandwidth test for almost an hour now with 0 dropped frames. Pretty irritating I'm going to have to now, on top of paying for my internet, also pay for a VPN to be able to use the internet for what I want.
09-29-2017 10:16 AM
OK so that's at least 2 of us that can prove a VPN works around this issue. @Token1are you in a single home or a multi unit apartment?
09-29-2017 10:01 PM
@RedLir I'm in a single home. Did a 5 hour stream with 0 dropped frames, so can confirm the 1 hour wasn't just a fluke.