Throttling RTMP to Twitch

[RedLir]

Hello,

I'm curious if any other Twitch broadcasters are experiencing this issue with Rogers.  I get 250 down / 20 up consistently and my upload to Twitch will start to drop frames/packets if I exceed around 3000 bps, which is not even a 1/4 of my upload.  

I've been troubleshooting the connection issues streaming to twitch, for almost a year now with support.   Modems have been replaced. Techs have come and replaced wire and tested to the curb etc.  Speedtests are fine.  The issue persists across multiple machines/cables and every Twitch ingest server that I can reasonably test location wise.

At some point, someone suggested I encapsulate my data by testing with a VPN to the local cities of the ingest servers I was testing.  This is when I realized, that Rogers may in fact be throttling my RTMP uploads by payload. When I use a VPN, and test to any of the ingest servers I had issues with, I get no packet loss or dropped frames to Twitch.  I can push the Twitch cap of 6000 bps without issue with the VPN enabled. ( testing to Toronto, NY, Chicago ingests primarily)  

I did another test.  I streamed my feed to restream.io ( which also uses RTMP) to rule out a rule specific to twitch completely.  I saw the exact same issue.  Without the VPN, I would get frame/packet loss, but with the VPN on, I would have no issue.

I've been running with the VPN for some time now as a workaround, but it's an additional cost of course, and one I shouldn't need to incur.  Ater a few months of solid streaming through the VPN, I tested again without it. As expected I immediately started to lose frames/packets intermittently through the broadcast, due to insufficient network when I disabled the VPN. 

I'm at the point where I think I'm going to need to change providers.  Calling front line support doesn't seem to get me anywhere.  They just want to replace a modem or close my ticket the next day without any feedback/consent or resolution.

 Is anyone else out there experiencing these types of issues uploading to twitch via rogers?  Advice on how I can escalate this to an appropriate resource within the support team who will look at it seriously?

Thanks!

***Edited Labels***

[Nadernt]

I've been having Twitch streaming issues ever since I switched to the CODA modems. I'm on the trial CODA modem right now and it still has the issue. I didn't have the issue with the previous rocket modem.

I have pretty terrible connection quality of all North American Twitch servers, and I even experience issues on the best server for me. 

The lag comes in spikes. It seems that either my connection or the CODA modem can't sustain a constant connection to Twitch. My theory is that it's not only upload but download as well that is affected, since I've been experiencing routine buffering whenever I watch a stream ever since I swapped to the CODA modem.  I have 10Mb up, and even with 3500 bitrate I experience issues. 

I'm gonna attempt disabling IPv6, opening the ports used by Twitch, set the DHCP reservation, and changed my computer's DNS to google's, and I'll report back if I'm getting frame drops still.

Edit: Changed the settings and after a few restarts I ran a 20 minute test stream. Still dropping frames (dropped about 6% of total frames). I ran a Twitch connectivity test using TwitchTest, and I get poor results to all of their North American servers. 

Anything else we can do to fix this?? 

[rfogarasi]

I switched back to the CGNM3552 (Rocket Modem) which I found at a Rogers center near me and it seems like the issue still persists. I had one night of 0% frames drop and everything being cleared up, but every other night has been awful. I can't tell if its the same issue because when I do speed tests the upload is way below my package is suppose to deliver which I'm not sure was the issue with the Coda modem? (From what I can remember it was mostly frame dropping rather than upload) I have a tech coming out this week  but something tells me they won't fix anything, and I just wanna get back to it. I even sent ina  ticket and "We have been unable to identify an issue at this time."

[bruce1234]

I have pretty much discovered that Rogers is throttling internet traffic by using my iPhone to do multiple different speed tests comparing Wifi mode over Rogers 75u vs Telus LTE.  I put a separate post together summarizing my test results in another thread.  I also just about 1 hour ago switched to TekSavvy, which uses the Rogers physical plant, and immediately all speedtest results that were showing slower than Telus LTE, started to perform at the same speed as Telus LTE.

The resident tech expert replied to my original post saying it's most likely backbone routing differences between Rogers and Telus that are responsible for the speed test differences, but that's unlikely the case for TekSavvy as they use the same network as Rogers.

So it's not 100% conclusive proof of throttling, but the results speak for themselves.

Anyone should be able to replicate my test if you have Rogers cable internet and a Bell or Telus Wireless plan.

[Datalink]

Actually, I think you'll find that Teksavvy doesn't use the same network as Rogers.  Rogers uses Cogent for moving data around.  So, a good portion of traffic that starts and ends in Canada routes thru the US, carried by Cogent, and most likely all or a majority of traffic that is US bound is probably handled by Cogent as well.  Rogers has been criticized in the past for traffic routing, but, to be fair, Bell and Shaw take hits from their users for wacky routing and latency that results from that routing.  

Your traffic starts on Rogers and from the CMTS is routed to the Teksavvy data center which I believe is located in London.  What Teksavvy does with it from there is anyone's guess.  So your results, comparing Rogers to Teksavvy should be much different.  That shouldn't be a surprise.  You could always post a question in the DSLReports forum and ask them how they route their traffic.  Don't know if you'll get an indepth response or not.  So, as @Gdkitty pointed out earlier, routing plays a big part in your test results.  What you've seen is that perhaps Teksavvy uses more intelligent routing, compared to the big ISPs in Canada.

[Tal3]

8 hour stream switching back to Start, nuff said.

[bruce1234]

Reply to Datalink, Rogers is by far the most expensive internet option where I live, if it's not by far the fastest, no reason to pay the premium. I was actually ok paying a bit more to get more, but why bother paying for 75u if poor routing or throttling is degrading me down to below 10Mb/s.

[lethalsniper]

So what Rogers is now throttling the speeds again no wonder gaming sucks with Rogers rogers with ea sports games is like water and oil the path Rogers takes to the ea servers are brutal

[Nadernt]

Ugh, I selected Toronto as my ingest server. 3500kbps is my bitrate. I was streaming earlier today, and had around <3% frame drops. I tried streaming now during prime time, and I'm getting 30% frame drops. I was assuming the problem is with the CODA modem since I wasn't getting these problems on the old modem, but now I'm just not sure.

[AhmedAskar]

Months and months and it's still an issue.

paying for 30mb upload, can't even get constant 5-10mb upload.

[Adelhart]

I'm having the same issues as well. Despite internet speeds being fine at Gigabit speeds and more than adequate upload speeds, my streams to Twitch are having extreme troubles, always buffering and losing frames. I can't stream for any period of time, and it's frustrating given that I was able to stream perfectly fine multiple times prior to a few weeks ago. 

[Nadernt]

So I got tired of not being able to stream at a good quality/without frame drops, so I went ahead and just bought a router so I can use the CODA in bridge mode.

When I first set it up, I was actually getting way worse of a connection to Twitch ingest servers than I was on the CODA. Upload speeds were very spotty, and the first couple of test streams resulted in really bad frame drops. 

However, I went ahead and followed the steps I think @Datalink posted (disabled IPv6 on the router, opened ports, DHCP reservation, etc) and after a little while, I stopped dropping frames (well, 30 total frames dropped over the course of 2 hours, which is half a seconds worth). I'm not sure which of these fixed the issue, but I disabled IPv6 last and that's when it started working well, so I think that was possibly the issue.

Using the Twitch Bandwidth Test desktop app, I'm getting very good quality numbers to a few servers, as well as maxing out on my upload to them as well, which is pretty good. 

I would get constant buffering while watching Twitch streams as well using the CODA in gateway mode, so I'll have to see if using it in bridge with a router has fixed that issue as well.

I'd still like to see the RTMP problems fixed with the CODA, since my router is a cheap, non-gigabit port, and if I upgrade my plan in the future, it would be nice to revert back to that, be able to stream, and also be able to take full advantage of the speeds.

[AhmedAskar]

Didn't work for me.

Using a router, disabled IPv6, opened ports.

Same old, dropping frames.

[RedLir]

OP here.  Still experiencing the same issue.   The VPN has been the only saving grace for me at least. Unfortunately no other workarounds offered have been effective.

It's disappointing that it's been 6+ months and over 10 pages of people experiencing this,  there has been 0 official Rogers involvement in this thread.  

I'm again opening another ticket with rogers.  We're in the 48 hour period.  I expect they will try the usual "found nothing, ticket closed" trick, and then the onsite visit to tell me nothing is wrong.  I informed the customer support analyst that this tactic has been tried time and time again, and asked them what the next step would be to escalate the issue if nothing was found ( again).  They had no response, other than to suggest I upgrade to a Rogers "Business Line" ...  ... ...  

If nothing is found again, I will be trying to escalate this first through Rogers support management, then publicly on Twitter.  If that yields no results, the next step will be head office and an official CRTC complaint.  It's unfortunate that such drastic measures need to be taken to have something looked at, but enough is enough at this point. 

[bruce1234]

They are throttling traffic after doing a double digit price increase, it's not accidental, it's a business strategy.  They will only change after enough subscribers switch to competitors.

After switching to TekSavvy, I have immediately seen a dramatic performance increase, and it's cheaper.

Forget CRTC and Rogers support, just switch.

[RedLir]

And around and around we go...

I called Rogers support again today, because as expected, I did not receive a notification on my ticket.  I was informed that it was closed as there was an 'outage' and it's now resolved.  Oddly enough, the issue is NOT resolved.  Surprised?

The new analyst had me do a speed test, which comes up fine, then proceeded to his L2?  I was informed that because my modem was in bridged mode ( which the previously requested by them to try) I was not supported and there was nothing they could do since the 'speed test' showed as fine.  I escalated.

Next up was a nice lady in a sales/retention team?  It wasn't really clear where they sent me to, but she informed me she was not technical but would however take all my information and escalate a ticket to the office of the president.  While we were there they also switched me to the 1000down/50up package as it was cheaper for the next 12 months, and would allow me to test that in parallel.  I removed my Asus RT-AC86U router, reset the white box to defaults and I ran a speed test.  I was getting 950 down / 24 up.  Less than half the rated upload speed.  I thanked her for her time, and am awaiting a call from the office of the president before the next step ( CRTC)

I called support back, to find out why exactly I would not be getting near the rated upload speed.  I spoke to a nice lady on the east coast.  She ran through the same tests, and proceeded to tell me that there is so much noise in my area, that it was quite alarming.  Even more alarming, is the fact that the 30 day window she had, showed this noise has been affecting my whole area for at least that many days.  Keep in mind, the VPN trick had been working until last week.  But now I'm thinking... why would the first analyst not have seen this same data, or the analyst before him?  This is also a familiar story that I heard many months ago before they sent an onsite visit, only to tell me everything is fine, and front line has no idea of what they're talking about...  Anyhow, she opened yet another ticket for this 'noise' issue.

When I hung up, I installed Wireshark.  I set the filter for tcp.port == 1935 && tcp.analysis.retransmission

I fired up the test stream.  Sure enough, everytime OBS Studio would tank and start dropping frames, the Wireshark window would be flooded with these retransmissions back to back.  There were 1000's of them over the course of just a few minutes.  I turned off the filter and I could see all the DUP ACK that go along with them.     I tested multiple twitch ingests, and the same pattern was there.  It seems like there is significant packet loss/dropping in the Rogers network.

Given the fact that the VPN trick no longer works, it's likely that there are 2 issues affecting my uploads right now.  First, possibly noise in the area ( we'll see).  Second, the usual failures that occrured where only a VPN connection would resolve the issue.

Unfortunately Rogers support seems to think a speedtest is indicitive of a positive end user experience.  In reality, there are so many dropped packets ( intentional or otherwise), there is no way to achieve a real time streaming experience.

I will let you know how things proceed.  There is at least a 48 hour waiting period on both tickets before I proceed with the next step. 

[Datalink]

Out of curiosity, run a trace to anywhere, it doesn't matter where, www.google.com for example.   Then use the second IP address, which is the CMTS address and run a ping test to that address just to see what the loss numbers look like after a period of time.  I'd run that for an hour for the first run and then for 24 hours for the second run.  When you stop the ping test, take note of the loss numbers.  

That test will show the condition of the modem to CMTS path.  You should only lose a handful of packets over the course of 24 hours.  So, it would be interesting to see what the results look like.  Any CSR can look at the noise figures and pull up the historical noise data, so, there shouldn't be a situation where one CSR doesn't look at the data and the next one does.  If any of the CSR's ran a signal check on the modem, and the noise problem is as bad as it sounds, the signal check should have failed automatically and the CSR should have arranged for a tech visit.  Looking out further, if the CSR checked your neighbours and worked back towards the CMTS, if the noise levels were consistently high, then a real Rogers tech and possibly a Maintenance Crew should have been dispatched.  

Can you confirm one item for me, you indicated that you were switched to the 1 gig down / 50 Mb/s up.  That plan hasn't been offered for several months now.  Was it the current 1 gig / 30 Mb/s plan instead?  Just curious. 

[RedLir]

Hello Datalink

Re the 1000/50 plan, that is a good question.  This is what I was 'told' I was getting, but I don't see any verbiage where this is explicitly written.  The account change email reads "Rogers Ignite Gigabit - UNLTD"   I think what you say makes sense, given that's what's listed on the site and the late night speed test I just did read 32 up.

Re the ping test.  I may have to wait until tomorrow.   I ran a ping to the 2nd and 3rd ip listed in the tracert to the Quebec Twitch ingest, but no failures came back.  Having said that, I'm running a stream test, and at 1am in the morning, I've only dropped a very very small amount of frames.  We are well off peak hours, so that probably makes sense.

Re the noise, the girl I spoke to made it sound like it was my neighbors and everyone else.  I'm a little suspicious but it's hard to know who to believe at this point, given they all have the same role.  She did log a ticket for it though, so I guess if there was a 'real' noise issue, hopefully a maintenance crew would investigate.  I guess I'll have to wait the 48 hours and find out.  ***EDIT  I JUST received an email from them at 1:15 AM stating "We were unable to identify an issue at this time" ....

I suspect come peak times on a Sunday morning/afternoon, the frames will start dropping as per usual.  I'll fire up the ping tests at the same time.

Oh and just another point.  Re all the Puma chipset stuff.  I've also ordered a TP-Link TC7650 (v1.0.3 Build 20161117 Rel358190) that apparently has a broadcom chipset, in case I need to test moving to TekSavvy or Start.ca as an alternative test.  I don't believe Rogers supports this officially themselves, but I could be mistaken.  In either case, if it was the modem, I would imagine I should see the same level of dropped frames at all hours? 

[Makaveli99]

I don't stream on twitch but wanted to see what my results would be

on 500u

[RedLir]

@Makaveli99 This tool is good for telling you a decent ingest for you, but it doesn't represent a real world test.  It's a short 10s burst, and not at a constant bit rate of 5000 over a stream length of time.  If I run that tool, I get similar results, but setting up a real stream in OBS or Xplit will reproduce the issue consistently for me.

@Datalink  The net seems to be stabilizing 'with' the VPN.  I was able to stream last night with the VPN and only had a brief moment where I started to lose packets.  I also let the ping test run during the stream.  In about 3 hours we lost 25 total pings.  The longest was 1500ms.

So at this point, they found nothing regarding the noise, I'm not sure what that was all about.  I'm contemplating calling back on that to see if a new analyst sees the same thing or if it was a fluke.

There is still a packet loss issue even with the VPN, but this could have been a hiccup in that route.  It was definitely nowhere as bad as it has been in the last week or so.  There is still the original issue I reported without the VPN, some months ago.  If I don't use a VPN, Wireshark shows DUP ACK and Rretransmits when OBS is starved and shows dropped frames. 

I'm awaiting the call from Rogers to see what the next step is.

[Ivan67]

Hi there,
I am not sure if i am experiencing the same issues as you but it seems to be the same case to me. I am not the most tech savvy or technical person.

But my issue is, I am only seeing issues when both me and another person in my house are both playing games and I am streaming to twitch through OBS studio. When i am the only one i am not seeing any issues only when another person connects and plays games also. However, I have the 500 download and 20 up package and i should have much more than enough bandwidth to stream to twitch even with another person connected as i believe. 

On OBS i have played around with many settings with the lowest bitrate set to 800 and highest being 2500. and tried multiple ingest servers (Toronto, Quebec, Chicago, New York).

The core issue is : About once every 40 min to an hour, when both pc are gaming and i am streaming, we experience a MASSIVE spike in ping on the network. Usually during game play we hover around 35ms-50ms depending on time of day and noise in network etc. but when it spikes i get anywhere up from 2000ms to 10000ms for about 10-20 sec.

now i am not sure if this is similar to your issue in terms of frames being dropped. But i have had tech people sent to my house for about 3 weeks replaced all wires from the box to my house and the wires in my house as well, along with a replacement modem, and being told that everything looks fine on their end and they see no issues recorded.
I still experience the same issues like clockwerk while streaming. It is very frustrating at this point and i have even tried the VPN trick @RedLir provided with no luck.

I can get some logs for reference later tonight when i get home.

[HughR]

These symptoms seem to suggest violations of network neutrality.  But another possible explanation is "buffer bloat": https://en.wikipedia.org/wiki/Bufferbloat

I found a few of the recent entries in this blog relevant: https://gettys.wordpress.com/

The most recent one is not on topic but it is interesting.