Other people with similar gaming issues across NA seem to have narrowed it down to IPv6, users who have been able to successfully cut off that link to their Xbox reported problems going away. And for myself it only popped up once I upgraded my modem/...
Disable IPv6 Connectivity to an Xbox (CODA -4582)
Unfortunately P2P gaming across different platforms/companies/servers etc hasn't caught on yet with IPv6 and often results in not being able to connect with other users. I had an older Hitron route...