IPv6 Routing Broken to Wightman (Local Ontario ISP)

[patheffernan]

Hello,

My dad uses Wightman for home internet and e-mail. Since Rogers has rolled out IPv6 on their LTE network, his Rogers LTE phone (Galaxy S5) cannot send/receive mail while on the Rogers LTE network (however he can when connected to home wireless). 

Yes, I have verified all phone settings are correct, mobile data is enabled, etc etc. I could also reproduce the issue using my laptop's mail client when tethered to the Rogers network, so it's not a problem with the phone.

I've done some troubleshooting to help, and it appears that Rogers routing to the Wightman prefix is broken:

$ host mail.wightman.ca

mail.wightman.ca is an alias for lvs0a.wightman.ca.

lvs0a.wightman.ca has address 68.69.147.20

lvs0a.wightman.ca has IPv6 address 2604:6400:0:1a::20

While tethered on my dad's phone on Rogers LTE, with an IPv6 address, tcp traceroute to the SMTP mailserver port routes to a prefix registered in China before being dropped.

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

ether 48:d7:05:d8:ba:0f

inet6 fe80::182b:3aa5:1523:4a39%en0 prefixlen 64 secured scopeid 0x4

inet6 2605:8d80:6a0:bddb:2f:6f9c:6d06:afe5 prefixlen 64 autoconf secured

inet6 2605:8d80:6a0:bddb:103:5378:a501:f50e prefixlen 64 autoconf temporary

nd6 options=201<PERFORMNUD,DAD>

media: autoselect

status: active

$ sudo mtr --tcp -P 587 -6 mail.wightman.ca

                                     My traceroute  [v0.86]

                                                   Sun Apr 30 20:41:16 2017

Resolver error: Received reply from unknown source: 2607:f798:18:10::uit

                                                       Packets               Pings

 Host                                                Loss%   Snt   Last   Avg  Best  Wrst StDev

 1. 2605:8d80:6a0:7c9b:a3d6:9344:99c7:515a            0.0%    47    2.3   5.4   1.8  86.6  13.0

 2. ???

 3. fdfe:dcba:9876:1152::66                          36.2%    47  218.7 163.2  41.0 378.9  76.3

    fdfe:dcba:9876:1150::2

 4. 2607:f230:0:115::1                                0.0%    47   34.1  46.4  25.3 215.1  34.5

    2607:f230:0:114::1

 5. 2607:f230:144:7::1                                0.0%    47   34.3  47.9  24.5 181.2  33.2

    2607:f230:144:6::1

 6. 2607:f230:0:10d::1                                0.0%    47   34.8  51.3  24.0 209.7  45.1

    2607:f230:0:10e::1

 7. 2607:f798:10:34::690:6324:8233                   46.8%    47   45.9  51.0  29.9 184.2  38.0

 8. 2607:f798:10:295::2                               0.0%    47   34.3  51.9  25.4 273.5  51.9

    2607:f798:10:293::2

 9. 2607:f798:14:a::2                                36.2%    47   77.0  64.1  43.4 153.9  26.7

10. 2001:550:2:8::44:2                               93.5%    47   59.4  55.2  48.9  59.4   5.4

11. 2001:550:2:8::44:2                               91.3%    47  3333. 3259. 3223. 3333.  50.6

12. ???

13. 2001:550:2:8::44:2                               93.5%    47  3273. 3272. 3239. 3304.  32.4

14. 2001:550:2:8::44:2                               97.9%    47  3181. 3181. 3181. 3181.   0.0

15. ???

16. 2001:550:2:8::44:2                               95.3%    44  3431. 3378. 3326. 3431.  74.2

17. ???

18. ???

19. 2001:550:2:8::44:2                               92.3%    27  3224. 3221. 3219. 3224.   3.9

20. ???

Whereas for v4, it's fine. I configured the Wightman SMTP and IMAP servers on my laptop and they also could not send/receive mail, until I disabled IPv6 on my laptop. Unfortunately disabling IPv6 on an android phone is not really an option.

Also notice that the packet loss between 2607:f798:10:295::2  and 2001:550:2:8::44:2  is >90%, which is also interesting, since it's a peering between Rogers and Cogent. However It seems strange that Rogers would route traffic to a local ontario ISP to a US ISP, so I suspect this is a red herring and the routing is the real problem. Of course, I could be wrong :).

I'm posting here in case someone can help. I've notified Wightman as well, since this could be a problem on either Wightman's or Roger's end, and I just want to help solve it because it's impacting my dad's ability to run his business.

That being said, IPv6 routing from some other locations in the US and EMEA I tried is also broken to mail.wightman.ca, so I'm not 100% sure this problem is exclusive to Rogers, unless Rogers is the only network Wightman peers with. In cases from the US my traffic always appeared to route through Cogent as well.

Anyway, any help is appreciated here. Thanks!

*Edited Labels*

[RogersDave · Accepted solution]

Good morning patheffernan,

I checked the routing for the hostname you provided (mail.wightman.ca) using IPv6. Rogers does not have direct connectivity with Wightman. In order to connect to them, we must go through a transit provider, which is in this case as you identified Cogent.

We hand over the traffic to Cogent in Toronto (they announce through routing protocol that they have connectivity back to Wightman) but once the traffic reaches Cogent, they don’t know how to route it further and drop the traffic.

The problem is either a routing policy at Cogent or an issue with the IPv6 interconnection between Cogent and Wightman. There is not much that can be done on our side but I will try to see if our Network Operation Centre can reach out to Cogent to have them investigate the situation further with Wightman.

As a temporary workaround, you can set the SMTP host in your Dad’s phone to the IPv4 address of the mail server (68.69.147.20) and it should work correctly.

Dave

[patheffernan]

Thanks for the quick response!

Makes sense that this would be Cogent, appreciate reaching out. I really appreciate you stepping up and helping me out here even though it's not a problem with your network.

I will try setting the mail server to the IP as a short term workaround (d'oh - I was so focused on the routing I didn't even think to do that), but I'd obviously like to see the routing fixed as well. I'll reach out to Wightman again with this information as well.