[add rfc5549 support] - WebUI doesn't allow IPv4 routes with IPv6 gateway

[Leseratte10]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

I've just set up a new instance of Opnsense 24.7 with the intention of replacing my old Linux router. I tried to replicate the following route that's working just fine on my current Linux system:

$ ip -4 r show 10.0.0.0
10.0.0.0/24 via inet6 fe80::1234:56ff:fe78:90ab dev eth3

but I can't manage to do that. I added fe80::1234:56ff:fe78:90ab as a Gateway, then tried to configure a route. But it just errors out with "Specify a valid gateway from the list matching the networks ip protocol."

I mean, yes, it is an unusual route. But all a "route" really is is a statement like "Please send all packets to network A to the Ethernet MAC address of the device that has IP address B", so there isn't really any reason for network A and IP B to belong to the same address family - is there?

If I add this very same route on the command line using # route add -net 10.0.0.0/24 -inet6 fe80::1234:56ff:fe78:90ab%re0, it adds the route just fine, it works, and it also shows up in the WebUI under System -> Routes -> Status, but it does not show up under Routes -> Configuration.

To Reproduce

Steps to reproduce the behavior:

1. Go to System -> Gateways -> Configuration
2. Add a new gateway with an IPv6 address
3. Go to System -> Routes -> Configuration
4. Try to add a route using this gateway to an IPv4 destination.

Expected behavior

I expected it to create the route, just like it would when adding the route manually on the CLI. But instead, it returns an error.

Describe alternatives you considered

A workaround would be to add an additional IPv4 address to that device so it can be reached with the same address family as the target network, but in my opinion the goal should be to move away from IPv4 whenever possible.

I could also just add the route using the command line instead of the GUI, but I fear that that might break at some time in the future or with an update or something.

Screenshots

Additional context

I also tried to "fool" the GUI by adding an IPv4 gateway, then adding the route, then editing the Gateway to be IPv6 instead. The route is then correctly displayed under System -> Routes -> Configuration but not under System -> Routes -> Status, and it doesn't seem to be active.

Also, when I DO manually add the route using the CLI, it looks like not even the system expects such a route, since when I enter "netstat -rn" to look at the routes, the columns are all messed up. But that's probably something that needs to be fixed in the underlying OS.

Internet:
Destination        Gateway            Flags     Netif Expire
10.0.0.0/24        fe80::1234:56ff:fe78:90ab%re0 UGS      re0
127.0.0.1          link#2             UH          lo0
192.168.1.1        link#2             UHS         lo0

Environment

OPNsense 24.7 (amd64).

[AdSchellevis]

At a first glance this looks like rfc5549 (https://datatracker.ietf.org/doc/html/rfc5549), which likely introduces some protocol mapping in between. I have no clue if current FreeBSD support it to be honest, but if you can enter the address, there's likely some glue in the kernel. The formatting thing isn't a huge issue, adding -W likely offers more readable output.

Upstream I found some differentials, but no clue what the actual state is, for reference in case you would like to do some more digging:

https://reviews.freebsd.org/D30398
https://reviews.freebsd.org/D18581

From the firewalls perspective (pf(4)), I have no clue what the effect would be, assuming one end talks ipv4 and the other end ipv6 the stateful firewall can't really make a lot out of this anymore.

For mapping between protocol families, you could have a look at Tayga, natively nat64/nat46 is currently not supported in OPNsense, by my knowledge these are most commonly used in such cases. (most common is still dual stack)

[Leseratte10]

Yes, the links you mentioned are about this feature, but it's not that one end talks IPv4 and the other end talks IPv6.

Tayga is to access an IPv4 destination over IPv6, this is not what I'm intending to do here. I'm using an IPv4 client with an IPv4 IP address to access an IPv4 destination - it just so happens that the next-hop router in-between the source and the destination has no IPv4 address, because there's no need to.

A client sends a request for 10.0.0.1 to OpnSense. OpnSense looks into the routing table, and finds that it needs to use the router at fe80::1234:56ff:fe78:90ab to reach this network. OpnSense does an NDP lookup to find the MAC address owned by this IPv6, and then just sends that packet, still addressed to 10.0.0.1, to the router's MAC address it found through ARP.

That router, without having an IPv4 address itself, will then receive the incoming ethernet packet (because it's addressed to its MAC), will see the destination of 10.0.1.1, and will then look it up in its own routing table and forward it to the destination.

There's no re-packing of IPv4 or IPv6 packets happening, it's just Ethernet packets getting forwarded to the proper MAC.

From the firewall's perspective, it's all just IPv4 traffic. And the route shows up in the IPv4 routing table. Every single packet on the wire is IPv4. The only thing that changes is that it didn't use ARP on the router's IPv4 to figure out the destination MAC, it instead uses NDP on the router's IPv6 to figure out the destination MAC.

If I set up such a route on Linux, it works. And I can also set it up on OpnSense through the command line interface using the command I mentioned in my report. I haven't yet tested if it actually works, but I have no reason to believe that it wouldn't. After all, the kernel accepts it, support was added with the two commits you linked, and it works just fine on Linux.

It's just the WebUI doesn't seem to support it / prevents me from adding such a route.

[AdSchellevis]

@Leseratte10 I haven't read the rfc, but this sounds like it might just work to be honest. If you can test it using your command line modifications, I don't mind converting this ticket into a feature request which implements the constraints on our end.

[Leseratte10]

Will do so once my new network card arrives, the one I intended to use in my machine is apparently not supported so I can't finish setting this up yet.

[Leseratte10]

@AdSchellevis I can confirm that when I set up a route like this in OPNsense 24.7.11_2 ...

# route add -net 10.0.0.0/24 -inet6 fe80::1234:56ff:fe78:90ab%re0

then local traffic generated on the OPNsense machine that's addressed to 10.0.0.0/24 (like a ping) will be sent from interface re0 to the corresponding MAC 10:34:56:78:90:ab and arrives on that device as intended.

I was not yet able to test this with traffic not generated on the router itself (so, traffic where the router would actually need to route something), but I have no reason to believe that the result would be different.

The other way around (IPv6 traffic routed to an IPv4 IP) does not seem to work, though, so the check probably shouldn't be removed but adapted to allow IPv4 traffic to an IPv6 gateway but not IPv6 traffic to an IPv4 gateway.

[AdSchellevis]

ok, let's flip this into a feature request then, it shouldn't be a huge change.

[AdSchellevis]

ok, this 52255d7 should do the trick, I've added the rfc number to the title as well for reference.

[Leseratte10]

Wow, that was faster than I expected. I'll give this a try and see if it works or if I see other issues in the webui.

[Leseratte10]

@AdSchellevis I just tested it, but it's still buggy, can you re-open this issue and/or take another look?

I can add the route in the WebUI, but a "route -rn" doesn't show it.

In the logs I can see the following error:

Error opnsense /usr/local/etc/rc.routing_configure: The command '/sbin/route add -inet '172.17.0.0/24' 'fd00:1234:5678::1'' returned exit code '68', the output was 'route: bad address: fd00:1234:5678::1'

The correct syntax should be one of these depending on the address type:

• route add -net 172.17.0.0/24 -inet6 fd00:1234:5678::1
• route add -net 172.17.0.0/24 -inet6 fe80::1234:5678:1%re0

The WebUI seems to try to use route add -inet 172.17.0.0/24 fd00:1234:5678::1 which isn't the correct syntax.

EDIT: Another bug: If I create the corresponding route using the command line, it is shown under System -> Routes -> Status, but then I click the Delete icon and confirm, the route doesn't get deleted. I can't find an error message for that in the logs, though. Maybe it's the same bug as with adding the route.

EDIT 2: Also, a general issue - why does the route even show up in the WebUI if it's not successfully added to the OS? Does the WebUI have its own list of routes it maintains? Sounds like a recipe for issues when the WebUI's routing table and the OS routing table happen to get out of sync, like in this case ...

[AdSchellevis]

@Leseratte10 ok, I was expecting only a validation, but backend code seems to need some modifications as well. Thanks for testing, I'll try this on my end as well.

[AdSchellevis]

@Leseratte10 can you try to add 37a3284 on top?

[Leseratte10]

@AdSchellevis Thanks for the quick update. It's way better, but not 100% fixed.

I can add a route in the WebUI and it gets added to the system. I can also delete the route and it goes away. I can even edit the Gateway and change it around between IPv4 and IPv6 and the route will update accordingly (which is great, didn't expect that to work).

There's three bugs still, though:

A) When I add the route through the WebUI (and it gets added to the system) and then I reboot the machine, after the reboot the route is still shown under Routes -> Configuration but no longer shows up under Routes -> Status and also doesn't show up on the command line. I need to click on "Apply" again to make it show up. So it looks like there's some code running on boot that reads the OPNsense routes and applies them to the system and that code isn't handling the new routes yet.

B) When I add a gateway with an IPv6 IP, then add a route for an IPv6 network towards that IP, this works just fine. However, when I then edit said gateway to be IPv4 instead, it tries to create a route routing that IPv6 network towards the IPv4 address (which is not supported by the kernel, only the other way around works).
Though I'm not sure what the best way to fix that would be - preventing people from changing the gateway protocol from IPv6 to IPv4 on an existing gateway (other way around is fine!) when such routes exist? Automatically disabling/deleting the route?
But that's probably not a bug in your new code, I think this bug has existed previously, too.

And C) I still found multiple ways to get the WebUI route config and the OS route table to de-sync (including when performing the steps needed for bugs A or B) but these also happen with a normal IPv6 route (routing an IPv6 network to an IPv6 IP) and is thus unrelated to your changes or to this RFC so it's probably better if I open a dedicated bug report for that.

[AdSchellevis]

Is there anything in the system log after boot that points to issues? the static routes should be applied using the same code on boot as from the gui.

Changing protocols on gateways will always be problematic, we can add constraints, but it's not very high on my list of things to do at the moment (if you want to take a stab at it, just open a pull-request to review)

[Leseratte10]

Is there anything in the system log after boot that points to issues? the static routes should be applied using the same code on boot as from the gui.

Hm. Now I tried to reproduce this issue again and now the route stays across reboots. I'm going to try a couple more times to see if that was just a result of me messing with the routes or if there's some kind of race condition ...

Changing protocols on gateways will always be problematic, we can add constraints, but it's not very high on my list of things to do at the moment (if you want to take a stab at it, just open a pull-request to review)

No problem, just wanted to mention it since I noticed it. Constraints for all the potential scenarios is probably going to be quite difficult to do reliably, but maybe a warning can be added when a gateway changes from IPv4 to IPv6 or vice versa that this could cause routes to stop working.

[AdSchellevis]

Hm. Now I tried to reproduce this issue again and now the route stays across reboots. I'm going to try a couple more times to see if that was just a result of me messing with the routes or if there's some kind of race condition ...

ok, on my end it also seems to boot with the proper routes installed, which is what I expected.

No problem, just wanted to mention it since I noticed it. Constraints for all the potential scenarios is probably going to be quite difficult to do reliably, but maybe a warning can be added when a gateway changes from IPv4 to IPv6 or vice versa that this could cause routes to stop working.

Since nothing goes horribly wrong, we ignore the fact for a while :)

[Leseratte10]

I managed to reproduce the route not being added.

I'm using the os-tailscale plugin that was released like two weeks ago, and this issue occurs when I add a route with a gateway in the tailscale network. And the issue also occurs with "normal" routes so again not an issue with your code changes.

I'll go report that to the tailscale plugin developers. I'm assuming that maybe the OS routes are added before the tailscale plugin is ready and thus it's unable to add a route pointing to the not-yet-connected tailscale network.

The rest seems to be working, or at least not more broken than before, so I'd say this is fixed. Thanks for the quick fix!