I would like to share the experience of combining networks in three geographically remote apartments, in each of which routers with OpenWRT are used as a gateway into one common network. When choosing a method for combining networks between L3 with routing subnets and L2 with bridging, when all nodes of the network will be on the same subnet, the second method was preferred, which was more difficult to configure, but giving more opportunities, since the network was planned to use transparent technologies Wake-on-Lan and DLNA.
Part 1: Background
OpenVPN was initially chosen as the protocol for implementing this task, because, firstly, it can create a tap device that can be easily added to the bridge, and secondly, OpenVPN supports TCP protocol, which was also important, because none of the apartments had a dedicated IP address, and I was not able to use STUN, because for some reason my provider blocks incoming connections via UDP from their networks, while TCP allowed me to forward the port of the VPN server to leased VPS using SSH. Yes, this approach gives a big load, because the data is encrypted twice, but I didn’t want to enter VPS into my private network, as there was a risk of third parties gaining control over it, therefore, having such a device in my home network was extremely undesirable and it was decided pay for security with a big overhead.
To forward the port on the router on which it was planned to deploy the server, the sshtunnel program was used. I will not describe the subtleties of its configuration – this is done quite easily, I just note that its task was to forward the TCP port 1194 from the router to VPS. Next, the OpenVPN server was configured on the tap0 device, which wound up in the br-lan bridge. After checking the connection to the server you just created from the laptop, it became clear that the idea of port forwarding had paid off and my laptop became a member of the router network, although I was not physically there.
Then it was necessary to distribute IP addresses in different apartments so that they did not conflict and configure routers as OpenVPN clients.
The following router IPs and DHCP server ranges were selected:
192.168.10.1 with a range of 192.168.10.2 – 192.168.10.80 for the server
192.168.10.100 with a range of 192.168.10.101 – 192.168.10.149 for the router in apartment No. 2
192.168.10.150 with a range of 192.168.10.151 – 192.168.10.199 for the router in apartment No. 3
It was also necessary to assign these addresses to the client routers of the OpenVPN server by adding the following lines to its configuration:
ifconfig-pool-persist /etc/openvpn/ipp.txt 0
and adding the following lines to the /etc/openvpn/ipp.txt file:
flat1_id 192.168.10.100 flat2_id 192.168.10.150
where flat1_id and flat2_id are the device names specified when creating certificates for connecting to OpenVPN
Next, OpenVPN clients were configured on the routers, tap0 devices on both were added to the br-lan bridge. At this stage, it seemed that everything was in order, since all three networks see each other and work as a whole. However, it turned out not a very pleasant detail: sometimes devices could not get an IP address from their router, with all the ensuing consequences. For some reason, the router in one of the apartments did not have time to respond in time to DHCPDISCOVER and the device did not receive its address. I realized that I need to filter such requests in tap0 on each of the routers, but, as it turned out, iptables cannot work with the device if it is part of the bridge and ebtables should come to my aid. Unfortunately, it wasn’t in my firmware and I had to rebuild the images for each device. Having done this and adding such lines to /etc/rc.local of each router, the problem was solved:
ebtables -A INPUT --in-interface tap0 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP ebtables -A INPUT --in-interface tap0 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP ebtables -A FORWARD --out-interface tap0 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP ebtables -A FORWARD --out-interface tap0 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
This configuration lasted for three years.
Part 2: WireGuard
Recently, on the Internet, they began to talk more and more about WireGuard, admiring the simplicity of its configuration, high transmission speed, low ping with comparable security. The search for additional information about him made it clear that they did not support either working as a bridge member or working with TCP, which made me think that there were still no alternatives to OpenVPN for me. So I put off getting to know WireGuard.
A few days ago, over the resources, one way or another connected with IT, the news flashed that WireGuard will finally be included in the Linux kernel, starting with version 5.6. News articles, as always, praised WireGuard. I again plunged into the search for ways to replace the good old OpenVPN. It talked about building an Ethernet tunnel over L3 using GRE. It remained unclear what to do with the UDP protocol. The search led me to articles on using socat in conjunction with an SSH tunnel to forward a UDP port, however, they noted that this approach only works in single connection mode, i.e. multiple VPN clients would be impossible. I came up with the idea of raising a VPN server on VPS, and setting up GRE for clients, but as it turned out, GRE does not support encryption, which will lead to the fact that in case of access to the server by third parties, all traffic between my networks is in their hands that did not suit me in general.
Again, a decision was made in favor of redundant encryption, by using a VPN over a VPN according to the following scheme:
First level VPN:
- VPS is a server with an internal address of 192.168.30.1
- MS is a VPS client with an internal address of 192.168.30.2
- MK2 is a VPS client with an internal address of 192.168.30.3
- MK3 is a VPS client with an internal address of 192.168.30.4
Second level VPN:
- MS is a server with an external address 192.168.30.2 and an internal 192.168.31.1
- MK2 is an MS client with an address of 192.168.30.2 and has an internal IP of 192.168.31.2
- MK3 is an MS client with an address of 192.168.30.2 and has an internal IP of 192.168.31.3
- MS – router-server in apartment 1, MK2 – router in apartment 2, MK3 – router in apartment 3
Device configurations are published in the spoiler at the end of the article.
So, pings between the nodes of the network 192.168.31.0/24 go, it is time to move on to setting up the GRE tunnel. Before this, in order not to lose access to the routers, it is worth setting up SSH tunnels to forward 22 ports on the VPS, so that_ for example, the router from apartment 2 will be available on the 10022nd VPS port, and on the 11122th port VPS will be available the router is from apartment 3. It is best to configure forwarding with the same sshtunnel, as it will restore the tunnel if it falls.
The tunnel is configured, you can connect to SSH through the forwarded port:
ssh root@VPS -p 10022
Next, disable OpenVPN:
/etc/init.d/openvpn stop
Now configure the GRE tunnel on the router from apartment 2:
ip link add grelan0 type gretap remote 192.168.31.1 local 192.168.31.2 ip link set grelan0 up
And add the created interface to the bridge:
brctl addif br-lan grelan0
Now perform the same procedure on the router-server:
ip link add grelan0 type gretap remote 192.168.31.2 local 192.168.31.1 ip link set grelan0 up
And, also, add the created interface to the bridge:
brctl addif br-lan grelan0
starting from this moment, pings begin to successfully go to the new network and I, with satisfaction, am going to drink coffee. Then, in order to evaluate how the network works on the other end of the wire, I try to connect via SSH to one of the computers in apartment 2, but the ssh client freezes without prompting for a password. I try to connect to this computer via telnet to port 22 and I see a line from which it can be understood that the connection is being established, the SSH server is responding, just for some reason does not offer me to log in.
$ telnet 192.168.10.110 22 SSH-2.0-OpenSSH_8.1
I try to connect to it via VNC and see a black screen. I assure myself that it is a remote computer, because I can easily connect to the router from this apartment at the internal address. However, I decide to connect to the SSH of this computer through a router and I am surprised to find that the connection is successful and the remote computer is working properly, but it also cannot connect to my computer.
I remove the grelan0 device from the bridge and run OpenVPN on the router in apartment 2 and make sure that the network works again as expected and the connections do not break. When I search, I come across forums where people complain about the same problems, where they are advised to raise the MTU.
However, until the MTU was set large enough – 7000 for gretap devices, either TCP disconnects or a low transmission rate were observed. Due to the high MTU for gretap, MTUs for WireGuard connections of the first and second level were set at 8000 and 7500, respectively.
I made a similar configuration on the router from apartment 3, with the only difference being that on the router the server added a second gretap interface named grelan1, which was also added to the br-lan bridge.
Everything works. Now you can put the gretap assembly at startup. For this:
Put these lines in /etc/rc.local on the router in apartment 2:
ip link add grelan0 type gretap remote 192.168.31.1 local 192.168.31.2 ip link set dev grelan0 mtu 7000 ip link set grelan0 up brctl addif br-lan grelan0
Added this to /etc/rc.local on the router in apartment 3:
ip link add grelan0 type gretap remote 192.168.31.1 local 192.168.31.3 ip link set dev grelan0 mtu 7000 ip link set grelan0 up brctl addif br-lan grelan0
And on the server router:
ip link add grelan0 type gretap remote 192.168.31.2 local 192.168.31.1 ip link set dev grelan0 mtu 7000 ip link set grelan0 up brctl addif br-lan grelan0 ip link add grelan1 type gretap remote 192.168.31.3 local 192.168.31.1 ip link set dev grelan1 mtu 7000 ip link set grelan1 up brctl addif br-lan grelan1
After rebooting the client routers, I found that for some reason they did not connect to the server. Having connected to their SSH (fortunately, I preconfigured sshtunnel for this), it was discovered that WireGuard was creating a route for the endpoint for some reason, but it was incorrect. So, for 192.168.30.2, the route table specified the route through the pppoe-wan interface, what means, via the Internet, although the route to it should have been routed through the wg0 interface. After deleting this route, the connection was restored. I could not find somewhere instructions on how to make WireGuard not create these routes. Moreover, I did not even understand, a feature is OpenWRT, or WireGuard itself. Without having to deal with this problem for a long time, I simply added a line to this router to the script looped by the timer to delete this route:
route del 192.168.30.2
Summary
I have not yet achieved a complete rejection of OpenVPN, since I sometimes need to connect to a new network from a laptop or phone, and setting up a gretap device on them is generally impossible, but despite this, I got an advantage in the data transfer speed between apartments and, for example, the use of VNC now does not cause inconvenience. Ping decreased slightly, but became more stable:
When using OpenVPN:
[r0ck3r@desktop ~]$ ping -c 20 192.168.10.110 PING 192.168.10.110 (192.168.10.110) 56(84) bytes of data. 64 bytes from 192.168.10.110: icmp_seq=1 ttl=64 time=133 ms ... 64 bytes from 192.168.10.110: icmp_seq=20 ttl=64 time=125 ms --- 192.168.10.110 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 19006ms rtt min/avg/max/mdev = 124.722/126.152/136.907/3.065 ms
When using WireGuard:
[r0ck3r@desktop ~]$ ping -c 20 192.168.10.110 PING 192.168.10.110 (192.168.10.110) 56(84) bytes of data. 64 bytes from 192.168.10.110: icmp_seq=1 ttl=64 time=124 ms ... 64 bytes from 192.168.10.110: icmp_seq=20 ttl=64 time=124 ms --- 192.168.10.110 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 19003ms rtt min/avg/max/mdev = 123.954/124.423/126.708/0.675 ms
It is more affected by high ping before VPS, which is approximately 61.5 ms.
However, the speed has increased significantly. So, in an apartment with a router-server, I have an Internet connection speed of 30 Mbps, and in other apartments – 5 Mbps. At the same time, while using OpenVPN, I was not able to achieve a data transfer rate between networks of more than 3.8 Mb / s according to iperf, while WireGuard “pumped” it to the same 5 Mb / s.
In the described configurations for the second-level VPN, I specify the 51821 port to WireGuard clients. In general, this is not necessary, since the client will establish a connection from any free unprivileged port, but I made it so that I could block all incoming connections on the wg0 interfaces of all routers, except incoming UDP connections to port 51821.
I hope the article will be useful to someone.