Tag: Linux Page 1 of 2

Trying to run a gocryptfs encrypted file system mount within a rclone mount remote file system itself accessing the remote host over SFTP, I ran into an annoying issue that writes would fail with a “Operation not permitted” error, but the file in question would appear within the gocryptfs file system (so clearly something was working).

Slothing through the output of running rclone mount with -vv but without --daemon to try to find clues as to what was actually going on, I came across these two lines:

yyyy/mm/dd hh:mm:ss ERROR : <encrypted filename>: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal

yyyy/mm/dd hh:mm:ss DEBUG : &{<encrypted filename> (w)}: >Read: read=0, err=operation not permitted

Well, there’s the error being returned, and the cause.

Per the documentation, rclone mount --vfs-cache-mode takes one of off, minimal, writes, full; and the default, lo and behold, is off, which is clearly less than minimal.

Adding a --vfs-cache-mode minimal to the end of the rclone mount command seems to have fixed it, insofar as the error is gone and writes appear to go through fully as intended.

A good software design principle is that of least surprise. Software should do what one can reasonably expect it to do in response to user actions and any configuration that has been made.

Another good design principle is to fail safely (or securely). If for some reason a program cannot perform a requested action, it should put itself, or the system, into a known-safe state. That state probably won’t be that which the user was seeking to achieve, but it should be one that does not cause the user to unexpectedly do anything dangerous or which might endanger the system further.

When used with VPN connections (both OpenVPN and Wireguard), the Linux NetworkManager tool unfortunately comes up horribly short in both areas.

In short: absent special configuration, DNS queries (and responses) can quite easily leak outside of the VPN tunnel to the DNS resolver provided by whatever network you are on. These DNS queries can allow whoever operates that DNS resolver to see the host names you are connecting to.

If you are on a trusted network, such as at home, while surprising, this is probably not a major issue.

If you are on an untrusted network, such as a public network at a café, an airport, or a hotel, where you might want to use a general-purpose VPN for traffic confidentiality, this can be a much larger issue.

By default, when connecting to a VPN, NetworkManager will combine the DNS servers provided by that network (either through fixed configuration or obtained dynamically via DHCP) with whatever resolver configuration existed previously, likely obtained when connecting to the network you are already on and connecting to the VPN through.

Consequently, if an attacker can disrupt the VPN traffic at the right moment, they can cause DNS requests to be made to the lower-priority DNS resolver: that on the local network, outside the VPN.

Additionally, if you mistype a host name, then it is possible that the search suffix can lead to a slight loss of anonymity against the operator of the DNS resolver that happens to be used.

Worse, there is no way to configure this behavior through the NetworkManager GUI.

Fortunately, it’s easy to configure through nmcli. Open a terminal window, and check the current settings:

$ nmcli con show "vpn connection name" | grep ipv.\.dns-priority

This will most likely show two lines of output, similar to:

ipv4.dns-priority:     50
ipv6.dns-priority:     50

(The exact value for both of these can vary.)

The actual encoding of the priority value is a little peculiar. In particular:

• Lower values are considered higher priority
• Negative values exclude configurations with higher values
• DNS configurations obtained through networks with the same priority value are combined

The value itself is a signed 32-bit integer, so the valid range is -2147483647 through +2147483647.

To force only the DNS servers obtained through this particular connection to be used when this connection is active, use nmcli to modify the connection to set both of these to the largest negative value possible: -2147483647. This ensures that no more negative priority value can exist on a different connection, causing the DNS configuration for this particular connection to always have priority.

$ nmcli con modify "vpn connection name" +ipv4.dns-priority -2147483647
$ nmcli con modify "vpn connection name" +ipv6.dns-priority -2147483647

For OpenVPN connections, after modifying the connection in this manner, you will need to provide the VPN user’s password (not your local login password) when connecting to it the next time.

Note that if you have multiple connections with the same value for the respective dns-priority properties, and connect to those networks simultaneously, the configurations are combined. Therefore, you do not want to set this on any potentially untrusted network that you might be connected to at the same time as the VPN connection.

Having made the configuration change, connect and disconnect the VPN connection repeatedly and observe the effect on the system name resolver configuration in /etc/resolv.conf:

$ watch cat /etc/resolv.conf

If everything is working as intended, you will see the set of search and nameserver directives being replaced as you connect and disconnect the VPN connection, instead of amended.

Port knocking is a technique to selectively allow for connections by sending a semi-secret sequence of packets, often called a “knocking” sequence.

While port knocking can very easily cut down on the amount of noise seen in logs, it’s important to keep in mind that it does not provide any significant level of security against a well-positioned adversary, as the knocking is done in the clear. The service that is hidden behind the port knocking still needs to be able to deal with being accessible from the outside network.

It used to be fairly complex on Linux to implement port knocking without relying on dedicated software to listen for the knocking packets and modify the firewall rules accordingly (which required that the listening software ran as root, which is usually something to be avoided if at all possible). Thankfully, with nftables, it’s relatively straight-forward to implement port knocking without ever leaving the firewall configuration, by using nftables’ set support.

The idea is to maintain two lists (sets) for a particular service: one of currently knocking clients, and one of clients that have successfully completed the knocking sequence.

In nftables syntax, it boils down to something similar to:

table inet filter {
  define ssh_knock_1 = 10000
  define ssh_knock_2 = 2345
  define ssh_knock_3 = 3456
  define ssh_knock_4 = 1234
  define ssh_service_port = 22

  set ssh_progress_ipv4 {
    type ipv4_addr . inet_service;
    flags timeout;
  }
  set ssh_clients_ipv4 {
    type ipv4_addr;
    flags timeout;
  }

  chain input {
    type filter hook input priority 0; policy drop;

    # ... other rules as needed ... #

    tcp dport $ssh_knock_1 update @ssh_progress_ipv4 { ip saddr . $ssh_knock_2 timeout 5s } drop
    tcp dport $ssh_knock_2 ip saddr . tcp dport @ssh_progress_ipv4 update @ssh_progress_ipv4 { ip saddr . $ssh_knock_3 timeout 5s } drop
    tcp dport $ssh_knock_3 ip saddr . tcp dport @ssh_progress_ipv4 update @ssh_progress_ipv4 { ip saddr . $ssh_knock_4 timeout 5s } drop
    tcp dport $ssh_knock_4 ip saddr . tcp dport @ssh_progress_ipv4 update @ssh_clients_ipv4 { ip saddr timeout 10s } drop

    ip saddr @ssh_clients_ipv4 tcp dport $ssh_service_port ct state new accept

    # ... other rules as needed ... #
  }
}

This works by, each time a port knock TCP connection attempt is received:

• check that this particular knock is in @ssh_progress_ipv4 (with the exception of the first knock in the sequence)
• for all but the last knock in the sequence, store the next expected knock in @ssh_progress_ipv4, and drop the knock packet (so to an outside observer, it looks no different from any other port)
• for the last knock in the sequence, store the connecting IP address in @ssh_clients_ipv4, and drop the knock packet
• when the actual connection attempt arrives, accept the connection only if the connecting IP address exists in @ssh_clients_ipv4

The knocking status is stored with a brief timeout (in the example above: 5 seconds during knocking, and 10 seconds on successful completion), ensuring that any lingering knockers are evicted promptly from the status sets.

The above example is for a four-port knocking sequence, but it could easily be both shorter and longer. The only state transition stanzas that are special is the very first and the last before the final decision stanza.

To also support port knocking and connections over IPv6, duplicate the two state sets (but use ipv6_addr for those instead of ipv4_addr), and duplicate the respective state transition and final decision stanzas (but use ip6 instead of ip).

Port knocking is a technique to selectively allow for connections by sending a semi-secret sequence of packets, often called a “knocking” sequence.

On a *nix system, nc (netcat) is useful for port knocking. Individual knocks can be sent by nc -w 1 -z host port; this will send a TCP connection attempt to the specified host and port, with a timeout of 1 second, and without sending any data.

To use nc to send a TCP knock sequence of ports 10000, 2345, 3456, 1234 to 192.0.2.234, you might do something like

nc -w 1 -z 192.0.2.234 10000
nc -w 1 -z 192.0.2.234 2345
nc -w 1 -z 192.0.2.234 3456
nc -w 1 -z 192.0.2.234 1234

Doing the same thing in Microsoft’s Powershell is rather more verbose:

Start-Job -ScriptBlock {Test-NetConnection -ComputerName "192.0.2.234" -Port 10000 -InformationLevel Quiet} | Wait-Job -Timeout 1

Start-Job -ScriptBlock {Test-NetConnection -ComputerName "192.0.2.234" -Port 2345 -InformationLevel Quiet} | Wait-Job -Timeout 1

Start-Job -ScriptBlock {Test-NetConnection -ComputerName "192.0.2.234" -Port 3456 -InformationLevel Quiet} | Wait-Job -Timeout 1

Start-Job -ScriptBlock {Test-NetConnection -ComputerName "192.0.2.234" -Port 1234 -InformationLevel Quiet} | Wait-Job -Timeout 1

A simple bash shell script to perform port knocking and then connect and hand a connected pipe to the calling process might look something like:

#!/bin/bash
host=$1
port=$2
nport=$3
while test -n "$nport"
do
  nc -w 1 -z $host $port
  shift
  port=$2
  nport=$3
done
test "$port" != "0" && exec nc $host $port

The above script takes the host name or IP address of the remote host as the first parameter, followed by a series of TCP port numbers; the last port number is the final connection port. This can be used for example with OpenSSH’s ProxyCommand directive:

$ cat .ssh/config
Host 192.0.2.234
  ProxyCommand ~/.local/bin/portknock-connect %h 10000 2345 3456 1234 %p
$

The difficulties of getting the combination of Linux KVM, host-side modern nftables packet filtering, and guest-side networking to work together without resorting to firewalld on the host are fairly well published; for example, here. The recommended solution usually involves going back to iptables on the host, and sometimes to define libvirt-specific nwfilter rules. While that might be tolerable for dedicated virtualization hosts, it’s less than ideal for systems that also see other uses, especially uses where nftables’ expressive power and relative ease of use is desired.

Fortunately, it can be worked around without giving up on nftables.

I’m assuming that you have already set up a typical basic nftables client-style ruleset on the host, something along the lines of:

#!/usr/bin/nft -f
flush ruleset
table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        ct state invalid drop
        ct state established accept
        ct state related accept
        iifname "lo" accept
    }
    chain forward {
        type filter hook forward priority 0; policy drop;
    }
    chain output {
        type filter hook output priority 0; policy accept;
    }
}

Start out by setting the KVM network to start automatically on boot. The network startup will also cause libvirt to create some NAT post-routing tables through iptables, which through the magic of conversion tools get transformed into a corresponding nftables table ip nat. This might cause an error to be displayed initially, but that’s OK for now. Reboot the host, run virsh net-list --all to check that the network is active, and nft list table ip nat to check to make sure that the table and chains were created. It should all look something like:

$ sudo virsh net-list --all
 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

$ sudo nft list table ip nat
table ip nat {
    chain LIBVIRT_PRT {
        ... a few moderately complex masquerading rules ...
    }
    chain POSTROUTING {
        type nat hook postrouting priority srcnat; policy accept;
        counter packets 0 bytes 0 jump LIBVIRT_PRT
    }
}
$

Letting libvirt’s magic and the iptables-to-nftables conversion tools handle the insertion of the routing filters makes it less likely that issues will develop later on due to for example changes in what rules newer versions need. An alternative approach, which works currently for me but might not work for you or in the future, is to manually create a postrouting chain; the nftables magic incantation can be reduced to something similar to:

table ip nat {
    chain postrouting {
        type nat hook postrouting priority 100; policy accept;
        ip saddr 192.168.122.0/24 masquerade
    }
}

(In the above snippet, 192.168.122.0/24 maps to the details from the <ip> node in the output of virsh net-dumpxml <name> for each network listed by virsh net-list earlier.)

You do, however, need to add some rules to the table inet filter to allow incoming and forwarded packets to pass through to and from the physical network interface (eth0 here; substitute as appropriate, ip addr sh will tell you the interface name):

table inet filter {
    chain input {
        # ... add at some appropriate location ...
        iifname "virbr0" accept
    }
    chain forward {
        # ... add at some appropriate location ...
        iifname "virbr0" oifname "eth0" accept
        iifname "eth0" oifname "virbr0" accept
    }
}

The forward chain rules probably aren’t necessary if your forward chain has the default accept policy, but it’s generally better to have a drop or reject policy and only allow the traffic that is actually needed.

The finishing touch is to make sure that sysctl net.ipv4.ip_forward = 1 on the host; without it, IPv4 forwarding won’t work at all.

Unfortunately, as KVM still tries to use iptables to create a NAT table when its network is started, and this can’t be done when a nftables NAT table exists, the table ip nat portion, if manually configured, needs to go into a nftables script that is loaded after the KVM network is started thus replacing the automatically generated chain, whereas most distributions are set up to load the normal nftables rule set quite early during the boot process, likely and hopefully before basic networking is even fully up and running (to close the window of opportunity for traffic to sneak through). The easiest way to deal with this is very likely to just let the iptables compatibility tools handle this for you when the KVM network is started and accept the need for a reboot during the early KVM configuration process. The most likely scenario in which this simple approach won’t work seems to be if you are already using nftables to do other IP forwarding magic as well; in that case, you may need to resort to a split nftables configuration and loading the post-routing NAT ruleset late during the boot process, such as perhaps through /etc/rc.local (which is typically executed very late during boot). If so, then it’s probably worth the trouble to rewrite one or the other in terms of nft add commands instead of a full-on, atomic nft -f script.

With all this in place, KVM guests should now be able to access the outside world over IPv4, NATed through the host, including after a reboot of the host.

A huge tip of the proverbial hat to user regox on the Gentoo forums, who posted what I was able to transform into most of the above.

For systems that commonly connect to untrusted networks, such as laptops, it can be useful to only allow outgoing traffic through a pre-configured, known-trusted (to the extent that such is a thing) VPN tunnel. This serves to ensure that unprotected traffic isn’t routed through a potentially unknown, potentially adversarial uplink provider.

Fortunately, Linux’s nftables functionality provides everything we need for that.

Usually, nftables is configured in such a way that incoming traffic is filtered, but outgoing traffic is implicitly trusted. Take, for example, Debian 11/Bullseye’s /usr/share/doc/nftables/examples/workstation.nft:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
	chain input {
		type filter hook input priority 0;

		# accept any localhost traffic
		iif lo accept

		# accept traffic originated from us
		ct state established,related accept

		# activate the following line to accept common local services
		#tcp dport { 22, 80, 443 } ct state new accept

		# accept neighbour discovery otherwise IPv6 connectivity breaks.
		ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit,  nd-router-advert, nd-neighbor-advert } accept

		# count and drop any other traffic
		counter drop
	}
}

This implicitly creates an output (and forward) chain as well:

table inet filter {
	chain output {
		type filter hook output priority 0;
	}
}

Since this chain doesn’t have any policy, the default policy accept applies. In other words, everything is allowed.

To block the unwanted traffic, we need to identify the traffic that does need to be allowed. There are three kinds of traffic that need to be allowed to flow outside of the VPN tunnel:

• Traffic for the purpose of bringing the interface up (DHCP, IPv6 neighbor discovery, …)
• Traffic for the purpose of bringing the VPN tunnel up (DNS)
• The VPN tunnel itself (Wireguard, OpenVPN, …)

Begin by determining on which interfaces you want to be able to establish an outgoing VPN connection. For some people this will be the wired interface, for some it might be the wireless interface, and for some, it might be both. Running ip addr sh in a terminal is one way to find the actual interface name, which will be needed in a moment. Also open the nftables configuration file (likely /etc/nftables.conf, but check your distribution’s documentation) in a text editor. If you don’t have one yet, you can start out with this, which is Debian’s example stripped of comments but the implicit chains included:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
	chain input {
		type filter hook input priority 0;
		iif "lo" accept
		ct state established,related accept
		ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit,  nd-router-advert, nd-neighbor-advert } accept
		counter drop
	}

	chain forward {
		type filter hook forward priority 0;
	}

	chain output {
		type filter hook output priority 0;
	}
}

For our purposes, we will be focused on the output chain, so I will be eliding the other parts of the configuration.

It’s useful to allow traffic that is routed locally on the host, for example for inter-process communication, so immediately after the type stanza, add a rule to allow traffic over the loopback interface (oif is output interface):

oif "lo" accept

Since all interfaces may not have been brought up yet by the time nftables rules are initially loaded, for the next several stanzas use oifname instead of oif. The use of oifname comes at a bit of a performance penalty, but it is more flexible especially with interfaces that aren’t always there.

First, allow DHCP traffic, which uses UDP with source and destination ports both either 67 or 68:

oifname { "en...", "wl..." } udp sport { 67, 68 } udp dport { 67, 68 } accept

Replace the "en...", "wl..." part with the name of the interface(s) in question.

Second, allow DNS traffic for initial name resolution, which uses UDP or TCP with a destination port of 53. If you configure your VPN tunnel with an IP address as a target instead of a DNS name, then you don’t need this.

oifname { "en...", "wl..." } meta l4proto { tcp, udp } th dport 53 accept

As an alternative, you can create two rules, one each for TCP and UDP; doing so will have the same effect, at a slight performance and maintenance penalty:

oifname { "en...", "wl..." } tcp dport 53 accept
oifname { "en...", "wl..." } udp dport 53 accept

Then add rules to allow traffic to the VPN concentrator. The more tightly scoped you can make this, the better. For example, if you know the IP address and the port used, you can add a stanza such as:

oifname { "en...", "wl..." } ip daddr 192.0.2.128 udp dport 29999 accept

If the VPN concentrator runs on either a standard port that is rarely used for other purposes (such as OpenVPN’s default 1194) or an uncommon port (as is often the case with Wireguard) but you don’t know its exact IP address ahead of time, you can either use a set, or elide the IP address specification:

oifname { "en...", "wl..." } ip daddr { 192.0.2.128/28, 198.51.100.0/27 } udp dport 29999 accept

or

oifname { "en...", "wl..." } udp dport 29999 accept

Then allow traffic as needed through the VPN tunnel interface. The exact name of this interface will vary with the VPN technology you’re using; for example, Wireguard tunnels typically allow you to specify the interface name, whereas OpenVPN tunnels use a semi-unpredictable interface name. For this, the ability of oifname to match a prefix by appending * can be useful. For example, for OpenVPN you might use:

oifname "tun*" accept

whereas for a Wireguard tunnel you might end up with:

oifname "wgmyvpn" accept

As a final touch, add a policy to block traffic not matched by other rules. Since all output rules specify on which interfaces traffic is allowed to flow, this blocks traffic outside of the VPN tunnel except for the traffic that is explicitly allowed to flow outside of the VPN tunnel.

The policy typically goes at the top, just below the type stanza, whereas the reject stanza must appear below all other rules.

policy drop;
reject

The purpose of also having a reject stanza is to provide more immediate feedback. In its absence, packets will simply be dropped, resulting in long wait times before attempts time out; with it, clients will be notified immediately that the connection failed and can report this back to the user.

The final output chain might look something like:

chain output {
	type filter hook output priority 0;
	policy drop;

	oif "lo" accept
	oifname { "en...", "wl..." } udp sport { 67, 68 } udp dport { 67, 68 } accept
	oifname { "en...", "wl..." } meta l4proto { tcp, udp } th dport 53 accept
	oifname { "en...", "wl..." } ip daddr { 203.0.113.113, 203.0.113.114 } udp dport 1194 accept
	oifname "tun*" accept
	reject
}

Reload the nftables rule set (sudo nft -f /etc/nftables.conf) and verify that you can connect to the VPN and access the Internet (or the remote network) through it. Disconnect the VPN and verify that traffic is blocked, for example by attempting to reload a web page.

Reboot the computer and verify that the network interface comes up and that you can connect to the VPN, access the Internet through it, and that traffic is again blocked when you disconnect from the VPN.

Keep in mind that this ruleset isn’t perfect. For example, if routes aren’t set up properly when starting the VPN tunnel, traffic can leak through ordinary DNS queries outside of it; and it relies on interface name matching which can match unexpected interfaces. Therefore, this does not serve as a proper “kill switch” for all traffic. However, it does form a decent second (or third) line of defense against unexpected but not actively malicious traffic leaks outside of the VPN tunnel, which for a system that would otherwise allow everything going out is very much an improvement.