Hello all, I am encountering an issue with virt-customize when the host kernel, on which virt-customize runs, has CONFIG_IPV6_SIT built-in. Below is the error output from the command: virt-customize -a noble-server-cloudimg-amd64.qcow2 --install isc-dhcp-client [

Using the -v verbose option, I discovered that the IP address assigned in the QEMU command line was not reflected inside the guest. Here is the relevant ip a output from the log:

+ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 brd 127.255.255.255 scope host lo valid_lft forever preferred_lft forever inet6 ::1 /128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe12:3456/64 scope link tentative valid_lft forever preferred_lft forever 3: sit0@NONE: mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 inet6 ::127.0.0.1/96 scope host valid_lft forever preferred_lft forever Further debugging revealed that when the host kernel has SIT-related configurations enabled (i.e., CONFIG_IPV6_SIT), the supermin appliance, which libguestfs uses, inherits the same kernel and initrd. Consequently, the guest environment booted by virt-customize includes the sit0 device. When the sit0 device is present, the primary NIC (eth0) does not receive the static IP address assigned by libguestfs in the QEMU command line: /usr/bin/qemu-system-x86_64 \ -global virtio-blk-pci.scsi=off \ ... -netdev user,id=usernet,net=169.254.2.15/16 \ -device virtio-net-pci,netdev=usernet \ ..

I verified this issue on another machine where the host kernel had no SIT-related configurations enabled. On that machine, the same virt-customize command with the same Noble image worked correctly. I have attached verbose logs for both working and non-working configurations. A diff of these logs shows that the sit0 device causes the IP assignment issue with the eth0 interface. I have the following queries and would appreciate any responses: 1. Why does the user network IP not get applied when the sit0 device is present in the appliance? 2. Is there a way to append parameters to the libguestfs QEMU command line to blacklist the sit module in such cases? 3. Can we override the kernel and initrd used by libguestfs via the virt-customize command line?

libguestfs: command: run: passt --help sh: 1: passt: not found

On Fri, 20 Jun 2025 14:26:11 +0100 "Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote: > On Fri, Jun 20, 2025 at 05:12:27PM +0530, Aithal, Srikanth wrote: >> Hello all, >> >> I am encountering an issue with virt-customize when the host kernel, on which >> virt-customize runs, has CONFIG_IPV6_SIT built-in. Below is the error output >> from the command: >> >> virt-customize -a noble-server-cloudimg-amd64.qcow2 --install isc-dhcp-client [ > ... >> Using the -v verbose option, I discovered that the IP address assigned in the >> QEMU command line was not reflected inside the guest. Here is the relevant ip a >> output from the log: > You would not expect the IP inside the appliance to be the same as > outside. We're using two tools called libslirp & passt to create a > user-level network. By the way, just to avoid confusion: passt's default behaviour (stand-alone, not started by libguestfs) is to advertise to the guest (DHCP / NDP / DHCPv6) addresses and routes copied from the host interface with the (first) default route. But with libguestfs, for compatibility with libslirp, passt is also configured with a fixed address, which won't be the same as the host. You would see that in passt's own command-line (it's a separate tool, so you won't see that in QEMU's command line). >> + ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 >> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 brd >> 127.255.255.255 scope host lo valid_lft forever preferred_lft forever inet6 ::1 >> /128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc >> pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd >> ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe12:3456/64 scope link tentative >> valid_lft forever preferred_lft forever 3: sit0@NONE: mtu 1480 qdisc noqueue >> state UNKNOWN group default qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 inet6 >> ::127.0.0.1/96 scope host valid_lft forever preferred_lft forever >> >> Further debugging revealed that when the host kernel has SIT-related >> configurations enabled (i.e., CONFIG_IPV6_SIT), the supermin appliance, which >> libguestfs uses, inherits the same kernel and initrd. Consequently, the guest >> environment booted by virt-customize includes the sit0 device. >> >> When the sit0 device is present, the primary NIC (eth0) does not receive the >> static IP address assigned by libguestfs in the QEMU command line: >> >> /usr/bin/qemu-system-x86_64 \ -global virtio-blk-pci.scsi=off \ ... -netdev >> user,id=usernet,net=169.254.2.15/16 \ -device virtio-net-pci,netdev=usernet \ > In this case it would use libslirp. I'm pretty sure slirp doesn't > support IPv6 at all which is the problem here. But also ... It kind of does, but it's not enabled by default, and it doesn't support SLAAC plus a number of mandatory functionalities, at least not ouf of the box, so, yes, see below... >> I verified this issue on another machine where the host kernel had no >> SIT-related configurations enabled. On that machine, the same virt-customize >> command with the same Noble image worked correctly. I have attached verbose >> logs for both working and non-working configurations. A diff of these logs >> shows that the sit0 device causes the IP assignment issue with the eth0 >> interface. >> >> I have the following queries and would appreciate any responses: >> >> 1. Why does the user network IP not get applied when the sit0 device is >> present in the appliance? I think the DHCP client tries to configure sit0 as it's the first interface reported by the kernel and can't do it with the information provided by libslirp's DHCP server. No idea about 2. and 3. >> 2. Is there a way to append parameters to the libguestfs QEMU command line to >> blacklist the sit module in such cases? >> 3. Can we override the kernel and initrd used by libguestfs via the >> virt-customize command line? >> libguestfs: command: run: passt --help >> sh: 1: passt: not found > If you use passt (which supports IPv6) it might work better, or at > least differently ... ...that would be my suggestion as well (apt install passt). It's not exactly up to date on Ubuntu 24.04, and doesn't look very maintained (see e.g. https://bugs.launchpad.net/ubuntu/+source/passt/+bug/2077158, I maintain the Debian package only), but it should work better than libslirp anyway, as the appliance can use SLAAC as well.

On Tue, Jun 24, 2025 at 01:25:47PM +0530, Aithal, Srikanth wrote: > libguestfs: command: run: \ --pid /tmp/libguestfsJamIlZ/passt1.pid ... > Don't run as root. Changing to nobody... ... > PID file open: Permission denied > libguestfs: trace: launch = -1 (error) In libguestfs we already work around qemu changing its user when we are running as root: https://github.com/libguestfs/libguestfs/blob/0991b4dc2124a8d6c3d232663ea... However I think because passt is creating the file, it cannot write into the 0755 directory. Honestly (just as with libvirt / qemu) unilaterally changing the user ID when running as root is not helping anyone nor adding any security. As for working around the bug, just don't run virt-customize as root. There's no need to run guestfs tools as root, unless for some reason you need to edit a disk image which is only accessible by root. Rich.

On Tue, 24 Jun 2025 09:12:33 +0100 "Richard W.M. Jones&quot;&lt;rjones(a)redhat.com&gt; wrote: > On Tue, Jun 24, 2025 at 01:25:47PM +0530, Aithal, Srikanth wrote: >> libguestfs: command: run: \ --pid /tmp/libguestfsJamIlZ/passt1.pid > ... >> Don't run as root. Changing to nobody... > ... >> PID file open: Permission denied I think that might be due to an issue in the AppArmor policy from the version shipped with Ubuntu 24.04 (that's a rather old version), but I'm not entirely sure what it might be. Aithal, could you have a look at /var/log/audit/audit.log (say, "tail -f /var/log/audit/audit.log") while you're running this? AppArmor access denials are logged there, as well as in the system log.

>> libguestfs: trace: launch = -1 (error) > In libguestfs we already work around qemu changing its user when we > are running as root: > > https://github.com/libguestfs/libguestfs/blob/0991b4dc2124a8d6c3d232663ea... > > However I think because passt is creating the file, it cannot write > into the 0755 directory. This is something we fixed: https://passt.top/passt/commit/?id=c9b24134656925e53fea3cde0b33ab143dcd84af the fix is not available in 0.0~git20240220.1e6f92b-1, and I'm generally having little luck with requesting backports, e.g.: https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2077158 but in any case that's not the issue Aithal is hitting here, because things don't work as non-root anyway. > Honestly (just as with libvirt / qemu) unilaterally changing the user > ID when running as root is not helping anyone nor adding any security. I'd say that, minus that kind of issue which is obviously detrimental to security, not being able to issue a number of system calls as root once a guest connects is still decreasing the surface for possible attacks.

On Tue, 24 Jun 2025 15:46:13 +0530 "Aithal, Srikanth" <sraithal(a)amd.com&gt; wrote: > On 6/24/2025 3:04 PM, Stefano Brivio wrote: > > On Tue, 24 Jun 2025 09:12:33 +0100 > > "Richard W.M. Jones&quot;&lt;rjones(a)redhat.com&gt; wrote: > > > >> On Tue, Jun 24, 2025 at 01:25:47PM +0530, Aithal, Srikanth wrote: > >>> libguestfs: command: run: \ --pid /tmp/libguestfsJamIlZ/passt1.pid > >> ... > >>> Don't run as root. Changing to nobody... > >> ... > >>> PID file open: Permission denied > > I think that might be due to an issue in the AppArmor policy from the > > version shipped with Ubuntu 24.04 (that's a rather old version), but > > I'm not entirely sure what it might be. > > > > Aithal, could you have a look at /var/log/audit/audit.log (say, > > "tail -f /var/log/audit/audit.log") while you're running this? > > > > AppArmor access denials are logged there, as well as in the system log. > > I updated my Ubuntu 24.04 system to the latest version using apt > upgrade. Following the update, I can now run virt-customize as a > non-root user with passt, resolving the previous permission issue. > > However, I am still encountering the original issue I faced with > libslirp and now with passt: the network interfaces inside the > libguestfs appliance are not being assigned IP addresses. > > I have attached the new logs for your review. Thank you for your time > and assistance. > > > /# dpkg -l | grep passt/ > ii  passt 0.0~git20240220.1e6f92b-1                  amd64        > user-mode networking daemons for virtual machines and namespaces > > > /# sudo -u amd virt-customize -v -x -a > /home/amd/noble-server-cloudimg-amd64.qcow2 --install isc-dhcp-client/ > > .. > > libguestfs: command: run: passt --help > Usage: passt [OPTION]... > >   -d, --debug\t\tBe verbose >       --trace\t\tBe extra verbose, implies --debug > .. > > .. > > libguestfs: command: run: passt > libguestfs: command: run: \ --one-off > libguestfs: command: run: \ --socket /tmp/libguestfsTs3w5r/passt.sock > libguestfs: command: run: \ --pid /tmp/libguestfsTs3w5r/passt1.pid > libguestfs: command: run: \ --address 169.254.2.15 > libguestfs: command: run: \ --netmask 16 > libguestfs: command: run: \ --mac-addr 52:56:00:00:00:02 > libguestfs: command: run: \ --gateway 169.254.2.2 > No routable interface for IPv6: IPv6 is disabled > Template interface: enp97s0 (IPv4) > MAC: >     host: 52:56:00:00:00:02 > DHCP: >     assign: 169.254.2.15 >     mask: 255.255.0.0 >     router: 169.254.2.2 > DNS: >     169.254.2.2 > DNS search list: >     amd.com > UNIX domain socket bound at /tmp/libguestfsTs3w5r/passt.sock ...I didn't realise you don't have IPv6 connectivity on the host, sorry. Then passt won't really help bypassing IPv4. Rich, I think the issue here is that appliance/init does: iface=$(ls -I all -I default -I lo /proc/sys/net/ipv4/conf) and that will pick sit0 over eth0, but it shouldn't. If requesting jq(1) is an option, perhaps: iface=$(ip -j link show | jq -rM '.[] | select(.link_type == "ether").ifname') would be somewhat more accurate.

On Tue, 24 Jun 2025 12:33:17 +0100 "Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote: > I'm a bit confused here. How does host IPv4 or IPv6 affect what > happens in the appliance? The appliance still needs to connect to hosts using IPv4 or IPv6 addresses. My hope was simply that availability of IPv6 would work around the issue, because IPv4 isn't configured here (due to the issue I'm pointing out).

The interface appliance/init selects to configure IPv4 via DHCP is sit0 instead of eth0. > For example if the host only had IPv6, it should still work wouldn't > it? With IPv6 only, appliance/init would at least need to bring up the interface to configure the interface via SLAAC. I don't see it being done, so, unless it's done implicitly somewhere, that shouldn't work, either. I guess that should be fixed as well, with, say: ip link set "${iface}" up > Also I'm confused where sit0 comes from (or even what it is, really). > Why does it appear inside the appliance at all? See original report from Aithal for full details (and in my opinion a reasonable request to offer a way to disable the module inside the appliance).

Long story short, you configure the kernel with CONFIG_IPV6_SIT, and the kernel will automatically create the sit0 interface at boot, which causes appliance/init to ask dhclient to configure it (instead of eth0). SIT stands for "Simple Internet Transition" and it's essentially the 6to4 (https://en.wikipedia.org/wiki/6to4) implementation on Linux. It's not supposed to be used in the appliance at all, but the interface appears merely because the kernel is configured with it. > My mental model is that passt provides an IPv4 address inside the host > on eth0. If the host has, say, only IPv6, then requests would be > proxied out over that. We don't care how that happens precisely. Protocol families are not simply proxied between each other because there's no shared address space between IPv4 and IPv6.

Let's say you connect to 88.198.0.164 from the appliance, but there's no IPv4 routing available on the host: passt will *not* reverse-resolve 88.198.0.164 to passt.top and connect you to 2a01:4f8:222:904::2 instead. That would be kind of surprising and add convoluted DNS requirements. We have a request for CLAT: https://bugs.passt.top/show_bug.cgi?id=43 where one could do something like that, but you would need to configure it explicitly (the day it's implemented). In any case, this has all little to do with the actual issue. The problem is that the appliance runs dhclient or dhcpcd on sit0 instead of eth0, because excluding "all", "default", and "lo" is not enough. If you prefer a minimal fix: iface=$(ls -I all -I default -I lo -I sit0 /proc/sys/net/ipv4/conf)

would also work. I can send a patch, too, but it might take me a while before I find a moment to prepare the test setup.

On Tue, 24 Jun 2025 13:15:58 +0100 "Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote: > If passt is like slirp, then it would originate connections itself > (ie. calling connect(2) on the host side), so I don't understand how > the address family would be relevant. Indeed, it does that, but it needs to pass an address (including family) to connect(2), and that's the destination address of the original connection as seen from the guest. You can't connect to 88.198.0.164 using IPv6.