virtdf outputs on host differs from df in guest
hi guys, happy new year!!! I encountered a problem when we used virt-df command to display my guest disks useage. the virt-df command from pkg libguestfs-tools-c-1.36.3-6.el7_4.3.x86_64,version is virt-df --version : virt-df 1.36.3rhel=7,release=6.el7_4.3,libvirt the output in guest with the method referred in http://libguestfs.org/virt-df.1.html#STATVFS NUMBERS is: [root@localhost ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/centos-root 23544596 4335172 19209424 19% / devtmpfs 931220 0 931220 0% /dev tmpfs 941860 0 941860 0% /dev/shm tmpfs 941860 8612 933248 1% /run tmpfs 941860 0 941860 0% /sys/fs/cgroup /dev/sda1 508588 133328 375260 27% /boot tmpfs 188376 0 188376 0% /run/user/0 using scripts: python -c 'import os; s = os.statvfs ("/"); print s' posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=5886149, f_bfree=4802342, f_bavail=4802342, f_files=23556096, f_ffree=23435372, f_favail=23435372, f_flag=4096, f_namemax=255) python -c 'import os; s = os.statvfs ("/boot"); print s' posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=127147, f_bfree=93815, f_bavail=93815, f_files=512000, f_ffree=511626, f_favail=511626, f_flag=4096, f_namemax=255) and then on host using virt-df showing: # sudo virt-df rpm-build-for-7.2 Filesystem 1K-blocks Used Available Use% rpm-build-for-7.2:/dev/sda1 508588 107728 400860 22% rpm-build-for-7.2:/dev/centos/root 23544596 4298384 19246212 19% using scripts: # guestfish --ro -d rpm-build-for-7.2 -i statvfs / bsize: 4096 frsize: 4096 blocks: 5886149 bfree: 4810534 bavail: 4810534 files: 23556096 ffree: 23435372 favail: 23435372 fsid: 64513 flag: 4097 namemax: 255 # sudo guestfish --ro -d rpm-build-for-7.2 -i statvfs /boot bsize: 4096 frsize: 4096 blocks: 127147 bfree: 100215 bavail: 100215 files: 512000 ffree: 511626 favail: 511626 fsid: 2049 flag: 4097 namemax: 255 we can saw that in guest the statvfs get the value was the same with the "df" command output. but the bfree/bavail values are different between using guestfish statvfs on guest and using statvfs in guest. I want to know whether is this a bug? or Does same reason cause this appearance？ Thanks, Chen
On Thu, Jan 04, 2018 at 12:58:40PM +0800, Chen Fan wrote: [In guest]
python -c 'import os; s = os.statvfs ("/"); print s' posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=5886149, f_bfree=4802342, f_bavail=4802342, f_files=23556096, f_ffree=23435372, f_favail=23435372, f_flag=4096, f_namemax=255)
python -c 'import os; s = os.statvfs ("/boot"); print s' posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=127147, f_bfree=93815, f_bavail=93815, f_files=512000, f_ffree=511626, f_favail=511626, f_flag=4096, f_namemax=255)
[From the host via libguestfs]
# guestfish --ro -d rpm-build-for-7.2 -i statvfs /
bsize: 4096 frsize: 4096 blocks: 5886149 bfree: 4810534 bavail: 4810534 files: 23556096 ffree: 23435372 favail: 23435372 fsid: 64513 flag: 4097 namemax: 255
# sudo guestfish --ro -d rpm-build-for-7.2 -i statvfs /boot bsize: 4096 frsize: 4096 blocks: 127147 bfree: 100215 bavail: 100215 files: 512000 ffree: 511626 favail: 511626 fsid: 2049 flag: 4097 namemax: 255
Was the guest quiescent and fully sync()d before you ran the python commands? In any case all that libguestfs is doing is executing the statvfs(2) system call on a copy of the disk: https://github.com/libguestfs/libguestfs/blob/e6e23c4d77ee15779f833fd4fbe6c5... So if there's any difference then it's something to do with the guest kernel vs the host kernel calculations. I wouldn't expect the numbers to be exactly the same, but libguestfs doesn't alter the numbers, it just passes them through. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
hi Jones, Thanks for your reply. At 2018-01-04 19:41:27, "Richard W.M. Jones" <rjones@redhat.com> wrote:
On Thu, Jan 04, 2018 at 12:58:40PM +0800, Chen Fan wrote: [In guest]
python -c 'import os; s = os.statvfs ("/"); print s' posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=5886149, f_bfree=4802342, f_bavail=4802342, f_files=23556096, f_ffree=23435372, f_favail=23435372, f_flag=4096, f_namemax=255)
python -c 'import os; s = os.statvfs ("/boot"); print s' posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=127147, f_bfree=93815, f_bavail=93815, f_files=512000, f_ffree=511626, f_favail=511626, f_flag=4096, f_namemax=255)
[From the host via libguestfs]
# guestfish --ro -d rpm-build-for-7.2 -i statvfs /
bsize: 4096 frsize: 4096 blocks: 5886149 bfree: 4810534 bavail: 4810534 files: 23556096 ffree: 23435372 favail: 23435372 fsid: 64513 flag: 4097 namemax: 255
# sudo guestfish --ro -d rpm-build-for-7.2 -i statvfs /boot bsize: 4096 frsize: 4096 blocks: 127147 bfree: 100215 bavail: 100215 files: 512000 ffree: 511626 favail: 511626 fsid: 2049 flag: 4097 namemax: 255
Was the guest quiescent and fully sync()d before you ran the python
commands?
yes, almost the guest is quiescent, and after invoke sync(), the results have no different.
In any case all that libguestfs is doing is executing the statvfs(2) system call on a copy of the disk:
https://github.com/libguestfs/libguestfs/blob/e6e23c4d77ee15779f833fd4fbe6c5...
So if there's any difference then it's something to do with the guest
kernel vs the host kernel calculations.
I saw that when doing virt-df, virt-df created a temporary VM with the host kernel, so I recreate a temporary QEMU VM with command line: /usr/libexec/qemu-kvm -cpu host -m 1024 -nographic -kernel /var/tmp/.guestfs-0/appliance.d/kernel -initrd /var/tmp/.guestfs-0/appliance.d/initrd -append "console=ttyS0 root=/dev/vda guestfs_rescue=1" -device virtio-blk-pci,drive=disk0 -drive file=/var/tmp/.guestfs-0/appliance.d/root,id=disk0,if=none -device virtio-serial -chardev socket,id=charchannel0,path=/tmp/a,server,nowait -device virtserialport,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -device virtio-blk-pci,drive=disk1 -drive file=/srv/imgs/centos-7.2.qcow2,if=none,id=disk1 then the console showing:
<rescue> mount /dev/vdb1 /sysroot/ [ 72.552185] SGI XFS with ACLs, security attributes, no debug enabled [ 72.568377] XFS (vdb1): Mounting V4 Filesystem [ 73.379679] XFS (vdb1): Starting recovery (logdev: internal) [ 73.406398] XFS (vdb1): Ending recovery (logdev: internal) <rescue> statvfs bash: statvfs: command not found <rescue> stat -f /sysroot File: "/sysroot" ID: fd1100000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 127147 Free: 93815 Available: 93815 Inodes: Total: 512000 Free: 511626
here the mount /dev/vdb1 device is the /boot partition from my guest, we can saw that the free blocks are the same as python command, so I think the host kernel does not affect the result. if I am wrong, pls correct me :) Thanks, Chen
I wouldn't expect the numbers to be exactly the same, but libguestfs doesn't alter the numbers, it just passes them through.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
To summarise:
[In guest] python -c 'import os; s = os.statvfs ("/boot"); print s' posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=127147, f_bfree=93815, f_bavail=93815, f_files=512000, f_ffree=511626, f_favail=511626, f_flag=4096, f_namemax=255)
[From the host via libguestfs] # sudo guestfish --ro -d rpm-build-for-7.2 -i statvfs /boot bsize: 4096 frsize: 4096 blocks: 127147 bfree: 100215 bavail: 100215 files: 512000 ffree: 511626 favail: 511626 fsid: 2049 flag: 4097 namemax: 255
[In virt-rescue]
Block size: 4096 Fundamental block size: 4096 Blocks: Total: 127147 Free: 93815 Available: 93815 Inodes: Total: 512000 Free: 511626
The block stats are the same as the "in guest" results above, but slightly different from libguestfs. - * - * - * - I wanted to see if I could reproduce this with a very simple and reproducible example. I made the assumption that this guest is using XFS. (1) Make an XFS filesystem and use libguestfs to query the statvfs: $ guestfish -N fs:xfs -m /dev/sda1 statvfs / bsize: 4096 frsize: 4096 blocks: 24713 bfree: 23347 bavail: 23347 files: 51136 ffree: 51133 favail: 51133 fsid: 2049 flag: 4096 namemax: 255 (2) Open the same disk image in virt-rescue and examine with stat command: $ virt-rescue -a test1.img
<rescue> mount /dev/sda1 /sysroot <rescue> stat -f /sysroot File: "/sysroot" ID: 80100000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 24713 Free: 23347 Available: 23347 Inodes: Total: 51136 Free: 51133
In this case the numbers are the same ... I would really like a bit more information though: (a) Are you using XFS or another filesystem? (b) Can you make a small independent reproducer like mine above? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
hi Jones, Sorry for the late reply. when I used your method to execute command: $guestfish -N fs:xfs -m /dev/sda1 statvfs / libguestfs: error: mkfs: xfs: /dev/sda1: mkfs.xfs: No such file or directory guestfish: error creating prepared disk image 'fs:xfs' on 'test1.img': failed to create filesystem (xfs): mkfs: xfs: /dev/sda1: mkfs.xfs: No such file or directory does I need to rebuild the iniramfs to include mkfs.xfs, is this the root reason that cause the different numbers ? then I used virt-rescure method , the filesystem type was ext2/ext3:
<rescue> mount /dev/sda1 /sysroot/ mount: /dev/sda1 is write-protected, mounting read-only mount: unknown filesystem type '(null)' <rescue> stat -f /sysroot File: "/sysroot" ID: ab0d9f3277e010fd Namelen: 255 Type: ext2/ext3 Block size: 4096 Fundamental block size: 4096 Blocks: Total: 1032112 Free: 941307 Available: 888879 Inodes: Total: 262144 Free: 250996 <rescue>
Thanks, Chen At 2018-01-05 00:22:46, "Richard W.M. Jones" <rjones@redhat.com> wrote:
To summarise:
[In guest] python -c 'import os; s = os.statvfs ("/boot"); print s' posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=127147, f_bfree=93815, f_bavail=93815, f_files=512000, f_ffree=511626, f_favail=511626, f_flag=4096, f_namemax=255)
[From the host via libguestfs] # sudo guestfish --ro -d rpm-build-for-7.2 -i statvfs /boot bsize: 4096 frsize: 4096 blocks: 127147 bfree: 100215 bavail: 100215 files: 512000 ffree: 511626 favail: 511626 fsid: 2049 flag: 4097 namemax: 255
[In virt-rescue]
Block size: 4096 Fundamental block size: 4096 Blocks: Total: 127147 Free: 93815 Available: 93815 Inodes: Total: 512000 Free: 511626
The block stats are the same as the "in guest" results above, but slightly different from libguestfs.
- * - * - * -
I wanted to see if I could reproduce this with a very simple and reproducible example. I made the assumption that this guest is using XFS.
(1) Make an XFS filesystem and use libguestfs to query the statvfs:
$ guestfish -N fs:xfs -m /dev/sda1 statvfs / bsize: 4096 frsize: 4096 blocks: 24713 bfree: 23347 bavail: 23347 files: 51136 ffree: 51133 favail: 51133 fsid: 2049 flag: 4096 namemax: 255
(2) Open the same disk image in virt-rescue and examine with stat command:
$ virt-rescue -a test1.img
<rescue> mount /dev/sda1 /sysroot <rescue> stat -f /sysroot File: "/sysroot" ID: 80100000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 24713 Free: 23347 Available: 23347 Inodes: Total: 51136 Free: 51133
In this case the numbers are the same ...
I would really like a bit more information though:
(a) Are you using XFS or another filesystem?
(b) Can you make a small independent reproducer like mine above?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Sun, Jan 07, 2018 at 08:11:58PM +0800, 陈帆 wrote:
Sorry for the late reply. when I used your method to execute command: $guestfish -N fs:xfs -m /dev/sda1 statvfs / libguestfs: error: mkfs: xfs: /dev/sda1: mkfs.xfs: No such file or directory guestfish: error creating prepared disk image 'fs:xfs' on 'test1.img': failed to create filesystem (xfs): mkfs: xfs: /dev/sda1: mkfs.xfs: No such file or directory
You need to install ‘libguestfs-xfs’. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
after install libguestfs_xfs, all results are: [using guestfish] guestfish -N fs:xfs -m /dev/sda1 statvfs / bsize: 4096 frsize: 4096 blocks: 24713 bfree: 23391 bavail: 23391 files: 51136 ffree: 51133 favail: 51133 fsid: 2049 flag: 4096 namemax: 255 [using virt-rescure] virt-rescue -a test1.img
<rescue> mount /dev/sda1 /sysroot <rescue> stat -f /sysroot File: "/sysroot" ID: 80100000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 24713 Free: 23391 Available: 23391 Inodes: Total: 51136 Free: 51133
the free numbers are the same. but the "virt-df" output is not changed. Thanks, Chen 在 2018-01-07 20:22:28，"Richard W.M. Jones" <rjones@redhat.com> 写道：
On Sun, Jan 07, 2018 at 08:11:58PM +0800, 陈帆 wrote:
Sorry for the late reply. when I used your method to execute command: $guestfish -N fs:xfs -m /dev/sda1 statvfs / libguestfs: error: mkfs: xfs: /dev/sda1: mkfs.xfs: No such file or directory guestfish: error creating prepared disk image 'fs:xfs' on 'test1.img': failed to create filesystem (xfs): mkfs: xfs: /dev/sda1: mkfs.xfs: No such file or directory
You need to install ‘libguestfs-xfs’.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
There are some strange things going on with the kernel XFS driver. Opening a filesystem for write or read-only makes a difference to the stats returned. With bit-for-bit identical filesystems:
<rescue> mount /dev/sda1 /sysroot <rescue> stat -f /sysroot File: "/sysroot" ID: 80100000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 24713 Free: 23347 Available: 23347 Inodes: Total: 51136 Free: 51133
vs:
<rescue> mount -o ro /dev/sda1 /sysroot <rescue> stat -f /sysroot File: "/sysroot" ID: 80100000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 24713 Free: 24653 Available: 24653 Inodes: Total: 51136 Free: 51133
virt-df always uses the ‘-o ro’ option. Despite this virt-df itself is working fine: $ virt-df -a test1.img Filesystem 1K-blocks Used Available Use% test1.img:/dev/sda1 98852 240 98612 1% Using the calculations from here: https://github.com/libguestfs/libguestfs/blob/ab0a3e0276656965021413194380ae... blocks * bsize / 1024 = 24713 * 4096 / 1024 = 98852 (blocks - bfree) * bsize / 1024 = (24713-24653) * 4096 / 1024 = 240 bavail * bsize / 1024 = 24653 * 4096 / 1024 = 98612 So everything appears to be working fine, besides possibly odd stuff going on in the kernel XFS driver. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
The answer is attached. In summary it's not a bug in virt-df, nor df, nor the kernel. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
participants (3)
-
Chen Fan -
Richard W.M. Jones -
陈帆