[PATCH] make-fs: Don't use du --apparent-size to estimate input size.
When calculating the initial size of the disk we must estimate how much space is taken by the input. This is quite difficult. For directories we used ‘du --apparent-size -bs DIR’. This is wrong because ’-b’ implies ‘--apparent-size --block-size=1’. But also ‘--apparent-size’ causes du to count the file size rather than number of blocks used by files. If you have a directory containing many small files this usually underestimates resulting in disk sizes which are far too small to actually contain the files. There's no really good answer here because du can't exactly do what we want, but we can at least remove this flag. This causes much larger estimates and therefore much larger virtual disks. Thanks: Nikolay Ivanets --- make-fs/make-fs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/make-fs/make-fs.c b/make-fs/make-fs.c index 5d8c3a385..386142280 100644 --- a/make-fs/make-fs.c +++ b/make-fs/make-fs.c @@ -393,7 +393,7 @@ static int estimate_input (const char *input, uint64_t *estimate_rtn, char **ifmt_rtn) { struct stat statbuf; - const char *argv[6]; + const char *argv[5]; CLEANUP_UNLINK_FREE char *tmpfile = NULL; CLEANUP_FCLOSE FILE *fp = NULL; char line[256]; @@ -424,11 +424,10 @@ estimate_input (const char *input, uint64_t *estimate_rtn, char **ifmt_rtn) } argv[0] = "du"; - argv[1] = "--apparent-size"; - argv[2] = "-b"; - argv[3] = "-s"; - argv[4] = input; - argv[5] = NULL; + argv[1] = "--block-size=1"; + argv[2] = "-s"; + argv[3] = input; + argv[4] = NULL; if (exec_command ((char **) argv, tmpfile) == -1) return -1; -- 2.24.1
вт, 18 лют. 2020 о 10:48 Richard W.M. Jones <rjones@redhat.com> пише:
When calculating the initial size of the disk we must estimate how much space is taken by the input. This is quite difficult.
For directories we used ‘du --apparent-size -bs DIR’. This is wrong because ’-b’ implies ‘--apparent-size --block-size=1’. But also ‘--apparent-size’ causes du to count the file size rather than number of blocks used by files.
If you have a directory containing many small files this usually underestimates resulting in disk sizes which are far too small to actually contain the files.
There's no really good answer here because du can't exactly do what we want, but we can at least remove this flag. This causes much larger estimates and therefore much larger virtual disks.
Thanks: Nikolay Ivanets --- make-fs/make-fs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/make-fs/make-fs.c b/make-fs/make-fs.c index 5d8c3a385..386142280 100644 --- a/make-fs/make-fs.c +++ b/make-fs/make-fs.c @@ -393,7 +393,7 @@ static int estimate_input (const char *input, uint64_t *estimate_rtn, char **ifmt_rtn) { struct stat statbuf; - const char *argv[6]; + const char *argv[5]; CLEANUP_UNLINK_FREE char *tmpfile = NULL; CLEANUP_FCLOSE FILE *fp = NULL; char line[256]; @@ -424,11 +424,10 @@ estimate_input (const char *input, uint64_t *estimate_rtn, char **ifmt_rtn) }
argv[0] = "du"; - argv[1] = "--apparent-size"; - argv[2] = "-b"; - argv[3] = "-s"; - argv[4] = input; - argv[5] = NULL; + argv[1] = "--block-size=1"; + argv[2] = "-s"; + argv[3] = input; + argv[4] = NULL;
if (exec_command ((char **) argv, tmpfile) == -1) return -1; -- 2.24.1
That didn't help either. And with sparse files even made things worse: holes are not taken into account now at all. I repeat my thoughts here again: it is all like "cats and rats game". We cannot properly handle all the possible cases. I think we can agree that virt-make-fs MAY fail and it is expected situation. Just let the user know that he need to adjust size manually. -- Mykola Ivanets
On Tue, Feb 18, 2020 at 01:18:54PM +0200, Nikolay Ivanets wrote:
вт, 18 лют. 2020 о 10:48 Richard W.M. Jones <rjones@redhat.com> пише:
When calculating the initial size of the disk we must estimate how much space is taken by the input. This is quite difficult.
For directories we used ‘du --apparent-size -bs DIR’. This is wrong because ’-b’ implies ‘--apparent-size --block-size=1’. But also ‘--apparent-size’ causes du to count the file size rather than number of blocks used by files.
If you have a directory containing many small files this usually underestimates resulting in disk sizes which are far too small to actually contain the files.
There's no really good answer here because du can't exactly do what we want, but we can at least remove this flag. This causes much larger estimates and therefore much larger virtual disks.
Thanks: Nikolay Ivanets --- make-fs/make-fs.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/make-fs/make-fs.c b/make-fs/make-fs.c index 5d8c3a385..386142280 100644 --- a/make-fs/make-fs.c +++ b/make-fs/make-fs.c @@ -393,7 +393,7 @@ static int estimate_input (const char *input, uint64_t *estimate_rtn, char **ifmt_rtn) { struct stat statbuf; - const char *argv[6]; + const char *argv[5]; CLEANUP_UNLINK_FREE char *tmpfile = NULL; CLEANUP_FCLOSE FILE *fp = NULL; char line[256]; @@ -424,11 +424,10 @@ estimate_input (const char *input, uint64_t *estimate_rtn, char **ifmt_rtn) }
argv[0] = "du"; - argv[1] = "--apparent-size"; - argv[2] = "-b"; - argv[3] = "-s"; - argv[4] = input; - argv[5] = NULL; + argv[1] = "--block-size=1"; + argv[2] = "-s"; + argv[3] = input; + argv[4] = NULL;
if (exec_command ((char **) argv, tmpfile) == -1) return -1; -- 2.24.1
That didn't help either. And with sparse files even made things worse: holes are not taken into account now at all.
I repeat my thoughts here again: it is all like "cats and rats game". We cannot properly handle all the possible cases. I think we can agree that virt-make-fs MAY fail and it is expected situation. Just let the user know that he need to adjust size manually.
I think it's a difficult problem and one which won't be solved using du. For comparison we have some nbdkit plugins for creating filesystems and they are far more sophisticated, especially this one: https://github.com/libguestfs/nbdkit/tree/master/plugins/floppy I would note that the reason the test fails was nothing to do with free space. It was actually running out of inodes. With ext2/3/4 the number of inodes in the metadata is fixed at creation and based on the size of the disk (see /etc/mke2fs.conf inode_ratio and the mke2fs -i and -N options). Because of the small size of the disk, mke2fs was choosing < 100 inodes, but because the test had 100+ files it failed when creating the later files even though there was enough disk space. There's no easy way to fix this. I think it could even be considered as a bug in mke2fs. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
I think it's a difficult problem and one which won't be solved using du. For comparison we have some nbdkit plugins for creating filesystems and they are far more sophisticated, especially this one: https://github.com/libguestfs/nbdkit/tree/master/plugins/floppy
I would note that the reason the test fails was nothing to do with free space. It was actually running out of inodes. With ext2/3/4 the number of inodes in the metadata is fixed at creation and based on the size of the disk (see /etc/mke2fs.conf inode_ratio and the mke2fs -i and -N options). Because of the small size of the disk, mke2fs was choosing < 100 inodes, but because the test had 100+ files it failed when creating the later files even though there was enough disk space. There's no easy way to fix this. I think it could even be considered as a bug in mke2fs.
Rich.
I've did extra testing and the issue really appears for small file systems composed from 100 x 10-bytes files or so because of lack of free inodes. Nevertheless we cannot swear that virt-make-fs will work for any possible file systems or set of files.--size=+N is for the rescue in such cases. Your patch made things worse because of sparse files. I think we cannot apply it. -- Mykola Ivanets
On Tue, Feb 18, 2020 at 04:15:36PM +0200, Nikolay Ivanets wrote:
I think it's a difficult problem and one which won't be solved using du. For comparison we have some nbdkit plugins for creating filesystems and they are far more sophisticated, especially this one: https://github.com/libguestfs/nbdkit/tree/master/plugins/floppy
I would note that the reason the test fails was nothing to do with free space. It was actually running out of inodes. With ext2/3/4 the number of inodes in the metadata is fixed at creation and based on the size of the disk (see /etc/mke2fs.conf inode_ratio and the mke2fs -i and -N options). Because of the small size of the disk, mke2fs was choosing < 100 inodes, but because the test had 100+ files it failed when creating the later files even though there was enough disk space. There's no easy way to fix this. I think it could even be considered as a bug in mke2fs.
Rich.
I've did extra testing and the issue really appears for small file systems composed from 100 x 10-bytes files or so because of lack of free inodes. Nevertheless we cannot swear that virt-make-fs will work for any possible file systems or set of files.--size=+N is for the rescue in such cases.
Your patch made things worse because of sparse files. I think we cannot apply it.
Sure, I've dropped it, it doesn't address the actual problem. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
participants (2)
-
Nikolay Ivanets -
Richard W.M. Jones