10:18 a.m.
Hi all,
I hope I am in the right place, I found this email by following the
link in the project README. We are collecting use cases for xfs dev
projects, one of which would be enhancing xfs protofiles, which is a
type of script that facilitates filesystem generation. I am not very
familiar with libguestfs, but a coworker suggested that this might be
of interest to your project. More specifically, the task would be to
add attributes to the files generated by the protofile script. Does
this sound like something that might be of use to libguestfs? I did
take a quick glance through the libguestfs code, and didnt notice any
current usage of xfs protofiles, but I thought it was worth asking if protofiles might
become of interest if we did add this feature. Thanks!
Allison
1:45 p.m.
On Sat, Apr 02, 2022 at 08:18:52AM -0700, Alli wrote:
Hi all,
I hope I am in the right place, I found this email by following the
link in the project README. We are collecting use cases for xfs dev
projects, one of which would be enhancing xfs protofiles, which is a
type of script that facilitates filesystem generation. I am not
very familiar with libguestfs, but a coworker suggested that this
might be of interest to your project. More specifically, the task
would be to add attributes to the files generated by the protofile
script. Does this sound like something that might be of use to
libguestfs? I did take a quick glance through the libguestfs code,
and didnt notice any current usage of xfs protofiles, but I thought
it was worth asking if protofiles might become of interest if we did
add this feature. Thanks!
Hello Allison,
I have to admit I'm not sure what any of this means. Also the
information about protofiles is a bit thin. There is some information
in the mkfs.xfs(8) man page, but even that is a bit unclear.
Is it like the (very useful) "-d" option of mke2fs?
Libguestfs does not currently provide a binding around mkfs.xfs.
Instead we rely on generic mkfs support and that only encodes a few
baseline mkfs features like blocksize:
https://github.com/libguestfs/libguestfs/blob/3115726aca7d898be3b1191d322...
https://github.com/libguestfs/libguestfs/blob/3115726aca7d898be3b1191d322...
This is unusual. We do support other mkfs-like tools more directly, eg mke2fs:
https://github.com/libguestfs/libguestfs/blob/3115726aca7d898be3b1191d322...
https://github.com/libguestfs/libguestfs/blob/3115726aca7d898be3b1191d322...
And we also currently have support for other xfs_* tools like
xfs_info, xfs_growfs. Just not (yet) any support for mkfs.xfs.
I've copied Eric Sandeen in on this email as he usually knows a lot
more than I do.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
9:45 a.m.
On Mon, 2022-04-04 at 19:45 +0100, Richard W.M. Jones wrote:
On Sat, Apr 02, 2022 at 08:18:52AM -0700, Alli wrote:
> Hi all,
> I hope I am in the right place, I found this email by following the
> link in the project README. We are collecting use cases for xfs
> dev
> projects, one of which would be enhancing xfs protofiles, which is
> a
> type of script that facilitates filesystem generation. I am not
> very familiar with libguestfs, but a coworker suggested that this
> might be of interest to your project. More specifically, the task
> would be to add attributes to the files generated by the protofile
> script. Does this sound like something that might be of use to
> libguestfs? I did take a quick glance through the libguestfs code,
> and didnt notice any current usage of xfs protofiles, but I thought
> it was worth asking if protofiles might become of interest if we
> did
> add this feature. Thanks!
Hello Allison,
I have to admit I'm not sure what any of this means. Also the
information about protofiles is a bit thin. There is some
information
in the mkfs.xfs(8) man page, but even that is a bit unclear.
Yes, there's just
that bit under the "-p protofile" option
Is it like the (very useful) "-d" option of mke2fs?
Not exactly. You
would use the -p flag and pass it a text file with
the protofile script in it. So here is example of using the protofile
from the man page:
root@garnet:/home/achender/work_area/study_group_demos# cat proto.txt
/stand/diskboot
4872 110
d--777 3 1
usr d--777 3 1
sh ---755 3 1 /bin/sh
ken d--755 6 1
$
b0 b--644 3 1 0 0
c0 c--644 3 1 0 0
fifo p--644 3 1
slink l--644 3 1 /a/symbolic/link
: This is a comment line
$
$
root@garnet:/home/achender/work_area/study_group_demos# dd if=/dev/zero
of=image.bin bs=1048576 count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.60282 s, 413 MB/s
root@garnet:/home/achender/work_area/study_group_demos# mkfs.xfs -p
proto.txt -d name=image.bin
meta-data=image.bin isize=512 agcount=4, agsize=65536
blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1,
rmapbt=0
= reflink=1 bigtime=1 inobtcount=1
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-
count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@garnet:/home/achender/work_area/study_group_demos# mount image.bin
mntPt/
root@garnet:/home/achender/work_area/study_group_demos# ls mntPt/
usr
root@garnet:/home/achender/work_area/study_group_demos# ls mntPt/usr/
b0 c0 fifo ken sh slink
root@garnet:/home/achender/work_area/study_group_demos#
Libguestfs does not currently provide a binding around mkfs.xfs.
Instead we rely on generic mkfs support and that only encodes a few
baseline mkfs features like blocksize:
https://urldefense.com/v3/__https://github.com/libguestfs/libguestfs/blob...
https://urldefense.com/v3/__https://github.com/libguestfs/libguestfs/blob...
This is unusual. We do support other mkfs-like tools more directly,
eg mke2fs:
https://urldefense.com/v3/__https://github.com/libguestfs/libguestfs/blob...
https://urldefense.com/v3/__https://github.com/libguestfs/libguestfs/blob...
And we also currently have support for other xfs_* tools like
xfs_info, xfs_growfs. Just not (yet) any support for mkfs.xfs.
I see, generally
protofiles are of interest when you want to create an
image with a particular root file system in it. Since there's no
journaling, it's a lot faster than creating and populating an empty fs.
It looks like there's a couple areas in libguestfs where a folder is
created, though not a huge tree. Perhaps not enough to be motivating
right now, since this would only apply to an xfs type, but maybe
something to keep in mind should the use case arise :-)
I've copied Eric Sandeen in on this email as he usually knows a lot
more than I do.
Sure, Eric may have some more ideas about use cases for this as
well.
Thanks for the feedback!
Allison
Rich.
11:06 p.m.
On Tue, Apr 05, 2022 at 07:45:05AM -0700, Alli wrote:
On Mon, 2022-04-04 at 19:45 +0100, Richard W.M. Jones wrote:
> On Sat, Apr 02, 2022 at 08:18:52AM -0700, Alli wrote:
> > Hi all,
> > I hope I am in the right place, I found this email by following the
> > link in the project README. We are collecting use cases for xfs
> > dev
> > projects, one of which would be enhancing xfs protofiles, which is
> > a
> > type of script that facilitates filesystem generation. I am not
> > very familiar with libguestfs, but a coworker suggested that this
> > might be of interest to your project. More specifically, the task
> > would be to add attributes to the files generated by the protofile
> > script. Does this sound like something that might be of use to
> > libguestfs? I did take a quick glance through the libguestfs code,
> > and didnt notice any current usage of xfs protofiles, but I thought
> > it was worth asking if protofiles might become of interest if we
> > did
> > add this feature. Thanks!
>
> Hello Allison,
>
> I have to admit I'm not sure what any of this means. Also the
> information about protofiles is a bit thin. There is some
> information
> in the mkfs.xfs(8) man page, but even that is a bit unclear.
Yes, there's just that bit under the "-p protofile" option
>
> Is it like the (very useful) "-d" option of mke2fs?
Not exactly. You would use the -p flag and pass it a text file with
the protofile script in it. So here is example of using the protofile
from the man page:
root@garnet:/home/achender/work_area/study_group_demos# cat proto.txt
/stand/diskboot
4872 110
d--777 3 1
usr d--777 3 1
sh ---755 3 1 /bin/sh
ken d--755 6 1
$
b0 b--644 3 1 0 0
c0 c--644 3 1 0 0
fifo p--644 3 1
slink l--644 3 1 /a/symbolic/link
: This is a comment line
$
$
FWIW, protofiles date back to Unix 4th Edition from November 1973. IIRC
the original reason for creating mkfs (with protofiles) was to make it
easier for Ken Thompson to "deploy" Unix onto the disk packs that people
sent him, since before that he had to write the superblock and the root
directory by hand, and being able to copy *some* of his computer was
quite the time saver.
They're documented on page 274 here:
https://dspinellis.github.io/unix-v4man/v4man.pdf
XFS (and JFS, and AIX and HPUX) still use the protofile format as a
means of copying files into a newly formatted filesystem. I'm not sure
why e2fsprogs went their own way, though I'd imagine that modern deploy
image strategies create a chroot and import everything within that part
of the directory tree instead of cherry-picking files off an existing
filesystem.
Possibly the answer here is that protofiles have have their day, and
mkfs.xfs should just grow the means to import a directory tree into the
filesystem so that you can run the image creation tools as non-root.
OTOH, I /do/ have a protofile generator up my sleeve, and as far as XFS
goes it would be a bit less churn to add the one missing piece instead
of rewriting a ton of mkfs code:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/c...
If it would be more generally useful for libguestfs to be able to format
and populate filesystems, we could talk about committing xfs_protofile
into the upstream xfsprogs distribution, as well as enhancing it to pick
up extended attributes and fsxattrs from the files it's importing?
--D
root@garnet:/home/achender/work_area/study_group_demos# dd
if=/dev/zero
of=image.bin bs=1048576 count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.60282 s, 413 MB/s
root@garnet:/home/achender/work_area/study_group_demos# mkfs.xfs -p
proto.txt -d name=image.bin
meta-data=image.bin isize=512 agcount=4, agsize=65536
blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1,
rmapbt=0
= reflink=1 bigtime=1 inobtcount=1
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-
count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@garnet:/home/achender/work_area/study_group_demos# mount image.bin
mntPt/
root@garnet:/home/achender/work_area/study_group_demos# ls mntPt/
usr
root@garnet:/home/achender/work_area/study_group_demos# ls mntPt/usr/
b0 c0 fifo ken sh slink
root@garnet:/home/achender/work_area/study_group_demos#
>
> Libguestfs does not currently provide a binding around mkfs.xfs.
> Instead we rely on generic mkfs support and that only encodes a few
> baseline mkfs features like blocksize:
>
>
https://urldefense.com/v3/__https://github.com/libguestfs/libguestfs/blob...
>
https://urldefense.com/v3/__https://github.com/libguestfs/libguestfs/blob...
>
> This is unusual. We do support other mkfs-like tools more directly,
> eg mke2fs:
>
>
https://urldefense.com/v3/__https://github.com/libguestfs/libguestfs/blob...
>
https://urldefense.com/v3/__https://github.com/libguestfs/libguestfs/blob...
>
> And we also currently have support for other xfs_* tools like
> xfs_info, xfs_growfs. Just not (yet) any support for mkfs.xfs.
I see, generally protofiles are of interest when you want to create an
image with a particular root file system in it. Since there's no
journaling, it's a lot faster than creating and populating an empty fs.
It looks like there's a couple areas in libguestfs where a folder is
created, though not a huge tree. Perhaps not enough to be motivating
right now, since this would only apply to an xfs type, but maybe
something to keep in mind should the use case arise :-)
>
> I've copied Eric Sandeen in on this email as he usually knows a lot
> more than I do.
Sure, Eric may have some more ideas about use cases for this as well.
Thanks for the feedback!
Allison
>
> Rich.
>
9 a.m.
On Tue, Apr 12, 2022 at 09:06:42PM -0700, Darrick J. Wong wrote:
Possibly the answer here is that protofiles have have their day, and
mkfs.xfs should just grow the means to import a directory tree into the
filesystem so that you can run the image creation tools as non-root.
It would definitely be more useful. We use the mke2fs -d option
in quite a lot of places.
However it's my understanding (maybe wrong?) that the reason that
option exists in e2fsprogs is because that already has a nice
userspace library for creating and parsing ext4 filesystems (-lext2fs)
and that no such library exists for XFS. So that would need to be
implemented first which is a significant amount of work.
If it would be more generally useful for libguestfs to be able to
format
and populate filesystems, we could talk about committing xfs_protofile
into the upstream xfsprogs distribution, as well as enhancing it to pick
up extended attributes and fsxattrs from the files it's importing?
libguestfs can already format & populate filesystems, and because that
is generic it can do it for any filesystem including XFS, and for both
new and existing filesystems:
https://libguestfs.org/virt-make-fs.1.html
https://libguestfs.org/virt-format.1.html
https://libguestfs.org/virt-tar-in.1.html
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
988
days inactive
999
days old
4 comments
3 participants
participants (3)
-
Alli -
Darrick J. Wong -
Richard W.M. Jones