3:57 p.m.
Hi,
thank you for flying xorriso. :))
But as its author and looking at
https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml
i feel obliged to point out that using it as replacement for isoinfo
will increase the consumption of RAM and CPU cycles substantially.
xorriso command -indev loads the whole directory tree of the ISO filesystem
with names and metadata. Depending on that tree this can be dozens of
megabytes and lots of insertion operations in a not much optimized tree
model.
On the other hand it is quite uncomplicated to implement an own ISO 9660
PVD reader and to become independent of any external program for that
purpose.
The Primary Volume Descriptor begins at LBA 16 (= byte offset 32768).
Its layout is described in ECMA-119 8.4 (note: BP = byte offset + 1)
https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_editio...
or in HTML at
https://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor
E.g. the info missing in xorriso's output of -pvd_info is at offsets:
80 - 87 iso_volume_space_size
(unsigned 32 bit, little endian first, then again as big endian)
124 - 127 iso_volume_sequence_number
(unsigned 16 bit, little endian first, then again as big endian)
128 - 131 iso_logical_block_size
(unsigned 16 bit, little endian first, then again as big endian)
(If you don't read 2048 here, then you are in unchartered
territory.)
The date format for e.g. iso_volume_creation_t at offset 813 - 829
is a decimal digit string YYYMMDDhhmmsshh with a trailing binary byte
value for the time zone with 15 minutes granularity. ("hh" means
hundredths of seconds.)
See also
https://wiki.osdev.org/ISO_9660#Numerical_formats
https://wiki.osdev.org/ISO_9660#Date.2Ftime_format
Have a nice day :)
Thomas
4:22 p.m.
On Tue, Mar 30, 2021 at 10:57:07PM +0200, Thomas Schmitt wrote:
Hi,
thank you for flying xorriso. :))
But as its author and looking at
https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml
i feel obliged to point out that using it as replacement for isoinfo
will increase the consumption of RAM and CPU cycles substantially.
xorriso command -indev loads the whole directory tree of the ISO filesystem
with names and metadata. Depending on that tree this can be dozens of
megabytes and lots of insertion operations in a not much optimized tree
model.
"megabytes" may not be a problem - and in fact hasn't been in my
testing. Could it grow unbounded?
On the other hand it is quite uncomplicated to implement an own ISO
9660
PVD reader and to become independent of any external program for that
purpose.
The Primary Volume Descriptor begins at LBA 16 (= byte offset 32768).
Its layout is described in ECMA-119 8.4 (note: BP = byte offset + 1)
https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_editio...
or in HTML at
https://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor
E.g. the info missing in xorriso's output of -pvd_info is at offsets:
80 - 87 iso_volume_space_size
(unsigned 32 bit, little endian first, then again as big endian)
124 - 127 iso_volume_sequence_number
(unsigned 16 bit, little endian first, then again as big endian)
128 - 131 iso_logical_block_size
(unsigned 16 bit, little endian first, then again as big endian)
(If you don't read 2048 here, then you are in unchartered
territory.)
The date format for e.g. iso_volume_creation_t at offset 813 - 829
is a decimal digit string YYYMMDDhhmmsshh with a trailing binary byte
value for the time zone with 15 minutes granularity. ("hh" means
hundredths of seconds.)
See also
https://wiki.osdev.org/ISO_9660#Numerical_formats
https://wiki.osdev.org/ISO_9660#Date.2Ftime_format
OK, thanks.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
5:47 p.m.
Hi,
Richard W.M. Jones wrote:
"megabytes" may not be a problem - and in fact hasn't
been in my
testing. Could it grow unbounded?
It depends on the number of files, the lengths of the file names,
and the amount of some other meta data in the tree.
So if malice is involved: Yes, it can become nasty.
In a backup ISO with 78478 files with conservative names i see about 17 MB
of directory tree (because the first data content starts at LBA 8613).
The memory representation of the tree in libisofs underneath xorriso will
need some extra bytes per file as overhead.
ps -o rsz,vsz of the process shows the numbers 47880 67136 with a
dynamically linked Debian binary of xorriso, and 73836 99844 with my
statically linked development version of GNU xorriso.
The run time of -indev with this ISO from RAM cache is about 0.5 seconds
on a not really young 3.5 GHz Xeon system.
Direct reading of the PVD would just need to load 2048 bytes from LBA 16
and to pick from byte fields at well documented locations the strings (with
trailing blanks) and numbers (with both endianesses).
Have a nice day :)
Thomas
5:47 a.m.
On Wed, Mar 31, 2021 at 12:47:21AM +0200, Thomas Schmitt wrote:
Hi,
Richard W.M. Jones wrote:
> "megabytes" may not be a problem - and in fact hasn't been in my
> testing. Could it grow unbounded?
It depends on the number of files, the lengths of the file names,
and the amount of some other meta data in the tree.
So if malice is involved: Yes, it can become nasty.
In a backup ISO with 78478 files with conservative names i see about 17 MB
of directory tree (because the first data content starts at LBA 8613).
The memory representation of the tree in libisofs underneath xorriso will
need some extra bytes per file as overhead.
ps -o rsz,vsz of the process shows the numbers 47880 67136 with a
dynamically linked Debian binary of xorriso, and 73836 99844 with my
statically linked development version of GNU xorriso.
The run time of -indev with this ISO from RAM cache is about 0.5 seconds
on a not really young 3.5 GHz Xeon system.
Direct reading of the PVD would just need to load 2048 bytes from LBA 16
and to pick from byte fields at well documented locations the strings (with
trailing blanks) and numbers (with both endianesses).
Thanks - I pushed this commit which reads directly from the PVD:
https://github.com/libguestfs/libguestfs/commit/2f587bbaec718e414e46c7e6f...
We are still using xorriso for some things outside the appliance
like creating the test ISO.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
6:50 a.m.
Hi,
Richard W.M. Jones wrote:
I pushed this commit which reads directly from the PVD:
https://github.com/libguestfs/libguestfs/commit/2f587bbaec718e414e46c7e6f...
It is ambitious to deal with the timezone information in the PVD.
I am open to possible bug reports about ISOs made by xorriso. :))
We are still using xorriso for some things outside the appliance
like creating the test ISO.
So i can be glad that i did not chase away a user by my proposal.
Have a nice day :)
Thomas
11:28 a.m.
Hi,
regarding
https://github.com/libguestfs/libguestfs/commit/2f587bbaec718e414e46c7e6f...
i have to report that the OSDev wiki had a bug about the start address
of ISO 9660 property Abstract File Identifier.
I guess that libguestfs' daemon/isoinfo.ml should be changed like this
- let iso_copyright_file_id = sub 702 38 |> iso_parse_strD in
- let iso_abstract_file_id = sub 740 36 |> iso_parse_strD in
+ let iso_copyright_file_id = sub 702 37 |> iso_parse_strD in
+ let iso_abstract_file_id = sub 739 37 |> iso_parse_strD in
Reasoning:
According to ECMA-119 the field starts at byte offset 739 rather than 740:
"8.4.24 Abstract File Identifier (BP 740 to 776)"
"BP" means Byte Position and begins its counting at 1. So the byte offset
is one less. The length of the field is 37 bytes, like the length of its
neighbors.
For more arguments why the OSDev wiki was wrong, see:
https://lists.gnu.org/archive/html/bug-xorriso/2021-04/msg00005.html
I felt confident enough to make this change in the wiki:
https://wiki.osdev.org/index.php?title=ISO_9660&diff=25811&oldid=...
Have a nice day :)
Thomas
1:01 p.m.
On Sat, Apr 17, 2021 at 06:28:49PM +0200, Thomas Schmitt wrote:
Hi,
regarding
https://github.com/libguestfs/libguestfs/commit/2f587bbaec718e414e46c7e6f...
i have to report that the OSDev wiki had a bug about the start address
of ISO 9660 property Abstract File Identifier.
I guess that libguestfs' daemon/isoinfo.ml should be changed like this
- let iso_copyright_file_id = sub 702 38 |> iso_parse_strD in
- let iso_abstract_file_id = sub 740 36 |> iso_parse_strD in
+ let iso_copyright_file_id = sub 702 37 |> iso_parse_strD in
+ let iso_abstract_file_id = sub 739 37 |> iso_parse_strD in
Thanks - I pushed it:
https://listman.redhat.com/archives/libguestfs/2021-April/msg00058.html
Rich.
Reasoning:
According to ECMA-119 the field starts at byte offset 739 rather than 740:
"8.4.24 Abstract File Identifier (BP 740 to 776)"
"BP" means Byte Position and begins its counting at 1. So the byte offset
is one less. The length of the field is 37 bytes, like the length of its
neighbors.
For more arguments why the OSDev wiki was wrong, see:
https://lists.gnu.org/archive/html/bug-xorriso/2021-04/msg00005.html
I felt confident enough to make this change in the wiki:
https://wiki.osdev.org/index.php?title=ISO_9660&diff=25811&oldid=...
Have a nice day :)
Thomas
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
6:17 p.m.
Hi Rich,
if you are still interested, you can try iso-info utility from libcdio package
(distributed with RHEL 9).
Jiri
----- Original Message -----
From: "Richard W.M. Jones" <rjones(a)redhat.com>
To: "Thomas Schmitt" <scdbackup(a)gmx.net>
Cc: libguestfs(a)redhat.com, "Jiri Kucera" <jkucera(a)redhat.com>
Sent: Tuesday, March 30, 2021 11:22:49 PM
Subject: Re: [Libguestfs] Consider to replace isoinfo by own code rather than by xorriso
On Tue, Mar 30, 2021 at 10:57:07PM +0200, Thomas Schmitt wrote:
> Hi,
>
> thank you for flying xorriso. :))
>
> But as its author and looking at
>
> https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml
>
> i feel obliged to point out that using it as replacement for isoinfo
> will increase the consumption of RAM and CPU cycles substantially.
>
> xorriso command -indev loads the whole directory tree of the ISO filesystem
> with names and metadata. Depending on that tree this can be dozens of
> megabytes and lots of insertion operations in a not much optimized tree
> model.
"megabytes" may not be a problem - and in fact hasn't been in my
testing. Could it grow unbounded?
> On the other hand it is quite uncomplicated to implement an own ISO 9660
> PVD reader and to become independent of any external program for that
> purpose.
>
> The Primary Volume Descriptor begins at LBA 16 (= byte offset 32768).
> Its layout is described in ECMA-119 8.4 (note: BP = byte offset + 1)
>
https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_editio...
> or in HTML at
> https://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor
>
> E.g. the info missing in xorriso's output of -pvd_info is at offsets:
>
> 80 - 87 iso_volume_space_size
> (unsigned 32 bit, little endian first, then again as big
> endian)
>
> 124 - 127 iso_volume_sequence_number
> (unsigned 16 bit, little endian first, then again as big
> endian)
>
> 128 - 131 iso_logical_block_size
> (unsigned 16 bit, little endian first, then again as big
> endian)
> (If you don't read 2048 here, then you are in unchartered
> territory.)
>
> The date format for e.g. iso_volume_creation_t at offset 813 - 829
> is a decimal digit string YYYMMDDhhmmsshh with a trailing binary byte
> value for the time zone with 15 minutes granularity. ("hh" means
> hundredths of seconds.)
> See also
> https://wiki.osdev.org/ISO_9660#Numerical_formats
> https://wiki.osdev.org/ISO_9660#Date.2Ftime_format
OK, thanks.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
5:04 a.m.
On Wed, May 26, 2021 at 07:17:52PM -0400, Jiri Kucera wrote:
Hi Rich,
if you are still interested, you can try iso-info utility from
libcdio package (distributed with RHEL 9).
In the end we went with parsing the PVD directly:
https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml
Rich.
Jiri
----- Original Message -----
> From: "Richard W.M. Jones" <rjones(a)redhat.com>
> To: "Thomas Schmitt" <scdbackup(a)gmx.net>
> Cc: libguestfs(a)redhat.com, "Jiri Kucera" <jkucera(a)redhat.com>
> Sent: Tuesday, March 30, 2021 11:22:49 PM
> Subject: Re: [Libguestfs] Consider to replace isoinfo by own code rather than by
xorriso
>
> On Tue, Mar 30, 2021 at 10:57:07PM +0200, Thomas Schmitt wrote:
> > Hi,
> >
> > thank you for flying xorriso. :))
> >
> > But as its author and looking at
> >
> > https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml
> >
> > i feel obliged to point out that using it as replacement for isoinfo
> > will increase the consumption of RAM and CPU cycles substantially.
> >
> > xorriso command -indev loads the whole directory tree of the ISO filesystem
> > with names and metadata. Depending on that tree this can be dozens of
> > megabytes and lots of insertion operations in a not much optimized tree
> > model.
>
> "megabytes" may not be a problem - and in fact hasn't been in my
> testing. Could it grow unbounded?
>
> > On the other hand it is quite uncomplicated to implement an own ISO 9660
> > PVD reader and to become independent of any external program for that
> > purpose.
> >
> > The Primary Volume Descriptor begins at LBA 16 (= byte offset 32768).
> > Its layout is described in ECMA-119 8.4 (note: BP = byte offset + 1)
> >
https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_editio...
> > or in HTML at
> > https://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor
> >
> > E.g. the info missing in xorriso's output of -pvd_info is at offsets:
> >
> > 80 - 87 iso_volume_space_size
> > (unsigned 32 bit, little endian first, then again as big
> > endian)
> >
> > 124 - 127 iso_volume_sequence_number
> > (unsigned 16 bit, little endian first, then again as big
> > endian)
> >
> > 128 - 131 iso_logical_block_size
> > (unsigned 16 bit, little endian first, then again as big
> > endian)
> > (If you don't read 2048 here, then you are in unchartered
> > territory.)
> >
> > The date format for e.g. iso_volume_creation_t at offset 813 - 829
> > is a decimal digit string YYYMMDDhhmmsshh with a trailing binary byte
> > value for the time zone with 15 minutes granularity. ("hh" means
> > hundredths of seconds.)
> > See also
> > https://wiki.osdev.org/ISO_9660#Numerical_formats
> > https://wiki.osdev.org/ISO_9660#Date.2Ftime_format
>
> OK, thanks.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW
>
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
1304
days inactive
1362
days old
8 comments
3 participants
participants (3)
-
Jiri Kucera -
Richard W.M. Jones -
Thomas Schmitt