distutils use in hivex libdnet libnbd (and more) [was: Re: Help needed triaging build failures without distutils]
On Tue, Oct 18, 2022 at 11:27:47AM +0200, Miro Hrončok wrote:
Hey Pythonistas.
The Python standard library distutils module will be removed from Python 3.12+
Annoying, but OK ...
As preparatory work, we build all python packages in a Copr repository with Python 3.11 sans distutils:
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/
I've rebuilt all the failed builds again and also in a control-group copr:
https://copr.fedorainfracloud.org/coprs/g/python/python-with-distutils/
250 packages that failed to build without distuils but succeeded with distutils need to be examined and categorized into various different groups:
- package uses distutils only if sys.version_info < (3, 12) -- this is OK but still fails here - package uses distutils unconditionally and the package needs to be fixed - package uses another package that uses distutils unconditionally and the dependency needs to be fixed
I suspect most of the packages will fail to build with Python 3.12 (planned for Fedora 39, change proposal TBD). The python3-setutpools package provides a distutils module [^1], so sometimes "simply" adding BuildRequires: python3-setuptools might workaround the problem.
...
rjones hivex libdnet libnbd
I think your testing methodology might have been wrong because we use distutils in other packages that I maintain, notably: https://github.com/libguestfs/libguestfs/blob/master/m4/guestfs-python.m4 I'm not sure what could have happened here. The RPM would have failed to build if the Python bindings had been ./configure-d out. The link shows libguestfs as "not build yet / Disabled": https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/pa... but then there's a build which succeeded: https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/bu... and it has Python bindings. (The logs of the build don't seem to be available.) Anyway I'll see if I can fix this upstream as requested. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Tue, Oct 18, 2022 at 10:53:12AM +0100, Richard W.M. Jones wrote:
https://github.com/libguestfs/libguestfs/blob/master/m4/guestfs-python.m4
Actually some help for how to replace it would be useful too as sysconfig is not an exact replacement for distutils.sysconfig. In particular get_python_lib doesn't exist, and I can't find an exact replacement. Original code: $ python3 -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(1,0));' /usr/lib64/python3.11/site-packages Potential replacement: $ python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"));' /usr/local/lib64/python3.11/site-packages Maybe the original code was wrong, but I guess we don't want to install site packages in /usr/local when running under RPM at least. Note it's used here: https://github.com/libguestfs/libguestfs/blob/master/python/Makefile.am#L50-... (We do not -- and cannot -- use ordinary Python install mechanisms.) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
[Sending 2nd email again since it didn't reach the Python-devel list first time] On Tue, Oct 18, 2022 at 11:13:47AM +0100, Richard W.M. Jones wrote:
On Tue, Oct 18, 2022 at 10:53:12AM +0100, Richard W.M. Jones wrote:
https://github.com/libguestfs/libguestfs/blob/master/m4/guestfs-python.m4
Actually some help for how to replace it would be useful too as sysconfig is not an exact replacement for distutils.sysconfig. In particular get_python_lib doesn't exist, and I can't find an exact replacement.
Original code:
$ python3 -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(1,0));' /usr/lib64/python3.11/site-packages
Potential replacement:
$ python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"));' /usr/local/lib64/python3.11/site-packages
Maybe the original code was wrong, but I guess we don't want to install site packages in /usr/local when running under RPM at least. Note it's used here:
https://github.com/libguestfs/libguestfs/blob/master/python/Makefile.am#L50-...
(We do not -- and cannot -- use ordinary Python install mechanisms.)
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On 24. 10. 22 15:14, Richard W.M. Jones wrote:
Original code:
$ python3 -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(1,0));' /usr/lib64/python3.11/site-packages
Potential replacement:
$ python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"));' /usr/local/lib64/python3.11/site-packages
That is a correct replacement. It will return the same value in rpmbuild environment.
Maybe the original code was wrong, but I guess we don't want to install site packages in /usr/local when running under RPM at least.
$ python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"));' /usr/local/lib64/python3.11/site-packages $ RPM_BUILD_ROOT=/ python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"));' /usr/lib64/python3.11/site-packages -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok
On Mon, Oct 24, 2022 at 06:00:31PM +0200, Miro Hrončok wrote:
On 24. 10. 22 15:14, Richard W.M. Jones wrote:
Original code:
$ python3 -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(1,0));' /usr/lib64/python3.11/site-packages
Potential replacement:
$ python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"));' /usr/local/lib64/python3.11/site-packages
That is a correct replacement. It will return the same value in rpmbuild environment.
Maybe the original code was wrong, but I guess we don't want to install site packages in /usr/local when running under RPM at least.
$ python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"));' /usr/local/lib64/python3.11/site-packages
$ RPM_BUILD_ROOT=/ python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"));' /usr/lib64/python3.11/site-packages
Ah ha! Good one, thanks :-) Here's the fix for libguestfs: https://github.com/libguestfs/libguestfs/commit/26940f64a740676103b0ee49bf0c... I'll leave this fix to stew for a while and if it works I'll fix various other projects in a similar way. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
[Sending again since it didn't reach the Python-devel list first time] On Tue, Oct 18, 2022 at 10:53:12AM +0100, Richard W.M. Jones wrote:
On Tue, Oct 18, 2022 at 11:27:47AM +0200, Miro Hrončok wrote:
Hey Pythonistas.
The Python standard library distutils module will be removed from Python 3.12+
Annoying, but OK ...
As preparatory work, we build all python packages in a Copr repository with Python 3.11 sans distutils:
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/
I've rebuilt all the failed builds again and also in a control-group copr:
https://copr.fedorainfracloud.org/coprs/g/python/python-with-distutils/
250 packages that failed to build without distuils but succeeded with distutils need to be examined and categorized into various different groups:
- package uses distutils only if sys.version_info < (3, 12) -- this is OK but still fails here - package uses distutils unconditionally and the package needs to be fixed - package uses another package that uses distutils unconditionally and the dependency needs to be fixed
I suspect most of the packages will fail to build with Python 3.12 (planned for Fedora 39, change proposal TBD). The python3-setutpools package provides a distutils module [^1], so sometimes "simply" adding BuildRequires: python3-setuptools might workaround the problem.
...
rjones hivex libdnet libnbd
I think your testing methodology might have been wrong because we use distutils in other packages that I maintain, notably:
https://github.com/libguestfs/libguestfs/blob/master/m4/guestfs-python.m4
I'm not sure what could have happened here. The RPM would have failed to build if the Python bindings had been ./configure-d out. The link shows libguestfs as "not build yet / Disabled":
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/pa...
but then there's a build which succeeded:
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/bu...
and it has Python bindings. (The logs of the build don't seem to be available.)
Anyway I'll see if I can fix this upstream as requested.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On 24. 10. 22 12:46, Richard W.M. Jones wrote:
I'm not sure what could have happened here. The RPM would have failed to build if the Python bindings had been ./configure-d out. The link shows libguestfs as "not build yet / Disabled":
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/pa...
(I forgot to respond to this part of your email, so doing that now.) That is caused by the fact that I build packages in a custom "directory" which is a copr thing that allows me to build thousands of packages in copr without them "seeing" each other, but it has some UX downsides, such as not being visible as built on this page.
but then there's a build which succeeded:
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/bu...
and it has Python bindings. (The logs of the build don't seem to be available.)
Yeah, I only renewed failed builds for the logs, I had no idea people will want to examine succeeded packages. Looking at the latest succeeded root.log from Koji instead https://koji.fedoraproject.org/koji/buildinfo?buildID=2074440 https://kojipkgs.fedoraproject.org//packages/libguestfs/1.49.5/2.fc38/data/l... I see python3-setuptools was installed, which is the likely reason the build succeeded without distutils (as said in my announcement, setuptools provide a distutils module, which is 99%-compatible replacement for the removed standard library module). -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok
participants (2)
-
Miro Hrončok -
Richard W.M. Jones