Re: [Libguestfs] nbdkit build failure (was: Re: jplesnik's nbdkit-1.35.5-4.fc39 failed to build)
On Tue, Jul 11, 2023 at 07:31:07PM +0100, Richard W.M. Jones wrote:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2230411
It's a bit of a coincidence that this failed exactly in the Perl test & you're upgrading Perl. On the other hand, it really looks like it might have been a temporary failure.
Could you resubmit the build? If it happens again then I'll take a closer look.
Thanks for resubmitting this. It looks as if it failed again: https://koji.fedoraproject.org/koji/taskinfo?taskID=103238055 & the failure _is_ related to perl (surprisingly), or at least it fails in the same way. I will try to have a look at this, but if you would like to file a BZ then that will ensure that we don't forget about it. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On Tue, Jul 11, 2023 at 09:23:35PM +0100, Richard W.M. Jones wrote:
On Tue, Jul 11, 2023 at 07:31:07PM +0100, Richard W.M. Jones wrote:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2230411
It's a bit of a coincidence that this failed exactly in the Perl test & you're upgrading Perl. On the other hand, it really looks like it might have been a temporary failure.
Could you resubmit the build? If it happens again then I'll take a closer look.
Thanks for resubmitting this. It looks as if it failed again:
https://koji.fedoraproject.org/koji/taskinfo?taskID=103238055
& the failure _is_ related to perl (surprisingly), or at least it fails in the same way.
The next build failure looks identical as well: https://koji.fedoraproject.org/koji/taskinfo?taskID=103297793 So it seems like this is a problem in the perl plugin (or maybe with the new version of perl).
I will try to have a look at this, but if you would like to file a BZ then that will ensure that we don't forget about it.
Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
It's a crash in Perl code, although called from the nbdkit perl plugin (C bindings). Still debugging this ... Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -U /tmp/nbdkittPN34a/sock -P /tmp/nbdkittPN'. Program terminated with signal SIGSEGV, Segmentation fault. warning: Section `.reg-xstate/150177' in core file too small. #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32, format=0x7f3a1ef7e322 "_<(eval %lu)") at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248 Downloading source file /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c 5248 WITH_LC_NUMERIC_SET_TO_NEEDED( [Current thread is 1 (Thread 0x7f3a1e5fd6c0 (LWP 150177))] (gdb) bt #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32, format=0x7f3a1ef7e322 "_<(eval %lu)") at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248 #1 0x00007f3a1ed8545f in Perl_pp_entereval (my_perl=0xcb39c90) at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/pp_ctl.c:5107 #2 0x00007f3a1ed2d5e8 in Perl_runops_standard (my_perl=0xcb39c90) at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/run.c:41 #3 0x00007f3a1ec73ddc in Perl_eval_sv (my_perl=my_perl@entry=0xcb39c90, sv=sv@entry=0xcb3d2b8, flags=flags@entry=2) at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/perl.c:3269 #4 0x00007f3a1ec740e0 in Perl_eval_pv (my_perl=0xcb39c90, p=<optimized out>, croak_on_error=croak_on_error@entry=0) at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/perl.c:3359 #5 0x00007f3a1fc9e6ca in callback_defined ( perl_func_name=perl_func_name@entry=0x7f3a1fca21af "can_write") at /home/rjones/d/nbdkit/plugins/perl/perl.c:102 #6 0x00007f3a1fc9f28c in perl_boolean (handle=0xcb7a350, callback_name=0x7f3a1fca21af "can_write", fn_name=0x7f3a1fca21b3 "write") at /home/rjones/d/nbdkit/plugins/perl/perl.c:430 #7 0x000000000040e290 in plugin_can_write (c=<optimized out>) at plugins.c:488 #8 0x00000000004078fe in backend_can_write (c=0x7f3a18000fb0) at backend.c:462 #9 0x0000000000410ea9 in protocol_common_open ( exportsize=exportsize@entry=0x7f3a1e5fca00, flags=flags@entry=0x7f3a18000e84, exportname=exportname@entry=0x7f3a18000f90 "") at protocol-handshake.c:103 #10 0x00000000004112ab in finish_newstyle_options ( exportsize=exportsize@entry=0x7f3a1e5fca00, exportname_in=<optimized out>, exportnamelen=exportnamelen@entry=0) at protocol-handshake-newstyle.c:319 #11 0x00000000004125e1 in negotiate_handshake_newstyle_options () at protocol-handshake-newstyle.c:619 #12 protocol_handshake_newstyle () at protocol-handshake-newstyle.c:985 #13 0x0000000000410e15 in protocol_handshake () at protocol-handshake.c:55 #14 0x000000000040a474 in handle_single_connection (sockin=6, sockout=<optimized out>) at connections.c:180 #15 0x0000000000414baf in start_thread (datav=0xcb2c4f0) at sockets.c:377 #16 0x00007f3a1f68db37 in start_thread (arg=<optimized out>) at pthread_create.c:444 #17 0x00007f3a1f714bec in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On Thu, Jul 13, 2023 at 10:29:44AM +0100, Richard W.M. Jones wrote:
It's a crash in Perl code, although called from the nbdkit perl plugin (C bindings). Still debugging this ...
Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -U /tmp/nbdkittPN34a/sock -P /tmp/nbdkittPN'. Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/150177' in core file too small. #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32, format=0x7f3a1ef7e322 "_<(eval %lu)") at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248 Downloading source file /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c 5248 WITH_LC_NUMERIC_SET_TO_NEEDED( [Current thread is 1 (Thread 0x7f3a1e5fd6c0 (LWP 150177))] (gdb) bt #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32, format=0x7f3a1ef7e322 "_<(eval %lu)") at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
https://github.com/Perl/perl5/blob/ff7e37d5ca2e55a1e53e5b375e3cc9aedc9ce4ec/... Since its hardly obvious from the code, that macro expands to the monster below. I didn't bother to format it completely ... # 5245 "util.c" if (retval == -1) { do { void (*_restore_LC_NUMERIC_function)(PerlInterpreter* my_perl __attribute__((unused))) = ((void *)0); do { _Bool _in_lc_numeric = ((( (( _Bool ) ((my_perl->Icompiling).cop_hints & 0x00000004)) || ( ((_Bool) ((my_perl->Icompiling).cop_hints & 0x00000010)) && Perl__is_in_locale_category(my_perl, (1), ( 1 )))) || (((my_perl->Icurcop) && (((my_perl->Icurcop))->cop_hints + 0) & 0x00000004) || (((my_perl->Icurcop) && (((my_perl->Icurcop))->cop_hints + 0) & 0x00000010) && Perl__is_in_locale_category(my_perl, (0), ( 1 )))))); ((void)0); if (_in_lc_numeric) { if ((! (my_perl->Inumeric_underlying) && (my_perl->Inumeric_standard) < 2)) { Perl_set_numeric_underlying(my_perl); _restore_LC_NUMERIC_function = &Perl_set_numeric_standard; } } else { if ((! (my_perl->Inumeric_standard))) { Perl_set_numeric_standard(my_perl); _restore_LC_NUMERIC_function = &Perl_set_numeric_underlying; } } } while (0); retval = vsnprintf(buffer, len, format, ap);; do { if (_restore_LC_NUMERIC_function) { _restore_LC_NUMERIC_function(my_perl); } ((void)0); } while (0); } while (0); ... because the fundamental problem is: (gdb) print my_perl $13 = (PerlInterpreter *) 0x0 so that's not gonna work. Notice the assignment of my_perl = NULL in the macro expansion. I think that comes from a dTHX macro, but the macros are so nested that it's hard to tell where from. We are creating a Perl interpreter in our code. I'll come up with a minimal reproducer & file a BZ. Rich.
#1 0x00007f3a1ed8545f in Perl_pp_entereval (my_perl=0xcb39c90) at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/pp_ctl.c:5107 #2 0x00007f3a1ed2d5e8 in Perl_runops_standard (my_perl=0xcb39c90) at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/run.c:41 #3 0x00007f3a1ec73ddc in Perl_eval_sv (my_perl=my_perl@entry=0xcb39c90, sv=sv@entry=0xcb3d2b8, flags=flags@entry=2) at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/perl.c:3269 #4 0x00007f3a1ec740e0 in Perl_eval_pv (my_perl=0xcb39c90, p=<optimized out>, croak_on_error=croak_on_error@entry=0) at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/perl.c:3359 #5 0x00007f3a1fc9e6ca in callback_defined ( perl_func_name=perl_func_name@entry=0x7f3a1fca21af "can_write") at /home/rjones/d/nbdkit/plugins/perl/perl.c:102 #6 0x00007f3a1fc9f28c in perl_boolean (handle=0xcb7a350, callback_name=0x7f3a1fca21af "can_write", fn_name=0x7f3a1fca21b3 "write") at /home/rjones/d/nbdkit/plugins/perl/perl.c:430 #7 0x000000000040e290 in plugin_can_write (c=<optimized out>) at plugins.c:488 #8 0x00000000004078fe in backend_can_write (c=0x7f3a18000fb0) at backend.c:462 #9 0x0000000000410ea9 in protocol_common_open ( exportsize=exportsize@entry=0x7f3a1e5fca00, flags=flags@entry=0x7f3a18000e84, exportname=exportname@entry=0x7f3a18000f90 "") at protocol-handshake.c:103 #10 0x00000000004112ab in finish_newstyle_options ( exportsize=exportsize@entry=0x7f3a1e5fca00, exportname_in=<optimized out>, exportnamelen=exportnamelen@entry=0) at protocol-handshake-newstyle.c:319 #11 0x00000000004125e1 in negotiate_handshake_newstyle_options () at protocol-handshake-newstyle.c:619 #12 protocol_handshake_newstyle () at protocol-handshake-newstyle.c:985 #13 0x0000000000410e15 in protocol_handshake () at protocol-handshake.c:55 #14 0x000000000040a474 in handle_single_connection (sockin=6, sockout=<optimized out>) at connections.c:180 #15 0x0000000000414baf in start_thread (datav=0xcb2c4f0) at sockets.c:377 #16 0x00007f3a1f68db37 in start_thread (arg=<optimized out>) at pthread_create.c:444 #17 0x00007f3a1f714bec in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Thu, Jul 13, 2023 at 11:03:35AM +0100, Richard W.M. Jones wrote:
On Thu, Jul 13, 2023 at 10:29:44AM +0100, Richard W.M. Jones wrote:
It's a crash in Perl code, although called from the nbdkit perl plugin (C bindings). Still debugging this ...
Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -U /tmp/nbdkittPN34a/sock -P /tmp/nbdkittPN'. Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/150177' in core file too small. #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32, format=0x7f3a1ef7e322 "_<(eval %lu)") at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248 Downloading source file /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c 5248 WITH_LC_NUMERIC_SET_TO_NEEDED( [Current thread is 1 (Thread 0x7f3a1e5fd6c0 (LWP 150177))] (gdb) bt #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32, format=0x7f3a1ef7e322 "_<(eval %lu)") at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
https://github.com/Perl/perl5/blob/ff7e37d5ca2e55a1e53e5b375e3cc9aedc9ce4ec/...
Since its hardly obvious from the code, that macro expands to the monster below. I didn't bother to format it completely ...
# 5245 "util.c" if (retval == -1) { do { void (*_restore_LC_NUMERIC_function)(PerlInterpreter* my_perl __attribute__((unused))) = ((void *)0); do { _Bool _in_lc_numeric = ((( (( _Bool ) ((my_perl->Icompiling).cop_hints & 0x00000004)) || ( ((_Bool) ((my_perl->Icompiling).cop_hints & 0x00000010)) && Perl__is_in_locale_category(my_perl, (1), ( 1 )))) || (((my_perl->Icurcop) && (((my_perl->Icurcop))->cop_hints + 0) & 0x00000004) || (((my_perl->Icurcop) && (((my_perl->Icurcop))->cop_hints + 0) & 0x00000010) && Perl__is_in_locale_category(my_perl, (0), ( 1 )))))); ((void)0); if (_in_lc_numeric) { if ((! (my_perl->Inumeric_underlying) && (my_perl->Inumeric_standard) < 2)) { Perl_set_numeric_underlying(my_perl); _restore_LC_NUMERIC_function = &Perl_set_numeric_standard; } } else { if ((! (my_perl->Inumeric_standard))) { Perl_set_numeric_standard(my_perl); _restore_LC_NUMERIC_function = &Perl_set_numeric_underlying; } } } while (0); retval = vsnprintf(buffer, len, format, ap);; do { if (_restore_LC_NUMERIC_function) { _restore_LC_NUMERIC_function(my_perl); } ((void)0); } while (0); } while (0);
... because the fundamental problem is:
(gdb) print my_perl $13 = (PerlInterpreter *) 0x0
so that's not gonna work.
It turns out to be a bug in nbdkit! A close reading of the perlembed man page reveals that we need to set the interpreter context when calling into Perl from a new thread. The fix is: https://gitlab.com/nbdkit/nbdkit/-/commit/495708e9d1b396ff450df15d5789b4f6c5... I'll add this to Rawhide shortly. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
Thank you very much for your investigation. Perl 5.38 is merge to rawhide now. Regards; Jitka On 7/13/23 12:55, Richard W.M. Jones wrote:
On Thu, Jul 13, 2023 at 11:03:35AM +0100, Richard W.M. Jones wrote:
On Thu, Jul 13, 2023 at 10:29:44AM +0100, Richard W.M. Jones wrote:
It's a crash in Perl code, although called from the nbdkit perl plugin (C bindings). Still debugging this ...
Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -U /tmp/nbdkittPN34a/sock -P /tmp/nbdkittPN'. Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/150177' in core file too small. #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32, format=0x7f3a1ef7e322 "_<(eval %lu)") at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248 Downloading source file /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c 5248 WITH_LC_NUMERIC_SET_TO_NEEDED( [Current thread is 1 (Thread 0x7f3a1e5fd6c0 (LWP 150177))] (gdb) bt #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32, format=0x7f3a1ef7e322 "_<(eval %lu)") at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248 https://github.com/Perl/perl5/blob/ff7e37d5ca2e55a1e53e5b375e3cc9aedc9ce4ec/...
Since its hardly obvious from the code, that macro expands to the monster below. I didn't bother to format it completely ...
# 5245 "util.c" if (retval == -1) { do { void (*_restore_LC_NUMERIC_function)(PerlInterpreter* my_perl __attribute__((unused))) = ((void *)0); do { _Bool _in_lc_numeric = ((( (( _Bool ) ((my_perl->Icompiling).cop_hints & 0x00000004)) || ( ((_Bool) ((my_perl->Icompiling).cop_hints & 0x00000010)) && Perl__is_in_locale_category(my_perl, (1), ( 1 )))) || (((my_perl->Icurcop) && (((my_perl->Icurcop))->cop_hints + 0) & 0x00000004) || (((my_perl->Icurcop) && (((my_perl->Icurcop))->cop_hints + 0) & 0x00000010) && Perl__is_in_locale_category(my_perl, (0), ( 1 )))))); ((void)0); if (_in_lc_numeric) { if ((! (my_perl->Inumeric_underlying) && (my_perl->Inumeric_standard) < 2)) { Perl_set_numeric_underlying(my_perl); _restore_LC_NUMERIC_function = &Perl_set_numeric_standard; } } else { if ((! (my_perl->Inumeric_standard))) { Perl_set_numeric_standard(my_perl); _restore_LC_NUMERIC_function = &Perl_set_numeric_underlying; } } } while (0); retval = vsnprintf(buffer, len, format, ap);; do { if (_restore_LC_NUMERIC_function) { _restore_LC_NUMERIC_function(my_perl); } ((void)0); } while (0); } while (0);
... because the fundamental problem is:
(gdb) print my_perl $13 = (PerlInterpreter *) 0x0
so that's not gonna work. It turns out to be a bug in nbdkit! A close reading of the perlembed man page reveals that we need to set the interpreter context when calling into Perl from a new thread. The fix is:
https://gitlab.com/nbdkit/nbdkit/-/commit/495708e9d1b396ff450df15d5789b4f6c5...
I'll add this to Rawhide shortly.
Rich.
-- Jitka Plesnikova Senior Software Engineer Red Hat
participants (2)
-
Jitka Plesnikova -
Richard W.M. Jones