3:23 p.m.
On Tue, Jul 11, 2023 at 07:31:07PM +0100, Richard W.M. Jones wrote:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2230411
It's a bit of a coincidence that this failed exactly in the Perl test
& you're upgrading Perl. On the other hand, it really looks like it
might have been a temporary failure.
Could you resubmit the build? If it happens again then I'll take a
closer look.
Thanks for resubmitting this. It looks as if it failed again:
https://koji.fedoraproject.org/koji/taskinfo?taskID=103238055
& the failure _is_ related to perl (surprisingly), or at least it
fails in the same way.
I will try to have a look at this, but if you would like to file a BZ
then that will ensure that we don't forget about it.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
2:46 a.m.
New subject: nbdkit build failure (was: Re: jplesnik's nbdkit-1.35.5-4.fc39 failed to build)
On Tue, Jul 11, 2023 at 09:23:35PM +0100, Richard W.M. Jones wrote:
On Tue, Jul 11, 2023 at 07:31:07PM +0100, Richard W.M. Jones wrote:
>
> https://koji.fedoraproject.org/koji/buildinfo?buildID=2230411
>
> It's a bit of a coincidence that this failed exactly in the Perl test
> & you're upgrading Perl. On the other hand, it really looks like it
> might have been a temporary failure.
>
> Could you resubmit the build? If it happens again then I'll take a
> closer look.
Thanks for resubmitting this. It looks as if it failed again:
https://koji.fedoraproject.org/koji/taskinfo?taskID=103238055
& the failure _is_ related to perl (surprisingly), or at least it
fails in the same way.
The next build failure looks identical as well:
https://koji.fedoraproject.org/koji/taskinfo?taskID=103297793
So it seems like this is a problem in the perl plugin (or maybe with
the new version of perl).
I will try to have a look at this, but if you would like to file a
BZ
then that will ensure that we don't forget about it.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
4:29 a.m.
New subject: nbdkit build failure (was: Re: jplesnik's nbdkit-1.35.5-4.fc39 failed to build)
It's a crash in Perl code, although called from the nbdkit perl plugin
(C bindings). Still debugging this ...
Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -U /tmp/nbdkittPN34a/sock -P
/tmp/nbdkittPN'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/150177' in core file too small.
#0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32,
format=0x7f3a1ef7e322 "_<(eval %lu)")
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
Downloading source file /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c
5248 WITH_LC_NUMERIC_SET_TO_NEEDED(
[Current thread is 1 (Thread 0x7f3a1e5fd6c0 (LWP 150177))]
(gdb) bt
#0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32,
format=0x7f3a1ef7e322 "_<(eval %lu)")
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
#1 0x00007f3a1ed8545f in Perl_pp_entereval (my_perl=0xcb39c90)
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/pp_ctl.c:5107
#2 0x00007f3a1ed2d5e8 in Perl_runops_standard (my_perl=0xcb39c90)
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/run.c:41
#3 0x00007f3a1ec73ddc in Perl_eval_sv (my_perl=my_perl@entry=0xcb39c90,
sv=sv@entry=0xcb3d2b8, flags=flags@entry=2)
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/perl.c:3269
#4 0x00007f3a1ec740e0 in Perl_eval_pv (my_perl=0xcb39c90, p=<optimized out>,
croak_on_error=croak_on_error@entry=0)
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/perl.c:3359
#5 0x00007f3a1fc9e6ca in callback_defined (
perl_func_name=perl_func_name@entry=0x7f3a1fca21af "can_write")
at /home/rjones/d/nbdkit/plugins/perl/perl.c:102
#6 0x00007f3a1fc9f28c in perl_boolean (handle=0xcb7a350,
callback_name=0x7f3a1fca21af "can_write", fn_name=0x7f3a1fca21b3
"write")
at /home/rjones/d/nbdkit/plugins/perl/perl.c:430
#7 0x000000000040e290 in plugin_can_write (c=<optimized out>) at plugins.c:488
#8 0x00000000004078fe in backend_can_write (c=0x7f3a18000fb0) at backend.c:462
#9 0x0000000000410ea9 in protocol_common_open (
exportsize=exportsize@entry=0x7f3a1e5fca00,
flags=flags@entry=0x7f3a18000e84,
exportname=exportname@entry=0x7f3a18000f90 "") at protocol-handshake.c:103
#10 0x00000000004112ab in finish_newstyle_options (
exportsize=exportsize@entry=0x7f3a1e5fca00, exportname_in=<optimized out>,
exportnamelen=exportnamelen@entry=0) at protocol-handshake-newstyle.c:319
#11 0x00000000004125e1 in negotiate_handshake_newstyle_options ()
at protocol-handshake-newstyle.c:619
#12 protocol_handshake_newstyle () at protocol-handshake-newstyle.c:985
#13 0x0000000000410e15 in protocol_handshake () at protocol-handshake.c:55
#14 0x000000000040a474 in handle_single_connection (sockin=6,
sockout=<optimized out>) at connections.c:180
#15 0x0000000000414baf in start_thread (datav=0xcb2c4f0) at sockets.c:377
#16 0x00007f3a1f68db37 in start_thread (arg=<optimized out>)
at pthread_create.c:444
#17 0x00007f3a1f714bec in clone3 ()
at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
5:03 a.m.
New subject: nbdkit build failure (was: Re: jplesnik's nbdkit-1.35.5-4.fc39 failed to build)
On Thu, Jul 13, 2023 at 10:29:44AM +0100, Richard W.M. Jones wrote:
It's a crash in Perl code, although called from the nbdkit perl
plugin
(C bindings). Still debugging this ...
Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -U /tmp/nbdkittPN34a/sock -P
/tmp/nbdkittPN'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/150177' in core file too small.
#0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32,
format=0x7f3a1ef7e322 "_<(eval %lu)")
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
Downloading source file /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c
5248 WITH_LC_NUMERIC_SET_TO_NEEDED(
[Current thread is 1 (Thread 0x7f3a1e5fd6c0 (LWP 150177))]
(gdb) bt
#0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32,
format=0x7f3a1ef7e322 "_<(eval %lu)")
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
https://github.com/Perl/perl5/blob/ff7e37d5ca2e55a1e53e5b375e3cc9aedc9ce4...
Since its hardly obvious from the code, that macro expands to the
monster below. I didn't bother to format it completely ...
# 5245 "util.c"
if (retval == -1) {
do {
void (*_restore_LC_NUMERIC_function)(PerlInterpreter* my_perl __attribute__((unused)))
= ((void *)0);
do {
_Bool _in_lc_numeric = ((( ((
_Bool
) ((my_perl->Icompiling).cop_hints & 0x00000004)) || ( ((_Bool)
((my_perl->Icompiling).cop_hints & 0x00000010)) &&
Perl__is_in_locale_category(my_perl, (1), (
1
)))) || (((my_perl->Icurcop) && (((my_perl->Icurcop))->cop_hints +
0) & 0x00000004) || (((my_perl->Icurcop) &&
(((my_perl->Icurcop))->cop_hints + 0) & 0x00000010) &&
Perl__is_in_locale_category(my_perl, (0), (
1
))))));
((void)0);
if (_in_lc_numeric) {
if ((! (my_perl->Inumeric_underlying) &&
(my_perl->Inumeric_standard) < 2)) { Perl_set_numeric_underlying(my_perl);
_restore_LC_NUMERIC_function = &Perl_set_numeric_standard; }
} else {
if ((! (my_perl->Inumeric_standard))) { Perl_set_numeric_standard(my_perl);
_restore_LC_NUMERIC_function = &Perl_set_numeric_underlying; } }
} while (0);
retval = vsnprintf(buffer, len, format, ap);;
do { if (_restore_LC_NUMERIC_function) { _restore_LC_NUMERIC_function(my_perl); }
((void)0); } while (0);
} while (0);
... because the fundamental problem is:
(gdb) print my_perl
$13 = (PerlInterpreter *) 0x0
so that's not gonna work.
Notice the assignment of my_perl = NULL in the macro expansion. I
think that comes from a dTHX macro, but the macros are so nested that
it's hard to tell where from. We are creating a Perl interpreter in
our code.
I'll come up with a minimal reproducer & file a BZ.
Rich.
#1 0x00007f3a1ed8545f in Perl_pp_entereval (my_perl=0xcb39c90)
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/pp_ctl.c:5107
#2 0x00007f3a1ed2d5e8 in Perl_runops_standard (my_perl=0xcb39c90)
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/run.c:41
#3 0x00007f3a1ec73ddc in Perl_eval_sv (my_perl=my_perl@entry=0xcb39c90,
sv=sv@entry=0xcb3d2b8, flags=flags@entry=2)
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/perl.c:3269
#4 0x00007f3a1ec740e0 in Perl_eval_pv (my_perl=0xcb39c90, p=<optimized out>,
croak_on_error=croak_on_error@entry=0)
at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/perl.c:3359
#5 0x00007f3a1fc9e6ca in callback_defined (
perl_func_name=perl_func_name@entry=0x7f3a1fca21af "can_write")
at /home/rjones/d/nbdkit/plugins/perl/perl.c:102
#6 0x00007f3a1fc9f28c in perl_boolean (handle=0xcb7a350,
callback_name=0x7f3a1fca21af "can_write", fn_name=0x7f3a1fca21b3
"write")
at /home/rjones/d/nbdkit/plugins/perl/perl.c:430
#7 0x000000000040e290 in plugin_can_write (c=<optimized out>) at plugins.c:488
#8 0x00000000004078fe in backend_can_write (c=0x7f3a18000fb0) at backend.c:462
#9 0x0000000000410ea9 in protocol_common_open (
exportsize=exportsize@entry=0x7f3a1e5fca00,
flags=flags@entry=0x7f3a18000e84,
exportname=exportname@entry=0x7f3a18000f90 "") at protocol-handshake.c:103
#10 0x00000000004112ab in finish_newstyle_options (
exportsize=exportsize@entry=0x7f3a1e5fca00, exportname_in=<optimized out>,
exportnamelen=exportnamelen@entry=0) at protocol-handshake-newstyle.c:319
#11 0x00000000004125e1 in negotiate_handshake_newstyle_options ()
at protocol-handshake-newstyle.c:619
#12 protocol_handshake_newstyle () at protocol-handshake-newstyle.c:985
#13 0x0000000000410e15 in protocol_handshake () at protocol-handshake.c:55
#14 0x000000000040a474 in handle_single_connection (sockin=6,
sockout=<optimized out>) at connections.c:180
#15 0x0000000000414baf in start_thread (datav=0xcb2c4f0) at sockets.c:377
#16 0x00007f3a1f68db37 in start_thread (arg=<optimized out>)
at pthread_create.c:444
#17 0x00007f3a1f714bec in clone3 ()
at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
5:55 a.m.
New subject: nbdkit build failure (was: Re: jplesnik's nbdkit-1.35.5-4.fc39 failed to build)
On Thu, Jul 13, 2023 at 11:03:35AM +0100, Richard W.M. Jones wrote:
On Thu, Jul 13, 2023 at 10:29:44AM +0100, Richard W.M. Jones wrote:
> It's a crash in Perl code, although called from the nbdkit perl plugin
> (C bindings). Still debugging this ...
>
> Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -U /tmp/nbdkittPN34a/sock
-P /tmp/nbdkittPN'.
> Program terminated with signal SIGSEGV, Segmentation fault.
>
> warning: Section `.reg-xstate/150177' in core file too small.
> #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32,
> format=0x7f3a1ef7e322 "_<(eval %lu)")
> at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
> Downloading source file /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c
> 5248 WITH_LC_NUMERIC_SET_TO_NEEDED(
> [Current thread is 1 (Thread 0x7f3a1e5fd6c0 (LWP 150177))]
> (gdb) bt
> #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32,
> format=0x7f3a1ef7e322 "_<(eval %lu)")
> at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
https://github.com/Perl/perl5/blob/ff7e37d5ca2e55a1e53e5b375e3cc9aedc9ce4...
Since its hardly obvious from the code, that macro expands to the
monster below. I didn't bother to format it completely ...
# 5245 "util.c"
if (retval == -1) {
do {
void (*_restore_LC_NUMERIC_function)(PerlInterpreter* my_perl __attribute__((unused)))
= ((void *)0);
do {
_Bool _in_lc_numeric = ((( ((
_Bool
) ((my_perl->Icompiling).cop_hints & 0x00000004)) || ( ((_Bool)
((my_perl->Icompiling).cop_hints & 0x00000010)) &&
Perl__is_in_locale_category(my_perl, (1), (
1
)))) || (((my_perl->Icurcop) && (((my_perl->Icurcop))->cop_hints
+ 0) & 0x00000004) || (((my_perl->Icurcop) &&
(((my_perl->Icurcop))->cop_hints + 0) & 0x00000010) &&
Perl__is_in_locale_category(my_perl, (0), (
1
))))));
((void)0);
if (_in_lc_numeric) {
if ((! (my_perl->Inumeric_underlying) &&
(my_perl->Inumeric_standard) < 2)) { Perl_set_numeric_underlying(my_perl);
_restore_LC_NUMERIC_function = &Perl_set_numeric_standard; }
} else {
if ((! (my_perl->Inumeric_standard))) { Perl_set_numeric_standard(my_perl);
_restore_LC_NUMERIC_function = &Perl_set_numeric_underlying; } }
} while (0);
retval = vsnprintf(buffer, len, format, ap);;
do { if (_restore_LC_NUMERIC_function) { _restore_LC_NUMERIC_function(my_perl); }
((void)0); } while (0);
} while (0);
... because the fundamental problem is:
(gdb) print my_perl
$13 = (PerlInterpreter *) 0x0
so that's not gonna work.
It turns out to be a bug in nbdkit! A close reading of the perlembed
man page reveals that we need to set the interpreter context when
calling into Perl from a new thread. The fix is:
https://gitlab.com/nbdkit/nbdkit/-/commit/495708e9d1b396ff450df15d5789b4f...
I'll add this to Rawhide shortly.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
6:10 a.m.
New subject: nbdkit build failure (was: Re: jplesnik's nbdkit-1.35.5-4.fc39 failed to build)
Thank you very much for your investigation.
Perl 5.38 is merge to rawhide now.
Regards;
Jitka
On 7/13/23 12:55, Richard W.M. Jones wrote:
On Thu, Jul 13, 2023 at 11:03:35AM +0100, Richard W.M. Jones wrote:
> On Thu, Jul 13, 2023 at 10:29:44AM +0100, Richard W.M. Jones wrote:
>> It's a crash in Perl code, although called from the nbdkit perl plugin
>> (C bindings). Still debugging this ...
>>
>> Core was generated by `/home/rjones/d/nbdkit/server/nbdkit -U
/tmp/nbdkittPN34a/sock -P /tmp/nbdkittPN'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>>
>> warning: Section `.reg-xstate/150177' in core file too small.
>> #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32,
>> format=0x7f3a1ef7e322 "_<(eval %lu)")
>> at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
>> Downloading source file /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c
>> 5248 WITH_LC_NUMERIC_SET_TO_NEEDED(
>> [Current thread is 1 (Thread 0x7f3a1e5fd6c0 (LWP 150177))]
>> (gdb) bt
>> #0 Perl_my_snprintf (buffer=0x7f3a1e5fc630 "p\310_\036:\177", len=32,
>> format=0x7f3a1ef7e322 "_<(eval %lu)")
>> at /usr/src/debug/perl-5.38.0-499.fc39.x86_64/util.c:5248
>
https://github.com/Perl/perl5/blob/ff7e37d5ca2e55a1e53e5b375e3cc9aedc9ce4...
>
> Since its hardly obvious from the code, that macro expands to the
> monster below. I didn't bother to format it completely ...
>
> # 5245 "util.c"
> if (retval == -1) {
> do {
> void (*_restore_LC_NUMERIC_function)(PerlInterpreter* my_perl
__attribute__((unused))) = ((void *)0);
> do {
> _Bool _in_lc_numeric = ((( ((
> _Bool
> ) ((my_perl->Icompiling).cop_hints & 0x00000004)) || (
((_Bool) ((my_perl->Icompiling).cop_hints & 0x00000010)) &&
Perl__is_in_locale_category(my_perl, (1), (
> 1
> )))) || (((my_perl->Icurcop) &&
(((my_perl->Icurcop))->cop_hints + 0) & 0x00000004) || (((my_perl->Icurcop)
&& (((my_perl->Icurcop))->cop_hints + 0) & 0x00000010) &&
Perl__is_in_locale_category(my_perl, (0), (
> 1
> ))))));
> ((void)0);
> if (_in_lc_numeric) {
> if ((! (my_perl->Inumeric_underlying) &&
(my_perl->Inumeric_standard) < 2)) { Perl_set_numeric_underlying(my_perl);
_restore_LC_NUMERIC_function = &Perl_set_numeric_standard; }
> } else {
> if ((! (my_perl->Inumeric_standard))) { Perl_set_numeric_standard(my_perl);
_restore_LC_NUMERIC_function = &Perl_set_numeric_underlying; } }
> } while (0);
> retval = vsnprintf(buffer, len, format, ap);;
> do { if (_restore_LC_NUMERIC_function) { _restore_LC_NUMERIC_function(my_perl); }
((void)0); } while (0);
> } while (0);
>
> ... because the fundamental problem is:
>
> (gdb) print my_perl
> $13 = (PerlInterpreter *) 0x0
>
> so that's not gonna work.
It turns out to be a bug in nbdkit! A close reading of the perlembed
man page reveals that we need to set the interpreter context when
calling into Perl from a new thread. The fix is:
https://gitlab.com/nbdkit/nbdkit/-/commit/495708e9d1b396ff450df15d5789b4f...
I'll add this to Rawhide shortly.
Rich.
--
Jitka Plesnikova
Senior Software Engineer
Red Hat
499
days inactive
501
days old
5 comments
2 participants
participants (2)
-
Jitka Plesnikova -
Richard W.M. Jones