[nbdkit python]: unable to locate modules
I'm working on a python plugin for nbdkit that treats Google Cloud Storage as a block device. I have a library, gcs.py, that implements simple commands: list, get, put, delete; source is here: https://github.com/wspeirs/cldblkdev/blob/master/gcs.py When I attempt to run nbdkit I get the following error: nbdkit python script=/home/wspeirs/src/cldblkdev/cldblkdev.py Traceback (most recent call last): File "/home/wspeirs/src/cldblkdev/cldblkdev.py", line 5, in <module> import gcs File "/home/wspeirs/src/cldblkdev/gcs.py", line 2, in <module> import httplib2 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 930, in <module> class HTTPSConnectionWithTimeout(httplib.HTTPSConnection): AttributeError: 'module' object has no attribute 'HTTPSConnection' Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 63, in apport_excepthook from apport.fileutils import likely_packaged, get_recent_crashes File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 5, in <module> from apport.report import Report File "/usr/lib/python2.7/dist-packages/apport/report.py", line 16, in <module> from xml.parsers.expat import ExpatError File "/usr/lib/python2.7/xml/parsers/expat.py", line 4, in <module> from pyexpat import * ImportError: /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so: undefined symbol: _Py_ZeroStruct Original exception was: Traceback (most recent call last): File "/home/wspeirs/src/cldblkdev/cldblkdev.py", line 5, in <module> import gcs File "/home/wspeirs/src/cldblkdev/gcs.py", line 2, in <module> import httplib2 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 930, in <module> class HTTPSConnectionWithTimeout(httplib.HTTPSConnection): AttributeError: 'module' object has no attribute 'HTTPSConnection' nbdkit: error: /home/wspeirs/src/cldblkdev/cldblkdev.py: error running this script Everything seems to work outside of nbdkit. I'm assuming this is some path related issue... any thoughts on how to fix this? Thanks... Bill-
On Thu, May 21, 2015 at 12:10:13AM -0400, Bill Speirs wrote:
I'm working on a python plugin for nbdkit that treats Google Cloud Storage as a block device. I have a library, gcs.py, that implements simple commands: list, get, put, delete; source is here: https://github.com/wspeirs/cldblkdev/blob/master/gcs.py
When I attempt to run nbdkit I get the following error:
I get a slightly different error: Traceback (most recent call last): File "/tmp/gcs.py", line 1, in <module> import json File "/usr/lib64/python2.7/json/__init__.py", line 108, in <module> from .decoder import JSONDecoder File "/usr/lib64/python2.7/json/decoder.py", line 5, in <module> import struct File "/usr/lib64/python2.7/struct.py", line 1, in <module> from _struct import * ImportError: /usr/lib64/python2.7/lib-dynload/_struct.so: undefined symbol: PyFloat_Type nbdkit: error: /tmp/gcs.py: error running this script A smallest possible reproducer is to have a file that has a single line: import json Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Thu, May 21, 2015 at 09:32:48AM +0100, Richard W.M. Jones wrote:
On Thu, May 21, 2015 at 12:10:13AM -0400, Bill Speirs wrote:
I'm working on a python plugin for nbdkit that treats Google Cloud Storage as a block device. I have a library, gcs.py, that implements simple commands: list, get, put, delete; source is here: https://github.com/wspeirs/cldblkdev/blob/master/gcs.py
When I attempt to run nbdkit I get the following error:
I get a slightly different error:
Traceback (most recent call last): File "/tmp/gcs.py", line 1, in <module> import json File "/usr/lib64/python2.7/json/__init__.py", line 108, in <module> from .decoder import JSONDecoder File "/usr/lib64/python2.7/json/decoder.py", line 5, in <module> import struct File "/usr/lib64/python2.7/struct.py", line 1, in <module> from _struct import * ImportError: /usr/lib64/python2.7/lib-dynload/_struct.so: undefined symbol: PyFloat_Type nbdkit: error: /tmp/gcs.py: error running this script
A smallest possible reproducer is to have a file that has a single line:
import json
I can fix it (for my simple json case) by doing: LD_PRELOAD=/usr/lib64/python2.7/lib-dynload/_json.so nbdkit python script=/tmp/json.py I wonder why Python doesn't dlopen the correct shared libraries? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Judging by this comment, it seems to be a generic problem with Python: https://github.com/blankwall/Python_Pin/issues/2#issuecomment-70569507 Strange. Anyway, patches welcome if you manage to solve this properly. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
What version of Python were you running for your test? $ python --version Python 2.7.9 Also, has anyone written a complex plugin for nbdkit that leverages shared libraries? I was hoping to hack this together quickly to test performance... Thanks! Bill- On Thu, May 21, 2015 at 5:34 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
Judging by this comment, it seems to be a generic problem with Python:
https://github.com/blankwall/Python_Pin/issues/2#issuecomment-70569507
Strange. Anyway, patches welcome if you manage to solve this properly.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Thu, May 21, 2015 at 10:03:42AM -0400, Bill Speirs wrote:
What version of Python were you running for your test?
$ python --version Python 2.7.9
2.7.8 here.
Also, has anyone written a complex plugin for nbdkit that leverages shared libraries? I was hoping to hack this together quickly to test performance...
Nope. I've written substantial plugins in Perl & C only. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
OK... I *think* I've solved the issue. It has to do with the way plugins are loaded. Here's the bug I filed: https://bugzilla.redhat.com/show_bug.cgi?id=1223870 Let me know... Bill- On Thu, May 21, 2015 at 10:36 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, May 21, 2015 at 10:03:42AM -0400, Bill Speirs wrote:
What version of Python were you running for your test?
$ python --version Python 2.7.9
2.7.8 here.
Also, has anyone written a complex plugin for nbdkit that leverages shared libraries? I was hoping to hack this together quickly to test performance...
Nope. I've written substantial plugins in Perl & C only.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
participants (2)
-
Bill Speirs -
Richard W.M. Jones