On Thu, Nov 22, 2018 at 10:00 AM Nikolay Ivanets <stenavin@gmail.com> wrote:

Let me share my thoughts.

Thanks! :)

...

However, as one can see in the video, it takes a pretty long time to inspect the disk (in order to find the operating system) and to mount the file systems.

Do you really need to inspect disk image BEFORE mounting file system(s)? You can do inspection on demand if you want/need to provide some metadata about installed OS etc. And definitely inspection might take some time. Pay attention that it is better to provide all images to libguestfs if original installation was spread across multiple disks. And while it is not strongly required but is good idea to provide all disk images in the same order.

Well, I don't need any information about the operating system itself but ideally, when browsing the disk image the user sees the file system(s) as if the user would have connected (e.g., via ssh) to the guest. So let's say that I browse a disk image with two partitions, the root that is mounted to "/" and another one that is mounted to "/boot". I think it makes sense to show the content of the disk in a way that the second partition is mounted to "/boot", and it seems that this mapping is only returned by inspect-os, right?

I think you can use list-filesystems, mount, ll and friends to navigate file system on demand as user drills down disk image(s). E.g.:

1. User: selects disk image, double-click. You: launch libguestfs instance, call list-filesystems and display them.

Right, adding another level of file-systems is possible. The downside though is that I then stop seeing the directory-structure from the guest point of view, right? Maybe it's not that bad if it significantely improves the responsiveness...

2. User: selects some file system, double-click. You: mount file system, call ll (or friends) and display top-level file system hierarchy. 3. User: selects directory, double-click. You: call ll (or friends) to discover and display next level. And so on. 4. You can umount current file system when the user goes back to a very first level (list of file systems). 5. And you can terminate current libguestfs instance when user goes out of the current disk image on you UI.

About terminating the libguestfs instance, note that muCommander caches the browsed files internally to improve responsiveness. For example, imagine that the user has several open tabs - it makes sense for muCommander to keep their current location cached because of the likelihood the user will switch between those tabs. So I would rather rely on the framework to decide when to release resources and not to control that within the plugin.

Again, pay attention that with LVM, Windows dynamic disks, MD arrays you might need to provide all involved disk images to a guestfs to make things to work as expected.

But on the other hand, I do want to enable the user to browse a single disk image even if it was attached to the VM along with few others. What would be the risk in inspecting a single disk image in that case? shouldn't each disk image be a stand alone entity?

Another thing to keep in mind that original disk image might be altered if you launched guestfs instance in RW mode even it seemed you/user didn't make any modifications. That is because some file systems will apply journal on mount.

Oh, that might explain some troubles I'm facing: When implemented an extension for RAR files, I used the file modification time in order to check whether the file has changed and therefore I need to re-inspect the file. But with libguestfs, I've noticed that even when not changing anything on the disk, it's modification time changes. This could explain why that happens (so currently I assume that the disk is not modified by other tool while being browsed by muCommander).

...

So it takes me back to the filesystem_walk function as I read in [4] that it allows inspecting the metadata without mounting the disk partition(s). Unfortunately, I didn't manage to try that function due to the following error I get when trying to use it (tried on Fedora 28 & 29):

Exception in thread "main" com.redhat.et.libguestfs.LibGuestFSException: internal_filesystem_walk: feature 'libtsk' is not available in this build of libguestfs. Read 'AVAILABILITY' in the guestfs(3) man page for how to check for the availability of features. at com.redhat.et.libguestfs.GuestFS._filesystem_walk(Native Method) at com.redhat.et.libguestfs.GuestFS.filesystem_walk(GuestFS.java:6010) at test.Main.main(Main.java:27)

Is that right that the filesystem_walk doesn't require mounting the disk partition(s)? If that's true, can we publish a version that supports that in the Fedora repo?

Do you think filesystem_walk will help you? It might take even longer. I think it will return you a list of ALL the files and directories. And you have to build directory tree yourself to present it to a user on your UI. But if you want to: just check libtsk feature availability before using guestfs functionality (like filesystem_walk) which depends on it.

Yeah, I didn't want to make the original post too long so I dropped the part that could explain why that's exactly what I'm doing at the moment (inspecting the whole directory structure) :) Long story short, there are three type of files in mCcommander: - Local files - Protocol files, e.g., FTP, SMB. - Archive files, e.g., ZIP, RAR, TAR. Protocol files require the user to authenticate a remote place and then we browse the destination in a lazy way, i.e., as you wrote before - inspecting a directory when the user asks to browse it. In contrast, archieve files were assumed local and easier to inspect and so their whole structure is retrieved when you "enter the file". muCommander then caches the whole tree structure and refreshes it only when needed. Virtual disks don't really fit either of them. On the one hand, they cannot be considered protocol files as they reside within the disk. On the other hand, they should be inspected in a lazy way because they are typically quite big. As I wrote in [1], some of the assumptions we took 10-15 years ago might be irrelevant anymore and need to change. One of them is probably to start inspecting archieve files in a lazy way as well but that requires massive changes in the framework and so I would not want to do it right now.

You will always depends on guestfs available on a host system if you don't build and provide your own build alongside with muCommander. And you never know how guestfs was built and what features are available. So you should always check feature availability before using it. Moreover a lot of functionality of libguestfs very depends on a host kernel and available modules. My experience with libguestfs forces me to build it myself. I also build my custom guestfs appliance to not depend on a host kernel etc.

You can consider using mount_local but that is another story...

That's a very good point. I touched the deployment aspect also in [1] - by making it a plugin I can provide it only on linux and declare libguestfs as a dependency *of the plugin*. As the functionality I need from libguestfs is pretty basic, I really hope I could use the default build of libguestfs :) [1] http://ahadas.github.io/about-mucommander/

-- Mykola Ivanets

On Sat, Nov 24, 2018 at 9:36 PM Nikolay Ivanets <stenavin@gmail.com> wrote:

...

Well, I don't need any information about the operating system itself but ideally, when browsing the disk image the user sees the file system(s) as if the user would have connected (e.g., via ssh) to the guest. So let's say that I browse a disk image with two partitions, the root that is mounted to "/" and another one that is mounted to "/boot". I think it makes sense to show the content of the disk in a way that the second partition is mounted to "/boot", and it seems that this mapping is only returned by inspect-os, right?

Yes, that is right. You have to call inspect-os if you want to present disk (or I would rather say VM) content in the way the user expect it to be.

...

...

1. User: selects disk image, double-click. You: launch libguestfs instance, call list-filesystems and display them.

Right, adding another level of file-systems is possible. The downside though is that I then stop seeing the directory-structure from the guest point of view, right?

Right. But I want to stress you again: you MUST to add ALL VM disks (preferably in the same order) if you want to present file system from the guest point of view. Otherwise you will be unable to present LVM, MD and Windows LDM partitions. You also should be prepared that guestfs might not be able to mount ALL found file systems and guestfs might not be found all mappings. Even more: you will not get mappings

if guetsfs unable to detect OS or there are no OS at all on the

disk(s) you are browsing.

Right, ack. And side question: are you ready to the

situation where inspect-os find more then one OS?

...

Maybe it's not that bad if it significantely improves the responsiveness...

Anyway you need to prepared to display disk(s)/VM from guest(s) point of view as well as simply browsing found file systems.

Yeah, so how about adding another level as you suggested before but of "file systems" and "operating systems"? That way, if the user enters "file systems" we can display them quickly and when the user enters "operating systems" he will be presented with all the detected VMs. Would that make sense?

...

But on the other hand, I do want to enable the user to browse a single disk image even if it was attached to the VM along with few others. What would be the risk in inspecting a single disk image in that case? shouldn't each disk image be a stand alone entity?

Imagine: I have two disks and I have create physical volumes (PV), and volume group (VG) adding both PVs, and I've created logical volume (LV), and finally any file system atop (e.g. XFS). Any attempt to browse disks separately will not be successful. You must to add them both to a guestfs before launching. The similar things happens with soft array (MD) spread across multiple disks. I hope you've got an idea.

Alright, I see. The thing is that when exploring a specific image, I don't have the context of the VM that includes its additional disks, right? so I think it's ok not to support such cases when browsing a specific image. To support such cases, maybe it worth thinking of another plugin for "libvirt" - where we query all the VMs and when entering a particular VM, we attach all its disks and do our best to reflect the tree structure from the guest's perspective.

And definitely I would not made an attempt to modify disks of live VM.

So for a "libvirt" plugin - definitely. But when exploring a specific disk, do I have a way to prevent this? I saw a similar warning in the documentation of libguestfs, but I wonder - if qemu opened the disk with write permissions, wouldn't libguestfs fail to open it with write permissions at the same time?

-- +380979184774 Mykola Ivanets