[Libguestfs] [PATCH nbdkit] curl: Add effective-url flag

Probably best to read this first: https://bugzilla.redhat.com/show_bug.cgi?id=2013000 This adds an effective-url=true|false flag to nbdkit-curl-plugin. If true, the first time we fetch the URL, we fetch the "effective" URL (ie. the URL after all redirects were resolved) and use that for all future connections within the current nbdkit instance. The idea behind this patch is to do with the Fedora mirror system which is flakey. Some mirrors return errors or 404s. And the mirror system will give you a new redirect each time (within your geographical region). This results in transfers sometimes failing because a single read went to a bad mirror. After implementing this patch I'm not very happy with it. It is incomplete because we pass the original URL to cookie-script and header-script, so plugin/curl/scripts.c will also need to be modified. Once those modifications are done the change becomes quite invasive. Also I'm not convinced that it really solves any problem. In the manual change I wrote: Note use of this feature in long-lived nbdkit instances can cause subtle problems: • The effective URL persists across connections for the lifetime of the nbdkit instance. If nbdkit is used for a long time then it is possible for the redirected URL to become stale. • It will defeat some mirror load-balancing techniques. • If the mirror service sometimes redirects to a broken URL and it happens that the URL you fetch first is broken then nbdkit will no longer recover on subsequent connections (instead you will need to restart nbdkit). I suggested another way to solve this by using curl APIs to fetch the effective URL up front and passing that URL to nbdkit (see https://bugzilla.redhat.com/show_bug.cgi?id=2013000#c1), but apparently that solution isn't acceptable for unclear reasons. I can't think of any other way to solve this in the context of nbdkit (maybe have it detect when redirection is happening and retry the redirection on error?). So here's the patch. Rich.