Hi Christian -

Alas, the data is a client's and confidential.

for $remote in $paths
  let $name as xs:string := file:name($remote)
  let $target as xs:string := file:resolve-path($name,$targetBase)
  let $fetched as item() :=
      http:send-request(<http:request method='get' username='{$id}' password='{$pass}' />,
       $remote)[2]
  return
      if ($fetched instance of document-node())
    then file:write($target,$fetched)
 else if ($fetched instance of xs:base64Binary)
    then file:write-binary($target,$fetched)
    else file:write-text($target,$fetched)

works -- the query completes and files are written to disk.   I suspect that server ignores override-content-type.

If I don't check the returned type and try to write everything returned out with file:write-binary() to have something where I could pick the html files back off the disk, I got an error that the content wasn't binary, it was xs:untypedAtomic.  (Which might imply that file:write-binary complains that way when fed a document node.)

Which leads to "is there a way to get the type of an item?"  I don't think there is, but it seems like it would be extremely helpful for stuff like this where "figure out what the web server feels like doing" is a concern.

Thank you! It was a helpful hint.

-- Graydon

On Fri, Apr 8, 2022 at 9:44 AM Christian Grün <christian.gruen@gmail.com> wrote:
Hi Graydon,

Maybe it’s TagSoup that has problems to convert some specific HTML
files to XML. Did you try to write the responses to disk and parse
them in a second step?

If your input data is not confidential, could you possibly provide us
with an example that runs out of the box?

Best,
Christian


> I'm using the basexgui to run (minus some identifying actual values defined previously in the query)
>
> (: for each path, retrieve the document :)
> for $remote in $paths
>   let $name as xs:string := file:name($remote)
>   let $target as xs:string := file:resolve-path($name,$targetBase)
>   let $fetched :=
>     http:send-request(<http:request method='get' override-media-type='application/octet-stream' username='{$id}' password='{$pass}' />,
>      $remote)[2]
>   let $use as item() := try {
>     html:parse($fetched)
>   } catch * {
>     $fetched
>   }
>   return if ($use instance of document-node())
>      then file:write($target,$use)
>      else file:write-binary($target,$use)
>
> It works, in that I get exactly 100 documents retrieved.  (There are unfortunately 140+ documents in the list.)
>
> However, the query fails with an "out of main memory" error when using a recent 10.0 beta or 9.7 with Xmx set to 2g.  Setting Xmx to 16g with 9.7 produces the same "out of memory" error in the same length of time (about 5 minutes).
>
> java -version says
> 20:27 test % java -version
> openjdk version "11.0.14.1" 2022-02-08
> OpenJDK Runtime Environment 18.9 (build 11.0.14.1+1)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.14.1+1, mixed mode, sharing)
>
> It's entirely possible I'm going about fetching files off a web server the wrong way; it's possible there's something there that's rather large, but I doubt it's that large.
>
> What should I be doing instead?
>
> Thanks!
> Graydon