I’d like to be able to create Zip files from large docs stored in a database (i.e., Oxygen validation reports, which can be 10s of MBs as raw XML).
Looking at the Archive module it’s not immediately clear how to go about it (or how best to go about it)—it would make the most sense to construct the Zip as a stream and return the stream, as opposed to constructing the Zip in memory (or on disk) and then streaming it from there.
Or am I overthinking it or missing some obvious solution?
Thanks,
E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
Looking at the Archive module it’s not immediately clear how to go about it (or how best to go about it)—
What have you tried so far? Maybe the examples in the documentation are not self-explanatory enough?
it would make the most sense to construct the Zip as a stream and return the stream, as opposed to constructing the Zip in memory (or on disk) and then streaming it from there.
The streaming capabilities of the Archive Module have been improved over time, but if you need to serialize XML documents on the fly, and if your total output exceeds the amount of free memory, it might be better to temporarily write your files to disk. Another reason to write files to disk can be that your clients have a slow connection, and it would block too many server-side resources if a transaction is finalized only after the full download.
Your point about streaming from disk is well taken—I can see that that’s the only practical solution.
I did a really quick experiment with no time for deeper introspection or thought, so kind of embarrassed to show it, but we learn by doing…
Here’s my attempt, which did not succeed in that the downloaded result is a zero-length file:
declare function webutils:makeZipForDoc( $node as node(), $filename as xs:string ) as xs:base64Binary { let $debug := prof:dump('webutils:makeZipForDoc(): filename=''' || $filename || '''' ) return archive:create( archive:entry{$filename}</archive:entry>, serialize($node, map{'method' : 'xml'} ) ) };
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: Christian Grün christian.gruen@gmail.com Date: Monday, February 7, 2022 at 3:03 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Download As Binary for Docs in Databases? [External Email]
Looking at the Archive module it’s not immediately clear how to go about it (or how best to go about it)—
What have you tried so far? Maybe the examples in the documentation are not self-explanatory enough?
it would make the most sense to construct the Zip as a stream and return the stream, as opposed to constructing the Zip in memory (or on disk) and then streaming it from there.
The streaming capabilities of the Archive Module have been improved over time, but if you need to serialize XML documents on the fly, and if your total output exceeds the amount of free memory, it might be better to temporarily write your files to disk. Another reason to write files to disk can be that your clients have a slow connection, and it would block too many server-side resources if a transaction is finalized only after the full download.
I did a really quick experiment with no time for deeper introspection or thought, so kind of embarrassed to show it, but we learn by doing…
Just fine! I’do it similar. If the following endpoint…
declare %rest:path('test') function local:test() { archive:create('file.xml', serialize(<a/>)) };
…is called…
curl -o tmp.zip "http://localhost:8984/test"
…it should give you a ZIP file with a single XML file.
basex-talk@mailman.uni-konstanz.de