March 2014 - BaseX-Talk - mailman.uni-konstanz.de

stream binary responses from http:module
by Johan Mörén 10 Nov '14

10 Nov '14

Hi I have built a small client in XQuery that fetches zip files from another remote service using the http-module and stores them on disk with the help of the file-module. Some of the returned zip files are really large and BaseX seems to need to materialize the files in memory before writing them to disk resulting in out of memory errors for some files. Is there some way read the response in a more streamable fashion when using the http-module? I tried using the fetch:binary function successfully but i really need the more extended functionality of the http-module. #results in out of memory if the zip file is too large file:write-binary("file.zip", http:sendrequest(...)[2]) #works all the time. file:write-binary("file.zip", fetch:binary(...)) Regards, Johan

3 7

JSON or JSONML output formatting
by Zachary Pride 16 Jul '14

16 Jul '14

Hello, I've been playing with BaseX lately and all-in-all am very impressed. Something I'm wrestling with however is the JSON output options. I've got various web applications that would really be simplified by receiving JSON straight away for the obvious scripting benefits. The JSON method has been giving me a lot of grief, stipulating rules I'm not sure how to follow. right now I have an element that "is typed as string and cannot be nested." I've been able to get JSONML to output but the JSON is this morass of nested Arrays... Attributes seem to get transformed into Objects but elements/tags and their values are pushed into Arrays for some rationale - which I suppose is on purpose. My question is how can I get Objects for simple key:value properties from <tag>value</tag> 's. Are there options I'm not utilizing? Would it be feasible to transform the XML output to some middle form first - setting it up to become Objects, before serializing it to JSON? Thanks! Zach Pride Associate Software Engineer Ultra Electronics, ProLogic 1000 Green River Drive, Ste. 201 Fairmont, WV 26554 zachary.pride(a)ultra-prologic.com Tel: +1 304 333 2680 ext.308 www.ultra-prologic.com This e-mail from Ultra Electronics, ProLogic and any attachments to it are confidential to the intended recipient and may also be privileged or controlled by U.S. export control laws and regulations. No technical data, information or other items provided by Ultra Electronics, ProLogic in connection with this email shall be shared or provided to any foreign persons, entities, or subsidiaries without the expressed written authorization of Ultra Electronic, ProLogic. If you have received it in error please notify the sender and delete it from your system. If you are not the intended recipient you must not copy it or use it for any purpose nor disclose or distribute its contents to any other person. All communications may be subject to interception or monitoring for operational and/or security purposes. Please rely on your own virus checking as the sender cannot accept any liability for any damage arising from any bug or virus infection. This e-mail from Ultra Electronics, ProLogic and any attachments to it are confidential to the intended recipient and may also be privileged or controlled by U.S. export control laws and regulations. No technical data, information or other items provided by Ultra Electronics, ProLogic in connection with this email shall be shared or provided to any foreign persons, entities, or subsidiaries without the expressed written authorization of Ultra Electronic, ProLogic. If you have received it in error please notify the sender and delete it from your system. If you are not the intended recipient you must not copy it or use it for any purpose nor disclose or distribute its contents to any other person. All communications may be subject to interception or monitoring for operational and/or security purposes. Please rely on your own virus checking as the sender cannot accept any liability for any damage arising from any bug or virus infection. Please consider the environment before printing this email.

4 4

Error handling
by Marco Lettere 26 May '14

26 May '14

Hello all, the error interception functions are pretty useful. My "feature request" here is whether it could be possible to overcome: "XQuery runtime errors can be processed via /error annotations/. A single argument must be supplied, which represents the QName of the error to be caught. A wildcard |*| may be specified to catch all possible errors. A function can only have a single error annotation:" It could be useful to have at least a second level of wildcards on the QName prefix in order to be able to differentiate behaviours according to categories of errors. |%rest:error||(||"ns1:*"||)| A more fine grained filtering (i.e. comma separated list of exception codes) could also be useful but since I don't understand the difficulty of building this into the code I won't ask too much. The only alternative I see at the moment is a single function with a switch or a lot of replicated functions. What is your opinion on this? Thanks and regards, Marco.

2 2

Dynamic Module Loading
by Jean-philippe Magué 19 May '14

19 May '14

Hello, Exist-db has an util:import-module<http://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xq…>function that allows to dynamically import an XQuery module at run time. I haven't found anything equivalent in BaseX. Is that right ? I have also tried to see how to implement such a function directly in Java, but as far as I understand the module parsing process, functions prefix are checked at compile time. That would compromise any attempt to write such a function. Am I right ? Best, Jean-Philippe -- Jean-Philippe Magué Maître de Conférences en Humanités Numériques Responsable de l'Atelier des Humanités Numériques : ahn.ens-lyon.fr Ecole Normale Supérieure de Lyon +33 (0)4 26 23 38 08 -- "Using computers will therefore lead us to a more profound and systematic knowledge of human expression; in principle, it can help us to be more humanistic than before." Busa, 1980.

6 16

Creating a Basex database directly from tarred XML?
by Hondros, Constantine (ELS-AMS) 08 Apr '14

08 Apr '14

Hello all, Is there a known workaround for creating a db from tarred XML using the Database -> New menu? I can always untar the tarfile first, but it would be sort-of nice just to point at the tarfile and load. TIA, Constantine ________________________________ Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.

3 7

Multiple db:replace() = slower updates with 7.8
by France Baril 02 Apr '14

02 Apr '14

Hi, We run batch processes on our files in order to send them out to translation in xliff format (xml format for translations). I run a piece of code that: 1. applies transformations 2. file:writes the resulting files (for debug purposes) 3. and then db:replaces the resulting files. If I send 5 files into the process, it takes a few seconds before the files appear on the file system. Yet full process with the db:replace takes 2 minutes to complete. When I send 60 files... the process never ends. All files are available on the file system and are valid, which means I am fairly certain that the freeze doesn't occur because of an error in one of the files. There is no error message in the terminal window. Is possible that something was changed after 7.7 that would affect performance? On our production server, which is on 7.7 we can run 100+ files in just a few minutes. Maybe there are new server settings that we missed? For reference, here is the code. I understand that updates are all performed at the end of the process, which is why we get the debug files first, but the lag time between the time where we get all files on the file system and the db:replace seems huge. *declare* *updating* *function* *translation:transform-to-xliff*(*$menu-id* *as **xs:string*, *$lang* *as **xs:string**, *$package-number* *as **xs:string* ){ *let **$menu* := *menu:xml-get-menu*(*$menu-id*, 'en-us') (: Need to translate menu and its topics without getting duplicated topics twice:) (: Always send the reusable strings from writers and from dev... doesn't cost anything if no new string anyway :) *let* *$file-ids* := (*$menu-id*, *for **$id* *in **distinct-values*( *$menu*/descendant::*[*name*()='topicref']/*data*(*@id-ref*))[1] *return * *$id*, 'stringreferences', 'stringreferencestransform') *return *(*for **$file-id* *in **distinct-values*(*$file-ids*) *let* *$file* := *db:open*('en-us')/*[*@id*=*$file-id*] *let* *$skl-href* := 'skeletons/' || *$package-number* || '/'|| *substring-after*(*$file*/*base-uri*(), 'en-us/') *let* *$file-with-preserve* := *copy* *$copy* := *$file* *modify*(*if* (*exists*(*$copy*/*@xml:space*)) *then *() *else **insert* *node* *attribute* {'xml:space' }{'preserve'} *into* *$copy*) *return **$copy* *let* *$all-trans-files* := *xliff-transformer:create-translation-files*(*$file-with-preserve*, *$package-number*, *$menu-id*, *$lang*, *$skl-href*) *let* *$skeleton* := *$all-trans-files*[1] *let* *$xliffs* := *for **$xliff* *in **$all-trans-files*[ *position*() != 1] *return **$xliff* *let* *$debug* := *file:write*('Debug-xliff/skls/' || *tokenize*(*$skl-href*, '/')[*last*()], *$skeleton*) *return *(*db:replace*('Translation-management', *$skl-href*, *$skeleton*), *for **$xliff* *in **$xliffs* *let* *$target-lang* := *$xliff*/descendant::*[*name* ()='file']/*data*(*@target-language*) *let* *$lang-pack-name* := *concat*(*$target-lang*,'_', *$package-number*) *let* *$file-loc-end* := *$menu-id* || '/' || *$lang-pack-name* ||'/' || *substring-after*(*$file*/*base-uri*(), 'en-us/') (:let $debug := file:append('debug-ids.xml', '
' || $file-loc-end):) *let* *$file-loc* := *if* (*exists*(*$xliff*/ descendant::*[*@translate*='yes'])) *then *'xliff-to-translate/' || *$file-loc-end* *else *'xliff-no-new-segment/' || *$file-loc-end* *let* *$debug* := *file:write*('Debug-xliff/' || *tokenize*(*$file-loc*, '/')[*last*()], *$xliff*) *return **db:replace*('Translation-management', *$file-loc*, *$xliff*) ) ) }; -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com (514) 572-0341

2 6

bug or requirement
by Rob Stapper 30 Mar '14

30 Mar '14

Hi there, In order to maintain consistency in my solution-patterns I seem to end up with the need for cyclicly chained/imported modules: module "A" imports module "B" which imports module "A". Too bad that the "import module" command of Basex can't handle this type of structure. Now I can't access the functionality of "A" within module "B". What are the chances in solving this issue? Have a nice weekend, Rob Stapper --- Dit e-mailbericht bevat geen virussen en malware omdat avast! Antivirus-bescherming actief is. http://www.avast.com

2 1

Runnable jar not running properly..
by John Best 29 Mar '14

29 Mar '14

Dear Christian I have downloaded BaseX source code from BaseX Github. Then imported the source code into Eclipse IDE and configured Maven. When I run the project, it runs properly. Then I export the project to create a runnable jar th rough Eclipse. Jar file is created smoothly, but when I run the jar, GUI is not starting. NO ERROR messages.. !!! Can you put some light on this issue ? -- Have a nice day JBest

2 1

Is there any mechanism of version control of objects, kept in BaseX database?
by Moshe Zuisman 29 Mar '14

29 Mar '14

Hi. I am right now using BaseX to analyse My XML files that keep installations pseudo-code(XMLs of IzPack open source installer). Procedure is quite entangled. I keep my XML source in files.. I periodically recreate BaseX database from folder that contain all my XML files and use it to perform queries, that analyse my data. When I find something I need - I have to go back to file, and perform search there to find once more therexml element . already retrieved at BaseX query. I weight to move from files to BaseX database and keep all info there. But... Here is a catch: Files I can keep in version control system (IBM Rational Cynnergy system) Is there some way to keep versions track of data - kept in BaseX database? Some interface with standard Version control systems, or any other way?

2 1

Accessing DOCTYPE information after DB creation?
by Hondros, Constantine (ELS-AMS) 28 Mar '14

28 Mar '14

Hi all, I would really like to be able to query a large corpus of documents to get names and counts of the DTDs which are declared in the (somewhat old-fashioned now) DOCTYPE declaration: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE converted-article PUBLIC "-//ES//DTD journal article DTD version 4.5.2//EN//XML" "art452.dtd" [ ]> <converted-article>  Is there any way to get BaseX to preserve this information? Can I rewrite the doctype declaration into some sort of element node as the DB is being created so that this info can be queried? Thanks for any tips, Constantine. ________________________________ Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.

4 4