May 2013 - BaseX-Talk - mailman.uni-konstanz.de

JSON or JSONML output formatting
by Zachary Pride 16 Jul '14

16 Jul '14

Hello, I've been playing with BaseX lately and all-in-all am very impressed. Something I'm wrestling with however is the JSON output options. I've got various web applications that would really be simplified by receiving JSON straight away for the obvious scripting benefits. The JSON method has been giving me a lot of grief, stipulating rules I'm not sure how to follow. right now I have an element that "is typed as string and cannot be nested." I've been able to get JSONML to output but the JSON is this morass of nested Arrays... Attributes seem to get transformed into Objects but elements/tags and their values are pushed into Arrays for some rationale - which I suppose is on purpose. My question is how can I get Objects for simple key:value properties from <tag>value</tag> 's. Are there options I'm not utilizing? Would it be feasible to transform the XML output to some middle form first - setting it up to become Objects, before serializing it to JSON? Thanks! Zach Pride Associate Software Engineer Ultra Electronics, ProLogic 1000 Green River Drive, Ste. 201 Fairmont, WV 26554 zachary.pride(a)ultra-prologic.com Tel: +1 304 333 2680 ext.308 www.ultra-prologic.com This e-mail from Ultra Electronics, ProLogic and any attachments to it are confidential to the intended recipient and may also be privileged or controlled by U.S. export control laws and regulations. No technical data, information or other items provided by Ultra Electronics, ProLogic in connection with this email shall be shared or provided to any foreign persons, entities, or subsidiaries without the expressed written authorization of Ultra Electronic, ProLogic. If you have received it in error please notify the sender and delete it from your system. If you are not the intended recipient you must not copy it or use it for any purpose nor disclose or distribute its contents to any other person. All communications may be subject to interception or monitoring for operational and/or security purposes. Please rely on your own virus checking as the sender cannot accept any liability for any damage arising from any bug or virus infection. This e-mail from Ultra Electronics, ProLogic and any attachments to it are confidential to the intended recipient and may also be privileged or controlled by U.S. export control laws and regulations. No technical data, information or other items provided by Ultra Electronics, ProLogic in connection with this email shall be shared or provided to any foreign persons, entities, or subsidiaries without the expressed written authorization of Ultra Electronic, ProLogic. If you have received it in error please notify the sender and delete it from your system. If you are not the intended recipient you must not copy it or use it for any purpose nor disclose or distribute its contents to any other person. All communications may be subject to interception or monitoring for operational and/or security purposes. Please rely on your own virus checking as the sender cannot accept any liability for any damage arising from any bug or virus infection. Please consider the environment before printing this email.

4 4

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

indent parameter
by Hans-Juergen Rennau 28 Jun '13

28 Jun '13

Hello, it seems to me there is a couple of issues concerning the handling of indentation (serialization parameter "indent"). a) Using "indent=no" whitespace-only text nodes between element nodes are removed b) Using "indent=yes" the content of elements with xml:space="preserve" is indented, too I suppose both behaviors are in contraction to the spec ( http://www.w3.org/TR/xslt-xquery-serialization/#xml-indent ). As a particular consequence, (a) seems to preclude making an exact copy of a document (fragment) - all whitespace-only text nodes vanish. And as a consequence of (b) one has no possibility to locally protect the whitespace used in the source document, as required for example when reusing "programlisting" elements of docbook. Or did I overlook something? Kind regards, Hans-Juergen

2 3

Best practices for concurrency
by The Trainspotter 19 Jun '13

19 Jun '13

Hi, I've read through the online example on server concurrency, however, currently I'm not running a BaseXServer instance I'm just running things locally. I want multiple threads in my application to be able to query the database at once without treading on each others toes. What's the best approach for doing this? I have tried querying the database using different threads but I keep getting internal BaseX errors which leads me to think I'm going about this the wrong way. Cheers Gary

5 15

BaseX Script memory leak . . ?
by Christopher.R.Ball 12 Jun '13

12 Jun '13

I have a BaseX script (.bxs) I am running that does queries in batches (sets of 5k documents), but as it progresses it bogs down in speed, does not release memory between sets even if I force it to close and reopen the db between queries, and eventually runs out of memory. But, if I break the same BaseX script into separate files still doing the same exact batches it is extremely fast and memory efficient. Very suggestive of a memory leak . . . I am running on BaseX 7.6.1 Beta. Any thoughts? Is there a way to force the script to do garbage collection?

5 7

server time out
by Cerstin Elisabeth Mahlow 05 Jun '13

05 Jun '13

Hi, I run several web applications using various CGI scripts to read and update BaseX databases via various XQUERIES. I use the "basexserver -S" command to start the server on a CentOS Server. Maybe this is not the very best idea. I already split the data into various databases to avoid concurrent reading and writing. And I have a simple cron job running, that tests whether the BaseX server is still running and restarts it if it isn't running any more. However, my users get (or rather create) a time out from time to time. If the basex server then shut's down, the cron job would restart it and everything is fine again. In the last weeks, however, I discovered that after such a time out the Basex server seems to still be running, but doesn't answer any xquery any more. So I have to kill the process by hand and restart it again. Could you recommend a better solution to ensure smooth running of the Basex server even it sometimes a time out occurs? Thanks in advance and best regards Cerstin -- Dr. phil. Cerstin Mahlow Universität Basel Departement Sprach- und Literaturwissenschaften Fachbereich Deutsche Sprach- und Literaturwissenschaft Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow(a)unibas.ch Web: http://www.oldphras.net

2 1

Using an XSD with BaseX (to provide extra type information)
by Ludovic Kuty 05 Jun '13

05 Jun '13

Hello, I would like to know if BaseX will use the XSD referenced in an XML document to get extra informations about types (he PSVI) and if it will validate the document in the process. If it does so, where do I have to store the XSD ? I would like to store it inside the database itself (or another one). I could also store it in the Web application provided by BaseX and make my URL point to it. In my opinion, the use of the validation module should not be necessary as it should happen automatically. Thank in advance for your insights and informations, Ludovic Kuty

3 4

how to find a possibly corrupted node?
by Cerstin Elisabeth Mahlow 31 May '13

31 May '13

Hi, I have several databases basically consisting of several entries like this one: <entry> <lastChange>2013-04-30T08:20:00</lastChange> <user>marcel</user> <phraseme>Ad0190</phraseme> <annotated>no</annotated> <negation>?</negation> <genus>?</genus> <mediality>?</mediality> <meaning>?</meaning> <sentence>?</sentence> <question>?</question> <node>85750</node> <query>[text() contains text ('über' ftand 'den' ftand 'Hals' ftand 'kommen') distance at most 10 words]</query> <p>Gar gewiß hätte sie selbst der Teufel lebendig hingeführt, oder der Donner in die Asche gelegt, oder die Erde lebendig verschluckt, oder den wilden Thieren zum Raub worden, weil sie sich aber der Todten haben angenommen, so konnte sie kein zeitliches Unglück berühren. Nolite timere so fürchtet euch dann nicht, alle Liebhaber der armen Seelen im Fegfeuer, es kann euch so bald kein Unglück über den Hals kommen, die Todten helfen den Lebendigen.</p> <meta>?</meta> <semantics>?</semantics> <comment/> <merged>no</merged> <cite>?</cite> <sem>?</sem> <note/> </entry> For the web-application I run with ExtJS, I extract some data like the values for <user> etc. I also need the text stored in the <p> node, and I use the xquery stored in the <query> node as argument for ft:mark. So this part basically looks like this one: for $i at $p in //entry let $q := $i/query let $annotated := string($i/annotated) let $t := if ($i/p) then $i/p else $i/l let $ft :=concat("ft:mark($binding", $q, ")") let $bm := map{'$binding' := $t} return <div>{if ($annotated = "no") then xquery:eval($ft, $bm) else $t}</div> I get the query and the text and then apply the query to the text depending on the value for <annotation>. This works fine for most of the databases. However, for six or seven, I get a time-out, regardless of the size of the database. It works OK with DB consisting of several hundreds or only a dozen of such entries, but it does not work for the one I attach here as XML source (I created the DB via another xquery with whitespace chopping off and UPDINDEX ON, all indexes have been created. It is valid XML as far as I can see.) I tested it with the GUI, I use BaseX 7.7beta from April 24, with VM=-Xmx512m. If I change the return expression to: return <div>{if ($annotated = "no") then ($q, $t) else $t}</div> I get a result for all entries in about 35ms. And this query plan: Compiling: - simplifying descendant-or-self step(s) - converting descendant::*:entry to child steps - removing redundant $p as xs:integer cast. - atomic evaluation of ($annotated = "no") - inlining let $annotated := fn:string($i/annotated) - atomic evaluation of (fn:string($i/annotated) = "no") - removing variable $annotated - removing variable $ft - removing variable $bm - inlining let $q := $i/query - removing variable $q - inlining let $t := if($i/p) then $i/p else $i/l - removing variable $t Optimized Query: for $i at $p in document-node { "annotate-Ad0190.xml" }/*:root/*:entry return element div { (if((fn:string($i/annotated) = "no")) then ($i/query, if($i/p) then $i/p else $i/l) else if($i/p) then $i/p else $i/l) } Result: - Hit(s): 72 Items - Updated: 0 Items - Printed: 759 KB - Locking: local [annotate-Ad0190] Timing: - Parsing: 0.0 ms - Compiling: 1.14 ms - Evaluating: 34.44 ms - Printing: 7.8 ms - Total Time: 43.4 ms Query plan: <QueryPlan> <GFLWOR> <For> <Var name="$i" id="1"/> <at> <Var name="$p" id="0"/> </at> <IterPath> <DBNode name="annotate-Ad0190" pre="0"/> <IterStep axis="child" test="*:root"/> <IterStep axis="child" test="*:entry"/> </IterPath> </For> <CElem> <QNm value="div" type="xs:QName"/> <If> <CmpG op="="> <FNAcc name="string([item])"> <CachedPath> <VarRef> <Var name="$i" id="1"/> </VarRef> <IterStep axis="child" test="annotated"/> </CachedPath> </FNAcc> <Str value="no" type="xs:string"/> </CmpG> <List> <CachedPath> <VarRef> <Var name="$i" id="1"/> </VarRef> <IterStep axis="child" test="query"/> </CachedPath> <If> <CachedPath> <VarRef> <Var name="$i" id="1"/> </VarRef> <IterStep axis="child" test="p"/> </CachedPath> <CachedPath> <VarRef> <Var name="$i" id="1"/> </VarRef> <IterStep axis="child" test="p"/> </CachedPath> <CachedPath> <VarRef> <Var name="$i" id="1"/> </VarRef> <IterStep axis="child" test="l"/> </CachedPath> </If> </List> <If> <CachedPath> <VarRef> <Var name="$i" id="1"/> </VarRef> <IterStep axis="child" test="p"/> </CachedPath> <CachedPath> <VarRef> <Var name="$i" id="1"/> </VarRef> <IterStep axis="child" test="p"/> </CachedPath> <CachedPath> <VarRef> <Var name="$i" id="1"/> </VarRef> <IterStep axis="child" test="l"/> </CachedPath> </If> </If> </CElem> </GFLWOR> </QueryPlan> For the original query, i.e., applying ft:mark, I only get a "Out of main memory error" after roughly 60000ms. I don't see any suspicious going here. Do you have any idea what might be the reason for the time out? All other information I extract are returned in less than 100ms, the ft:mark seems to be the issue. Is there any way I can find out which of the entries is causing the time out? Thanks in advance and best regards Cerstin -- Dr. phil. Cerstin Mahlow Universität Basel Departement Sprach- und Literaturwissenschaften Fachbereich Deutsche Sprach- und Literaturwissenschaft Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow(a)unibas.ch Web: http://www.oldphras.net

1 0

Perl and BaseX question (add document to existing db)
by m4058＠aol.com 30 May '13

30 May '13

I'm trying to add a document to an existing dababase xmark. Any idea on how to do this with Perl? $session->execute("create db database"); (would like the existing database xmark in here) print $session->info()."\n"; # add document $session->add("world/World.xml"); Thanks Mark Shortt

2 1

namespaces and the doc() function: conflicting prefix bindings persist past their scope?
by C. M. Sperberg-McQueen 29 May '13

29 May '13

I'm having a puzzling experience with BaseX 7.6 running under Mac OS X 10.6.8. Consider the following document, in my file system as test.xml: <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="back"/> <xsd:element name="body"> <xsd:annotation xmlns:xsd="http://www.w3.org/2000/10/XMLSchema"/> </xsd:element> <xsd:element name="bold"/> </xsd:schema> The query doc('.../test.xml')//xs:element returns not three elements, but just the first two. The third element is encountered after the prefix 'xsd' was temporarily rebound to a different namespace, and it is not recognized as being in the same namespace as the other 'element' elements. (Further investigation shows that the system believes it to be in namespace http://www.w3.org/2000/10/XMLSchema.) Some observations: - The behavior has nothing to do with the specific namespace or the fact that the XSD namespace is predefined; analogous behavior occurs with the following document: <a:a xmlns:a="http://example.com/a"> <a:b/> <a:b> <x xmlns:a="http://example.com/x"/> </a:b> <a:b/> </a:a> - If a database is built using the document just given, the query declare namespace foo = "http://example.com/a"; //foo:b returns two elements, not three. In my tests, it appeared not to matter whether I checked or unchecked the 'Use internal XML parser' box; I got two elements either way. - The behavior changes if the document is given literally in the query; the following query returns three {http://example.com/a}b elements, not two: declare namespace foo = "http://example.com/a"; let $doc := <a:a xmlns:a="http://example.com/a"> <a:b/> <a:b> <x xmlns:a="http://example.com/x"/> </a:b> <a:b/> </a:a> return $doc//foo:b Is this a bug, or am I doing something wrong? It looks as if something were going wrong in the XML parser's management of namespace bindings. Thanks for any light you can shed on the matter. In the short run I suppose I can try to revise my XSLT to suppress the unnecessary namespace rebindings. -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

2 3