March 2013 - BaseX-Talk - mailman.uni-konstanz.de

JSON or JSONML output formatting
by Zachary Pride 16 Jul '14

16 Jul '14

Hello, I've been playing with BaseX lately and all-in-all am very impressed. Something I'm wrestling with however is the JSON output options. I've got various web applications that would really be simplified by receiving JSON straight away for the obvious scripting benefits. The JSON method has been giving me a lot of grief, stipulating rules I'm not sure how to follow. right now I have an element that "is typed as string and cannot be nested." I've been able to get JSONML to output but the JSON is this morass of nested Arrays... Attributes seem to get transformed into Objects but elements/tags and their values are pushed into Arrays for some rationale - which I suppose is on purpose. My question is how can I get Objects for simple key:value properties from <tag>value</tag> 's. Are there options I'm not utilizing? Would it be feasible to transform the XML output to some middle form first - setting it up to become Objects, before serializing it to JSON? Thanks! Zach Pride Associate Software Engineer Ultra Electronics, ProLogic 1000 Green River Drive, Ste. 201 Fairmont, WV 26554 zachary.pride(a)ultra-prologic.com Tel: +1 304 333 2680 ext.308 www.ultra-prologic.com This e-mail from Ultra Electronics, ProLogic and any attachments to it are confidential to the intended recipient and may also be privileged or controlled by U.S. export control laws and regulations. No technical data, information or other items provided by Ultra Electronics, ProLogic in connection with this email shall be shared or provided to any foreign persons, entities, or subsidiaries without the expressed written authorization of Ultra Electronic, ProLogic. If you have received it in error please notify the sender and delete it from your system. If you are not the intended recipient you must not copy it or use it for any purpose nor disclose or distribute its contents to any other person. All communications may be subject to interception or monitoring for operational and/or security purposes. Please rely on your own virus checking as the sender cannot accept any liability for any damage arising from any bug or virus infection. This e-mail from Ultra Electronics, ProLogic and any attachments to it are confidential to the intended recipient and may also be privileged or controlled by U.S. export control laws and regulations. No technical data, information or other items provided by Ultra Electronics, ProLogic in connection with this email shall be shared or provided to any foreign persons, entities, or subsidiaries without the expressed written authorization of Ultra Electronic, ProLogic. If you have received it in error please notify the sender and delete it from your system. If you are not the intended recipient you must not copy it or use it for any purpose nor disclose or distribute its contents to any other person. All communications may be subject to interception or monitoring for operational and/or security purposes. Please rely on your own virus checking as the sender cannot accept any liability for any damage arising from any bug or virus infection. Please consider the environment before printing this email.

4 4

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

indent parameter
by Hans-Juergen Rennau 28 Jun '13

28 Jun '13

Hello, it seems to me there is a couple of issues concerning the handling of indentation (serialization parameter "indent"). a) Using "indent=no" whitespace-only text nodes between element nodes are removed b) Using "indent=yes" the content of elements with xml:space="preserve" is indented, too I suppose both behaviors are in contraction to the spec ( http://www.w3.org/TR/xslt-xquery-serialization/#xml-indent ). As a particular consequence, (a) seems to preclude making an exact copy of a document (fragment) - all whitespace-only text nodes vanish. And as a consequence of (b) one has no possibility to locally protect the whitespace used in the source document, as required for example when reusing "programlisting" elements of docbook. Or did I overlook something? Kind regards, Hans-Juergen

2 3

Best practices for concurrency
by The Trainspotter 19 Jun '13

19 Jun '13

Hi, I've read through the online example on server concurrency, however, currently I'm not running a BaseXServer instance I'm just running things locally. I want multiple threads in my application to be able to query the database at once without treading on each others toes. What's the best approach for doing this? I have tried querying the database using different threads but I keep getting internal BaseX errors which leads me to think I'm going about this the wrong way. Cheers Gary

5 15

invalid collection name?
by C. M. Sperberg-McQueen 27 May '13

27 May '13

Using the GUI interface to BaseX 7.2, when I select Database / Open and Manage ..., I get a list of the databases I have built over the time I've been using BaseX. When I try to open the collection named MEP.bigbooks, though, I get an error box saying "Name 'MEP.bigbooks' is invalid." In the "Manage Databases" dialog box, the information about the collection appears normal and plausible (at least, the size is about what I expect, and so on). It's only when I try to open the collection that I run into the error. I get similar errors on other collections which have . in their name; I do not get errors when opening other collections built (as far as I can remember) at the same time from the same data, if their name does not include a full stop. When I try renaming the collection without the full stop, I get the same error message. When I try to issue the command ALTER DATABASE MEP.bigbooks MEPbigbooks from the server command area of the GUI, I get a short syntax summary in the Query Info area complaining about the syntax of the command. When I replace MEP.bigbooks with MEP\.bigbooks and "MEP.bigbooks", I also get syntax errors. If I close the GUI and just rename the directory within my BaseXData directory, the database appears to open normally. I think I have two questions: Q1 is just renaming the directory a safe way (assuming no server is currently working with the data) to rename a collection? Should I do the same for all the other collections I have that don't currently open? I assume that if the 7.2 GUI won't let me rename them, it also will have trouble deleting them, so a related question is: is just deleting a sub-directory of BaseXData a safe way to delete a collection? Q2 Did the rules for names of collections change? Or is there some other cause? Speaking for myself, I guess that as a user I haven't thought much about what characters should be allowed in collection names, but I may have assumed that if they are not unrestricted, they would be restricted to being legal NCNames. I do kind of like using dots in names, so you can count this as one vote for relaxing the naming rules at some convenient moment. It's not a really important point, of course. best, Michael -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

2 2

XqueryDebugger
by Rob Stapper 16 May '13

16 May '13

Hi, Making intensive use of mappings {} and high-order recursive functions, I really could do with a nice debugger for the Xquery in BaseX. Is a debugger in the planning? Thnx. Regards, Rob Stapper

5 11

Namespaces on element construction
by Arto Viitanen 14 May '13

14 May '13

Why does this query (on BaseX 7.5, basexgui) let $ns := namespace {''} {'the-namespace'} let $c1 := <c xmlns="the-namespace">text</c> let $c2 := <c>{ $ns,'text'}</c> let $c3 := element {'c'} {$ns,'text'} let $qn := QName($ns, 'c') let $c4 := element {$qn} {'text'} let $nl := "
" return ($c1,$nl, $c2,$nl, $c3,$nl, $c4,$nl, '|' || namespace-uri-from-QName(node-name($c1)) || '|',$nl, '|' || namespace-uri-from-QName(node-name($c2)) || '|',$nl, '|' || namespace-uri-from-QName(node-name($c3)) || '|',$nl, '|' || namespace-uri-from-QName(node-name($c4)) || '|',$nl) outputs <c xmlns="the-namespace">text</c> <c xmlns="the-namespace">text</c> <c xmlns="the-namespace">text</c> <c xmlns="the-namespace">text</c> |the-namespace| || || |the-namespace| I.e, the namespace seems to exist on all four cases when outputted, by node-name finds it only if the namespace is given in element as text, or as QName value in computed element. -- Arto Viitanen Microteam Oy Finland

2 2

BaseX Capacity
by Rajabrata Chaudhuri 06 May '13

06 May '13

Hello, First I'd like to thank you guys for all your great work on BaseX. I am fairly familiar with XML DBs and have done a significant amount of development on top of Mark Logic. I would like to ask some questions about capacity and scalability. I have reviewed the documentation and see that the biggest store is for SDMX @ approximately 8000 GB. So I am just trying to understand what this means better and would appreciate any of your expert advice for my questions below: 1. Is the expectation that you can query against 8 TB of XML data efficiently? 2. My requirements will be to query across probably 24 TB of XML data. Do you guys feel this is possible? 3. What is the method to scale horizontally and vertically? I.E. Would I be adding more servers, or starting more instances, etc.? 4. How does high availability work? I.E. Can I have multiple active-active nodes, or should it be active-passive, etc.? Any help anyone can render is greatly appreciated. Thanks Raj

3 5

xquery:eval yields NullPointerException
by Marco Lettere 29 Apr '13

29 Apr '13

Hi all, on the way to writing a small DSL I stumbled upon the following BaseX 7.6 issue. I'm not able to explain to myself why the first two expressions of the following xquery module return correctly whereas the last yields NullPointerException (as shown in the appended stacktrace). What I'm mtrying to do is to wrap up some modification code into userfriendly function calls that can be used in mini xquery scripts to formalize xml workflow. Thank you for any help, Marco. declare variable $globals := map { 'change' := local:change#1, 'generate' := local:gen#0 }; declare function local:gen(){ <b/> }; declare function local:change($doc){ copy $newdoc := $doc modify( insert node <a/> as last into $newdoc ) return $newdoc }; ( xquery:eval("copy $newdoc := <b/> modify( insert node <a/> as last into $newdoc ) return $newdoc",$globals), xquery:eval("$generate()", $globals), xquery:eval("$change($doc)", map:new( ($globals, map { "doc" := <b/>}))) ) The stacktrace and query plan: Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.6 Java: Oracle Corporation, 1.7.0_15 OS: Linux, i386 Stack Trace: java.lang.NullPointerException org.basex.query.up.expr.Transform.value(Transform.java:68) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.func.UserFunc.value(UserFunc.java:175) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:68) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:104) org.basex.query.func.DynamicFunc.value(DynamicFunc.java:56) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.QueryContext.value(QueryContext.java:254) org.basex.query.func.FNXQuery.eval(FNXQuery.java:95) org.basex.query.func.FNXQuery.eval(FNXQuery.java:71) org.basex.query.func.FNXQuery.iter(FNXQuery.java:37) org.basex.query.QueryContext.iter(QueryContext.java:285) org.basex.query.expr.List$1.next(List.java:100) org.basex.query.QueryContext.execute(QueryContext.java:524) org.basex.query.QueryProcessor.execute(QueryProcessor.java:96) org.basex.core.cmd.AQuery.query(AQuery.java:77) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.gui.GUI.exec(GUI.java:397) org.basex.gui.GUI$7.run(GUI.java:349) Compiling: - pre-evaluating map { "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } } Optimized Query: declare function local:change($doc) { copy $newdoc as node() := $doc modify insert node element a { () } into $newdoc return $newdoc }; declare function local:gen() { element b { () } }; (xquery:eval("copy $newdoc := $generate() modify( insert node <a/> as last into $newdoc ) return $newdoc", map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }), xquery:eval("$generate()", map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }), xquery:eval("$change($doc)", map:new((map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }, map { "doc":=element b { () } })))) Query plan: <QueryPlan> <UserFuncs> <UserFunc name="local:change" arg0="doc"> <Transform> <Let var="$newdoc as node()"> <VarRef> <Var name="$doc" id="2"/> </VarRef> </Let> <Insert> <VarRef> <Var name="$newdoc as node()" id="3"/> </VarRef> <CElem> <QNm value="a" type="xs:QName"/> </CElem> </Insert> <VarRef> <Var name="$newdoc as node()" id="3"/> </VarRef> </Transform> </UserFunc> <UserFunc name="local:gen"> <CElem> <QNm value="b" type="xs:QName"/> </CElem> </UserFunc> </UserFuncs> <VarStack> <Var name="$globals" id="0"/> </VarStack> <List> <FNXQuery name="eval(string[,bindings])"> <Str value="copy $newdoc := $generate()
 modify(
 insert node <a/> as last into $newdoc 
 )
 return $newdoc" type="xs:string"/> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> </FNXQuery> <FNXQuery name="eval(string[,bindings])"> <Str value="$generate()" type="xs:string"/> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> </FNXQuery> <FNXQuery name="eval(string[,bindings])"> <Str value="$change($doc)" type="xs:string"/> <FNMap name="new([maps[,coll]])"> <List> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> <LitMap> <Str value="doc" type="xs:string"/> <CElem> <QNm value="b" type="xs:QName"/> </CElem> </LitMap> </List> </FNMap> </FNXQuery> </List> </QueryPlan>

2 2

timezone formatting
by Marco Lettere 17 Apr '13

17 Apr '13

Hi again, when, in Basex 7.6, formatting a datetime with format-dateTime(current-dateTime(), "[Y0001][M01][D01][h01][m01][s01][Z0000]") I was expecting something like 20130314035118+0100 for my area (Central Europe) and instead I get 201303140351180060. Looks like minutes are not translated to hours and the sign is missing. For documentation I'm referring to http://www.w3.org/TR/xpath-functions-30/#rules-for-datetime-formatting. Thanks, M.

2 3