- BaseX-Talk - mailman.uni-konstanz.de

BaseX Error
by Rogers, Paul 06 Aug '15

06 Aug '15

Good afternoon, we're getting an error on BaseX rest and DBA connections: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> Version: BaseX 8.2.3 Java: Oracle Corporation, 1.7.0_01 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException Is there an issue with the service? Best Regards, Paul Rogers DevOps DBA Engineer Kuoni Global Travel Services Ueberlandstrasse 360 CH-8051 Zurich, Switzerland T +41 44 325 21 39 paul.rogers(a)ch.kuoni.com<mailto:paul.rogers@ch.kuoni.com> www.kuoniglobaltravelservices.com<http://www.kuoniglobaltravelservices.com/>

2 1

SIGPIPE error
by mar＠centrum.cz 06 Aug '15

06 Aug '15

Hi, I am having difficulties with populating BASEX database. I have plenty of XML files (~ half a million, with various sizes ranging from several kilobytes up to hundred of kilobytes). I use BASEX Java API and finally I call for each file org.basex.core.cmd.Add. I am trying to import them into BASEX database, in fact there are 22 types of files (22 XSD definitions) the files conform to, so I have 22 different databases in a single BASEX server. I have plenty of RAM and CPU power and I monitor the process (both -- the BASEX server and my client program) from within JVisualVM, the JVM reaches the CPU boundaries, but RAM is never exhausted. Before importing, I need to enhance the XML data with some additional information taken from SQL database. I have written a Groovy multithreaded program that uses BASEX Java API with heavy use of GPars library. Simply put, the program: 1. has several producer threads -- each producer reads given portion of the database and provides those additional information 2. has several consumer threads -- each consumer takes the original files, wraps it with additional information and finally calls org.basex.core.cmd.Add command. Various testing with less data (upto ~ several thousands of files) provides good results -- no loss of data, BASEX server and my client program behaves as it should. Unfortunately when trying to import all of the files, the program starts fine, but when it gets "warm" I got SIGPIPE errors in log from time to time (as I said, there is plenty of RAM and CPU available) (see attachment please). Comments to picture: 1. I am adding document with ID ISPOP_166007 -- this ID is indeed missing in the final database 2. just simple call to Add: Closure add = { session -> def cmd = new org.basex.core.cmd.Add(dsn, enhancedXml) session.execute(cmd) } 3. I am reusing the session, the session is bound to current thread and never gets closed until the thread (consumer) finishes There is nothing wrong in BASEX server log, other documents are added just fine, there is no trace about document ISPOP_166007. Just for reference the complete stack trace follows: - - - - ERROR basex.support.AddResourcesSupport - unable to consume ISPOP_166007 java.net.SocketException: Roura přerušena (SIGPIPE) at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_45] at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_45] at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_45] at org.basex.io.out.BufferOutput.flush(BufferOutput.java:60) ~[basex-8.2.jar!/:8.2] at org.basex.io.out.BufferOutput.write(BufferOutput.java:54) ~[basex-8.2.jar!/:8.2] at org.basex.io.out.PrintOutput.write(PrintOutput.java:66) ~[basex-8.2.jar!/:8.2] at java.io.OutputStream.write(OutputStream.java:116) ~[na:1.8.0_45] at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.8.0_45] at org.basex.api.client.ClientSession.send(ClientSession.java:238) ~[basex-8.2.jar!/:8.2] at org.basex.api.client.ClientSession.execute(ClientSession.java:160) ~[basex-8.2.jar!/:8.2] at org.basex.api.client.ClientSession.execute(ClientSession.java:167) ~[basex-8.2.jar!/:8.2] at org.basex.api.client.Session.execute(Session.java:36) ~[basex-8.2.jar!/:8.2] at org.basex.api.client.Session$execute.call(Unknown Source) ~[na:na] at basex.support.AddResourcesSupport$_consume_closure9$_closure17.doCall(AddResourcesSupport.groovy:255) ~[basex-1.0.jar!/:na] at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] at basex.BasexSessionRegistry.withThreadBoundSession(BasexSessionRegistry.groovy:79) ~[basex-1.0.jar!/:na] at basex.BasexSessionRegistry$withThreadBoundSession$0.call(Unknown Source) ~[na:na] at basex.support.AddResourcesSupport$_consume_closure9.doCall(AddResourcesSupport.groovy:257) ~[basex-1.0.jar!/:na] at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.BooleanReturningMethodInvoker.invoke(BooleanReturningMethodInvoker.java:51) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.BooleanClosureWrapper.call(BooleanClosureWrapper.java:53) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.DefaultGroovyMethods.find(DefaultGroovyMethods.java:3908) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.dgm$191.invoke(Unknown Source) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] at basex.support.AddResourcesSupport.consume(AddResourcesSupport.groovy:251) ~[basex-1.0.jar!/:na] - - - - Best Regards, Martin

3 9

Resolving http file with local repository.
by ankit kumar 06 Aug '15

06 Aug '15

Hi, I want to load document having a http url from local cache directory. Is there a way i can resolve http url to my local cache directory. For example let $file := doc("http://abc.com/example.xml") should resolve example.xml from c:\documents\repo\http\abc.com\example.xml Thanks, Anky

2 2

Running BaseX In A Docker Container?
by Eliot Kimber 05 Aug '15

05 Aug '15

As part of my DITA For Small Teams project we're setting up a set of coordinated Docker containers to manage the various components (git server, Jenkins server, BaseX server). We're setting up our own custom containers so that they are appropriately pre-configured. In the case of BaseX that means configured with the DFST Web app and supporting modules, configuration details, etc. Just curious if anyone has experience putting BaseX in a container and if there's anything we should look out for or anything we can contribute back out of this effort. Cheers, E. ---- Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 1

BaseX Package and Jetty
by Eckey, Florian 05 Aug '15

05 Aug '15

3 5

Destination of result-document from XSLT module
by Lizzi, Vincent 05 Aug '15

05 Aug '15

I'm trying to use the XSLT Module in BaseX 8.2.3 with Saxon 9.6 to run an XSLT that produces several output documents using xsl:result-document. I'm having trouble setting the location of the output documents. I want to have xsl:result-document create the output documents in a temporary folder because the documents need to be zipped together. According to Saxon's documentation, the a relative path in the href attribute of xsl:result-document will be resolved using either the path of the Destination, or the current directory. The XSLT Module does not appear have a way to provide a path for a destination document. What I'm seeing is that the result documents are created in BaseX's home directory. The XSLT works as expected when run using Saxon from the command line, where it's possible to set a destination path. Is there a way to specify a Base Output URI to the XSLT Module? Or, would it be possible to specify a file URI output location to a method like xslt:transform? One possible workaround is to provide an absolute path as a parameter to the XSLT, and use that parameter in the xsl:result-document href location. Here is a self-contained example code: declare function local:example($in, $xsl, $zipPath) { let $tempDir := file:create-temp-dir('test', 'example') let $x := xslt:transform-text($in, $xsl) return let $zip := archive:create-from($tempDir) return ( file:write-binary($zipPath, $zip) (: , file:delete($tempDir, true()) :) ) }; let $xsl := <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0"> <xsl:template match="/"> <xsl:result-document href="doc1.xml"> <test>this is a test 1 <xsl:apply-templates/></test> </xsl:result-document> <xsl:result-document href="doc2.xml"> <test>this is a test 2 <xsl:apply-templates/></test> </xsl:result-document> </xsl:template> </xsl:stylesheet> let $doc := <test>this is input</test> let $zipPath := 'report.zip' return local:example($doc, $xsl, $zipPath) The expected output is a zip file report.zip that contains doc1.xml and doc2.xml. However, what I'm seeing is that report.zip is created as an empty zip file and doc1.xml and doc2.xml are placed in BaseX's home directory. Thanks, Vincent

5 7

Performance with joins?
by Tim Thompson 04 Aug '15

04 Aug '15

Hello, I have a database containing two resources/documents: they both represent the same set of library catalog records (7728 "records" in each), but they each contain different data that I want to join. The first resource looks like this: <marc:collection xmlns:marc="http://www.loc.gov/MARC21/slim"> <marc:record> <marc:leader>01225cam a2200373Mi 4500</marc:leader> <marc:controlfield tag="001">5323084</marc:controlfield> <marc:datafield ind1="1" ind2="4" tag="245"> <marc:subfield code="a">Els teleclubs a les illes Balears :</marc:subfield> </marc:datafield> </marc:record> <marc:record> <marc:leader>01225cam a2200373Mi 4500</marc:leader> <marc:controlfield tag="001">5323084</marc:controlfield> <marc:datafield ind1="1" ind2="4" tag="245"> <marc:subfield code="a">Els teleclubs a les illes Balears :</marc:subfield> </marc:datafield> </marc:record> <marc:record> <marc:leader>00818cam a2200241Mi 4500</marc:leader> <marc:controlfield tag="001">6310976</marc:controlfield> <marc:datafield ind1="0" ind2="0" tag="245"> <marc:subfield code="a">Diccionari manual de sinònims i antònims de la llengua catalana /</marc:subfield> </marc:datafield> </marc:record> </marc:collection> The second one looks like this: <root> <row> <LANGUAGE>cat</LANGUAGE> <ITEM_ID>5912416</ITEM_ID> <BIB_ID>5323084</BIB_ID> <VENDOR_CODE>MXBKSMX</VENDOR_CODE> </row> <row> <LANGUAGE>cat</LANGUAGE> <ITEM_ID>5912416</ITEM_ID> <BIB_ID>5323084</BIB_ID> <VENDOR_CODE>PUVILL</VENDOR_CODE> </row> <row> <LANGUAGE>cat</LANGUAGE> <ITEM_ID>5935043</ITEM_ID> <BIB_ID>6310976</BIB_ID> <VENDOR_CODE>PUVILL</VENDOR_CODE> </row> </root> I have a simple query that joins the two using the value of the marc:controlfield[@tag = '001'] from resource 1 and the BIB_ID from resource 2. The query: declare namespace marc="http://www.loc.gov/MARC21/slim"; for $m in collection( "latin_hold_20150730" )/marc:collection/marc:record, $r in collection( "latin_hold_20150730" )/root/row[BIB_ID = $m/marc:controlfield[@tag = '001']] group by $key := $r/ITEM_ID return <test n="{ $key }">{ $m/marc:datafield[@tag = '245']/marc:subfield[@code = 'a']/string() }</test> When I run this using Saxon (substituting fn:doc for fn:collection), it only takes a second to execute. In BaseX 8.2.3 (using the GUI), however, it takes a very long time to execute (around 6 minutes!). The BaseX database has both attribute and text indexes enabled. Any idea what is causing it to take so long? Here is the BaseX Query Info: Total Time: 383756.3 ms Compiling: - pre-evaluating collection("latin_hold_20150730") - pre-evaluating collection("latin_hold_20150730") Query: declare namespace marc="http://www.loc.gov/MARC21/slim"; for $m in collection( "latin_hold_20150730" )/marc:collection/marc:record, $r in collection( "latin_hold_20150730" )/root/row[BIB_ID = $m/marc:controlfield[@tag = '001']] group by $key := $r/ITEM_ID return <test n="{ $key }">{ $m/marc:datafield[@tag = '245']/marc:subfield[@code = 'a']/string() }</test> Optimized Query: for $m_0 in (db:open-pre("latin_hold_20150730",0), ...)/marc:collection/marc:record for $r_1 in (db:open-pre("latin_hold_20150730",0), ...)/root/row[(BIB_ID = $m_0/marc:controlfield[(@tag = "001")])] let (: post-group :) $m_4 := $m_0 group by $key_2 := $r_1/ITEM_ID return element test { (attribute n { ($key_2) }, $m_4/marc:datafield[(@tag = "245")]/marc:subfield[(@code = "a")]/string()) } Result: - Hit(s): 7587 Items - Updated: 0 Items - Printed: 505 KB - Read Locking: local [latin_hold_20150730] - Write Locking: none Timing: - Parsing: 0.43 ms - Compiling: 1.96 ms - Evaluating: 383737.5 ms - Printing: 16.41 ms - Total Time: 383756.3 ms Query plan: <QueryPlan compiled="true"> <GFLWOR> <For> <Var name="$m" id="0"/> <IterPath> <DBNodeSeq size="2"> <DBNode name="latin_hold_20150730" pre="0"/> <DBNode name="latin_hold_20150730" pre="365692"/> </DBNodeSeq> <IterStep axis="child" test="marc:collection"/> <IterStep axis="child" test="marc:record"/> </IterPath> </For> <For> <Var name="$r" id="1"/> <IterPath> <DBNodeSeq size="2"> <DBNode name="latin_hold_20150730" pre="0"/> <DBNode name="latin_hold_20150730" pre="365692"/> </DBNodeSeq> <IterStep axis="child" test="root"/> <IterStep axis="child" test="row"> <CmpG op="="> <CachedPath> <IterStep axis="child" test="BIB_ID"/> </CachedPath> <IterPath> <VarRef> <Var name="$m" id="0"/> </VarRef> <IterStep axis="child" test="marc:controlfield"> <CmpG op="="> <CachedPath> <IterStep axis="attribute" test="tag"/> </CachedPath> <Str value="001" type="xs:string"/> </CmpG> </IterStep> </IterPath> </CmpG> </IterStep> </IterPath> </For> <GroupBy> <Spec> <Var name="$key" id="2"/> <IterPath> <VarRef> <Var name="$r" id="1"/> </VarRef> <IterStep axis="child" test="ITEM_ID"/> </IterPath> </Spec> </GroupBy> <CElem> <QNm value="test" type="xs:QName"/> <CAttr> <QNm value="n" type="xs:QName"/> <VarRef> <Var name="$key" id="2"/> </VarRef> </CAttr> <MixedPath> <VarRef> <Var name="$m" id="4"/> </VarRef> <IterStep axis="child" test="marc:datafield"> <CmpG op="="> <CachedPath> <IterStep axis="attribute" test="tag"/> </CachedPath> <Str value="245" type="xs:string"/> </CmpG> </IterStep> <IterStep axis="child" test="marc:subfield"> <CmpG op="="> <CachedPath> <IterStep axis="attribute" test="code"/> </CachedPath> <Str value="a" type="xs:string"/> </CmpG> </IterStep> <FnString name="string([item])"/> </MixedPath> </CElem> </GFLWOR> </QueryPlan> Thanks in advance! Tim -- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library

2 4

Round tripping JSON
by Marc van Grootel 04 Aug '15

04 Aug '15

Hi, I am working on some REST API code and I'm struggling with POSTing of JSON. What I want to achieve is to have the function receive an XDM map (from JSON using fn:parse-json). I also want to output from such a map (in effect roundtripping some JSON). The test POST body I'm using is: {"foo": "bar"} When reading the docs I'm led to believe that this should achieve it: declare %rest:POST("{$body}") %rest:path("/json") %rest:consumes("application/json;lax=yes;format=map") %rest:produces("application/json;format=map") function api:json($body) { $body }; No matter what I do I always get this as response: <json type="object"> <foo>bar</foo> </json> Somehow I cannot force the function to receive a map(*) as $body argument. When I wrap $body in a trace() I see that it gets XML and not a map(*). What am I missing? Or do I really have to go JSON > XML > map {....}? -- --Marc

3 18

File permissions on .basex files
by Amanda Galtman 03 Aug '15

03 Aug '15

Hi, It seems that the commands and functions that operate on the .basex files, such as ADD and db:add(...), make the .basex files writable by the creator but not by anyone else. On the other hand, executing a query requires write permission for at least some of the .basex files, even if the query does not attempt to update the database. As a result, if I create a database and want anyone other than me to be able to use it (via the GUI, for example), I have to change permissions on the .basex files after creating the database. Apparently, I have to change the permissions again if I later update the database and rebuild the indexes. Do folks on this list have a slick way to keep the permissions as open as needed? I tried searching the list archive but didn't find anything that looked relevent. Would it be a reasonable enhancement request to make this a bit smoother? For example, an option could let you specify that the .basex files being created or modified should be writable by all. Alternatively or in addition, perhaps it is possible to stop requiring write permission for non-updating queries; I'm not familiar with the reason why write permission is needed. Thanks, Amanda

2 4

Slow query
by Menashè Eliezer 03 Aug '15

03 Aug '15

Hello, I wonder if the attached query can be optimised. I'm attaching all relevant information. Basex 7.9, Debian, powerful server. This is just an example. The queries will be built based on a compilation of a search form. So reordering the conditions for having smaller subset right from the begging isn't relevant. Any help would be appreciated. 40 seconds are not acceptable. -- With kind regards, Menashè

2 45