September 2011 - BaseX-Talk - mailman.uni-konstanz.de

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

deadlock in BaseXServer 6.2.7
by Godmar Back 23 Jun '12

23 Jun '12

Hi, we're observing that our BaseXServer 6.2.7 deadlocks. The stack traces are here: http://top.cs.vt.edu/~gback/bx/deadlock-6.2.7/basex-deadlock.txt Could you take a look to see if these ring any bells? Has this problem been addressed in a more recent version? - Godmar

3 5

BaseX server deadlock
by Laurent Chevalier 03 Nov '11

03 Nov '11

Hi, A deadlock occurs in the following situation: a first client program opens an iterative query. For each iteration, this program does some processing and sends another reading request to BaseX (using another BaseX session). All works fine until a second client program (or another thread) sends an updating command to BaseX (like optimize for instance). This locks BaseX server. To unlock it, you have to kill the first program. I have read BaseX server code and found the reason for this behavior in the class org.basex.core.Lock: - with the iterative query, there is always at least one reader alive (readers=1). - when the updating query is received, it is put in the queue (index 0) and remains in it as long as there is a reading query running (that is to say, as long as the iterative reading query is running). - then a second reading request is received, it is put in the queue (index 1 as there is already the updating query in the queue). As it is only the second item of the queue, it remains in the queue as long as the first item in the queue (the updating query) has not been processed (BaseX processes the requests in the order of arrival, FIFO queue). But this first item can not be processed because there is the iterative reading query running. All queries are thus locked. Some may say that we should not send another query while we are in the loop of an iterative query but in our context of many sites being developed by several developers, it is possible that a developer codes this and we do not want BaseX to be locked in this case (whatever it is a mistake of the developer or not). I have found a solution to this problem by modifying the org.basex.core.Lock class. You will find my code hereafter. I do not use a queue anymore and i use a static mutex (called queueMutex) to synchronize all pending queries (threads). The "drawback" of this solution is that the queries are not processed anymore in the order of arrival but randomly. What do you think of this solution ? Do you plan to update BaseX locking mechanism ? I'm using BaseX 6.7.1 but I have seen that Lock.java has not been changed in BaseX 6.7.2. Here is my code : package org.basex.core; import java.util.Date; //import java.util.LinkedList; import java.util.Random; import org.basex.util.Util; /** * Management of executing read/write processes. * Supports multiple readers, limited by {@link MainProp#PARALLEL}, * and single writers (readers/writer lock). * * @author BaseX Team 2005-11, BSD License * @author Christian Gruen */ final class Lock { /** Queue for all waiting processes. */ // private final LinkedList<Object> queue = new LinkedList<Object>(); /** Mutex object. */ private final Object mutex = new Object(); /** Database context. */ private final Context ctx; /** Static mutex used to synchronize all pending queries. **/ private final static Object queueMutex = new Object(); /** Number of active readers. */ private int readers; /** Writer flag. */ private boolean writer; /** * Default constructor. * @param c context */ Lock(final Context c) { ctx = c; } /** * Modifications before executing a command. * @param w writing flag */ void lock(final boolean w) { synchronized(mutex) { int code = new Random(new Date().getTime()).nextInt(); // final Object o = new Object(); // queue.add(o); try { while(true) { synchronized(queueMutex) { // if(o == queue.get(0) && !writer) { if(!writer) { if(w) { if(readers == 0) { writer = true; break; } } else if(readers < Math.max(ctx.mprop.num(MainProp.PARALLEL), 1)) { ++readers; break; } } } mutex.wait(); } } catch(final InterruptedException ex) { Util.stack(ex); } // queue.remove(0); } } /** * Modifications after executing a command. * @param w writing flag */ synchronized void unlock(final boolean w) { synchronized(mutex) { if(w) { writer = false; } else { --readers; } mutex.notifyAll(); } } }

3 21

Problems in using BaseX REST server - IncompatibleClassChangeError javax.ws.rs.core.Response$Status does not implement the requested interface
by Pushkaraj Thorat 21 Oct '11

21 Oct '11

Hello, I am doing setup for REST enabled basex database with procedure mentioned in http://docs.basex.org/wiki/JAX-RX_API URL. Configuring & starting server and running failed due to class not found, but later I've modified the ./basexrest script as: (Included all the jars in lib folder) ######START #!/bin/bash # Path to this script PWD=`dirname $0` # Paths to distributed files or source directories BASEX=$PWD/BaseX.jar BASEXAPI=$PWD/../lib/basex-api.jar # Classpath LIB=$PWD/../lib LIB_CP="" for JAR_FILE in $LIB/*.jar do LIB_CP="$LIB/$JAR_FILE:$LIB_CP" done CP=$BASEX:$BASEXAPI:$LIB_CP # Options for virtual machine VM=-Xmx1g # Run REST server java -cp "$CP" $VM org.basex.api.jaxrx.JaxRxServer "$@" ######END after this, server started successfully, hit the URL - http://localhost:8984/basex/jax-rx I am getting following exception: java.lang.IncompatibleClassChangeError: Class javax.ws.rs.core.Response$Status does not implement the requested interface javax.ws.rs.core.Response$StatusType at com.sun.jersey.spi.container.ContainerResponse.getStatus(ContainerResponse.java:548) at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.commitWrite(ContainerResponse.java:156) at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.write(ContainerResponse.java:133) at org.basex.io.BufferedOutput.flush(BufferedOutput.java:50) at org.basex.io.PrintOutput.flush(PrintOutput.java:126) at org.basex.data.XMLSerializer.cls(XMLSerializer.java:206) at org.basex.data.Serializer.close(Serializer.java:219) at org.basex.api.jaxrx.BXJaxRx$1.code(BXJaxRx.java:96) at org.basex.api.jaxrx.BXCode.run(BXCode.java:53) at org.basex.api.jaxrx.BXOutput.write(BXOutput.java:53) at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71) at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57) at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:299) Your help is appreciated. Thanks, Pushkaraj

3 6

Snapshot of basex-6.8.war
by Christian Grün 09 Oct '11

09 Oct '11

...beta testers wanted! BaseX is now also available as web application archive (war): http://files.basex.org/releases/latest/basex-6.8-SNAPSHOT.war The latest documentation on our REST API is found at: http://docs.basex.org/wiki/REST We're looking forward to your feedback, Christian ___________________________ BaseX Team Christian Grün Uni KN, Box 188 D-78457 Konstanz http://www.basex.org

3 5

node.js interface?
by Andy Bunce 07 Oct '11

07 Oct '11

Hi, I wonder if you have any plans to add a Node.js interface to the basex-api client set? I think this could be very useful. In design terms I imagine it might look very similar the Redis client https://github.com/bnoguchi/redis-node. /Andy

2 4

csv export problem
by Kevin S. Clarke 26 Sep '11

26 Sep '11

Hi, Timely that CSV is being discussed. I have a question that has been stumping me... I want to be able to output something like: "text here", "<div><a href=""http://basex.org"">text, and more</a></div>", "text here again" What's stumping me is being able to double-quote escape the quotation marks that are in my markup. I've tried serializing my elements and then replacing the single quotes of the attributes to double quotes but then when I try to write that to a file (either serializing as text or xml) the tags are escaped to < etc. Here is a bit of what I've tried: declare option output:indent 'no'; declare variable $txtSer := <serialization-parameters xmlns="http://www.w3.org/2010/xslt-xquery-serialization"> <method value="text"/> </serialization-parameters>; declare variable $xmlSer := <serialization-parameters xmlns="http://www.w3.org/2010/xslt-xquery-serialization"> <method value="xml"/> <indent value="no"/> </serialization-parameters>; declare variable $outFile := 'out2.csv'; declare variable $xml := <sample><a href="http://basex.org">text, and more text</a></sample>; declare function local:writeEOL() { file:append($outFile, '
', $txtSer) }; (: Trying to output markup so I can eventually output something like: "text", "<div><a href=""http://basex.org"">text, and more</a></div>", "text" :) ( file:append($outFile, $xml, $txtSer), (: only text of xml out :) local:writeEOL(), file:append($outFile, $xml, $xmlSer), (: xml out but can't escape quotes :) local:writeEOL(), file:append($outFile, serialize($xml), $txtSer), (: tags escaped :) local:writeEOL(), file:append($outFile, serialize($xml, $txtSer), $txtSer), (: only text out :) local:writeEOL(), file:append($outFile, serialize($xml), $xmlSer), (: tags escaped :) local:writeEOL(), file:append($outFile, serialize($xml), $txtSer), (: tags escaped :) local:writeEOL(), file:append($outFile, concat('<![CDATA[', serialize($xml), ']]>'), $xmlSer), (: tags escaped :) local:writeEOL(), file:append($outFile, concat('<![CDATA[', serialize($xml), ']]>'), $txtSer), (: tags escaped :) local:writeEOL(), file:append($outFile, '<,<', $txtSer) (: both are escaped :) ) I don't seem to be able to output the markup in either text or xml mode. What am I doing wrong? Thanks! Kevin

2 3

export to CSV, or even Excel
by Andreas Karpf 26 Sep '11

26 Sep '11

Hello, I have got question regarding the export function of Basex. I used BaseX to load the Reuters Corpus Volume 1 which is a corpus of around 800k files, unzipped nealry 3GB in size. Is there any way to export some parts of these files (with xpath), parsed into a Basex database, to csv? I am aware that those kind of questions would probably fall under the support which costs. But if there is any simpler solution for that it would be very nice if you could post it. It would help me a lot! Best regards, Andreas

3 2

Builds full-text information.
by Toshio HIRAI 26 Sep '11

26 Sep '11

Hi. Although I was trying implements of Japanese Lexer to 6.8 beta, I have noticed that the result of a ft:mark() function shifts. 55 line of https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/query/… final FTLexer lex = new FTLexer(ftopt).sc().init(nd.atom()); FTLexer will generate the tokenizer of the default language instead of an original language, if it does not perform in this way. How is it? In code which I am correcting, ft:mark() operated normally by carrying out like this. thanks. -- Toshio HIRAI

2 1

Query question
by Wiard Vasen 26 Sep '11

26 Sep '11

Dear All, Is it possible to formulate a query which gives the distinct documents from a repository containing a specific word: let $range := 1 to 640 for $doc in collection('brievenvangogh') let $uri := base-uri($doc), $num := substring($uri, string-length($uri) - 6, 3) where $num castable as xs:integer and xs:integer($num) = $range return <document uri='{$uri}'>{ for $n score $s in $doc//*[text() contains text 'cezanne'] return <hit score='{$s}'>{ $n }</hit> }</document> In this query I get the number of hits of all documents in the repository. Thanks in advance! Regards, Wiard

1 0