October 2011 - BaseX-Talk - mailman.uni-konstanz.de

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

deadlock in BaseXServer 6.2.7
by Godmar Back 23 Jun '12

23 Jun '12

Hi, we're observing that our BaseXServer 6.2.7 deadlocks. The stack traces are here: http://top.cs.vt.edu/~gback/bx/deadlock-6.2.7/basex-deadlock.txt Could you take a look to see if these ring any bells? Has this problem been addressed in a more recent version? - Godmar

3 5

BaseX 7.0.1: Release early, release often..
by Christian Grün 06 Nov '11

06 Nov '11

Hi all, we are glad to announce BaseX 7.0.1, which is basically the result of your initial feedback on 7.0. Those are the most important changes: DISTRIBUTIONS: - Windows installer was updated to support latest features - ZIP file was updated (initial config & directories added) - Short directory names are chosen if config file resides in app.dir. - Start scripts have been improved XQUERY: - much faster execution of count() when applied to opened databases SERVER (http://docs.basex.org/wiki/Startup_Options#BaseX_HTTP_Server) - Flag -c connects to an existing database server - Flag -s specifies a port for stopping the HTTP server (Jetty) - Flag -S starts the HTTP server as a service - running write operations will be completed before server is stopped API: - Ruby, Python, PHP, Java: clients updated Please check out the well-known links: - Homepage: http://basex.org/ - Documentation: http://docs.basex.org/ - Snapshots: http://files.basex.org/releases/latest/ - Sources: https://github.com/BaseXdb Enjoy the update! Christian BaseX Team

5 17

Windows fast, linux very slow
by Sven Rega 03 Nov '11

03 Nov '11

Hy, i'm using the basexserver 6.7.1 on a windows (i7; 8GB; 2,8GHz) and on different linux machines (i7; 16GB, 3,4GHz). the same xml file (~2,5GByte) and one simple query has i big difference in speed between both machines and i don't know, where the bottleneck is coming from. on the windows machine i get througput of ~600 to 1000 xml entities per second. On the linux machine i only get max. 110 xml entities per second. I use the same jvm setting on both machines: -server -Xmx3g I also use the index (text, path, attribtue) on both machines. The query: "declare function local:ProductValues() { for $n in //Product[(@ID='A' or @ID='D' or @ID='G')]/Values return <pv><utid>{data($n/../@ID)}</utid><id>{data($n/../@SecondID)}</id>{$n}</pv> }; local:ProductValues()" I read from the basex server over lan with 1GBit. Does anybody have a idea, what could be the reason for such speed difference between windows and linux? king regards Sven

5 17

BaseX server deadlock
by Laurent Chevalier 03 Nov '11

03 Nov '11

Hi, A deadlock occurs in the following situation: a first client program opens an iterative query. For each iteration, this program does some processing and sends another reading request to BaseX (using another BaseX session). All works fine until a second client program (or another thread) sends an updating command to BaseX (like optimize for instance). This locks BaseX server. To unlock it, you have to kill the first program. I have read BaseX server code and found the reason for this behavior in the class org.basex.core.Lock: - with the iterative query, there is always at least one reader alive (readers=1). - when the updating query is received, it is put in the queue (index 0) and remains in it as long as there is a reading query running (that is to say, as long as the iterative reading query is running). - then a second reading request is received, it is put in the queue (index 1 as there is already the updating query in the queue). As it is only the second item of the queue, it remains in the queue as long as the first item in the queue (the updating query) has not been processed (BaseX processes the requests in the order of arrival, FIFO queue). But this first item can not be processed because there is the iterative reading query running. All queries are thus locked. Some may say that we should not send another query while we are in the loop of an iterative query but in our context of many sites being developed by several developers, it is possible that a developer codes this and we do not want BaseX to be locked in this case (whatever it is a mistake of the developer or not). I have found a solution to this problem by modifying the org.basex.core.Lock class. You will find my code hereafter. I do not use a queue anymore and i use a static mutex (called queueMutex) to synchronize all pending queries (threads). The "drawback" of this solution is that the queries are not processed anymore in the order of arrival but randomly. What do you think of this solution ? Do you plan to update BaseX locking mechanism ? I'm using BaseX 6.7.1 but I have seen that Lock.java has not been changed in BaseX 6.7.2. Here is my code : package org.basex.core; import java.util.Date; //import java.util.LinkedList; import java.util.Random; import org.basex.util.Util; /** * Management of executing read/write processes. * Supports multiple readers, limited by {@link MainProp#PARALLEL}, * and single writers (readers/writer lock). * * @author BaseX Team 2005-11, BSD License * @author Christian Gruen */ final class Lock { /** Queue for all waiting processes. */ // private final LinkedList<Object> queue = new LinkedList<Object>(); /** Mutex object. */ private final Object mutex = new Object(); /** Database context. */ private final Context ctx; /** Static mutex used to synchronize all pending queries. **/ private final static Object queueMutex = new Object(); /** Number of active readers. */ private int readers; /** Writer flag. */ private boolean writer; /** * Default constructor. * @param c context */ Lock(final Context c) { ctx = c; } /** * Modifications before executing a command. * @param w writing flag */ void lock(final boolean w) { synchronized(mutex) { int code = new Random(new Date().getTime()).nextInt(); // final Object o = new Object(); // queue.add(o); try { while(true) { synchronized(queueMutex) { // if(o == queue.get(0) && !writer) { if(!writer) { if(w) { if(readers == 0) { writer = true; break; } } else if(readers < Math.max(ctx.mprop.num(MainProp.PARALLEL), 1)) { ++readers; break; } } } mutex.wait(); } } catch(final InterruptedException ex) { Util.stack(ex); } // queue.remove(0); } } /** * Modifications after executing a command. * @param w writing flag */ synchronized void unlock(final boolean w) { synchronized(mutex) { if(w) { writer = false; } else { --readers; } mutex.notifyAll(); } } }

3 21

context for util:eval
by Andy Bunce 01 Nov '11

01 Nov '11

I notice Basex has a util:eval function similar to the eXist db one<http://demo.exist-db.org/exist/functions/util/eval>however it is not clear for the BaseX version what the inherited context, if any, is. The example on the eXist site: let $a := "Hello" return util:eval("$a") Fails with an undefined variable $a with BaseX 7.0.1 /Andy

4 6

very common attribute values/custom indexing
by Ross Judson 31 Oct '11

31 Oct '11

I have a database with the following characteristics: Database Properties Name: merge Size: 1165 MB Nodes: 45217101 Resources: 9 Timestamp: 30.10.2011 18:15:04 On my database, executing a query such as count(//summary[@totalDirs=0]) executes in about 30 seconds, returning a count of about 777,000 items. Executing count(//file[@length=0] also takes about 30 seconds, returning about 12,000. The query plans look like: <FNAggr name="count(item)"> <AxisPath> <RangeAccess data="merge" min="0" max="0" type="ATTRIBUTE"/> <IterStep axis="self" test="*:totalDirs"/> <IterStep axis="parent" test="*:summary"/> </AxisPath> </FNAggr> It's clear that BaseX goes to the attribute index, scans every entry there for the 0-0 range, then checks for the appropriate attribute and parent types. What this means is that very common attribute values can take a much longer time to run, even when the number of results, when discriminated by attribute name, is relatively small. The keys for the attribute index are simple values. Since this is an ordered/balanced structure, you can compound the key for free (sort of). Instead of using the attribute value as the key, use the index as a prefix map by compounding attribute value+attribute name (something like index.index(Token.concat(data.text(pre, text), data.name(pre)), pre);). I'm wondering if you've tried and rejected compounding the indexes like this. It's a common technique to use when indexing in triple stores (a few compound indexes allow you to have complete one-step indexing). With the additional discrimination provided you'd probably get a big jump in speed for the vast majority of queries, which won't be searching for an unknown attribute with a certain value. The query plan would become something like this: <FNAggr name="count(item)"> <AxisPath> <RangeAccess data="merge" min="0.self" max="0.self" type="ATTRIBUTE"/> <IterStep axis="parent" test="*:summary"/> </AxisPath> </FNAggr> where 0.self is the binary compound key. There's no reason to stop at compounding in the attribute name. You could also compound in the name of the element that owns the attribute, resulting in: <FNAggr name="count(item)"> <AxisPath> <RangeAccess data="merge" min="0.self.summary" max="0.self.summary" type="ATTRIBUTE"/> </AxisPath> </FNAggr> The down side to such an index would be increased key sizes in the attribute index, increased complexity handling numeric data, and some additional processing time. The up side would be an index with much more ability to discriminate across common attribute values. -- RJ

2 1

Wrong namespace for the EXPath HTTP Client function?
by Florent Georges 30 Oct '11

30 Oct '11

Hi, The following query: import module namespace http = "http://expath.org/ns/http-client"; http:send-request( <http:request href="http://google.com/" method="get"/>) returns the following error: Error: [XQST0059] Unknown module for namespace "http://expath.org/ns/http-client". The following one though is successful (even though it is not correct according to the HTTP Client spec) (note the element namespace and the function namespace are different): import module namespace http = "http://expath.org/ns/http"; declare namespace h = "http://expath.org/ns/http-client"; http:send-request( <h:request href="http://google.com/" method="get"/>) Regards, -- Florent Georges http://fgeorges.org/ http://h2oconsulting.be/

2 2

REST example wrong on wiki?
by Florent Georges 30 Oct '11

30 Oct '11

Hi, If I try the first POST example in the "Command Line" examples section at the end of http://docs.basex.org/wiki/REST, I get the following response from the REST server: HTTP/1.1 400 Bad Request Content-Length: 24 Server: Jetty(6.1.26) Unknown parameter: count For the records, the command is: curl -i -X POST -H "Content-Type: application/xml" \ -d "<query xmlns='http://www.basex.org/rest'><text>//city/name</text><parameter name='count' value='5'/></query>" \ "admin:admin@localhost:8984/rest/factbook" Regards, -- Florent Georges http://fgeorges.org/ http://h2oconsulting.be/

2 1

Database 'Products' contains more than one document.
by Erol Akarsu 27 Oct '11

27 Oct '11

This database has been working find with previous basex. but I just installed 7.0.1 version and receiving this error. Query: let $doc := fn:doc("Products") return $doc Error: [BASX0008] Database 'Products' contains more than one document. Erol Akarsu

2 3