March 2012 - BaseX-Talk - mailman.uni-konstanz.de

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

Empty Initial Document Inconsistencies
by Dave Glick 05 Jul '12

05 Jul '12

Hello again, I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. Dave

4 7

deadlock in BaseXServer 6.2.7
by Godmar Back 23 Jun '12

23 Jun '12

Hi, we're observing that our BaseXServer 6.2.7 deadlocks. The stack traces are here: http://top.cs.vt.edu/~gback/bx/deadlock-6.2.7/basex-deadlock.txt Could you take a look to see if these ring any bells? Has this problem been addressed in a more recent version? - Godmar

3 5

Detecting abandoned locks
by Charles Duffy 09 Apr '12

09 Apr '12

Howdy -- I have a BaseX database with a pin-* file in its directory, reporting as locked and unavailable. However, OS-level tools such as fuser report that no programs have open file handles for any processes in this directory. What mechanism does BaseX use for detecting abandoned locks? It would be nice to use OS-level advisory locking where available, such that the lock is automatically released whenever the program holding it exits (or otherwise ceases to hold a handle on the file).

3 5

Concurrency problems with REST API
by Tobias Hofer 01 Apr '12

01 Apr '12

Hi, I'm using the REST API of BaseX 7.2 and run into concurrency problems. If I add new documents in parallel then the server will stop working after adding a few documents. My command template looks like: <?xml version="1.0" encoding="UTF-8"?> <command xmlns="http://basex.org/rest"><text><![CDATA[ ADD TO $KEY$ <node id="$KEY$" /> ]]></text></command> I call the REST API using curl: for ((i=0; i<10; i++)); do (cp tx-template tx$i; replace '$KEY$' $i -- tx$i; curl -i -X POST -H "Content-Type: application/xml" -T tx$i "admin:admin@localhost:8984/rest/QDEV") & done After four or less insert the following error occurs: HTTP/1.1 400 Bad Request Content-Type: text/plain; charset=utf-8 Content-Length: 2247 Server: Jetty(6.1.26) java.lang.NullPointerExceptionjava.lang.NullPointerExceptionjava.lang.NullPointerExceptionorg.xml.sax.SAXParseException: cvc-type.3.1.2: Element 'command' is a simple type, so it must have no element information item [children].org.xml.sax.SAXParseException: cvc-type.3.1.2: Element 'command' is a simple type, so it must have no element information item [children].Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.2 Java: Sun Microsystems Inc., 1.6.0_26 OS: Linux, amd64 Stack Trace: java.lang.NullPointerException org.basex.core.cmd.Add.run(Add.java:92) org.basex.core.Command.run(Command.java:345) org.basex.core.Command.exec(Command.java:324) org.basex.core.Command.execute(Command.java:76) org.basex.server.LocalSession.execute(LocalSession.java:134) org.basex.server.LocalSession.execute(LocalSession.java:126) org.basex.server.Session.execute(Session.java:51) org.basex.http.rest.RESTCommand.run(RESTCommand.java:40) org.basex.http.rest.RESTPost.run(RESTPost.java:129) org.basex.http.rest.RESTServlet.run(RESTServlet.java:14) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49) org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) org.mortbay.jetty.Server.handle(Server.java:326) org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) Now, the server cannot be accessed any more. Any further call will return "Database 'MYDB' is currently opened by another client. A restart helps. The error does not happen if the commands are executed in sequence. Please help. Regards, Tobias Hofer

2 1

possible bug? or opportunity for documentation enhancement? [2d sending]
by C. M. Sperberg-McQueen 01 Apr '12

01 Apr '12

[2d posting - this time with the promised attachment] I got some error messages today which seem to indicate a problem, either in BaseX or in (my understanding of) the documentation. Since the error messages (in the attached log) suggest contacting you about it, I am doing so. The document at http://www.w3.org/TR/xsd-unicode-blocknames/ includes a table of Unicode block names created by querying a database consisting of XML representations of the various versions of the Unicode Database. Today I noticed that a new version of the Unicode Database was issued in January, and so I downloaded the XML data and added the document to my existing BaseX database of Unicode information. The command seemed to run successfully, but when I ran my old query, I got an error message saying "Data access out of bounds". I installed version 7.2 then, which gave the same error message. Rebuilding the database from scratch allowed me to run the query without any problems, so I don't actually have a problem at the moment. But I'm sending this problem report in the hopes that it may help you diagnose either a problem with dynamic adds to a database or perhaps a problem with the server command language documentation (it's entirely possible that I did something wrong when adding the new document to the database). If there is any information I can provide that would be helpful, please let me know. best, Michael Sperberg-McQueen -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

2 1

possible bug? or opportunity for documentation enhancement?
by C. M. Sperberg-McQueen 30 Mar '12

30 Mar '12

I got some error messages today which seem to indicate a problem, either in BaseX or in (my understanding of) the documentation. Since the error messages (in the attached log) suggest contacting you about it, I am doing so. The document at http://www.w3.org/TR/xsd-unicode-blocknames/ includes a table of Unicode block names created by querying a database consisting of XML representations of the various versions of the Unicode Database. Today I noticed that a new version of the Unicode Database was issued in January, and so I downloaded the XML data and added the document to my existing BaseX database of Unicode information. The command seemed to run successfully, but when I ran my old query, I got an error message saying "Data access out of bounds". I installed version 7.2 then, which gave the same error message. Rebuilding the database from scratch allowed me to run the query without any problems, so I don't actually have a problem at the moment. But I'm sending this problem report in the hopes that it may help you diagnose either a problem with dynamic adds to a database or perhaps a problem with the server command language documentation (it's entirely possible that I did something wrong when adding the new document to the database). If there is any information I can provide that would be helpful, please let me know. best, Michael Sperberg-McQueen -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

1 0

Module import times and caching
by Charles Duffy 30 Mar '12

30 Mar '12

import module namespace reporting="http://dyfis.net/reporting"; 1+1 Timing: - Parsing: 4.33 ms - Compiling: 4205.89 ms <<< (!) - Evaluating: 7.55 ms - Printing: 6.39 ms - Total Time: 4224.18 ms ...after some analysis and optimization this is down to ~115ms, but that's still a fairly high floor for query execution time of anything using this module (which almost everything I write does). Said module includes several lines of the form declare variable $reporting:foo_db := db:open("foo", "db") ...and a lot of function declarations, some of which are fairly complex. Have there been thoughts to a module cache, even if enabled only via pragma? It would be nice to be able to reuse the optimizer's results and variable definitions (even if we completely expired the cache on writes to any database referenced during the module's compilation phase).

1 0

Java interop - Unexpected FORG0006 ("Invalid arguments")
by Charles Duffy 30 Mar '12

30 Mar '12

Howdy -- I have a thin wrapper around the 3rd-party svnkit library. At present, it provides two functions: module namespace svnkit = "http://dyfis.net/svnkit-wrapper"; declare namespace j="java:net.dyfis.svnkit_wrapper.SvnWrapper"; declare function svnkit:cat($path as xs:string) as xs:string { j:cat($path) }; declare function svnkit:cat($path as xs:string, $rev as xs:integer) as xs:string { j:cat($path, $rev) }; SvnWrapper (written in Clojure) provides both versions of cat as static methods: (ns net.dyfis.svnkit-wrapper.core (:gen-class :name net.dyfis.svnkit_wrapper.SvnWrapper :methods [#^{:static true} [cat [java.lang.String] java.lang.String] #^{:static true} [cat [java.lang.String int] java.lang.String]])) svnkit:cat($path) works fine. svnkit:cat($path, $revno) results in the following: [FORG0006] Invalid arguments: SvnWrapper.cat(xs:string, xs:integer) found. SvnWrapper.cat(xs:string, xs:integer) is exactly what I *expect* it to find, so the error doesn't make a great deal of sense to me. Any hints?

2 3

BaseX XQuery vs. python / lxml performance
by Ronny Möbius 30 Mar '12

30 Mar '12

Hi, we use now for a couple of years XML files for organizing the schedule of lectures in our institute. The amount data for one semester is about one megabyte. For presenting the schedule on the web we use currently python in combination with the C-based lxml XPath implementation. Only basic selecting nodes happens on the layer of lxml, but the most of intelligence is implemented in filtering python lists and lists of dictionaries. We read the xml data from hard disk and parse them into lxml objects. >From time to time we thought about storing the data in a XML database as this seems intuitive. One main point is, that we need some sort of transaction management, because different applications may manipulate the files simultaneously. Now I gave BaseX a try and implemented one basic output of our online schedule in XQuery. Soon I noticed, that all of the intelligence should be managed inside XQuery as it should return the full-prepared html for presenting online. Unfortunately we now have the impression, that we did not gain speed - to the contrary the query itself needs more execution time than the whole corresponding ready interface (including the user interface etc.) I'm now interested in your general opinion about this: Is it surprising, that the XQuery implementation than the lxml/python one (For me it is, as I thought the indices etc. created when importing the data should decrease computational affords in searching the tree)? Is there some catch in my approach? May the reason be bad designing the query? I would appreciate to hear from you, Ronny

5 7