June 2012 - BaseX-Talk - mailman.uni-konstanz.de

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

invalid collection name?
by C. M. Sperberg-McQueen 27 May '13

27 May '13

Using the GUI interface to BaseX 7.2, when I select Database / Open and Manage ..., I get a list of the databases I have built over the time I've been using BaseX. When I try to open the collection named MEP.bigbooks, though, I get an error box saying "Name 'MEP.bigbooks' is invalid." In the "Manage Databases" dialog box, the information about the collection appears normal and plausible (at least, the size is about what I expect, and so on). It's only when I try to open the collection that I run into the error. I get similar errors on other collections which have . in their name; I do not get errors when opening other collections built (as far as I can remember) at the same time from the same data, if their name does not include a full stop. When I try renaming the collection without the full stop, I get the same error message. When I try to issue the command ALTER DATABASE MEP.bigbooks MEPbigbooks from the server command area of the GUI, I get a short syntax summary in the Query Info area complaining about the syntax of the command. When I replace MEP.bigbooks with MEP\.bigbooks and "MEP.bigbooks", I also get syntax errors. If I close the GUI and just rename the directory within my BaseXData directory, the database appears to open normally. I think I have two questions: Q1 is just renaming the directory a safe way (assuming no server is currently working with the data) to rename a collection? Should I do the same for all the other collections I have that don't currently open? I assume that if the 7.2 GUI won't let me rename them, it also will have trouble deleting them, so a related question is: is just deleting a sub-directory of BaseXData a safe way to delete a collection? Q2 Did the rules for names of collections change? Or is there some other cause? Speaking for myself, I guess that as a user I haven't thought much about what characters should be allowed in collection names, but I may have assumed that if they are not unrestricted, they would be restricted to being legal NCNames. I do kind of like using dots in names, so you can count this as one vote for relaxing the naming rules at some convenient moment. It's not a really important point, of course. best, Michael -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

2 2

Performance of Add command
by Manuel Bernhardt 09 Jul '12

09 Jul '12

Hi, I'm doing some testing before migration one of our customers to a new version of our platform that uses BaseX in order to store documents. They have approx. 4M documents, and I'm running an import operation on a 1 M document collection on my laptop. The way I'm inserting documents is by firing off one Add command per document, based on a stream of the document, at a different (unique) path for each document, and flushing every at 10K Adds. Since most CPU usage (for one of the cores, the other ones being untouched) is taken by the BaseX server, I fired up YourKit out of curiosity to see where the CPU time was spent. My machine is a 2*4 core MacBook Pro with 8GB of RAM and SSD, so I think hardware-wise it should do pretty fine. YourKit shows that what seems to use up most time is the Namespaces.update method: Thread-12 [RUNNABLE] CPU time: 2h 7m 9s org.basex.data.Namespaces.update(NSNode, int, int, boolean, Set) org.basex.data.Namespaces.update(int, int, boolean, Set) org.basex.data.Data.insert(int, int, Data) org.basex.core.cmd.Add.run() org.basex.core.Command.run(Context, OutputStream) org.basex.core.Command.exec(Context, OutputStream) org.basex.core.Command.execute(Context, OutputStream) org.basex.core.Command.execute(Context) org.basex.server.ClientListener.execute(Command) org.basex.server.ClientListener.add() org.basex.server.ClientListener.run() I'm not really sure what that method does - it's a recursive function and seems to be triggered by Data.insert: // NSNodes have to be checked for pre value shifts after insert nspaces.update(ipre, dsize, true, newNodes); The whole set of records should have no more than 5 different namespaces in total. Thus I'm wondering if there would perhaps be some potential for optimization here? Note that I'm completely ignorant as to what the method does and what its exact purpose is. Thanks, Manuel PS: the import is now finished: Storing 1001712 records into BaseX took 9285008 ms

2 11

BaseX client event protocol
by Andy Bunce 08 Jul '12

08 Jul '12

Hi, I am looking at adding WATCH and UNWATCH to my Node.js BaseX interface and I have a couple of questions before I start. 1) The documentation at http://docs.basex.org/wiki/Server_Protocol shows nothing in the Server Response column. This seems not to be the case. It looks like a standard BaseX protocol type response is to be expected? 2) The example at https://github.com/BaseXdb/basex-examples/blob/master/src/main/java/org/bas… session2 watching and then executing. Does the server protocol identify "event pushes" in some way. That is when I receive a message how can I tell if is a event or a command response? /Andy

2 3

Performance regression from 7.2 to 7.3 ?
by Nicolas Labrot 07 Jul '12

07 Jul '12

Hello, I have upgraded BaseX from 7.2 to 7.3 but I have a severe performance drop on my query. My query is in 2 parts : - The first part extracts values from an xpath and store them in a map (the map contains 40000 entries) - The second part extracts values from an xpath and test if they are contains in the map (xpath return around 30000 entries). On 7.2 a typical query run on 2s and on 7.3 I have no result after 10 minutes Is there modification on BaseX 7.3 which can explain this performance drop ? Thanks for your help, Regards, Nicolas

2 6

Empty Initial Document Inconsistencies
by Dave Glick 05 Jul '12

05 Jul '12

Hello again, I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. Dave

4 7

normalize-unicode function
by Toshio HIRAI 02 Jul '12

02 Jul '12

Hi, I tried some Japanese letters in fn:normalize-unicode. FULLWIDTH DIGIT, FULLWIDTH LATIN CAPITAL LETTER: fn:normalize-unicode("１２３４５６７８９０ａｂｃｄｅｆｇ", "NFKC") returns 1234567890abcdefg FULLWIDTH EXCLAMATION MARK: fn:normalize-unicode("！", "NFKC") returns ! FULLWIDTH LESS-THAN SIGN, ?FULLWIDTH GREATER-THAN SIGN: fn:normalize-unicode("＜＞", "NFKC") returns <> FULLWIDTH AMPERSAND; fn:normalize-unicode("＆", "NFKC") returns?& These are normal. but, FULLWIDTH QUOTATION MARK, FULLWIDTH APOSTROPHE: fn:normalize-unicode("”’", "NFKC") returns ”’ In this case, I think that "' should be returned. Best regards, Toshio HIRAI

3 5

REST service problems - 401-Header sent
by MyNew EMail 02 Jul '12

02 Jul '12

Hi, I am using the REST service of BaseX. I'm using POST with the following body to send requests (XPath/XQuery): <query xmlns="http://basex.org/rest"> <text><![CDATA[ (//city/name)[position() <= 5] ]]></text></query> (it's the example from the documentation) Most of the time it works well, but when the XQuery request is to long I get a 401 status code (not authenticated) while I am authenticated. Is there any solution for this problem? Thanks @all

2 1

Full-text search and mixed content
by Michael Piotrowski 27 Jun '12

27 Jun '12

Hi, In "XQuery and XPath Full Text 1.0 Use Cases" [1] it says: Querying across element boundaries is similar to an XQuery and XPath character string function converting the sub-tree under an element into a string by removing all markup. However, I'm having trouble to get this to work in BaseX 7.2.1. For example, given this document: <doc> Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet </doc> How do I query, for example, for p elements that contain the string "sit amet"? //p[. contains text 'sit amet'] returns only paragraphs 1 and 2. I've tried a number of variations, but I've been unable to come up with a query that returns all four p elements. Are my queries incorrect or is this a bug in BaseX? Thanks and best regards Footnotes: [1] <http://www.w3.org/TR/xpath-full-text-10-use-cases/> -- Dr.-Ing. Michael Piotrowski, M.A. <mxp(a)cl.uzh.ch> Institute of Computational Linguistics, University of Zurich Phone +41 44 63-54313 | OpenPGP public key ID 0x1614A044 * OUT NOW: Systems and Frameworks for Computational Morphology * <http://www.springeronline.com/978-3-642-23137-7>

5 27

how to formulate an excluding xquery?
by Cerstin Mahlow 27 Jun '12

27 Jun '12

Hi, in my project, users look for text nodes containing certain words. However, some of these queries result in a very large number of hits, with a lot of irrelevant ones. Is it possible to search with excluding xqueries? For example in this document A B C D E F A B D E F C A B D E F A B D F when looking for [text() contains contains text "A" ftand "B" ftand "E" ordered distance at most 3 words] I would like to make explicit, that "C" should not occur at all, to ensure to find only the third node, not the first nor second one -- i.e., "C" should not occur outside the sequence, nor inside the sequence. Is this possible at all? Can I also specify a query where "C" would be allowed to occur outside the sequence, like in the second node, but not inside like in the first node? Or the other way around? Best regards Cerstin -- Dr. phil. Cerstin Mahlow Universität Basel Departement Sprach- und Literaturwissenschaften Fachbereich Deutsche Sprach- und Literaturwissenschaft Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow(a)unibas.ch Web: http://www.oldphras.net ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

2 4