September 2012 - BaseX-Talk - mailman.uni-konstanz.de

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

invalid collection name?
by C. M. Sperberg-McQueen 27 May '13

27 May '13

Using the GUI interface to BaseX 7.2, when I select Database / Open and Manage ..., I get a list of the databases I have built over the time I've been using BaseX. When I try to open the collection named MEP.bigbooks, though, I get an error box saying "Name 'MEP.bigbooks' is invalid." In the "Manage Databases" dialog box, the information about the collection appears normal and plausible (at least, the size is about what I expect, and so on). It's only when I try to open the collection that I run into the error. I get similar errors on other collections which have . in their name; I do not get errors when opening other collections built (as far as I can remember) at the same time from the same data, if their name does not include a full stop. When I try renaming the collection without the full stop, I get the same error message. When I try to issue the command ALTER DATABASE MEP.bigbooks MEPbigbooks from the server command area of the GUI, I get a short syntax summary in the Query Info area complaining about the syntax of the command. When I replace MEP.bigbooks with MEP\.bigbooks and "MEP.bigbooks", I also get syntax errors. If I close the GUI and just rename the directory within my BaseXData directory, the database appears to open normally. I think I have two questions: Q1 is just renaming the directory a safe way (assuming no server is currently working with the data) to rename a collection? Should I do the same for all the other collections I have that don't currently open? I assume that if the 7.2 GUI won't let me rename them, it also will have trouble deleting them, so a related question is: is just deleting a sub-directory of BaseXData a safe way to delete a collection? Q2 Did the rules for names of collections change? Or is there some other cause? Speaking for myself, I guess that as a user I haven't thought much about what characters should be allowed in collection names, but I may have assumed that if they are not unrestricted, they would be restricted to being legal NCNames. I do kind of like using dots in names, so you can count this as one vote for relaxing the naming rules at some convenient moment. It's not a really important point, of course. best, Michael -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

2 2

Documents added by db:add(...) not visible for other existing database connections?
by Arjan van Bentem 11 Nov '12

11 Nov '12

Hi all, I noticed that TCP/IP clients are not seeing new documents that are created using db:add(...) in another client: Once a query has been executed, subsequent queries using the very same database connection do not see/return new documents that were meanwhile created using another connection. When creating some (different) document using one's own existing connection, then everything seems to be in-sync for a moment. But right after that, any new documents are again not seen. Updates made in other clients are visible just fine. So, the documents that are returned are up-to-date. It's just that not all documents are retrieved. I can see that in the query plans, the size in <DBNodeSeq size="..."> does not change, and is lower than the size that clients with a newer connection would show. Should it be okay to keep a connection open? Or is the expected behaviour to need a new connection to see new documents? Details below. Thanks, Arjan. 1. Start the server and a client in debugging mode: ./basexserver -d -S ./basexclient -U admin -P admin -d -x 2. Create and open a database: create database db1 open db1 3. Keep the client open, and make another client insert some documents. Like every second for about 2 minutes, each time using a new connection: for i in {1..120}; \ do doc="<test><a>`uuidgen`</a></test>"; \ echo $i, $doc; \ ./basexclient -U admin -P admin "db:add('db1', '$doc', 'dummy')"; \ sleep 1; \ done 4. While Step 3 is running, back in the first client, execute: xquery count(/test) The log in data/.logs shows: REQUEST XQUERY count(/test) OK Query executed in 25.74 ms. 223.77 ms So far, so good. 5. Repeat Step 4 to see the result NOT change, while the loop from Step 3 is still adding new documents. Logs show a faster result, but otherwise are the same. Also the query plans are always the same. (Typically: the size in <DBNodeSeq size="..."> does not change.) REQUEST XQUERY count(/test) OK Query executed in 5.48 ms. 7.29 ms 6. Using "close" followed by "open db1" gets one a more up-to-date value for "count". Also, using the connection to add some random other type of document, updates the value for "count(/test)" as well: xquery db:add('db1', '<another-test><xyz>123</xyz></another-test>', 'dummy') 7. It's not just "count" that does not change its value. Also, one can only query documents up to that count as returned by Step 4. Like if Step 4 returned "15", then the following would return 5 results, not 10: xquery (/test)[position()=10 to 20] Also, the following would only return the count as returned earlier: xquery /test Like noted, UPDATES made in other clients are visible just fine: xquery replace value of node /another-test/xyz with '456' xquery for $n in /test/a return replace value of node $n with fn:generate-id($n) Tested in both 7.3 and 7.5-20120929.201251 from http://files.basex.org/releases/latest/, in Oracle Java 7 on Mac OS X 10.8.2. I first noticed that my running application did not see new documents created by a test script. But the command line tools show the same issue. Any clue?

2 2

RegExp Bug S{n,m}
by Andy Bunce 24 Oct '12

24 Oct '12

Hi This expression: replace("aaaa bbbbbbbb ddd ",'(.{6,15}) ','$1@') results in [FORX0002] Invalid pattern: 'Illegal quantifier, lower > upper bound: {6,1}'. Could it be that *S*{n,m} is only accepting single digit n and m? This causes http://stackoverflow.com/questions/3767452/xquery-word-wrap to wildly overwrap. Regards /Andy

2 2

XQJ: Exception using Saxon9 TransformerFactoryImpl
by David Oguns 06 Oct '12

06 Oct '12

I am trying to perform a basic document insert using the XQJ API communicating with a standalone BaseXServer. I have a lot of dependencies on my classpath (I'm sure this affects JAXP's behavior), including Saxon, Xerces, Xalan, and probably a few versions of some of them. When I set the transformer factor explicitly to Xalan, there is no exception, but otherwise it appears Saxon is being used and it down throw an exception. I have a very high opinion of Saxon so I have my doubts that it's doing anything illegal with respect to the standards, so I thought I should ask what is going on. This is the property I can set to get rid of the problem: System.setProperty("javax.xml.transform.TransformerFactory", "org.apache.xalan.processor.TransformerFactoryImpl"); Here is the exception and stack trace: javax.xml.xquery.XQException: XQJTO013 - Values of 'standalone' and 'omit-xml-declaration' conflict Root Cause: net.sf.saxon.trans.XPathException: Values of 'standalone' and 'omit-xml-declaration' conflict at MyClass.main(MyClass.java:82) at net.sf.saxon.serialize.XMLEmitter.writeDeclaration(XMLEmitter.java:226) at net.sf.saxon.serialize.XMLEmitter.openDocument(XMLEmitter.java:143) at net.sf.saxon.serialize.XMLEmitter.startElement(XMLEmitter.java:303) at net.sf.saxon.event.NamespaceReducer.startElement( NamespaceReducer.java:72) at net.sf.saxon.event.ReceivingContentHandler.startElement( ReceivingContentHandler.java:290) at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at net.sf.saxon.event.Sender.sendSAXSource(Sender.java:400) at net.sf.saxon.event.Sender.send(Sender.java:154) at net.sf.saxon.IdentityTransformer.transform(IdentityTransformer.java:35) at MyClass.main(MyClass.java:82) And here is the relevant code: XQDataSource xqs = new BaseXXQDataSource(); xqs.setProperty("serverName", "localhost"); xqs.setProperty("port", "1984"); XQConnection conn = xqs.getConnection("admin", "admin"); try { XQStaticContext context = conn.getStaticContext(); XMLStreamReader xmlReader = XMLInputFactory.newInstance().createXMLStreamReader(newFileReader(args[0])); XQPreparedExpression xqpe = conn.prepareExpression("declare namespace output = \"http://www.w3.org/2010/xslt-xquery-serialization\";declare option output:omit-xml-declaration \"no\";declare variable $doc external;\nfn:put($doc,\"jc003.xml\")", context); //xquery doc('oguns/jc001.xml') try{ xqpe.bindDocument(new QName("doc"), xmlReader, conn.createDocumentType()); XQResultSequence xqrs = xqpe.executeQuery(); } finally { xqpe.close(); } } finally { conn.close(); }

3 4

Silent installation of BaseX on Windows7
by Christian Steinherr 28 Sep '12

28 Sep '12

Dear BaseX Users, i just wanted to ask, if anyone tried to do a silent installation. I'm in charge of computer lab at the university and i was asked to install basex on all of the win7-pcs. Is there a hidden option, where all nessescary parameters can added at the beginning? Often Programms come with an option for silent or unattended installation. If you know such, please tell me, it would help me a lot. Kind regards christian

2 1

Keeping track of XSD versions? And multiple/single huge documents, or many smaller documents?
by Arjan van Bentem 28 Sep '12

28 Sep '12

Hi all, We're planning to use an XML database for a model that we know will change often over time. Like today a <User> might just have a <Name> and <Address>, but in the future we might also require <Telephone>. We will also be updating the data itself a lot, using XQuery Update through the xqj.net drivers. I have read [1] that the number of distinct namespace URIs in BaseX is limited to 256. We were thinking to keep keep track of "releases" of our XSDs through versioning of namespaces, but then 256 seems too low a number. - Any smart ideas about this? Also, I wonder if it would be better to use many small documents, or use a single document that hold all the nodes. Like: <Users> with many <User> nodes, or just many <User> documents. And many <Review> documents, or a single <Reviews> holding many <Review> nodes. Or even one big <Data> document to hold both <Users> and <Reviews>. One reason to use many documents is because, as described above, we will have different versions/releases, which seems easier to maintain in multiple documents. I've seen Christian's research [2], but that does not cover updates. So: - Does anyone know of real life examples of one approach, or the other? - Would updating a small document be faster than updating a node in a larger document? - Would it matter if we update multiple documents, like both a <User> and a <Review>, in a single XQJ statement? Also, one of the developers of eXist-db once wrote [3]: > [...] there are certainly some practical considerations. Storing a very > large document can block the db more than storing many small ones. > It requires a single transaction (and sufficient disk space for the > transaction log). > > [...] I know other users have to deal with much larger documents (many > gigabytes), but if you have a choice, I would definitely recommend to > split your data in smaller chunks, which are easier to handle. - Does anyone know if the same applies to BaseX? Any (other) input appreciated! Thanks, Arjan. [1] http://docs.basex.org/wiki/Statistics [2] http://kops.ub.uni-konstanz.de/bitstream/handle/urn:nbn:de:bsz:352-opus-127… [3] http://exist-open.markmail.org/thread/byuzpoj7cyfmlqdo

1 0

Re: [basex-talk] BUG# Multiple Thread try to execute some Query
by Samuel Lacmene 28 Sep '12

28 Sep '12

3 2

User rights: does using db:add really need the GLOBAL write permission?
by Arjan van Bentem 27 Sep '12

27 Sep '12

Hi all, In 7.3, I am using db:add (through a XQJ XQPreparedExpression) to add documents to a database. For a BaseX user to do that, I need to grant that database user the GLOBAL write permissions. With just LOCAL write permissions, I get: javax.xml.xquery.XQQueryException: [BASX0001]: WRITE permission required. Is this the expected behaviour? This is troublesome, as the GLOBAL write permission also gives that user read and write permissions on ALL other databases, unless these permissions are explicitly revoked. So, as a workaround, I can grant all user GLOBAL write permissions, and then explicitly deny all users LOCAL permissions to all existing databases that are not theirs. (Reading is not a problem with just local permissions. And as an aside: using "-i" on the command line even seems to require global CREATE permissions. But I don't really use that.) Any thoughts? Details below. Arjan. # 1. Create database and a user, with password "secret": ./basexclient -U admin -P admin > create db db1 Database 'db1' created in 11.05 ms. > create user user1 5ebe2294ecd0e0f08eab7690d2a6ee69 User 'user1' created. > grant write on db1 to user1 WRITE granted to 'user1' on 'db1'. > show users on db1 Username Read Write --------------------- user1 X X 1 Users. # 2. Let's this user access their own database. # Odd, using the command line "-i" even needs CREATE permissions? ./basexclient -U user1 -P secret -i db1 CREATE permission needed. # 3. Let's ignore that for now; using "open" works just fine: ./basexclient -U user1 -P secret > open db1 Database 'db1' was opened in 0.91 ms. > xquery db:add("db1", "<test><a>created by user1</a></test>", "test-doc-1") Stopped at line 1, column 67: [BASX0001] WRITE permission required. # 4. Okay, let's give the user the GLOBAL write permissions ./basexclient -U admin -P admin > grant write to user1 WRITE granted to 'user1'. > show users Username Read Write Create Admin ------------------------------------- admin X X X X user1 X X 2 Users. # 5. All fine for user1 now: ./basexclient -U user1 -P secret > xquery db:add("db1", "<test><a>created by user1</a></test>", "test-doc-1") Query executed in 6.18 ms. # 6. Let's create another user, with their own database, and the GLOBAL write as well: ./basexclient -U admin -P admin > create db db2 Database 'db2' created in 5.09 ms. > create user user2 5ebe2294ecd0e0f08eab7690d2a6ee69 User 'user2' created. > grant write to user2 WRITE granted to 'user2'. > grant write on db2 to user2 WRITE granted to 'user2' on 'db2'. > show users Username Read Write Create Admin ------------------------------------- admin X X X X user1 X X user2 X X 3 Users. > show users on db1 Username Read Write --------------------- user1 X X 1 Users. > show users on db2 Username Read Write --------------------- user2 X X 1 Users. # 7. And prove that user2 can now access db1, due to the global WRITE permissions: > open db1 Database 'db1' was opened in 1.9 ms. > xquery db:add("db1", "<test><a>created by user2 in db1</a></test>", "test-doc-2") Query executed in 1.27 ms. > xquery / <test> <a>created by user1</a> </test><test> <a>created by user2 in db1</a> </test> Query executed in 0.72 ms. # 8. That's bad. Let's explicitly revoke permissions from user2: > grant none on db1 to user2 NONE granted to 'user2' on 'db1'. > show users on db1 Username Read Write --------------------- user1 X X user2 2 Users. # 9. And have that user try again: ./basexclient -U user2 -P secret > open db1 READ permission needed. > xquery db:add("db1", "<test><a>created by user2 in db1</a></test>", "test-doc-3") Stopped at line 1, column 74: [BXDB0002] READ permission needed. # That's more like it. But: what when I forget that one day...? http://docs.basex.org/wiki/User_Management http://docs.basex.org/wiki/Database_Module#Updates

2 1

Adding a XML and fulltext index is gone
by Abhishek Jain 27 Sep '12

27 Sep '12

Hi Fellas, We are researching Basex for our upcoming solution to one of our customers. We are required to search and add XML documents to the data set. Just to brief you on the amount of data we receive around 8 million XML message per customer for a month. These set of data needs to be quickly searched and at the same time to expand it on receiving messages. I have created a fulltext index on the database. But as soon as I do insert a data item the fulltext index is gone and my searches are slowed down. How shall I proceed forward on this thread? Please help !! Regards, Abhishek

2 1