October 2012 - BaseX-Talk - mailman.uni-konstanz.de

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

Best practices for concurrency
by The Trainspotter 19 Jun '13

19 Jun '13

Hi, I've read through the online example on server concurrency, however, currently I'm not running a BaseXServer instance I'm just running things locally. I want multiple threads in my application to be able to query the database at once without treading on each others toes. What's the best approach for doing this? I have tried querying the database using different threads but I keep getting internal BaseX errors which leads me to think I'm going about this the wrong way. Cheers Gary

5 15

invalid collection name?
by C. M. Sperberg-McQueen 27 May '13

27 May '13

Using the GUI interface to BaseX 7.2, when I select Database / Open and Manage ..., I get a list of the databases I have built over the time I've been using BaseX. When I try to open the collection named MEP.bigbooks, though, I get an error box saying "Name 'MEP.bigbooks' is invalid." In the "Manage Databases" dialog box, the information about the collection appears normal and plausible (at least, the size is about what I expect, and so on). It's only when I try to open the collection that I run into the error. I get similar errors on other collections which have . in their name; I do not get errors when opening other collections built (as far as I can remember) at the same time from the same data, if their name does not include a full stop. When I try renaming the collection without the full stop, I get the same error message. When I try to issue the command ALTER DATABASE MEP.bigbooks MEPbigbooks from the server command area of the GUI, I get a short syntax summary in the Query Info area complaining about the syntax of the command. When I replace MEP.bigbooks with MEP\.bigbooks and "MEP.bigbooks", I also get syntax errors. If I close the GUI and just rename the directory within my BaseXData directory, the database appears to open normally. I think I have two questions: Q1 is just renaming the directory a safe way (assuming no server is currently working with the data) to rename a collection? Should I do the same for all the other collections I have that don't currently open? I assume that if the 7.2 GUI won't let me rename them, it also will have trouble deleting them, so a related question is: is just deleting a sub-directory of BaseXData a safe way to delete a collection? Q2 Did the rules for names of collections change? Or is there some other cause? Speaking for myself, I guess that as a user I haven't thought much about what characters should be allowed in collection names, but I may have assumed that if they are not unrestricted, they would be restricted to being legal NCNames. I do kind of like using dots in names, so you can count this as one vote for relaxing the naming rules at some convenient moment. It's not a really important point, of course. best, Michael -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

2 2

Re: [basex-talk] BUG# Multiple Thread try to execute some Query
by Samuel Lacmene 18 Nov '12

18 Nov '12

2 1

Documents added by db:add(...) not visible for other existing database connections?
by Arjan van Bentem 11 Nov '12

11 Nov '12

Hi all, I noticed that TCP/IP clients are not seeing new documents that are created using db:add(...) in another client: Once a query has been executed, subsequent queries using the very same database connection do not see/return new documents that were meanwhile created using another connection. When creating some (different) document using one's own existing connection, then everything seems to be in-sync for a moment. But right after that, any new documents are again not seen. Updates made in other clients are visible just fine. So, the documents that are returned are up-to-date. It's just that not all documents are retrieved. I can see that in the query plans, the size in <DBNodeSeq size="..."> does not change, and is lower than the size that clients with a newer connection would show. Should it be okay to keep a connection open? Or is the expected behaviour to need a new connection to see new documents? Details below. Thanks, Arjan. 1. Start the server and a client in debugging mode: ./basexserver -d -S ./basexclient -U admin -P admin -d -x 2. Create and open a database: create database db1 open db1 3. Keep the client open, and make another client insert some documents. Like every second for about 2 minutes, each time using a new connection: for i in {1..120}; \ do doc="<test><a>`uuidgen`</a></test>"; \ echo $i, $doc; \ ./basexclient -U admin -P admin "db:add('db1', '$doc', 'dummy')"; \ sleep 1; \ done 4. While Step 3 is running, back in the first client, execute: xquery count(/test) The log in data/.logs shows: REQUEST XQUERY count(/test) OK Query executed in 25.74 ms. 223.77 ms So far, so good. 5. Repeat Step 4 to see the result NOT change, while the loop from Step 3 is still adding new documents. Logs show a faster result, but otherwise are the same. Also the query plans are always the same. (Typically: the size in <DBNodeSeq size="..."> does not change.) REQUEST XQUERY count(/test) OK Query executed in 5.48 ms. 7.29 ms 6. Using "close" followed by "open db1" gets one a more up-to-date value for "count". Also, using the connection to add some random other type of document, updates the value for "count(/test)" as well: xquery db:add('db1', '<another-test><xyz>123</xyz></another-test>', 'dummy') 7. It's not just "count" that does not change its value. Also, one can only query documents up to that count as returned by Step 4. Like if Step 4 returned "15", then the following would return 5 results, not 10: xquery (/test)[position()=10 to 20] Also, the following would only return the count as returned earlier: xquery /test Like noted, UPDATES made in other clients are visible just fine: xquery replace value of node /another-test/xyz with '456' xquery for $n in /test/a return replace value of node $n with fn:generate-id($n) Tested in both 7.3 and 7.5-20120929.201251 from http://files.basex.org/releases/latest/, in Oracle Java 7 on Mac OS X 10.8.2. I first noticed that my running application did not see new documents created by a test script. But the command line tools show the same issue. Any clue?

2 2

Database housekeeping for newbies
by Sam Eklund 05 Nov '12

05 Nov '12

Hey all! I am considering BaseX to hold event information for a broadcast graphics play out system. This system gets a lot of event data from the planning system. Each segment can have up to 12 events, and a day in a typical commercial tv-channel has about 120-160 segments. The commercial breaks not included. Every time a full day of programming is made for a channel, an xml file is produced containing a lot of (badly formatted) EVENT nodes, below a typical "coming next" event: <EVENT IDdeko="109213416" DATE="2012-09-21" CHANNEL_ID="123" VIDEOFORMAT="2"> <SCENARIO ID="29"> <METADATA pos="0" value="00:20"/> <METADATA pos="1" value="The Godfather pt III"/> <METADATA pos="2" value="American drama from 1990 with Al Pacino and Andy Garcia"/> </SCENARIO> </EVENT> I have manually created a database from one of these export xml files and manged to call a BaseXClient dll that I compiled from your examples to retrieve data from the database. All is well with that. Now to my question, since I'm not really familiar with databases in general... Is it possible to set up a "watch folder" for the incoming xml files that will add them to a specified database or would I have to create an application for this? The second question is about housekeeping, since we are handling about 20 channels with this database, the number of events in the database will increase quite fast. Is it possible to specify an xpath search that would match all events with a date more than X days back in time and delete them from the database, i.e delete data from the database without calling the exact file the data was imported with? Best regards SamTheSwede Stockholm

2 4

BaseX Scala Client 0.5 released, this time with documentation
by Manuel Bernhardt 04 Nov '12

04 Nov '12

Hi all, I finally found / forcefully took the time and documented the Scala Client library. The new release mostly brings documentation and uses BaseX 7.3. We have been using it for 5 months now and it works nicely. One interesting feature if you're dealing with queries that return large quantities of documents (and use Scala...) is the StreamingClientSession which doesn't cache incoming results but directly passes them on to be consumed by the client code. Check it out here: https://github.com/delving/basex-scala-client Cheers, Manuel

4 6

Setting User-Agent for http:send-request?
by Erol Akarsu 01 Nov '12

01 Nov '12

Hi, I am trying to set User-Agent header attribute for an HTTP request with following code snippet. let $content := http:send-request(<http:request method='get' status-only='false' User-Agent='Mozilla/5.0'/>, $url) let $body := $content[2] return $body But It is sending "Java/1.7.0_02" as User-Agent header attribute. This has been taken from wireshark: GET / HTTP/1.1 User-Agent: Java/1.7.0_02 Host: istanbul.olx.com.tr Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Connection: keep-alive Do we have any way to set the user-agent attribute? Thanks Erol Akarsu

2 3

Calling static Java function from XQuery
by The Trainspotter 01 Nov '12

01 Nov '12

Hi, I followed the docs and examples on calling Java functions from XQuery, but when I try and add a package to the repository that calls one of my own functions BaseX complains that it can't find the function. I am running BaseX locally and the class I want to call is in my project's classpath. Any ideas as to what I'm doing wrong? Cheers Gary

2 7

Re: [basex-talk] Database housekeeping for newbies
by pw＠themail.co.uk 31 Oct '12

31 Oct '12

Think about using the Mule ESB [ http://www.mulesoft.com/mule-esb-open-source-esb ] or similar and then feed in to webservices from the service bus Peter ---- Original Message ---- From: gaerber(a)axxepta.de To: basex-talk(a)mailman.uni-konstanz.de Subject: Re: [basex-talk] Database housekeeping for newbies Date: Wed, 31 Oct 2012 15:00:09 +0100 >Hi, > >I don't know of any hot-folder mechanism in basex but you might be >interested in the webdav api: http://docs.basex.org/wiki/WebDAV > >Concerning the housekeeping, I'd create new databases containing a >defined time period (e.g. one week or so). You can query several >databases at once, so you always have the option to: > >- query over single or multiple time periods >- drop a complete time period > >Changing a lot of nodes (or deleting) inside a single database is >sometimes slow. But that might be due the structure of my data. > >Regards, > >Max > > >2012/10/31 Sam Eklund <sam.eklund(a)gluebox.se>: >> Hey all! >> >> >> >> I am considering BaseX to hold event information for a broadcast >graphics >> play out system. This system gets a lot of event data from the >planning >> system. Each segment can have up to 12 events, and a day in a >typical >> commercial tv-channel has about 120-160 segments. The commercial >breaks not >> included. >> >> >> >> Every time a full day of programming is made for a channel, an xml >file is >> produced containing a lot of (badly formatted) EVENT nodes, below a >typical >> coming next event: >> >> >> >> <EVENT IDdeko="109213416" DATE="2012-09-21" CHANNEL_ID="123" >> VIDEOFORMAT="2"> >> >> <SCENARIO ID="29"> >> >> <METADATA pos="0" value="00:20"/> >> >> <METADATA pos="1" value="The Godfather pt III"/> >> >> <METADATA pos="2" value="American drama from 1990 with Al >Pacino and >> Andy Garcia"/> >> >> </SCENARIO> >> >> </EVENT> >> >> >> >> I have manually created a database from one of these export xml >files and >> manged to call a BaseXClient dll that I compiled from your examples >to >> retrieve data from the database. All is well with that. Now to my >question, >> since Im not really familiar with databases in general >> >> >> >> Is it possible to set up a watch folder for the incoming xml >files that >> will add them to a specified database or would I have to create an >> application for this? >> >> >> >> The second question is about housekeeping, since we are handling >about 20 >> channels with this database, the number of events in the database >will >> increase quite fast. Is it possible to specify an xpath search that >would >> match all events with a date more than X days back in time and >delete them >> from the database, i.e delete data from the database without >calling the >> exact file the data was imported with? >> >> >> >> >> >> Best regards >> >> >> >> SamTheSwede >> >> Stockholm >> >> >> >> >> >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk(a)mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> > > > >-- > >Maximilian Gärber > >axxepta solutions GmbH >Postfach 51 02 38 >13362 Berlin > >Tel +49 (0)30 499 147 66 >Fax +49 (0)30 499 147 67 >Mail gaerber(a)axxepta.de >_______________________________________________ >BaseX-Talk mailing list >BaseX-Talk(a)mailman.uni-konstanz.de >https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

1 0