November 2012 - BaseX-Talk - mailman.uni-konstanz.de

Out of memory bug
by Ivan Lagunov 31 Jul '13

31 Jul '13

Hello, I've got 'Out of memory' error and corrupted database again, this time it's for 6.3 version. Steps: 1. Started server, started terminal client, checked there are no text and attribute indexes. 2. Started my Java application that runs several tests on database: querying, adding, removing. 3. Then I forgot to reopen terminal client and used the one that I opened in Step 1. 4. Run optimize command for the collection and got error. 5. Cannot open the collection any more as I'm getting 'Out of memory'. It's clear that I shouldn't have used the terminal client that I opened before execution of my Java application. But sometimes it's hard to remember. And it's completely unacceptable that these actions result in the corrupted database. You really should catch ArrayIndexOutOfBoundsException everywhere in order to save database from being corrupted. Terminal client stacktrace: > optimize Possible bug? Feedback is welcome: basex-talk(a)mailman.uni-konstanz.de BaseX 6.3: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.data.PathSummary.add(PathSummary.java:68) org.basex.core.cmd.Optimize.stats(Optimize.java:68) org.basex.core.cmd.Optimize.run(Optimize.java:34) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) > open products Out of Main Memory. Best regards, Ivan

4 8

indent parameter
by Hans-Juergen Rennau 28 Jun '13

28 Jun '13

Hello, it seems to me there is a couple of issues concerning the handling of indentation (serialization parameter "indent"). a) Using "indent=no" whitespace-only text nodes between element nodes are removed b) Using "indent=yes" the content of elements with xml:space="preserve" is indented, too I suppose both behaviors are in contraction to the spec ( http://www.w3.org/TR/xslt-xquery-serialization/#xml-indent ). As a particular consequence, (a) seems to preclude making an exact copy of a document (fragment) - all whitespace-only text nodes vanish. And as a consequence of (b) one has no possibility to locally protect the whitespace used in the source document, as required for example when reusing "programlisting" elements of docbook. Or did I overlook something? Kind regards, Hans-Juergen

2 3

Best practices for concurrency
by The Trainspotter 19 Jun '13

19 Jun '13

Hi, I've read through the online example on server concurrency, however, currently I'm not running a BaseXServer instance I'm just running things locally. I want multiple threads in my application to be able to query the database at once without treading on each others toes. What's the best approach for doing this? I have tried querying the database using different threads but I keep getting internal BaseX errors which leads me to think I'm going about this the wrong way. Cheers Gary

5 15

invalid collection name?
by C. M. Sperberg-McQueen 27 May '13

27 May '13

Using the GUI interface to BaseX 7.2, when I select Database / Open and Manage ..., I get a list of the databases I have built over the time I've been using BaseX. When I try to open the collection named MEP.bigbooks, though, I get an error box saying "Name 'MEP.bigbooks' is invalid." In the "Manage Databases" dialog box, the information about the collection appears normal and plausible (at least, the size is about what I expect, and so on). It's only when I try to open the collection that I run into the error. I get similar errors on other collections which have . in their name; I do not get errors when opening other collections built (as far as I can remember) at the same time from the same data, if their name does not include a full stop. When I try renaming the collection without the full stop, I get the same error message. When I try to issue the command ALTER DATABASE MEP.bigbooks MEPbigbooks from the server command area of the GUI, I get a short syntax summary in the Query Info area complaining about the syntax of the command. When I replace MEP.bigbooks with MEP\.bigbooks and "MEP.bigbooks", I also get syntax errors. If I close the GUI and just rename the directory within my BaseXData directory, the database appears to open normally. I think I have two questions: Q1 is just renaming the directory a safe way (assuming no server is currently working with the data) to rename a collection? Should I do the same for all the other collections I have that don't currently open? I assume that if the 7.2 GUI won't let me rename them, it also will have trouble deleting them, so a related question is: is just deleting a sub-directory of BaseXData a safe way to delete a collection? Q2 Did the rules for names of collections change? Or is there some other cause? Speaking for myself, I guess that as a user I haven't thought much about what characters should be allowed in collection names, but I may have assumed that if they are not unrestricted, they would be restricted to being legal NCNames. I do kind of like using dots in names, so you can count this as one vote for relaxing the naming rules at some convenient moment. It's not a really important point, of course. best, Michael -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

2 2

XQuery update: updating inside a closure & how to modify a node and insert it in a DB
by Daniel Kvasnička 10 Jan '13

10 Jan '13

Hi folks, I'm trying to implement a secure RESTXQ service like this: declare %rest:path("api/users/init") %restxq:PUT("{$data}") %output:method("text") function page:init-user($data) { sec:secure(function() { let $profile := json:parse-ml(convert:binary-to-string($data)) return ( insert node attribute id { sec:get-current-user-id() } into $profile, insert node $profile as last into db:open("db")/s:root ) }) }; And I have two problems: - when executing it like this, inside of a closure, nothing happens. No modifications of the database and no errors (%updating missing) either - which is weird. Is this construct supported in 7.5? - when I remove the sec:secure call and leave only the rest (adding %updating to the declaration) the second insert runs but the first one doesn't - the node is saved without a "id" attribute. Does this have something to do with the PUL order? (http://docs.basex.org/wiki/Update#Example) Thanks for your hints! I feel like I'm overlooking something obvious... Daniel -- danielkvasnicka.net

4 9

Upcoming: XMLPrague 2013
by Christian Grün 14 Dec '12

14 Dec '12

Hi all! The XMLPrague Conference is the most important European event for XML specialists, developers and users. Next year, it will take place from Feb 8-10: http://www.xmlprague.cz/ Once again, our team will be present there, and it will be a pleasure for us to see you and talk to you live! What may be even more interesting: we are planning yet another BaseX User Meeting on the first day (Friday). We have just started to compile our agenda. As many of you are using BaseX for both exciting projects and inn productive environments, it would be great to have some of you talk about your experiences with BaseX. Would you be interested to contribute? We are looking forward to your feedback! Feel free to ask for more details, Christian

2 2

Re: [basex-talk] new to basex
by Christian Grün 30 Nov '12

30 Nov '12

Hi Bill (cc to the list), the current fulltext global indexing options (specified via SET) are applied and stored in the target database whenever an index is created via CREATE INDEX. If OPTIMIZE is used, these database options will be used for recreating the index. Both the database options and index structures should not get lost if you restart your server. The following (non-server) example may help to demonstrate that fulltext options such as stemming will be stored along with the database: basex -c "set stemming on;set ftindex on; create db db <a>A</a>; optimize; close; set stemming off; open db;info index fulltext" Btw, which BaseX version do you use? It would be interesting to see what happens with our latest stable snapshot [1]; the final version is long overdue, but it will finally be released in December. Christian [1] http://files.basex.org/releases/latest/ ___________________________ > HI Christian, > > Thanks for the fast reply. > > I tried your steps, with no change. I am curious about a couple of things I notice: > > Using these command I enabled fulltext with stemming: > >>set stemming true >>create index fulltext > > And info index shows these to be true. But if I stop the server and restart, these are all off, and the fulltext index is gone (the collection and it's files are still there). And yet, the same flawed search still works. > > So my questions are: > Do we need to reset globals and reindex each time the server is restart? It our implimentation it seems so (running standalone mode). > > Thanks. > > -bb > > > On Nov 30, 2012, at 8:50 AM, Christian Grün wrote: > >> Hi Bill, >> >>> I'm having trouble figuring out how exactly to enable fulltext XQuery searches. We have it working, but quite a few keywords are not being indexed. [...] >> >> as you already indicated, the fulltext index gets lost when performing >> updates. If you set FTNDEX to TRUE before creating your database, your >> fulltext index will always be recreated when calling OPTIMIZE: >> >> set ftindex true >> create db dorr_index >> add ... >> optimize >> >>> for $i in collection(PATH)//item >>> let $s := $i/mhs:sort >>> where $i contains text ("QUERY" using stemming) >>> return ... >> >> I know too less about your xml structure, so it's difficult to tell >> what/if sth. goes wrong. Does the behavior change when following the >> steps suggested above? >> >> Cheers, >> Christian >> ___________________________ >> >>> for $i in collection(PATH)//item >>> let $s := $i/mhs:sort >>> where $i contains text ("QUERY" using stemming) >>> return ... >>> >>> Are there any steps missing? Also, any tutorial out there to follow? >>> >>> Thanks very much in advance! >>> >>> -Bill B >>> -- >>> Bill Beck, Web Development Specialist >>> Massachusetts Historical Society >>> 1154 Boylston Street, Boston, MA 02215 >>> Tel: 617-646-0505, Fax: 617-859-0074 >>> www.masshist.org - America's Oldest Historical Society - Founded 1791 >>> >>> >>> In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/. >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk(a)mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > -- > Bill Beck, Web Development Specialist > Massachusetts Historical Society > 1154 Boylston Street, Boston, MA 02215 > Tel: 617-646-0505, Fax: 617-859-0074 > www.masshist.org - America's Oldest Historical Society - Founded 1791 > > > In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/.

1 0

new to basex
by Bill Beck 30 Nov '12

30 Nov '12

Hi all, Here that the Massachusetts Historical Society we're exploring using BaseX for some new transcriptions of newspapers from around the American Revolution, as collected by a fellow named Harbottle Dorr. I'm having trouble figuring out how exactly to enable fulltext XQuery searches. We have it working, but quite a few keywords are not being indexed. From the command line tool, we do: >create database dorr_index >create index fulltext Then we have a PHP tool (using the class found on the BaseX site) that calls $session->execute to open the collection and $session->add to place the files. But when I then run >info db from the command line tool, it shows: Database Properties Name: dorr_index Size: 1390 KB Nodes: 32411 Resources: 24 Timestamp: 30.11.2012 08:00:11 Resource Properties Timestamp: 30.11.2012 08:03:21 Encoding: UTF-8 Whitespace Chopping: ON Indexes Up-to-date: false Path Summary: ON Text Index: OFF Attribute Index: OFF Full-Text Index: OFF if I then run the command >create index fulltext again, info db shows full-text index on, but up-to-date is still false, and our search results still have many missing hits. our XQuery is: for $i in collection(PATH)//item let $s := $i/mhs:sort where $i contains text ("QUERY" using stemming) return ... Are there any steps missing? Also, any tutorial out there to follow? Thanks very much in advance! -Bill B -- Bill Beck, Web Development Specialist Massachusetts Historical Society 1154 Boylston Street, Boston, MA 02215 Tel: 617-646-0505, Fax: 617-859-0074 www.masshist.org - America's Oldest Historical Society - Founded 1791 In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/.

3 3

using fuzzy in oxygen "invalid XQuery syntax, syntax does not pass static validation."
by Alex Muir 30 Nov '12

30 Nov '12

Hi, Just noticing that this using fuzzy example works in the basex gui however running in oxygen gives an error. It doesn't like the "using fuzzy" text. declare namespace db="http://basex.org/modules/db"; let $x := db:open('F-DDEX')//MessageHeader/MessageThreadId[text() contains text '893747' using fuzzy] return $x Oxygen Error System ID: /mnt/basex_volume/i4Base-Basex7.5/xquery/indevelopment/basex-connectiing.xq Scenario: all XQuery file: /mnt/basex_volume/i4Base-Basex7.5/xquery/indevelopment/basex-connectiing.xq Engine name: Basex-23.23.82.41 Severity: error Description: Invalid XQuery syntax, syntax does not pass static validation. Would that be an xqj issue? -- - Alex G. Muir Software Engineering Consultant Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 Love African Kora Music? Take a moment to listen to Gambia's - Amadu Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com Your support keeps Africa's griot tradition alive... Cheers!

2 1

Time query takes looking for element with text() ID
by Alex Muir 29 Nov '12

29 Nov '12

Hi, Sorry for all the questions of late first of all... I've been using basex for a while now and wrote some nice xquery that gather metrics from datasets ~7000 files querying the whole corpus to create statistics and it quite fast at that but were thinking this query below feels slow at 34 seconds so we thought we should ask for you thoughts on it's duration. Query: let $content := db:open('F-DDEX')//MessageHeader/MessageThreadId[text() eq '8937478'] return $content Content: So every file has a message header with a MessageThreadId. <MessageHeader xmlns:ern="http://ddex.net/xml/2011/ern-main/33" xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance"> <MessageThreadId>8937478</MessageThreadId> <MessageId>C2C977FDFDHF98DHF9D8FHEURYX</MessageId> <MessageSender> <PartyId>PA47F93H54HU93HJSFDINF</PartyId> <PartyName> <FullName>Warner Music Group</FullName> </PartyName> </MessageSender> <MessageRecipient> <PartyId>3G3E</PartyId> <PartyName> <FullName>3G3E-YADS</FullName> </PartyName> </MessageRecipient> <MessageCreatedDateTime>2012-06-18T05:35:54Z</MessageCreatedDateTime> </MessageHeader> Given the following database with text, attribute and fulltext indexes on. Database Properties Name: F-DDEX Size: 5251 MB Nodes: 239945615 Documents: 7954 Specs Server: $ cat /proc/meminfo | grep MemTotal MemTotal: 7633876 kB $ cat /proc/cpuinfo | grep name model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz Thoughts? Thanks -- - Alex G. Muir Software Engineering Consultant Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 Love African Kora Music? Take a moment to listen to Gambia's - Amadu Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com Your support keeps Africa's griot tradition alive... Cheers!

2 2