- BaseX-Talk - mailman.uni-konstanz.de

Performance of Add command
by Manuel Bernhardt 10 Jul '12

10 Jul '12

Hi, I'm doing some testing before migration one of our customers to a new version of our platform that uses BaseX in order to store documents. They have approx. 4M documents, and I'm running an import operation on a 1 M document collection on my laptop. The way I'm inserting documents is by firing off one Add command per document, based on a stream of the document, at a different (unique) path for each document, and flushing every at 10K Adds. Since most CPU usage (for one of the cores, the other ones being untouched) is taken by the BaseX server, I fired up YourKit out of curiosity to see where the CPU time was spent. My machine is a 2*4 core MacBook Pro with 8GB of RAM and SSD, so I think hardware-wise it should do pretty fine. YourKit shows that what seems to use up most time is the Namespaces.update method: Thread-12 [RUNNABLE] CPU time: 2h 7m 9s org.basex.data.Namespaces.update(NSNode, int, int, boolean, Set) org.basex.data.Namespaces.update(int, int, boolean, Set) org.basex.data.Data.insert(int, int, Data) org.basex.core.cmd.Add.run() org.basex.core.Command.run(Context, OutputStream) org.basex.core.Command.exec(Context, OutputStream) org.basex.core.Command.execute(Context, OutputStream) org.basex.core.Command.execute(Context) org.basex.server.ClientListener.execute(Command) org.basex.server.ClientListener.add() org.basex.server.ClientListener.run() I'm not really sure what that method does - it's a recursive function and seems to be triggered by Data.insert: // NSNodes have to be checked for pre value shifts after insert nspaces.update(ipre, dsize, true, newNodes); The whole set of records should have no more than 5 different namespaces in total. Thus I'm wondering if there would perhaps be some potential for optimization here? Note that I'm completely ignorant as to what the method does and what its exact purpose is. Thanks, Manuel PS: the import is now finished: Storing 1001712 records into BaseX took 9285008 ms

2 11

Incorrect use of timezone Z instead of implicit timezone
by Gunther Rademacher 09 Jul '12

09 Jul '12

As explained in http://www.w3.org/TR/xquery-operators/#func-subtract-dates the implicit timezone should be used for augmenting an xs:date without a timezone before subtracting it from one with a timezone. BaseX however appears to be using timezone Z in this case. The following xquery demonstrates this: current-date() - xs:date(substring(string(current-date()), 1, 10)) It should return a zero duration, but BaseX returns -PT2H I am in timezone +02:00. Best regards Gunther

2 1

FW: Problem with RunQueries.java example!!
by Anand Chiney (AIR) 09 Jul '12

09 Jul '12

Dear BaseX Team, The below code is working fine and I am getting the required output - for $x in doc('input.xml')//li return $x/text() i.e; ------------------------Output with XML File---------------- Example 1 Example 2 Example 3 ------------------------------------------------------------ But when I try the same with BaseX DB , I am not getting the required output ;-( for $x in doc('input')//li return $x/text() ------------------------Output with DB---------------------- Example 1Example 2Example 3 ------------------------------------------------------------ Now the first output can easily be populated in the Combo box, but what about the second..? The length of the output is not fixed in real application... I have attached the input.xml file. Regards, Anand Chiney -----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, July 07, 2012 5:42 PM To: Anand Chiney (AIR); BaseX Subject: Re: Problem with RunQueries.java example!! Dear Anand, > Thanks for the reply. The serialize() did work. But what I was expecting > Is pure textual output and I am getting XML output. The output which I am > getting is - I strongly recommend you to spend some more time to get into XPath and XQuery, as those issues can all be realized fairly easily with the languages themselves. Some straightforward tutorials are e.g. found at [1,2]. Here are some examples (among many others) how you could rewrite your query to get the desired results: for $x in doc('input.xml')//li return $x/text() doc('input.xml')//li/text() doc('input.xml')//li/string() doc('input.xml')//li/data() You can also compose the whole combobox within XQuery. Your query could then look like <select>{ for $x in doc('input.xml')//li return <option>{ $x/text() }</option> }</select> or element select { doc('input.xml')//li ! element option { text() } } Hope this helps, Christian [1] http://www.w3schools.com/xpath [2] http://www.w3schools.com/xquery > > ----------------------------------------------------------- > * Convert each result to its Java representation: > Iterator > <li>Exercise 1</li> > <li>Exercise 2</li> > ----------------------------------------------------------- > > But I am expecting this is - > > Exercise 1 > Exercise 2 > > Actually I want to populate a JComboBox which is a part of my front-end. > So I thought that this piece of code, where it is said "Java > representation", would gimme the output as I required. > > Can you help me a little more..? > Which method will gimme the plain output, without the XML Tags...? So that > I can populate a JComboBox... !! > > > Regards, > Anand Chiney > > > -----Original Message----- > From: Christian Grün [mailto:christian.gruen@gmail.com] > Sent: Saturday, July 07, 2012 5:02 PM > To: Anand Chiney (AIR); BaseX > Subject: Re: Problem with RunQueries.java example!! > > Here's a second feedback (sorry for being distracted): Your query does > return standard Java DOM elements, the toString() method of which > returns "[li: null]". As you are probably interested in the string > representation of the elements, you'll need to replace "toJava()" with > "serialize()": > > System.out.println(item.serialize()); > > Hope this helps, > Christian > ___________________________ > > On Sat, Jul 7, 2012 at 1:26 PM, Christian Grün > <christian.gruen(a)gmail.com> wrote: >> Your query will yield the expected results if you remove the explicit >> cq.execute(); call. >> >> In the latest snapshot [1], an exception will be returned if the >> result has already been requested. >> >> Hope this helps, >> Christian >> >> [1] http://files.basex.org/releases/latest/ >> ___________________________ >> >>> Dear Christian >>> >>> Please find attached 2 files, MyRunQueries.java and input.xml >>> >>> The .java file is an edited version of RunQueries.java version and the >>> >>> I hope this would be sufficient for you to test. >>> >>> >>> >>> Regards >>> Anand chiney >>> >>> -----Original Message----- >>> From: Christian Grün [mailto:christian.gruen@gmail.com] >>> Sent: Saturday, July 07, 2012 4:34 PM >>> To: Sandeep Yohans >>> Cc: Anand Chiney; basex-talk(a)mailman.uni-konstanz.de >>> Subject: Re: Problem with RunQueries.java example!! >>> >>> One again I'm sorry Please send us a (small !) example that we can >>> try by ourselves. >>> >>> Thank you in advance, >>> Christian >>> ___________________________ >>> >>> On Sat, Jul 7, 2012 at 1:02 PM, Sandeep Yohans <sandeepy.air(a)gmail.com> >>> wrote: >>>> Hello Christian, >>>> >>>> We are extracting names from a very long list of names. The > item.toJava() >>>> method call in iterate(query) is returning null. >>>> example output: >>>> >>>> [li: null] >>>> >>>> Rest of the methods are printing output fine. >>>> >>>> >>>> -- >>>> Thanks and Regards >>>> >>>> Sandeep Yohans >>>> Software Developer >>>> Systems - AIR Infotech >>>> Nagpur - India >>>> www.airinfotech.in >>>> Mobile: +91-997-086-5520 >>>> >

2 1

BaseX with a Booleen Column
by Scott Clayton 09 Jul '12

09 Jul '12

Hey... I know this may be getting out of the native XML database talk but it would be great if we could add a column and mark as true or false when we grab the data from an external source and mark as gotten. Anyways...just a thought. Maybe there is a plugin for this or something to the effect? Best, Scott PS - I sent a similar email to the default contact at BaseX.org site and after I sent it there was a suggestion to send feature inquiries to this email... I could not find that suggextion until after I had already sent the email. :-)

2 1

BaseX client event protocol
by Andy Bunce 08 Jul '12

08 Jul '12

Hi, I am looking at adding WATCH and UNWATCH to my Node.js BaseX interface and I have a couple of questions before I start. 1) The documentation at http://docs.basex.org/wiki/Server_Protocol shows nothing in the Server Response column. This seems not to be the case. It looks like a standard BaseX protocol type response is to be expected? 2) The example at https://github.com/BaseXdb/basex-examples/blob/master/src/main/java/org/bas… session2 watching and then executing. Does the server protocol identify "event pushes" in some way. That is when I receive a message how can I tell if is a event or a command response? /Andy

2 3

Re: [basex-talk] Problem with RunQueries.java example!!
by Christian Grün 07 Jul '12

07 Jul '12

Dear Anand, > Thanks for the reply. The serialize() did work. But what I was expecting > Is pure textual output and I am getting XML output. The output which I am > getting is - I strongly recommend you to spend some more time to get into XPath and XQuery, as those issues can all be realized fairly easily with the languages themselves. Some straightforward tutorials are e.g. found at [1,2]. Here are some examples (among many others) how you could rewrite your query to get the desired results: for $x in doc('input.xml')//li return $x/text() doc('input.xml')//li/text() doc('input.xml')//li/string() doc('input.xml')//li/data() You can also compose the whole combobox within XQuery. Your query could then look like… <select>{ for $x in doc('input.xml')//li return <option>{ $x/text() }</option> }</select> …or… element select { doc('input.xml')//li ! element option { text() } } Hope this helps, Christian [1] http://www.w3schools.com/xpath [2] http://www.w3schools.com/xquery > > ----------------------------------------------------------- > * Convert each result to its Java representation: > Iterator > <li>Exercise 1</li> > <li>Exercise 2</li> > ----------------------------------------------------------- > > But I am expecting this is - > > Exercise 1 > Exercise 2 > > Actually I want to populate a JComboBox which is a part of my front-end. > So I thought that this piece of code, where it is said "Java > representation", would gimme the output as I required. > > Can you help me a little more..? > Which method will gimme the plain output, without the XML Tags...? So that > I can populate a JComboBox... !! > > > Regards, > Anand Chiney > > > -----Original Message----- > From: Christian Grün [mailto:christian.gruen@gmail.com] > Sent: Saturday, July 07, 2012 5:02 PM > To: Anand Chiney (AIR); BaseX > Subject: Re: Problem with RunQueries.java example!! > > Here's a second feedback (sorry for being distracted): Your query does > return standard Java DOM elements, the toString() method of which > returns "[li: null]". As you are probably interested in the string > representation of the elements, you'll need to replace "toJava()" with > "serialize()": > > System.out.println(item.serialize()); > > Hope this helps, > Christian > ___________________________ > > On Sat, Jul 7, 2012 at 1:26 PM, Christian Grün > <christian.gruen(a)gmail.com> wrote: >> Your query will yield the expected results if you remove the explicit >> cq.execute(); call. >> >> In the latest snapshot [1], an exception will be returned if the >> result has already been requested. >> >> Hope this helps, >> Christian >> >> [1] http://files.basex.org/releases/latest/ >> ___________________________ >> >>> Dear Christian >>> >>> Please find attached 2 files, MyRunQueries.java and input.xml >>> >>> The .java file is an edited version of RunQueries.java version and the >>> >>> I hope this would be sufficient for you to test. >>> >>> >>> >>> Regards >>> Anand chiney >>> >>> -----Original Message----- >>> From: Christian Grün [mailto:christian.gruen@gmail.com] >>> Sent: Saturday, July 07, 2012 4:34 PM >>> To: Sandeep Yohans >>> Cc: Anand Chiney; basex-talk(a)mailman.uni-konstanz.de >>> Subject: Re: Problem with RunQueries.java example!! >>> >>> One again I'm sorry… Please send us a (small !) example that we can >>> try by ourselves. >>> >>> Thank you in advance, >>> Christian >>> ___________________________ >>> >>> On Sat, Jul 7, 2012 at 1:02 PM, Sandeep Yohans <sandeepy.air(a)gmail.com> >>> wrote: >>>> Hello Christian, >>>> >>>> We are extracting names from a very long list of names. The > item.toJava() >>>> method call in iterate(query) is returning null. >>>> example output: >>>> >>>> [li: null] >>>> >>>> Rest of the methods are printing output fine. >>>> >>>> >>>> -- >>>> Thanks and Regards >>>> >>>> Sandeep Yohans >>>> Software Developer >>>> Systems - AIR Infotech >>>> Nagpur - India >>>> www.airinfotech.in >>>> Mobile: +91-997-086-5520 >>>> >

1 0

Re: [basex-talk] Problem with RunQueries.java example!!
by Christian Grün 07 Jul '12

07 Jul '12

Your query will yield the expected results if you remove the explicit cq.execute(); call. In the latest snapshot [1], an exception will be returned if the result has already been requested. Hope this helps, Christian [1] http://files.basex.org/releases/latest/ ___________________________ > Dear Christian > > Please find attached 2 files, MyRunQueries.java and input.xml > > The .java file is an edited version of RunQueries.java version and the > > I hope this would be sufficient for you to test. > > > > Regards > Anand chiney > > -----Original Message----- > From: Christian Grün [mailto:christian.gruen@gmail.com] > Sent: Saturday, July 07, 2012 4:34 PM > To: Sandeep Yohans > Cc: Anand Chiney; basex-talk(a)mailman.uni-konstanz.de > Subject: Re: Problem with RunQueries.java example!! > > One again I'm sorry… Please send us a (small !) example that we can > try by ourselves. > > Thank you in advance, > Christian > ___________________________ > > On Sat, Jul 7, 2012 at 1:02 PM, Sandeep Yohans <sandeepy.air(a)gmail.com> > wrote: >> Hello Christian, >> >> We are extracting names from a very long list of names. The item.toJava() >> method call in iterate(query) is returning null. >> example output: >> >> [li: null] >> >> Rest of the methods are printing output fine. >> >> >> -- >> Thanks and Regards >> >> Sandeep Yohans >> Software Developer >> Systems - AIR Infotech >> Nagpur - India >> www.airinfotech.in >> Mobile: +91-997-086-5520 >>

1 1

Re: [basex-talk] Problem with RunQueries.java example!!
by Sandeep Yohans 07 Jul '12

07 Jul '12

Hello Christian, We are extracting names from a very long list of names. The item.toJava() method call in iterate(query) is returning null. example output: [li: null] Rest of the methods are printing output fine. -- Thanks and Regards Sandeep Yohans Software Developer Systems - AIR Infotech Nagpur - India www.airinfotech.in Mobile: +91-997-086-5520

2 1

Problem with RunQueries.java example!!
by Anand Chiney (AIR) 07 Jul '12

07 Jul '12

Dear BaseX Team, The method iterate() which is used to convert the XML result to its Java representation is retuning [null]. I tried this code with the default query and also with 2-3 more XML files. All are giving same, [null] output..!! Have a nice day :) Regards Anand Chiney

2 1

Re: [basex-talk] Empty Initial Document Inconsistencies
by Dave Glick 07 Jul '12

07 Jul '12

Christian, Thanks again for this! I still haven't had a chance to do any rigorous testing, but in my initial use everything looks great (and all my unit tests still pass). I think more importantly, this resolves an undocumented inconsistency for users of the low-level API - with the guarantee that the database will behave the same way regardless of state, a lot of conditional code can now be removed. Proof that good things come to those who wait :). Dave -----Original message----- From: "Christian Grün" <christian.gruen(a)gmail.com> To: Dave Glick <dglick(a)dracorp.com> Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> Sent: Tue, Jul 3, 2012 00:13:58 GMT+00:00 Subject: Re: [basex-talk] Empty Initial Document Inconsistencies Hi Dave, here's finally some public feedback for your outstanding feature request: I've rewritten our internal database structures, such that an empty database will now really be empty (i.e., contain no dummy document node anymore). As a result, some convenience methods have disappeared from the Data class (Data.isEmpty(), Data.single()), as they were only offered to hide the former inconsistency. The changes have now been merged into the BaseX master branch. While I believe that we have tested the changes quite well, I am always glad for any feedback. Christian ___________________________ On Thu, Dec 29, 2011 at 4:36 PM, Dave Glick <dglick(a)dracorp.com> wrote: > Hello again, > > I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: > > - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. > - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. > - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. > - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). > > In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. > > At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. > > Dave > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

4 12