- BaseX-Talk - mailman.uni-konstanz.de

Fwd: Replication with BaseX
by Dirk Kirsten 11 Jul '12

11 Jul '12

Once again forwarding to the list as I am apparently unable to correctly use this Gmail-client... So for the sake of completeness see attached mails ---------- Forwarded message ---------- From: Dirk Kirsten <dk(a)basex.org> Date: Wed, Jul 11, 2012 at 12:20 PM Subject: Re: [basex-talk] Replication with BaseX To: Usama Dar <munir.usama(a)gmail.com> Oh, yeah, sure. You are absolutely right, sorry for misguiding you there. Yes, the indexes are on disk and will be transferred, so using rsync should be pretty performant. Cheers, Dirk On Wed, Jul 11, 2012 at 9:31 AM, Usama Dar <munir.usama(a)gmail.com> wrote: > Thanks Dirk. I noticed that all data structures are stored in the data > directory , so if i copy over the full data directory on a remote machine > and then continuously rsync them , do you think there would still be some > index re-building required? i thought the indexes like everything else are > on disk already? > > Best Regards, > Usama Dar > > > On Wed, Jul 11, 2012 at 9:02 AM, Dirk Kirsten <dk(a)basex.org> wrote: > >> Dear Usama, >> >> Currently there is no recommended strategy how to this. So there is no >> single right answer to your question. >> >> I think your strategy using rsync sounds very reasonable. Rsync has the >> nice advantage that it will be able to just transfer the incremental >> change, so you don't have to transfer the whole 500K documents all the >> time. However, you then still have to update the indexes, so depending on >> your data and your update interval this could take some time. >> >> Another very simple approach could be by using the BaseX Client-Server >> Infrastructure to send all commands and queries to both servers and trying >> to keep them synchronized. This could be more efficient, but this highly >> depends on your data and your queries (i.e. you use case). >> >> Cheers, >> Dirk >> >> On Tue, Jul 10, 2012 at 8:12 PM, Usama Dar <munir.usama(a)gmail.com> wrote: >> >>> Hi Guys, >>> >>> I am new to BaseX and i am going to use it to store 500K XML documents >>> which will be edited, Queried and stuff. I have already built the main >>> application. I have need to be able to run BaseX in a >>> high availability environment. For that i will initially have two servers >>> and i plan to use rsync on baseX data directory to replicate data. >>> >>> My question basically is are there any recommended strategies to do >>> this? is using rsync going to do the job? i can copy files over in the data >>> directory in a running instance? or the other server should be shutdown >>> while i copy files? any tool/ strategy you might know of? >>> >>> Thanks for your help >>> >>> Usama Dar. >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk(a)mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >> >

1 0

Re: [basex-talk] Displaying transformed XML into JEditorPane
by Anand Chiney (AIR) 11 Jul '12

11 Jul '12

Dear Christian, Yes there is SIMPLE GENERAL answer to the question asked by Sandeep. :-) Actually we are storing thousands of resumes in BaseX. These are stored as a single XML document with following format. <Resumes> <Resume>No 1</Resume> <Resume>No 2</Resume> <Resume>No </Resume> <Resume>No 4</Resume> <Resume></Resume> <Resumes> There are various child elements for <Resume>, which I think is not essential for this generalized answer. Till now I had explained about the Data. Now coming to front-end. We are using Swing based application to search and view these resumes. We want to display the resumes one-by-one in JEditorPane. For this we first transformed the XML to HTML using a XSLT. The BaseX editor can do this using XSLT Module. Christian, do you remember that, earlier there were 3 examples on this and we faced trouble using them. You then corrected these and it helped us a lot, But these were to work with BaseX editor. We need to perform this transformation using Java. Yesterday, Sandeep asked the question on this. We did found a good solution in these 2 examples, here https://github.com/BaseXdb/basex/blob/master/src/test/java/org/basex/test/qu ery/func/FNXsltTest.java https://github.com/BaseXdb/basex/blob/master/src/test/java/org/basex/test/qu ery/AdvancedQueryTest.java But this was not exactly what we were looking for. We need to modify this as per our requirement. And we did it :-) I am not going into deep to explain how we did and why due to less time. Rather I would simply paste the method which worked for us. ----------------------------- Method from the above Example (FNXsltTest.java )------------------------------------------------------------ @Test public void xsltTransform() { check(_XSLT_TRANSFORM); final String doc = "<a/>"; String style = wrap("<xsl:template match='/'><X/></xsl:template>"); query(_XSLT_TRANSFORM.args(doc, style), "<X/>"); query(_XSLT_TRANSFORM.args(doc, '"' + style + '"'), "<X/>"); style = wrap("<xsl:param name='t'/><xsl:template match='/'>" + "<X><xsl:value-of select='$t'/></X></xsl:template>"); final String param = "<xslt:parameters><xslt:t>1</xslt:t></xslt:parameters>"; query(_XSLT_TRANSFORM.args(doc, style, param), "<X>1</X>"); } ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- ---------------- ----------------------------- Our Method ------------------------------------------------------------ public String xsltTransform() { return query(_XSLT_TRANSFORM.args(getDoc(), getStyleSheet())); } Explanation- getDoc() will send the query which will retrieve the required Resumes based on user search getStyleSheet() for resume.xslt let $in := /Resumes/Resume[WorkExpr/Java > "5"] let $style := doc('resume.xslt') return (xslt:transform($in, $style)) ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- ---------------- ----------------------------- Method from the above Example (AdvancedQueryTest.java)------------------------------------------------- protected static String query(final String query) { final QueryProcessor qp = new QueryProcessor(query, CONTEXT); qp.ctx.sc.baseURI("."); try { return qp.execute().toString(); } catch (QueryException ex) { Logger.getLogger(DBTransform.class.getName()).log(Level.SEVERE, null, ex); return ex.getMessage(); } } We used this method as it is rather copy and paste Other methods after certain R&D, we found not useful in our case (!!) ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- ---------------- So wrote a class with these 2 methods (for transformation) and other methods which were useful and it did work. But with a little trouble. This piece of code is NOT PRESERVING THE SPACES which we are giving through our stylesheet. Now for this either you can help us or we need to find a way out of it. Thanks for your kind and timely help. And we will trouble you more in future ;) Regards Anand Chiney _____ From: Sandeep Yohans [mailto:sandeepy.air@gmail.com] Sent: Wednesday, July 11, 2012 4:24 PM To: Anand Chiney Subject: Fwd: [basex-talk] Displaying transformed XML into JEditorPane ---------- Forwarded message ---------- From: Christian Grün <christian.gruen(a)gmail.com> Date: Wed, Jul 11, 2012 at 4:09 PM Subject: Re: [basex-talk] Displaying transformed XML into JEditorPane To: Sandeep Yohans <sandeepy.air(a)gmail.com> Cc: basex-talk(a)mailman.uni-konstanz.de Dear Sandeep, there is no simple general answer to your question, so I'd be glad to if you could prepare a little example [1] that demonstrates the problem. It often takes some time to generate a good and minimized example, but it's often helpful for other readers, too. Thanks in advance (and sorry for all the wise instructions ;), Christian [1] http://mindprod.com/jgloss/sscce.html _____________________________ On Tue, Jul 10, 2012 at 6:31 AM, Sandeep Yohans <sandeepy.air(a)gmail.com> wrote: > Hello, > > Below is the code to transform XML file to HTML. When tried with BaseX > working fine. > > > > let $in := /Resumes/Resume[WorkExpr/Java > "5"] > > let $style := doc('resume.xslt') > > return (xslt:transform($in, $style)) > > > > The aobve XSLT file will tramsform the output to HTML. > > Now I want to display this HTML output in JEditorPane. > > > > > -- > Thanks and Regards > > Sandeep Yohans > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > -- Thanks and Regards Sandeep Yohans Software Developer Systems - AIR Infotech Nagpur - India www.airinfotech.in <http://www.airinfotech.in/> Mobile: +91-997-086-5520

2 1

Fwd: Displaying transformed XML into JEditorPane
by Dirk Kirsten 11 Jul '12

11 Jul '12

By mistake I answered privately to Sandeep, so for the sake of documentation here is my answer as well on the mailing list. ---------- Forwarded message ---------- From: Dirk Kirsten <dk(a)basex.org> Date: Wed, Jul 11, 2012 at 8:45 AM Subject: Re: [basex-talk] Displaying transformed XML into JEditorPane To: Sandeep Yohans <sandeepy.air(a)gmail.com> Dear Sandeep, I don't see how your question relates to BaseX? If the transformation is working fine and you have well-formed HTML output, then the rest is up to the application. Seems like you want to do some Swing-programming; displaying HTML in JEditorPane should work out-of-the-box, but please have a look at the Java-API, e.g.: http://docs.oracle.com/javase/7/docs/api/javax/swing/JEditorPane.html Cheers, Dirk On Tue, Jul 10, 2012 at 6:31 AM, Sandeep Yohans <sandeepy.air(a)gmail.com>wrote: > Hello, > > Below is the code to transform XML file to HTML. When tried with BaseX > working fine. > > > > let $in := /Resumes/Resume[WorkExpr/Java > "5"] > > let $style := doc('resume.xslt') > > return (xslt:transform($in, $style)) > > > > *The aobve XSLT file will tramsform the output to HTML.* > > *Now I want to display this HTML output in JEditorPane.* > > * * > ** > -- > Thanks and Regards > > Sandeep Yohans > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

1 0

Displaying transformed XML into JEditorPane
by Sandeep Yohans 11 Jul '12

11 Jul '12

Hello, Below is the code to transform XML file to HTML. When tried with BaseX working fine. let $in := /Resumes/Resume[WorkExpr/Java > "5"] let $style := doc('resume.xslt') return (xslt:transform($in, $style)) *The aobve XSLT file will tramsform the output to HTML.* *Now I want to display this HTML output in JEditorPane.* * * ** -- Thanks and Regards Sandeep Yohans

2 1

usability issue in GUI
by Godmar Back 11 Jul '12

11 Jul '12

Hi, is it possible to connect the GUI to an existing server? If not, I think it really should be, rather than requiring than the GUI start the server. - Godmar

2 3

improve BaseX error message
by Godmar Back 10 Jul '12

10 Jul '12

It should be "outdated" or "out-of-date", not "out-of-dated". Looks like moving to 7.3 is a bigger effort than thought... - Godmar java.io.IOException: Protocol for query iteration is out-of-dated. at org.libx.support.basex.BaseXSessionSupport$Query.exec(BaseXSessionSupport.java:324) at org.libx.support.basex.BaseXSessionSupport$Query.more(BaseXSessionSupport.java:270) at org.libx.support.basex.BaseXSession$1.next(BaseXSession.java:104) at org.libx.support.basex.XMLDatabaseClient$1.after(XMLDatabaseClient.java:53) at org.libx.support.basex.BaseXSession.runQuery(BaseXSession.java:100) at org.libx.support.basex.XMLDatabaseClient.runQuery(XMLDatabaseClient.java:40) at org.libx.support.basex.XMLDatabaseClient.runQueryAsString(XMLDatabaseClient.java:95) at org.libx.support.basex.XMLDatabaseOperation$2.run(XMLDatabaseOperation.java:65) at org.libx.support.basex.XMLDatabaseOperation$2.run(XMLDatabaseOperation.java:62) at org.libx.support.basex.XMLDatabaseOperation.execute(XMLDatabaseOperation.java:46) at org.libx.support.basex.XMLDatabaseOperation.runQueryAsString(XMLDatabaseOperation.java:62)

2 2

problem using BaseX GUI
by Godmar Back 10 Jul '12

10 Jul '12

I'm using the most recent BaseX GUI; I'm trying to change the value of an attribute in an XML document. This results in the following exception: java.lang.ArrayIndexOutOfBoundsException: -1 org.basex.util.list.IntList.get(IntList.java:65) org.basex.gui.view.info.InfoView.setInfo(InfoView.java:173) org.basex.gui.GUI.exec(GUI.java:402) org.basex.gui.GUI$6.run(GUI.java:343) (see attached screenshot). I can't provide a reproducible test case; my second attempt to change the attribute value succeeded, see log below, but perhaps this backtrace helps the developers fix this issue. - Godmar // log of successful execution upon second attempt Query: replace value of node db:open-pre("feed_libx_dot_editions_at_gmail_dot_com_core", 827) with "161" Compiling: - pre-evaluating db:open-pre("feed_libx_dot_editions_at_gmail_dot_com_core", 827) Result: replace value of node attribute src { " libx.editions(a)gmail.com/core/161" } with "161" Timing: - Parsing: 0.66 ms - Compiling: 0.22 ms - Evaluating: 3.29 ms - Printing: 0.06 ms - Total Time: 4.25 ms Result: - Hit(s): 0 Items - Updated: 1 Item - Printed: 0 Bytes Query plan: <QueryPlan> <Replace> <DBNode name="feed_libx_dot_editions_at_gmail_dot_com_core" pre="827"/> <Str value="161" type="xs:string"/> </Replace> </QueryPlan>

2 1

Replication with BaseX
by Usama Dar 10 Jul '12

10 Jul '12

Hi Guys, I am new to BaseX and i am going to use it to store 500K XML documents which will be edited, Queried and stuff. I have already built the main application. I have need to be able to run BaseX in a high availability environment. For that i will initially have two servers and i plan to use rsync on baseX data directory to replicate data. My question basically is are there any recommended strategies to do this? is using rsync going to do the job? i can copy files over in the data directory in a running instance? or the other server should be shutdown while i copy files? any tool/ strategy you might know of? Thanks for your help Usama Dar.

1 0

Performance of Add command
by Manuel Bernhardt 10 Jul '12

10 Jul '12

Hi, I'm doing some testing before migration one of our customers to a new version of our platform that uses BaseX in order to store documents. They have approx. 4M documents, and I'm running an import operation on a 1 M document collection on my laptop. The way I'm inserting documents is by firing off one Add command per document, based on a stream of the document, at a different (unique) path for each document, and flushing every at 10K Adds. Since most CPU usage (for one of the cores, the other ones being untouched) is taken by the BaseX server, I fired up YourKit out of curiosity to see where the CPU time was spent. My machine is a 2*4 core MacBook Pro with 8GB of RAM and SSD, so I think hardware-wise it should do pretty fine. YourKit shows that what seems to use up most time is the Namespaces.update method: Thread-12 [RUNNABLE] CPU time: 2h 7m 9s org.basex.data.Namespaces.update(NSNode, int, int, boolean, Set) org.basex.data.Namespaces.update(int, int, boolean, Set) org.basex.data.Data.insert(int, int, Data) org.basex.core.cmd.Add.run() org.basex.core.Command.run(Context, OutputStream) org.basex.core.Command.exec(Context, OutputStream) org.basex.core.Command.execute(Context, OutputStream) org.basex.core.Command.execute(Context) org.basex.server.ClientListener.execute(Command) org.basex.server.ClientListener.add() org.basex.server.ClientListener.run() I'm not really sure what that method does - it's a recursive function and seems to be triggered by Data.insert: // NSNodes have to be checked for pre value shifts after insert nspaces.update(ipre, dsize, true, newNodes); The whole set of records should have no more than 5 different namespaces in total. Thus I'm wondering if there would perhaps be some potential for optimization here? Note that I'm completely ignorant as to what the method does and what its exact purpose is. Thanks, Manuel PS: the import is now finished: Storing 1001712 records into BaseX took 9285008 ms

2 11

Incorrect use of timezone Z instead of implicit timezone
by Gunther Rademacher 09 Jul '12

09 Jul '12

As explained in http://www.w3.org/TR/xquery-operators/#func-subtract-dates the implicit timezone should be used for augmenting an xs:date without a timezone before subtracting it from one with a timezone. BaseX however appears to be using timezone Z in this case. The following xquery demonstrates this: current-date() - xs:date(substring(string(current-date()), 1, 10)) It should return a zero duration, but BaseX returns -PT2H I am in timezone +02:00. Best regards Gunther

2 1