April 2015 - BaseX-Talk - mailman.uni-konstanz.de

] Database files across multiple directories
by Goetz Heller 29 Apr '15

29 Apr '15

Small correction; I should have said "reference each database with doc()" instead of "open each database". My apologies. Kind regards, Goetz

1 0

Re: [basex-talk] Database files across multiple directories
by Goetz Heller 29 Apr '15

29 Apr '15

Hi Julia, the Xml files I'm using are also in different directories, one for each day. I create one database on each of these directories, each containing between several hundred and up to 6000 documents. Usually, queries need to take in account only one day, so the search space is considerably smaller than for one large database. If access to documents of more than one day is required, you can create the range of database identifiers (supposing there is an algorithm to do so) and then open each database in a for loop, thus getting a sequence of top nodes. Also, for more complicated queries, I can choose to (re)create some the databases in-memory in order to improve performance. Since creating a database of this size takes just a couple of seconds, this is an approach which would possibly not be feasible if all the documents were in one huge database. Kind regards, Goetz

1 0

Reg : Map Reduce Frame Work
by Chandrasekhar 29 Apr '15

29 Apr '15

Dear Team, Information required about map reduce framework for xquery performence.Suggest us to best practice documents on this. Thanks&Regards ChandraSekhar The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited.

2 1

Database files across multiple directories
by julia.jacobson＠arcor.de 28 Apr '15

28 Apr '15

Hello everybody out there using BaseX, Is there a possibility to have xml files in multiple subdirectories? I would like to sort binary files according to date into different directories and write the corresponding metadata next to them in an xml file. Alternative solutions are also appreciated. Kind regards, Julia

2 1

XLink and XPointer
by Eckey, Florian 27 Apr '15

27 Apr '15

Hello, i am actually writing my master thesis using basex for xml processing. I need to create consistency links between two xml documents, especially basex databases and its content. So I am certainly looking for xlink and xpointer implementations in basex. Are there any examples to do this with basex or if not, has anyone implemented an xquery version to execute it? I recoginzed that there are capabilities with xinclude. But I does not find any references of xinclude in basex, unless this: http://docs.basex.org/wiki/Options#XINCLUDE So has anyone ideas or experiences using xlink and especially xpointer with basex or xquery? Regards, Florian ------------------------------------------------------- >>> business. people. technology. <<< ------------------------------------------------------- adesso AG mit Sitz in Dortmund Vorstand: Michael Kenfenheuer (Co-Vors.), Dr. Rüdiger Striemer (Co-Vors.), Christoph Junge, Andreas Prenneis Vorsitzender des Aufsichtsrates: Prof. Dr. Volker Gruhn Amtsgericht Dortmund HRB 20663

2 6

Large memory basex instances
by cmarchand＠oxiane.com 25 Apr '15

25 Apr '15

Hi ! in J2EE servers, it is a performance fault to make instances with more than 2Gb Xmx, because of the GC pauses. In baseX, uge xqueries on uge databases need a lot of memory. But looking at memory consumption (Runtime.freeMemory), it appears that a lot of GC are run during a query. GC pauses are time-lost, and I'm looking for a way to accelerate queries. Does someone has experience on giving large quantities of memory (above 8Gb) to a basex server ? Does someone has experience on using no-GC-pauses JVM (like Azul Zing) with baseX ? Best regards, Christophe

3 2

Running BaseX on Solaris
by Goetz Heller 25 Apr '15

25 Apr '15

Are there any experiences with running BaseX on Solaris? Kind regards Goetz

1 0

Fwd: IllegalMonitorStateException at org.basex.core.locks.DBLocking
by Simon Chatelain 25 Apr '15

25 Apr '15

Oups, forgot to put the mailing list in copy. Hello Christian, After some testing on my side, I didn't see the ConcurrentModificationException any more. That's the good news. However, when running a slightly modified version of the small test case you wrote from my sample application, I faced another problem. The test can run for a few seconds to almost an hour but eventually, the following exception is thrown. java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) at org.basex.core.locks.DBLocking.release(DBLocking.java:215) at org.basex.core.Context.unregister(Context.java:287) at org.basex.core.Command.execute(Command.java:105) at org.basex.api.client.LocalSession.execute(LocalSession.java:132) at org.basex.api.client.Session.execute(Session.java:36) at basextest.BaseXTestDBLocking$1.run(BaseXTestDBLocking.java:43) I attach my new test case (BaseXTestAdd.java), but the main modification is that in each created thread I also add documents to the collections instead of only opening it. I was also able to see that in the call to getOrCreateLock() in DBLocking#release(final Proc pr) (line 212) the lock is created while it should already be in the locks Map, but I really cannot understand how this is possible. It would mean that the lock was removed by another thread but for that the usage value must be wrong in the lockUsage map, and I cannot find any sequence of operation that could lead to such a situation. Trying to pin-point more precisely the problem, I wrote another test ( BaseXTestDBLocking.java) that calls directly the acquire and release methods of the DBLocking class. The problem seems to happen more quickly with this test. Any thoughts ? Regards Simon On Tue, Apr 14, 2015 at 3:31 PM, Simon Chatelain <schatela(a)gmail.com> wrote: > That was fast :-) . Thank you very much. I will test it right away. > > Regards > > Simon > > On Tue, Apr 14, 2015 at 3:28 PM, Christian Grün <christian.gruen(a)gmail.com > > wrote: > >> Simon, >> >> I am glad to report the concurrency bug in the user access code has >> been fixed [1,2]. BaseX 8.1.1 may already be released this week. Of >> course, please report back to me if you should encounter additional >> bugs. >> >> Thanks for testing, >> Christian >> >> [1] >> https://github.com/BaseXdb/basex/commit/b5d1716e662a300d484cbbb19d1bf23cbef… >> [2] http://files.basex.org/releases/latest >> >> >> >> On Tue, Apr 14, 2015 at 2:44 PM, Christian Grün >> <christian.gruen(a)gmail.com> wrote: >> > I have minimized your example a bit (attached), and I agree it seems >> > to be the user iterator that causes surprises. >> > >> > Looking at it.. >> > Christian >> > >> > >> > >> > On Tue, Apr 14, 2015 at 1:48 PM, Christian Grün >> > <christian.gruen(a)gmail.com> wrote: >> >> Hi Simon, >> >> >> >> Thanks for the updated code. It's still quite a lot code.. Is all of >> >> it required to reproduce the bug? E.g., does the bug only occur with >> >> the specified options etc? Could you possibly reduce it even more? >> >> Idealle, we can later rewrite it to a JUnit test once the problem is >> >> fixed (and we try to keep our tests as concise as possible). >> >> >> >> But I'll have some thoughts about your helpful assumptions in parallel. >> >> >> >> Thanks in advance, >> >> Christian >> >> >> >> >> >> On Tue, Apr 14, 2015 at 1:10 PM, Simon Chatelain <schatela(a)gmail.com> >> wrote: >> >>> Hello Christian, >> >>> >> >>> I checked again my code, but I am quite sure that each LocalSession is >> >>> created and used by only one thread. >> >>> I attach the test application code. >> >>> >> >>> Each LocalSession is used to execute several commands, but that >> shouldn't be >> >>> a problem, right ? >> >>> >> >>> I also had a look in BaseX code, especially in the >> org.basex.core.users.User >> >>> class. >> >>> >> >>> What happens in my test application is that I create several Writer >> that >> >>> will add XML document each in a specific collection, but using the >> same >> >>> user. >> >>> Each time I create a new Writer it will first create the collection >> then >> >>> grant read and write permission to this collection to the user >> "USER", and >> >>> then start adding XML document in this collection. >> >>> >> >>> >> >>> My suspicion is that the problem occurs when a new Writer is created, >> and it >> >>> is granting permission to the newly created collection while another >> Writer >> >>> (in another thread) is adding a new XML document in its own >> collection. >> >>> If, during the execution of the Grant command, the method >> User#perm(final >> >>> Perm prm, final String pattern) is called while the execution of a >> >>> concurrent Write command is at the same time executing the >> User#find(final >> >>> String db) method, then we have a ConcurrentModificationException as >> perm >> >>> method will put a new entry in locals field while the find method is >> >>> iterating on the locals entrySet of the same User, the same User >> instance >> >>> being shared by all subcontext created when opening a new >> LocalSession. >> >>> For test purposes, I synchronized the access to locals in the perm(), >> find() >> >>> and remove() methods of the class User, in that case I do not have the >> >>> ConcurrentModificationException any more. >> >>> >> >>> What do you think, is this possible ? >> >>> >> >>> Regards >> >>> >> >>> Simon >> >>> >> >>> On Mon, Apr 13, 2015 at 11:04 AM, Christian Grün < >> christian.gruen(a)gmail.com> >> >>> wrote: >> >>>> >> >>>> > But the java.util.ConcurrentModificationException >> >>>> > at org.basex.core.users.User.find >> >>>> > is still happening. >> >>>> >> >>>> I am pretty sure that any of the concurrent command calls are not >> >>>> moved to their own thread yet. >> >>>> >> >>>> But feel free to pass me on your code (ideally, a version that's as >> >>>> small as possible). >> >>>> >> >>>> Best, >> >>>> Christian >> >>>> >> >>>> >> >>>> >> >>>> > >> >>>> > What I did is basically, in the test application, replace all >> session >> >>>> > creations >> >>>> > >> >>>> > try (final Session session = new LocalSession(rootContext, ...)) { >> >>>> > ... >> >>>> > } >> >>>> > catch {...} >> >>>> > >> >>>> > by >> >>>> > >> >>>> > Thread t = new Thread(new Runnable() { public void run() { >> >>>> > try (final Session session = new LocalSession(rootContext, ...)) { >> >>>> > ... >> >>>> > } >> >>>> > catch {...} >> >>>> > >> >>>> > } >> >>>> > }); >> >>>> > >> >>>> > Not very elegant but should be enough to make sure every session is >> >>>> > executed >> >>>> > in its own Java thread. >> >>>> > >> >>>> > >> >>>> > Am I still missing something ? >> >>>> > >> >>>> > Thank you very much for your help, it is very appreciated. >> >>>> > >> >>>> > Regards >> >>>> > >> >>>> > Simon >> >>>> > >> >>>> > >> >>>> > On Tue, Apr 7, 2015 at 12:16 PM, Christian Grün >> >>>> > <christian.gruen(a)gmail.com> >> >>>> > wrote: >> >>>> >> >> >>>> >> Hi Simon, >> >>>> >> >> >>>> >> Thanks for the detailed description of your setup and your test >> code. >> >>>> >> >> >>>> >> In BaseX, every session needs to be executed in a separate Java >> >>>> >> thread. The exceptions will e.g. disappear if you pass on your >> admin >> >>>> >> session to the store() and updateMetaStore() functions (or if you >> run >> >>>> >> these operations in a separate thread). >> >>>> >> >> >>>> >> Hope this helps, >> >>>> >> Christian >> >>>> >> >> >>>> >> >> >>>> >> On Thu, Apr 2, 2015 at 1:58 PM, Simon Chatelain < >> schatela(a)gmail.com> >> >>>> >> wrote: >> >>>> >> > Hello all, >> >>>> >> > >> >>>> >> > I have the following problem. >> >>>> >> > >> >>>> >> > Using: >> >>>> >> > >> >>>> >> > BaseX 8.1.1 beta cf713e2 (20150328.133759) >> >>>> >> > Java 1.8.0_31 >> >>>> >> > Windows 7 >> >>>> >> > >> >>>> >> > The context: I have several sources of data from which I >> receive XML >> >>>> >> > documents. My application will create a collection for each of >> these >> >>>> >> > sources >> >>>> >> > and store each received document into the collection >> corresponding to >> >>>> >> > the >> >>>> >> > source, and then update some kind of metadata about the >> collection. >> >>>> >> > The >> >>>> >> > metadata of all collections are stored into a common >> collection. Each >> >>>> >> > source >> >>>> >> > is handled by a different thread. >> >>>> >> > I am using an embedded BaseX, and access to it through >> LocalSession. >> >>>> >> > >> >>>> >> > Now the problem is that after a certain amount of time (not >> constant, >> >>>> >> > could >> >>>> >> > be seconds or minutes) there is some exceptions. >> >>>> >> > Either: java.lang.IllegalMonitorStateException >> >>>> >> > at org.basex.core.locks.DBLocking.acquire >> >>>> >> > and org.basex.core.locks.DBLocking.release >> >>>> >> > >> >>>> >> > Or: java.util.ConcurrentModificationException >> >>>> >> > at org.basex.core.users.User.find >> >>>> >> > >> >>>> >> > I put the full stack trace in attachment. >> >>>> >> > >> >>>> >> > I build a test application that mimic what happens in the real >> >>>> >> > application >> >>>> >> > and it seems to exhibit the same problem. The source of this >> test >> >>>> >> > application is attached >> >>>> >> > >> >>>> >> > My question: Am I doing something wrong, or is there a bug in >> BaseX? >> >>>> >> > >> >>>> >> > Thanks a lot >> >>>> >> > >> >>>> >> > Simon >> >>>> > >> >>>> > >> >>> >> >>> >> > >

3 5

Basex xquery import module classpath lookup
by Karel Hovorka 24 Apr '15

24 Apr '15

Hi, I am trying to run xquery file on basex, but it depends on xquery module located in classpath. I have found parameter "org.basex.QUERYPATH", that configures module lookup on filesystem. Is there way to lookup modules in classpath? Thanks, Karel Hovorka, Semantico LTD

3 25

Best Forum for Generate XQuery How To Discussions?
by Eliot Kimber 23 Apr '15

23 Apr '15

What is the best forum for asking general how-to XQuery questions that are not BaseX specific? Thanks, Eliot ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 1