- BaseX-Talk - mailman.uni-konstanz.de

Reg : Map Reduce Frame Work
by Chandrasekhar 29 Apr '15

29 Apr '15

Dear Team, Information required about map reduce framework for xquery performence.Suggest us to best practice documents on this. Thanks&Regards ChandraSekhar The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited.

2 1

Database files across multiple directories
by julia.jacobson＠arcor.de 28 Apr '15

28 Apr '15

Hello everybody out there using BaseX, Is there a possibility to have xml files in multiple subdirectories? I would like to sort binary files according to date into different directories and write the corresponding metadata next to them in an xml file. Alternative solutions are also appreciated. Kind regards, Julia

2 1

XLink and XPointer
by Eckey, Florian 27 Apr '15

27 Apr '15

Hello, i am actually writing my master thesis using basex for xml processing. I need to create consistency links between two xml documents, especially basex databases and its content. So I am certainly looking for xlink and xpointer implementations in basex. Are there any examples to do this with basex or if not, has anyone implemented an xquery version to execute it? I recoginzed that there are capabilities with xinclude. But I does not find any references of xinclude in basex, unless this: http://docs.basex.org/wiki/Options#XINCLUDE So has anyone ideas or experiences using xlink and especially xpointer with basex or xquery? Regards, Florian ------------------------------------------------------- >>> business. people. technology. <<< ------------------------------------------------------- adesso AG mit Sitz in Dortmund Vorstand: Michael Kenfenheuer (Co-Vors.), Dr. Rüdiger Striemer (Co-Vors.), Christoph Junge, Andreas Prenneis Vorsitzender des Aufsichtsrates: Prof. Dr. Volker Gruhn Amtsgericht Dortmund HRB 20663

2 6

Large memory basex instances
by cmarchand＠oxiane.com 26 Apr '15

26 Apr '15

Hi ! in J2EE servers, it is a performance fault to make instances with more than 2Gb Xmx, because of the GC pauses. In baseX, uge xqueries on uge databases need a lot of memory. But looking at memory consumption (Runtime.freeMemory), it appears that a lot of GC are run during a query. GC pauses are time-lost, and I'm looking for a way to accelerate queries. Does someone has experience on giving large quantities of memory (above 8Gb) to a basex server ? Does someone has experience on using no-GC-pauses JVM (like Azul Zing) with baseX ? Best regards, Christophe

3 2

Running BaseX on Solaris
by Goetz Heller 25 Apr '15

25 Apr '15

Are there any experiences with running BaseX on Solaris? Kind regards Goetz

1 0

Fwd: IllegalMonitorStateException at org.basex.core.locks.DBLocking
by Simon Chatelain 25 Apr '15

25 Apr '15

Oups, forgot to put the mailing list in copy. Hello Christian, After some testing on my side, I didn't see the ConcurrentModificationException any more. That's the good news. However, when running a slightly modified version of the small test case you wrote from my sample application, I faced another problem. The test can run for a few seconds to almost an hour but eventually, the following exception is thrown. java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) at org.basex.core.locks.DBLocking.release(DBLocking.java:215) at org.basex.core.Context.unregister(Context.java:287) at org.basex.core.Command.execute(Command.java:105) at org.basex.api.client.LocalSession.execute(LocalSession.java:132) at org.basex.api.client.Session.execute(Session.java:36) at basextest.BaseXTestDBLocking$1.run(BaseXTestDBLocking.java:43) I attach my new test case (BaseXTestAdd.java), but the main modification is that in each created thread I also add documents to the collections instead of only opening it. I was also able to see that in the call to getOrCreateLock() in DBLocking#release(final Proc pr) (line 212) the lock is created while it should already be in the locks Map, but I really cannot understand how this is possible. It would mean that the lock was removed by another thread but for that the usage value must be wrong in the lockUsage map, and I cannot find any sequence of operation that could lead to such a situation. Trying to pin-point more precisely the problem, I wrote another test ( BaseXTestDBLocking.java) that calls directly the acquire and release methods of the DBLocking class. The problem seems to happen more quickly with this test. Any thoughts ? Regards Simon On Tue, Apr 14, 2015 at 3:31 PM, Simon Chatelain <schatela(a)gmail.com> wrote: > That was fast :-) . Thank you very much. I will test it right away. > > Regards > > Simon > > On Tue, Apr 14, 2015 at 3:28 PM, Christian Grün <christian.gruen(a)gmail.com > > wrote: > >> Simon, >> >> I am glad to report the concurrency bug in the user access code has >> been fixed [1,2]. BaseX 8.1.1 may already be released this week. Of >> course, please report back to me if you should encounter additional >> bugs. >> >> Thanks for testing, >> Christian >> >> [1] >> https://github.com/BaseXdb/basex/commit/b5d1716e662a300d484cbbb19d1bf23cbef… >> [2] http://files.basex.org/releases/latest >> >> >> >> On Tue, Apr 14, 2015 at 2:44 PM, Christian Grün >> <christian.gruen(a)gmail.com> wrote: >> > I have minimized your example a bit (attached), and I agree it seems >> > to be the user iterator that causes surprises. >> > >> > Looking at it.. >> > Christian >> > >> > >> > >> > On Tue, Apr 14, 2015 at 1:48 PM, Christian Grün >> > <christian.gruen(a)gmail.com> wrote: >> >> Hi Simon, >> >> >> >> Thanks for the updated code. It's still quite a lot code.. Is all of >> >> it required to reproduce the bug? E.g., does the bug only occur with >> >> the specified options etc? Could you possibly reduce it even more? >> >> Idealle, we can later rewrite it to a JUnit test once the problem is >> >> fixed (and we try to keep our tests as concise as possible). >> >> >> >> But I'll have some thoughts about your helpful assumptions in parallel. >> >> >> >> Thanks in advance, >> >> Christian >> >> >> >> >> >> On Tue, Apr 14, 2015 at 1:10 PM, Simon Chatelain <schatela(a)gmail.com> >> wrote: >> >>> Hello Christian, >> >>> >> >>> I checked again my code, but I am quite sure that each LocalSession is >> >>> created and used by only one thread. >> >>> I attach the test application code. >> >>> >> >>> Each LocalSession is used to execute several commands, but that >> shouldn't be >> >>> a problem, right ? >> >>> >> >>> I also had a look in BaseX code, especially in the >> org.basex.core.users.User >> >>> class. >> >>> >> >>> What happens in my test application is that I create several Writer >> that >> >>> will add XML document each in a specific collection, but using the >> same >> >>> user. >> >>> Each time I create a new Writer it will first create the collection >> then >> >>> grant read and write permission to this collection to the user >> "USER", and >> >>> then start adding XML document in this collection. >> >>> >> >>> >> >>> My suspicion is that the problem occurs when a new Writer is created, >> and it >> >>> is granting permission to the newly created collection while another >> Writer >> >>> (in another thread) is adding a new XML document in its own >> collection. >> >>> If, during the execution of the Grant command, the method >> User#perm(final >> >>> Perm prm, final String pattern) is called while the execution of a >> >>> concurrent Write command is at the same time executing the >> User#find(final >> >>> String db) method, then we have a ConcurrentModificationException as >> perm >> >>> method will put a new entry in locals field while the find method is >> >>> iterating on the locals entrySet of the same User, the same User >> instance >> >>> being shared by all subcontext created when opening a new >> LocalSession. >> >>> For test purposes, I synchronized the access to locals in the perm(), >> find() >> >>> and remove() methods of the class User, in that case I do not have the >> >>> ConcurrentModificationException any more. >> >>> >> >>> What do you think, is this possible ? >> >>> >> >>> Regards >> >>> >> >>> Simon >> >>> >> >>> On Mon, Apr 13, 2015 at 11:04 AM, Christian Grün < >> christian.gruen(a)gmail.com> >> >>> wrote: >> >>>> >> >>>> > But the java.util.ConcurrentModificationException >> >>>> > at org.basex.core.users.User.find >> >>>> > is still happening. >> >>>> >> >>>> I am pretty sure that any of the concurrent command calls are not >> >>>> moved to their own thread yet. >> >>>> >> >>>> But feel free to pass me on your code (ideally, a version that's as >> >>>> small as possible). >> >>>> >> >>>> Best, >> >>>> Christian >> >>>> >> >>>> >> >>>> >> >>>> > >> >>>> > What I did is basically, in the test application, replace all >> session >> >>>> > creations >> >>>> > >> >>>> > try (final Session session = new LocalSession(rootContext, ...)) { >> >>>> > ... >> >>>> > } >> >>>> > catch {...} >> >>>> > >> >>>> > by >> >>>> > >> >>>> > Thread t = new Thread(new Runnable() { public void run() { >> >>>> > try (final Session session = new LocalSession(rootContext, ...)) { >> >>>> > ... >> >>>> > } >> >>>> > catch {...} >> >>>> > >> >>>> > } >> >>>> > }); >> >>>> > >> >>>> > Not very elegant but should be enough to make sure every session is >> >>>> > executed >> >>>> > in its own Java thread. >> >>>> > >> >>>> > >> >>>> > Am I still missing something ? >> >>>> > >> >>>> > Thank you very much for your help, it is very appreciated. >> >>>> > >> >>>> > Regards >> >>>> > >> >>>> > Simon >> >>>> > >> >>>> > >> >>>> > On Tue, Apr 7, 2015 at 12:16 PM, Christian Grün >> >>>> > <christian.gruen(a)gmail.com> >> >>>> > wrote: >> >>>> >> >> >>>> >> Hi Simon, >> >>>> >> >> >>>> >> Thanks for the detailed description of your setup and your test >> code. >> >>>> >> >> >>>> >> In BaseX, every session needs to be executed in a separate Java >> >>>> >> thread. The exceptions will e.g. disappear if you pass on your >> admin >> >>>> >> session to the store() and updateMetaStore() functions (or if you >> run >> >>>> >> these operations in a separate thread). >> >>>> >> >> >>>> >> Hope this helps, >> >>>> >> Christian >> >>>> >> >> >>>> >> >> >>>> >> On Thu, Apr 2, 2015 at 1:58 PM, Simon Chatelain < >> schatela(a)gmail.com> >> >>>> >> wrote: >> >>>> >> > Hello all, >> >>>> >> > >> >>>> >> > I have the following problem. >> >>>> >> > >> >>>> >> > Using: >> >>>> >> > >> >>>> >> > BaseX 8.1.1 beta cf713e2 (20150328.133759) >> >>>> >> > Java 1.8.0_31 >> >>>> >> > Windows 7 >> >>>> >> > >> >>>> >> > The context: I have several sources of data from which I >> receive XML >> >>>> >> > documents. My application will create a collection for each of >> these >> >>>> >> > sources >> >>>> >> > and store each received document into the collection >> corresponding to >> >>>> >> > the >> >>>> >> > source, and then update some kind of metadata about the >> collection. >> >>>> >> > The >> >>>> >> > metadata of all collections are stored into a common >> collection. Each >> >>>> >> > source >> >>>> >> > is handled by a different thread. >> >>>> >> > I am using an embedded BaseX, and access to it through >> LocalSession. >> >>>> >> > >> >>>> >> > Now the problem is that after a certain amount of time (not >> constant, >> >>>> >> > could >> >>>> >> > be seconds or minutes) there is some exceptions. >> >>>> >> > Either: java.lang.IllegalMonitorStateException >> >>>> >> > at org.basex.core.locks.DBLocking.acquire >> >>>> >> > and org.basex.core.locks.DBLocking.release >> >>>> >> > >> >>>> >> > Or: java.util.ConcurrentModificationException >> >>>> >> > at org.basex.core.users.User.find >> >>>> >> > >> >>>> >> > I put the full stack trace in attachment. >> >>>> >> > >> >>>> >> > I build a test application that mimic what happens in the real >> >>>> >> > application >> >>>> >> > and it seems to exhibit the same problem. The source of this >> test >> >>>> >> > application is attached >> >>>> >> > >> >>>> >> > My question: Am I doing something wrong, or is there a bug in >> BaseX? >> >>>> >> > >> >>>> >> > Thanks a lot >> >>>> >> > >> >>>> >> > Simon >> >>>> > >> >>>> > >> >>> >> >>> >> > >

3 5

Basex xquery import module classpath lookup
by Karel Hovorka 24 Apr '15

24 Apr '15

Hi, I am trying to run xquery file on basex, but it depends on xquery module located in classpath. I have found parameter "org.basex.QUERYPATH", that configures module lookup on filesystem. Is there way to lookup modules in classpath? Thanks, Karel Hovorka, Semantico LTD

3 25

Best Forum for Generate XQuery How To Discussions?
by Eliot Kimber 23 Apr '15

23 Apr '15

What is the best forum for asking general how-to XQuery questions that are not BaseX specific? Thanks, Eliot ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 1

multi-language full-text indexing
by Goetz Heller 23 Apr '15

23 Apr '15

Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g. CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( (path_a)/PART_A, (path_b)/ PART_B,… ) Kind regards, Goetz -----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing > It is desirable to have > documents indexed by locale-specific parts, e.g. I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture. As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). Hope this helps, Christian [1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg04837.ht… [2] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06089.ht… > > > > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( > > (path_a)/LOCALIZED_PART_A[@LANG=$lang], > > (path_b)/LOCALIZED_PART_B[@LG=$lang],… > > ) FOR LANGUAGE $lang IN ( > > BG, > > DN, > > DE WITH STOPWORDS filepath_de WITH STEM = YES, > > EN WITH STOPWORDS filepath_en, > > FR, … > > ) [USING language_code_map] > > and then to write full-text retrieval queries with a clause such as > ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller > and full-text retrieval therefore much faster. The language codes > would be mapped somehow to standard values recognized by BaseX in the > language_code_map file. > > Are there any efforts towards such a feature?

4 4

Creation of Full-Text-Index failed
by Goetz Heller 22 Apr '15

22 Apr '15

For the task at hand I need to create a database on a daily base from file packages I received. The language taken here is German, however the files contain lots of international characters as well. Usually this does not harm, and I don't know if this is the real cause of failure in this case. Actually, the database was created, but an error message occurred which was not very specific: "file xxx could not be parsed". File "xxx" was the last file of the package, and it was accessible for xQuery search. However, no full-text index was created as with the other packages. Trying to create the index directly resulted in a different message: "Improper use? . Stack Trace: java.lang.ArrayIndexOutOfBoundsException". The package can be downloaded from http://www.hellerim.de/downloads/BaseX/20150203_023.7z. This does not look like a problem with the data but rather like a bug in BaseX. If I'm wrong, however, I would prefer to get a message which points me to the problem so I can try to solve it. Kind regards, Goetz

2 2