January 2010 - BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] Clustered XML database
by Christian Grün 12 Jan '10

12 Jan '10

Hi Erol, it's true that one database and one single XQuery call won't be enough to process data of that size. As BaseX does not natively support distributed storage and parallel queries, I would suggest the classical (manual) way: split the data to multiple databases, run the query on each instance and eventually merge the textual results. This could be done in a few lines of code, provided that the queries are not sophisticated, the database contents are disjunct, no concurrency issues have to be considered, etc. Hope this helps, Christian On Tue, Jan 12, 2010 at 3:22 AM, Erol Akarsu <eakarsu(a)gmail.com> wrote: > Hi Christian, > > I appreciate a new release of BaseX. > > I have one question on storing a huge data,say 10-50 petabytes data, with > Basex. I expect current version may not handle it properly. > Can we distribute XML data across clusters of nodes and have a parallel > xquery to be run across nodes and get result? > I would like to know how it can be implemented such a big clustered xml > database and have efficient xquery on this cluster. > > Regards > > Erol Akarsu > > On Sun, Jan 10, 2010 at 4:49 PM, Christian Grün <christian.gruen(a)gmail.com> > wrote: >> >> Dear all, >> >> we are excited to finally announce the release of BaseX 6.0! It offers >> a bunch of new features, such as: >> >> - a full implementation of XQuery Update, reaching 100% in the W3C Test >> Suite. >> - an enhanced client/server architecture with ACID-safe transactions, >> user and password management and transaction logging. >> - extended XQuery Full Text features, supporting TF/IDF scoring, stop >> word lists, fuzzy querying, etc. >> - the improved creation of index structures, using constant memory. >> - optimizations to increase the use of index structures and iterative >> query processing. >> >> As usual, please find more information at http://basex.org/ >> >> All the best, >> >> Christian Grün >> BaseX Team >> _______________________________________________ >> BaseX-Announce mailing list >> BaseX-Announce(a)mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-announce > >

1 0

Re: [basex-talk] Initial observation on BaseX 6.0 release
by Christian Grün 12 Jan '10

12 Jan '10

Thanks Phil, indeed I forgot to mention that all index full-text options, that have been chosen for index creation, must be specified in the XQuery as well; see this: http://www.inf.uni-konstanz.de/dbis/basex/xquery.php#ftindex We took this approach to be fully compliant with the xqft language extension - which, by default, does not activate any of the available match options. Hope this helps, Christian On Tue, Jan 12, 2010 at 12:04 AM, Phil Oliver <phil3(a)olivercomputing.com> wrote: > After some experimentation, I've identified the fulltext search index > problem. Hopefully it's something easily fixed. > > I did have the FT indexing turned on, but basically with all of the various > options. I decided to experiment with the minimal index and then to > systematically add options. It turns out that the query does use the FT > index in the instance where there are no options, or with wildcards turned > on. However, the index is no longer used once stemming, case sensitivity, or > diacritics are turned on in the FT database indexing options. > > The speed is impressive with the FT indexing enabled, I think good enough > for me to plan to use it in development. However, the other FT options are > potentially important (to me and others) and I hope it's a priority to fix > the indexing problem. > > Thanks for your help. > > Phil ___________________________ Christian Gruen Universitaet Konstanz Department of Computer & Information Science D-78457 Konstanz, Germany Tel: +49 (0)7531/88-4449, Fax: +49 (0)7531/88-3577 http://www.inf.uni-konstanz.de/~gruen

1 0

Initial observation on BaseX 6.0 release
by Phil 10 Jan '10

10 Jan '10

I've been looking forward to the new release because I really want to use an XQuery Fulltext system for a search engine for a reasonably large (200,000 printed page equivalent) XML structured text-oriented database. However, my initial test, on a large fraction of the total database (certainly over 120,000 pages printed equivalent) yields disappointing results. The XML schema is roughly based on Docbook, so it is not unusual. Most of the actual text is within PARA elements. My test query was: //para[. ftcontains "sumter"] On a dual quad Xeon system the query took about 28 seconds to run. This is completely unusable in my context. I am pretty sure I know the reason for the poor performance. The derived query plan resolves to: <QueryPlan> <IterPath> <Root/> <IterStep axis="descendant" test="*:para"> <FTContains> <Context/> <FTWords>sumter</FTWords> </FTContains> </IterStep> </IterPath> </QueryPlan> Now, there are *millions* of PARA elements in the database - but not so many (hundreds) of references to the word "sumter". And this is by no means an uncommon XML structure for a text-oriented system. The problem is obvious. There are an absolutely enormous number of PARA elements and very few actual text hits. The IterStep is evidently hitting every single PARA element. The query optimization should be checking the fulltext search index and using those results as an initial filter. Then the IterStep over PARAs from that filtered set would be highly productive. Is there a way to get that kind of (properly) optimized query from the system, or is this hopelessly built into the current architecture? I do hope that I get some kind of response this time, because at least two previous queries on earlier releases of BaseX went unanswered.

2 1

inaugurating the new mailing list
by Christian Grün 10 Jan '10

10 Jan '10

Hi everyone, c'est voilà.. the new BaseX mailing list for talks and discussions, which might get the new Platzhirsch and gradually replace info(a)basex.org. Depending on the nature and number of future subscribers, English might emerge as the predominant communication language (although Mandarin, Hindi, Urdu etc. are welcome as well). As most of you know, BaseX 6.0 will be released this very night. By accident, nearly all todos have already been worked at. If, however, you still have some incredibly or surprisingly much time left you're more than welcome and invited to - check the new homepage http://www.inf.uni-konstanz.de/dbis/basex2/ for typos and other embarrassments - deliver me and all of us with updated screenshots & videos - create handsome 16x16, 32x32, 48x48, ... (16/32bit) versions of the BaseX logo. Check out the sources at: https://svn.uni-konstanz.de/dbis/basex/trunk/basex/etc/images/ - notify everyone about remaining tacky, persistent bugs in the code (well.. just in case) ありがとう！ Christian

1 0