- BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] BaseX Questions..
by Christian Grün 18 Jan '10

18 Jan '10

Dear Tyler, you might also be interested in looking into the following lecture/seminar slides: http://www.inf.uni-konstanz.de/dbis/teaching/ws0708/xml/24-Native.pdf http://www.inf.uni-konstanz.de/dbis/teaching/ws0708/dbissem/BaseXInside.pdf Apart from that and the listed publications.. http://www.inf.uni-konstanz.de/dbis/basex/publications.php we cannot provide you with more details on our implementation (yet). Basically, we are storing all data in a flat table representation. Table entries take 16 bytes each, and all entries are stored in blocks of 4kb, the order of which is referenced by a main memory directory. The nodes' pre values serve as node ids, which are unique during a single update transaction. If you need to know more, feel free to ask. Best, Christian FYI: I've forwarded this mail to the basex-talk mailing list. On Mon, Jan 18, 2010 at 4:21 PM, Tyler Corbin <094568c(a)acadiau.ca> wrote: > I don’t know if you can disclose this type of information – But I’m really > interested in the type of structures you’ve implemented for your BaseX and > how you achieved such quick updating and querying time. I’m doing some heavy > research into how different native XML Databases go about this, and it seems > BaseX is near the top (On performance). > > Do you have a more recent publication, which covers on how you go about > updating XML Structures? I’ve read one of your more recent publications on > your Database Architecture and some other things... but it doesn’t really > touch on your implementation of updates. > > > > - Since you don’t update the index structures for single updates, > do you have some type of “pending list” where all updates that are not > officially added to the index structures can still be accessed by the > processor? If not, can you briefly explain the data structure (if allowed)? > > - Also, how much compression do you get when your Database > Architecture is performed? > > > > > > I also have a few more questions, but I’m positive your recent publication > covers it. > > > > Thank you for your time! > > ~ Tyler > > > > > > > > > >

1 0

Re: [basex-talk] Another index question
by Christian Grün 18 Jan '10

18 Jan '10

Hi Phil, in nearly all cases, the query optimizer will try to apply the existing index structures. As the XQuery syntax is much more versatile than SQL, this won't always succeed. - However, your query can be easily written. Three alternative writings, that all should access the index, look as follows: Query: //page[para[text() contains text "indiana"] and para[text() contains text "ninth"]] Alternative 1: //page[para/text() contains text "indiana"] [para/text() contains text "ninth"] Alternative 2: //page[para/text() contains text "indiana" and para/text() contains text "ninth"] Alternative 3: //page[para/text() contains text "indiana" ftand "ninth"] All three queries will yield different execution plans, and - to be strict - the third plan is not equivalent to the first two, as it expects the two query terms to occur in the very same text node. - Check out all of the alternatives to see which one works out best for you. Christian On Mon, Jan 18, 2010 at 7:25 PM, Phil Oliver <phil3(a)olivercomputing.com> wrote: > Another question regarding usage of full-text indices. My test database has > the full text indexing enabled with no options currently for simplicity > (e.g. wildcards, stemming, etc. disabled). > > A simple query such as: > > let $p:=//para[text() contains text "indiana"] > return $p > > executes rapidly using the FT index selection. However, if I try a more > complex query (designed to find two words on the same page): > > let $p:=//page[para[text() contains text "indiana"] and para[text() contains > text "ninth"]] > return $p > > the query plan does *not* use the FT index and takes a long time to execute. > > While this is a more complex query, it doesn't seem *really* complex, and > would be a common use case. Can you give me some idea as to why the FT > indices are not used in this instance, and if possible a suggestion on a > rewritten query that would use them? Is there a way to give hints to the > optimizer to always use them? > >

1 0

Question on xQuery Updates.
by Tyler Corbin 12 Jan '10

12 Jan '10

If you could answer these questions or give a URL to my questions, It would be greatly appreciated... Can you explain how this program goes about dealing with xQuery Updates? Because Ive been getting some insanely quick inserting / deletion times and out of every open-source database system Ive tinkered with, Ive never seen anything as fast as this. I have ideas on how you go about this (havent had time to look at the source code) and I couldnt find anything in the documentation (other than the indexing processes.) (Sorry for such a short question, I was in a rush typing this.) Much appreciated and great release! Tyler.

2 1

Re: [basex-talk] Clustered XML database
by Christian Grün 12 Jan '10

12 Jan '10

Hi Erol, it's true that one database and one single XQuery call won't be enough to process data of that size. As BaseX does not natively support distributed storage and parallel queries, I would suggest the classical (manual) way: split the data to multiple databases, run the query on each instance and eventually merge the textual results. This could be done in a few lines of code, provided that the queries are not sophisticated, the database contents are disjunct, no concurrency issues have to be considered, etc. Hope this helps, Christian On Tue, Jan 12, 2010 at 3:22 AM, Erol Akarsu <eakarsu(a)gmail.com> wrote: > Hi Christian, > > I appreciate a new release of BaseX. > > I have one question on storing a huge data,say 10-50 petabytes data, with > Basex. I expect current version may not handle it properly. > Can we distribute XML data across clusters of nodes and have a parallel > xquery to be run across nodes and get result? > I would like to know how it can be implemented such a big clustered xml > database and have efficient xquery on this cluster. > > Regards > > Erol Akarsu > > On Sun, Jan 10, 2010 at 4:49 PM, Christian Grün <christian.gruen(a)gmail.com> > wrote: >> >> Dear all, >> >> we are excited to finally announce the release of BaseX 6.0! It offers >> a bunch of new features, such as: >> >> - a full implementation of XQuery Update, reaching 100% in the W3C Test >> Suite. >> - an enhanced client/server architecture with ACID-safe transactions, >> user and password management and transaction logging. >> - extended XQuery Full Text features, supporting TF/IDF scoring, stop >> word lists, fuzzy querying, etc. >> - the improved creation of index structures, using constant memory. >> - optimizations to increase the use of index structures and iterative >> query processing. >> >> As usual, please find more information at http://basex.org/ >> >> All the best, >> >> Christian Grün >> BaseX Team >> _______________________________________________ >> BaseX-Announce mailing list >> BaseX-Announce(a)mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-announce > >

1 0

Re: [basex-talk] Initial observation on BaseX 6.0 release
by Christian Grün 12 Jan '10

12 Jan '10

Thanks Phil, indeed I forgot to mention that all index full-text options, that have been chosen for index creation, must be specified in the XQuery as well; see this: http://www.inf.uni-konstanz.de/dbis/basex/xquery.php#ftindex We took this approach to be fully compliant with the xqft language extension - which, by default, does not activate any of the available match options. Hope this helps, Christian On Tue, Jan 12, 2010 at 12:04 AM, Phil Oliver <phil3(a)olivercomputing.com> wrote: > After some experimentation, I've identified the fulltext search index > problem. Hopefully it's something easily fixed. > > I did have the FT indexing turned on, but basically with all of the various > options. I decided to experiment with the minimal index and then to > systematically add options. It turns out that the query does use the FT > index in the instance where there are no options, or with wildcards turned > on. However, the index is no longer used once stemming, case sensitivity, or > diacritics are turned on in the FT database indexing options. > > The speed is impressive with the FT indexing enabled, I think good enough > for me to plan to use it in development. However, the other FT options are > potentially important (to me and others) and I hope it's a priority to fix > the indexing problem. > > Thanks for your help. > > Phil ___________________________ Christian Gruen Universitaet Konstanz Department of Computer & Information Science D-78457 Konstanz, Germany Tel: +49 (0)7531/88-4449, Fax: +49 (0)7531/88-3577 http://www.inf.uni-konstanz.de/~gruen

1 0

Initial observation on BaseX 6.0 release
by Phil 10 Jan '10

10 Jan '10

I've been looking forward to the new release because I really want to use an XQuery Fulltext system for a search engine for a reasonably large (200,000 printed page equivalent) XML structured text-oriented database. However, my initial test, on a large fraction of the total database (certainly over 120,000 pages printed equivalent) yields disappointing results. The XML schema is roughly based on Docbook, so it is not unusual. Most of the actual text is within PARA elements. My test query was: //para[. ftcontains "sumter"] On a dual quad Xeon system the query took about 28 seconds to run. This is completely unusable in my context. I am pretty sure I know the reason for the poor performance. The derived query plan resolves to: <QueryPlan> <IterPath> <Root/> <IterStep axis="descendant" test="*:para"> <FTContains> <Context/> <FTWords>sumter</FTWords> </FTContains> </IterStep> </IterPath> </QueryPlan> Now, there are *millions* of PARA elements in the database - but not so many (hundreds) of references to the word "sumter". And this is by no means an uncommon XML structure for a text-oriented system. The problem is obvious. There are an absolutely enormous number of PARA elements and very few actual text hits. The IterStep is evidently hitting every single PARA element. The query optimization should be checking the fulltext search index and using those results as an initial filter. Then the IterStep over PARAs from that filtered set would be highly productive. Is there a way to get that kind of (properly) optimized query from the system, or is this hopelessly built into the current architecture? I do hope that I get some kind of response this time, because at least two previous queries on earlier releases of BaseX went unanswered.

2 1

inaugurating the new mailing list
by Christian Grün 10 Jan '10

10 Jan '10

Hi everyone, c'est voilà.. the new BaseX mailing list for talks and discussions, which might get the new Platzhirsch and gradually replace info(a)basex.org. Depending on the nature and number of future subscribers, English might emerge as the predominant communication language (although Mandarin, Hindi, Urdu etc. are welcome as well). As most of you know, BaseX 6.0 will be released this very night. By accident, nearly all todos have already been worked at. If, however, you still have some incredibly or surprisingly much time left you're more than welcome and invited to - check the new homepage http://www.inf.uni-konstanz.de/dbis/basex2/ for typos and other embarrassments - deliver me and all of us with updated screenshots & videos - create handsome 16x16, 32x32, 48x48, ... (16/32bit) versions of the BaseX logo. Check out the sources at: https://svn.uni-konstanz.de/dbis/basex/trunk/basex/etc/images/ - notify everyone about remaining tacky, persistent bugs in the code (well.. just in case) ありがとう！ Christian

1 0