- BaseX-Talk - mailman.uni-konstanz.de

Apache Lucene for full text search ?
by Erol Akarsu 13 Apr '11

13 Apr '11

I have another question. Does Basex use Apache Lucene for full text search or have its own search engine? Thanks On Mon, Apr 11, 2011 at 12:35 PM, Erol Akarsu <eakarsu(a)gmail.com> wrote: > Ok, > > I was able to run full text search with another XML Database. > > I am primarily interested in how Basex will play with big XML file > Wikipedia. > > Actually, Database create of wikipedia is fine. But when I add full tet > search and indexes for it, it always throws out of memory exception error. > > I have changed -Xmx with 6GB that is still not enough to generate indexes > for Wikipedia. > > Can you help me on how to generate indexes with a machine that case 6GB > for Basex process? > > Thanks > > Erol Akarsu > > > > On Sun, Apr 10, 2011 at 4:07 AM, Andreas Weiler < > andreas.weiler(a)uni-konstanz.de> wrote: > >> Hi, >> >> the following query could work for you: >> >> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >> for $i in doc("enwiki-latest-pages-articles")//w:sitename >> return $i[. contains text "Wikipedia"]/.. >> >> -- Andreas >> >> Am 09.04.2011 um 20:43 schrieb Erol Akarsu: >> >> Hi >> >> I am having difficulty in running full text operators. This script gives >> siteinfo below >> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >> let $d := fn:doc ("enwiki-latest-pages-articles") >> return ($d//w:siteinfo)[1] >> >> But return $d//w:siteinfo[w:sitename contains text 'Wikipedia'] does NOT >> give same node >> Why "contains" ft operator behave incorrectly? I remember it was working >> fine. I just dropped and recreated database and turn all indexes. Can you >> help me? >> Query info is here: >> >> Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >> Compiling: >> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >> - adding text() step >> - optimizing descendant-or-self step(s) >> - removing path with no index results >> - pre-evaluating (())[1] >> - binding static variable $res >> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >> - binding static variable $d >> - adding text() step >> - optimizing descendant-or-self step(s) >> - removing path with no index results >> - simplifying flwor expression >> Result: () >> Timing: >> - Parsing: 0.46 ms >> - Compiling: 0.42 ms >> - Evaluating: 0.17 ms >> - Printing: 0.1 ms >> - Total Time: 1.15 ms >> Query plan: >> <sequence size="0"/> >> >> >> >> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" xmlns:xsi=" >> http://www.w3.org/2001/XMLSchema-instance"> >> <sitename>Wikipedia</sitename> >> <base>http://en.wikipedia.org/wiki/Main_Page</base> >> <generator>MediaWiki 1.17wmf1</generator> >> <case>first-letter</case> >> <namespaces> >> <namespace key="-2" case="first-letter">Media</namespace> >> <namespace key="-1" case="first-letter">Special</namespace> >> <namespace key="0" case="first-letter"/> >> <namespace key="1" case="first-letter">Talk</namespace> >> <namespace key="2" case="first-letter">User</namespace> >> <namespace key="3" case="first-letter">User talk</namespace> >> <namespace key="4" case="first-letter">Wikipedia</namespace> >> <namespace key="5" case="first-letter">Wikipedia talk</namespace> >> <namespace key="6" case="first-letter">File</namespace> >> <namespace key="7" case="first-letter">File talk</namespace> >> <namespace key="8" case="first-letter">MediaWiki</namespace> >> <namespace key="9" case="first-letter">MediaWiki talk</namespace> >> <namespace key="10" case="first-letter">Template</namespace> >> <namespace key="11" case="first-letter">Template talk</namespace> >> <namespace key="12" case="first-letter">Help</namespace> >> <namespace key="13" case="first-letter">Help talk</namespace> >> <namespace key="14" case="first-letter">Category</namespace> >> <namespace key="15" case="first-letter">Category talk</namespace> >> <namespace key="100" case="first-letter">Portal</namespace> >> <namespace key="101" case="first-letter">Portal talk</namespace> >> <namespace key="108" case="first-letter">Book</namespace> >> <namespace key="109" case="first-letter">Book talk</namespace> >> </namespaces> >> </siteinfo> >> >> On Mon, Apr 4, 2011 at 7:31 AM, Erol Akarsu <eakarsu(a)gmail.com> wrote: >> >>> I imported wikipedia xml into basex and tried to search it. >>> >>> But searching it takes longer. >>> >>> I tried to search one element that is first child of whole document and >>> it took 52 sec. >>> I know the XML file is very big 31GB. How can I optimize the search? >>> >>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>> >>> let $d := fn:doc ("enwiki-latest-pages-articles")//w:siteinfo >>> return $d >>> >>> Database info: >>> >>> > open enwiki-latest-pages-articles >>> Database 'enwiki-latest-pages-articles' opened in 778.49 ms. >>> > info database >>> Database Properties >>> Name: enwiki-latest-pages-articles >>> Size: 23356 MB >>> Nodes: 228090153 >>> Height: 6 >>> >>> Database Creation >>> Path: /mnt/hgfs/C/tmp/enwiki-latest-pages-articles.xml >>> Time Stamp: 03.04.2011 12:29:15 >>> Input Size: 30025 MB >>> Encoding: UTF-8 >>> Documents: 1 >>> Whitespace Chopping: ON >>> Entity Parsing: OFF >>> >>> Indexes >>> Up-to-date: true >>> Path Summary: ON >>> Text Index: ON >>> Attribute Index: ON >>> Full-Text Index: OFF >>> > >>> >>> >>> Timing info: >>> >>> Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>> Compiling: >>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>> - optimizing descendant-or-self step(s) >>> - binding static variable $d >>> - removing variable $d >>> - simplifying flwor expression >>> Result: element siteinfo { ... } >>> Timing: >>> - Parsing: 1.4 ms >>> - Compiling: 52599.0 ms >>> - Evaluating: 0.28 ms >>> - Printing: 0.62 ms >>> - Total Time: 52601.32 ms >>> Query plan: >>> <DBNode name="enwiki-latest-pages-articles" pre="5"/> >>> >>> >>> >>> Result of query: >>> >>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" xmlns:xsi=" >>> http://www.w3.org/2001/XMLSchema-instance"> >>> <sitename>Wikipedia</sitename> >>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >>> <generator>MediaWiki 1.17wmf1</generator> >>> <case>first-letter</case> >>> <namespaces> >>> <namespace key="-2" case="first-letter">Media</namespace> >>> <namespace key="-1" case="first-letter">Special</namespace> >>> <namespace key="0" case="first-letter"/> >>> <namespace key="1" case="first-letter">Talk</namespace> >>> <namespace key="2" case="first-letter">User</namespace> >>> <namespace key="3" case="first-letter">User talk</namespace> >>> <namespace key="4" case="first-letter">Wikipedia</namespace> >>> <namespace key="5" case="first-letter">Wikipedia talk</namespace> >>> <namespace key="6" case="first-letter">File</namespace> >>> <namespace key="7" case="first-letter">File talk</namespace> >>> <namespace key="8" case="first-letter">MediaWiki</namespace> >>> <namespace key="9" case="first-letter">MediaWiki talk</namespace> >>> <namespace key="10" case="first-letter">Template</namespace> >>> <namespace key="11" case="first-letter">Template talk</namespace> >>> <namespace key="12" case="first-letter">Help</namespace> >>> <namespace key="13" case="first-letter">Help talk</namespace> >>> <namespace key="14" case="first-letter">Category</namespace> >>> <namespace key="15" case="first-letter">Category talk</namespace> >>> <namespace key="100" case="first-letter">Portal</namespace> >>> <namespace key="101" case="first-letter">Portal talk</namespace> >>> <namespace key="108" case="first-letter">Book</namespace> >>> <namespace key="109" case="first-letter">Book talk</namespace> >>> </namespaces> >>> </siteinfo> >>> >>> >>> >>> Thanks >>> >>> Erol Akarsu >>> >>> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk(a)mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >> >

2 1

Basex performance
by Erol Akarsu 11 Apr '11

11 Apr '11

I imported wikipedia xml into basex and tried to search it. But searching it takes longer. I tried to search one element that is first child of whole document and it took 52 sec. I know the XML file is very big 31GB. How can I optimize the search? declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; let $d := fn:doc ("enwiki-latest-pages-articles")//w:siteinfo return $d Database info: > open enwiki-latest-pages-articles Database 'enwiki-latest-pages-articles' opened in 778.49 ms. > info database Database Properties Name: enwiki-latest-pages-articles Size: 23356 MB Nodes: 228090153 Height: 6 Database Creation Path: /mnt/hgfs/C/tmp/enwiki-latest-pages-articles.xml Time Stamp: 03.04.2011 12:29:15 Input Size: 30025 MB Encoding: UTF-8 Documents: 1 Whitespace Chopping: ON Entity Parsing: OFF Indexes Up-to-date: true Path Summary: ON Text Index: ON Attribute Index: ON Full-Text Index: OFF > Timing info: Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; Compiling: - pre-evaluating fn:doc("enwiki-latest-pages-articles") - optimizing descendant-or-self step(s) - binding static variable $d - removing variable $d - simplifying flwor expression Result: element siteinfo { ... } Timing: - Parsing: 1.4 ms - Compiling: 52599.0 ms - Evaluating: 0.28 ms - Printing: 0.62 ms - Total Time: 52601.32 ms Query plan: <DBNode name="enwiki-latest-pages-articles" pre="5"/> Result of query: <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance"> <sitename>Wikipedia</sitename> <base>http://en.wikipedia.org/wiki/Main_Page</base> <generator>MediaWiki 1.17wmf1</generator> <case>first-letter</case> <namespaces> <namespace key="-2" case="first-letter">Media</namespace> <namespace key="-1" case="first-letter">Special</namespace> <namespace key="0" case="first-letter"/> <namespace key="1" case="first-letter">Talk</namespace> <namespace key="2" case="first-letter">User</namespace> <namespace key="3" case="first-letter">User talk</namespace> <namespace key="4" case="first-letter">Wikipedia</namespace> <namespace key="5" case="first-letter">Wikipedia talk</namespace> <namespace key="6" case="first-letter">File</namespace> <namespace key="7" case="first-letter">File talk</namespace> <namespace key="8" case="first-letter">MediaWiki</namespace> <namespace key="9" case="first-letter">MediaWiki talk</namespace> <namespace key="10" case="first-letter">Template</namespace> <namespace key="11" case="first-letter">Template talk</namespace> <namespace key="12" case="first-letter">Help</namespace> <namespace key="13" case="first-letter">Help talk</namespace> <namespace key="14" case="first-letter">Category</namespace> <namespace key="15" case="first-letter">Category talk</namespace> <namespace key="100" case="first-letter">Portal</namespace> <namespace key="101" case="first-letter">Portal talk</namespace> <namespace key="108" case="first-letter">Book</namespace> <namespace key="109" case="first-letter">Book talk</namespace> </namespaces> </siteinfo> Thanks Erol Akarsu

5 8

Re: [basex-talk] BaseX 6.6.1 Update - Soft-Files.com
by Alexander Holupirek 11 Apr '11

11 Apr '11

Hi Gary, the changelog can be found here: https://github.com/BaseXdb/basex/blob/master/changelog Kind regards, Alex On 11.04.2011, at 14:56, Gary Larsen wrote: > Could someone point me to a change log to learn what’s in this update? > > > > Thanks, > > gary > > > > > > > > From: basex-talk-bounces(a)mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] On Behalf Of Soft-Files Team > Sent: Monday, April 11, 2011 1:00 AM > To: basex-talk(a)mailman.uni-konstanz.de > Subject: [basex-talk] BaseX 6.6.1 Update - Soft-Files.com > > > > Dear Developer, > > Your product BaseX 6.6.1 has been updated. > > Your product review page is located at: http://www.soft-files.com/basex_72311.html. > > Sincerely, > The Soft-Files Team > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Alexander Holupirek |-- http://www.informatik.uni-konstanz.de/~holupire |-- Database & Information Systems Group, U Konstanz `-- Room E 209, 0049 7531 88 4449 (phone) 3577 (fax)

1 1

BaseX 6.6.1 Update - Soft-Files.com
by Soft-Files Team 11 Apr '11

11 Apr '11

Dear Developer, Your product BaseX 6.6.1 has been updated. Your product review page is located at: http://www.soft-files.com/basex_72311.html. Sincerely, The Soft-Files Team

2 1

Puzzling XQuery result
by Meredith Gregory 09 Apr '11

09 Apr '11

Dear BaseXers, i've got a result from an XQuery that i don't understand and am looking for insight and guidance. i've got a collection called 'Squibble' with three documents in it. They are listed below. The query for $rcrd in collection( 'Squibble' )//record where $rcrd/*[1] = <contentChannel><String>email</String></contentChannel> return $rcrd returns record elements from the first *two* documents. This is surprising to me because the first child of the first record element in the first document is <contactMeUsing>...</contactMeUsing>, not <contentChannel>...</contentChannel>. i was expecting it to return just the record element from the second document. i confess, i don't know anything about XQuery. So, it could be that i don't understand how the where clause is supposed to work. Any clues would be most appreciated. i'm running my tests using - BaseX 6.6 - on Mac OS X 10.6.7 Best wishes, --greg <?xml version="1.0" encoding="UTF-8"?> <record> <contactMeUsing> <String>email</String> </contactMeUsing> <String>{&quot;com.biosimilarity.lift.model.store.MonadicTermTypes_-Ground&quot;:{&quot;v&quot;:{&quot;@class&quot;:&quot;string&quot;,&quot;$&quot;:&quot;for the win Mikail.Mauz(a)gmail.com &quot;},&quot;_-outer&quot;:{&quot;@class &quot;:&quot;com.biosimilarity.lift.model.store.PersistedMonadicTS$TheMTT$&quot;}}}</String> </record> <?xml version="1.0" encoding="UTF-8"?> <record> <contentChannel> <String>email</String> </contentChannel> <String>{&quot;com.biosimilarity.lift.model.store.MonadicTermTypes_-Ground&quot;:{&quot;v&quot;:{&quot;@class&quot;:&quot;string&quot;,&quot;$&quot;:&quot;for the get Donald.Duque(a)yahoo.com &quot;},&quot;_-outer&quot;:{&quot;@class &quot;:&quot;com.biosimilarity.lift.model.store.PersistedMonadicTS$TheMTT$&quot;}}}</String> </record> <?xml version="1.0" encoding="UTF-8"?> <record> <phone> <var>cell</var> </phone> <String>{&quot;com.biosimilarity.lift.model.store.MonadicTermTypes_-Ground&quot;:{&quot;v&quot;:{&quot;@class &quot;:&quot;string&quot;,&quot;$&quot;:&quot;2xx.6xx.3xxx&quot;},&quot;_-outer&quot;:{&quot;@class &quot;:&quot;com.biosimilarity.lift.model.store.PersistedMonadicTS$TheMTT$&quot;}}}</String> </record> -- L.G. Meredith Managing Partner Biosimilarity LLC 7329 39th Ave SW Seattle, WA 98136 +1 206.650.3740 http://biosimilarity.blogspot.com

2 7

tf/idf
by Wiard Vasen 06 Apr '11

06 Apr '11

Dear sirs of Basex, I am doing my Master thesis on the letters of Vincent van Gogh at the University of Amsterdam. For that purpose I use BaseX to analyze the letters. I wonder, is there the possibility to generate a tf/idf score automatically? In your faq I noticed there needs to be a special term like 'SET SCORING 0' to be able to get a tf/idf score. This information I get from the following page: http://docs.basex.org/wiki/Full-Text Could you help me with this? I would be very grateful. Kind regards,

4 25

Trigger!
by Truong An Nguyen 06 Apr '11

06 Apr '11

Hi, Does Basex have trigger? If not, can you please show me how to recognize which documents are changed after the execution of an xquery update? Thanks. Cheers, An

2 1

Re: [basex-talk] Updates and indexing
by Gary Larsen 05 Apr '11

05 Apr '11

Hi, Just wondering if there is something available to test from the index update effort. Thanks, Gary From: basex-talk-bounces(a)mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] On Behalf Of Gary Larsen Sent: Tuesday, March 15, 2011 3:09 PM To: basex-talk(a)mailman.uni-konstanz.de Subject: Re: [basex-talk] Updates and indexing Hi Dimitar, Thanks for the info. I see that adding a document changes the database indexing status to off. A query to locate a document by node value in a database with 54k documents went from 3 seconds to 4 minutes. I can image its difficult to balance performance in different aspects of the application and maintain database stability at the same time. For the past couple of weeks Ive tested query performance on a half dozen databases and BaseX queries were almost always quickest and never crashed even when running with lower memory. Depending on how things go with my research I may be able to help test the index updating feature when you are ready for that. Thanks, Gary >Hi Gary, >I'm the one who's working on the index updating. As Michael said it's a work >in progress, but I expect to finish my work soon. Most probably it will be >released at some point after 6.6 (which should be available on March 23). >Best regards, >Dimitar Am Dienstag 15 März 2011, 10:01:20 schrieb Michael Seiferle: > Hi Gary, > > yes this is true either way, still in 6.5.1. > We are currently working on incremental index updates, but I unfortunately > can't give you any hint on a release date. > > For most use-cases recreating the indexes (OPTIMIZE [1]) works fine in day > to day use, but we are aware that this might get a little annoying under > some circumstances. > > Thanks a lot for your feedback, feel free to discuss or ask for more > information. -- > > Kind regards > Michael From: Gary Larsen [mailto:gary.larsen@envisn.com] Sent: Monday, March 14, 2011 8:08 PM To: 'basex-talk(a)mailman.uni-konstanz.de' Subject: Updates and indexing Hi, I was reading a thread in the mail list from Oct 2010 which states updates are not indexed: https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-October/000707.htm l Is this still the case with version 6.5.1? When talking of updates does that mean whole document replacement (or deletion) as well as XQuery updates? Thanks for any clarification, gary

1 0

using variables along with namespace prefix
by Sony Vijay 05 Apr '11

05 Apr '11

Hi, I am using the sample xml document(xyz) shown below: <entry xmlns:libx="http://libx.org/xml/libx2"> <libx:libapp> First libapp </libx:libapp> <libx:libapp> Second libapp </libx:libapp> </entry> I am using BaseX 6.5.2 and am trying to use a variable along with a namespace prefix in my sample xquery like so: declare variable $libx_ns as xs:string external; declare namespace libx=$libx_ns; let $type := 'libapp' return doc('xyz')/entry*[libx:{$type}]*[1] When I pass the value "http://libx.org/xml/libx2" to the parameter libx_ns and run the above xquery I see the following exception: [XPST0003] Incomplete expression. However, the following xquery works: declare variable $libx_ns as xs:string external; declare namespace libx=$libx_ns; let $type := 'libapp' return doc('xyz')/entry*[libx:libapp]*[1] The child nodes could be libx:libapp or libx:module etc. So, I will have to use a variable rather than hard wiring the actual child name. But, I am not sure how to use a variable along with a namespace prefix. Am I doing something wrong or is this a bug ? Thanks, Sony

2 1

Re: [basex-talk] tf/idf
by Andreas Weiler 03 Apr '11

03 Apr '11

You can change the input type in the left combobox to XQuery and then enter your full text query. However, I would suggest to use the XQuery view, because it offers detailed error feedback and syntax highlighting. -- Andreas Am 02.04.2011 um 18:01 schrieb Wiard Vasen: > Thanks a lot Andreas for your answer on my question. > I will look at your solution this week. > > I was in Berlin this weekend. > > Is it possible to use set scoring * and then enter full text queries in the upper text-bar? > > Thank you for the live-demo as well. > > If I succeed I come back to you. > > Regards, > > Wiard Vasen > > Vrije Universiteit van Amsterdam > Masterthesis: Deep diving into Van Gogh > > 2011/3/31 Andreas Weiler <andreas.weiler(a)uni-konstanz.de> > Dear Wiard Vasen, > > you just need to set the scoring property once. > If you work in the GUI: > > Go to the top input bar, choose command and type: > > set scoring * > > as * set the scoring algorithm you like. > > In the console just type: set scoring * > > After setting this you can use the score function, like in the 8th query of our online demo (basex.org/products/live-demo) > > let $names := ('Jack London', 'Jack', 'Jim Beam', 'Jack Daniels') > for $name at $pos score $score in $names[. contains text 'Jack'] > order by $score descending > return <name pos='{ $pos }'>{ $name }</name> > > Don't hesitate to ask for more, > Andreas > > Am 30.03.2011 um 22:02 schrieb Wiard Vasen: > >> Dear sirs of Basex, >> >> I am doing my Master thesis on the letters of Vincent van Gogh at the University of Amsterdam. >> For that purpose I use BaseX to analyze the letters. >> >> I wonder, is there the possibility to generate a tf/idf score automatically? >> In your faq I noticed there needs to be a special term like 'SET SCORING 0' to be able to get a tf/idf score. >> >> This information I get from the following page: http://docs.basex.org/wiki/Full-Text >> >> Could you help me with this? >> >> I would be very grateful. >> >> Kind regards, >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk(a)mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

1 0