Clustering of BaseX servers - BaseX-Talk - mailman.uni-konstanz.de

29 Apr 2011


      Hi,
Have we thought on clustering basex servers so we can partition xml
documents?
Here, I am only interested in a global partitioning:
Let's say if we have xml document like this, I would like to partition
RECORDS content so that each host will have equal number of RECORD elements.
Then, we need to aggregate results of individual hosts.
Can we implement this simple clustering framework with Basex?
<RECORDS>
   <RECORD>
.....
  </RECORD>
<RECORD>
.....
  </RECORD>
</RECORDS>
On Mon, Apr 11, 2011 at 12:35 PM, Erol Akarsu eakarsu@gmail.com wrote:
...
Ok,
I was able to run full text search with another XML Database.
I am primarily interested in how Basex will play with big XML file
Wikipedia.
Actually, Database create of wikipedia is fine. But when I add full tet
search and indexes for it, it always throws out of memory exception error.
I have changed -Xmx with 6GB that is still not enough to generate indexes
for Wikipedia.
Can you help me on how to generate indexes with a machine that case  6GB
for Basex process?
Thanks
Erol Akarsu
On Sun, Apr 10, 2011 at 4:07 AM, Andreas Weiler <
andreas.weiler@uni-konstanz.de> wrote:
...
Hi,
the following query could work for you:
declare namespace w="http://www.mediawiki.org/xml/export-0.5/";
for $i in doc("enwiki-latest-pages-articles")//w:sitename
return $i[. contains text "Wikipedia"]/..
-- Andreas
Am 09.04.2011 um 20:43 schrieb Erol Akarsu:
Hi
I am having difficulty in running full text operators. This script gives
siteinfo below
declare namespace w="http://www.mediawiki.org/xml/export-0.5/";
 let $d :=  fn:doc ("enwiki-latest-pages-articles")
return ($d//w:siteinfo)[1]
But  return $d//w:siteinfo[w:sitename contains text 'Wikipedia'] does NOT
give same node
Why "contains" ft operator behave incorrectly? I remember it was working
fine. I just dropped and recreated database and turn all indexes. Can you
help me?
Query info is here:
Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/";
Compiling:

pre-evaluating fn:doc("enwiki-latest-pages-articles")
adding text() step
optimizing descendant-or-self step(s)
removing path with no index results
pre-evaluating (())[1]
binding static variable $res
pre-evaluating fn:doc("enwiki-latest-pages-articles")
binding static variable $d
adding text() step
optimizing descendant-or-self step(s)
removing path with no index results
simplifying flwor expression

Result: ()
Timing:

Parsing:  0.46 ms
Compiling:  0.42 ms
Evaluating:  0.17 ms
Printing:  0.1 ms
Total Time:  1.15 ms

Query plan:
<sequence size="0"/>
<siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance">
  <sitename>Wikipedia</sitename>
  <base>http://en.wikipedia.org/wiki/Main_Page</base>
  <generator>MediaWiki 1.17wmf1</generator>
  <case>first-letter</case>
  <namespaces>
    <namespace key="-2" case="first-letter">Media</namespace>
    <namespace key="-1" case="first-letter">Special</namespace>
    <namespace key="0" case="first-letter"/>
    <namespace key="1" case="first-letter">Talk</namespace>
    <namespace key="2" case="first-letter">User</namespace>
    <namespace key="3" case="first-letter">User talk</namespace>
    <namespace key="4" case="first-letter">Wikipedia</namespace>
    <namespace key="5" case="first-letter">Wikipedia talk</namespace>
    <namespace key="6" case="first-letter">File</namespace>
    <namespace key="7" case="first-letter">File talk</namespace>
    <namespace key="8" case="first-letter">MediaWiki</namespace>
    <namespace key="9" case="first-letter">MediaWiki talk</namespace>
    <namespace key="10" case="first-letter">Template</namespace>
    <namespace key="11" case="first-letter">Template talk</namespace>
    <namespace key="12" case="first-letter">Help</namespace>
    <namespace key="13" case="first-letter">Help talk</namespace>
    <namespace key="14" case="first-letter">Category</namespace>
    <namespace key="15" case="first-letter">Category talk</namespace>
    <namespace key="100" case="first-letter">Portal</namespace>
    <namespace key="101" case="first-letter">Portal talk</namespace>
    <namespace key="108" case="first-letter">Book</namespace>
    <namespace key="109" case="first-letter">Book talk</namespace>
  </namespaces>
</siteinfo>
On Mon, Apr 4, 2011 at 7:31 AM, Erol Akarsu eakarsu@gmail.com wrote:
...
I imported  wikipedia xml into basex and tried to search it.
But searching it takes longer.
I tried to search one element that is first child of whole document and
it took 52 sec.
I know the XML file is very big 31GB. How can I optimize the search?
declare namespace w="http://www.mediawiki.org/xml/export-0.5/";
let $d := fn:doc ("enwiki-latest-pages-articles")//w:siteinfo
return $d
Database info:
...
open enwiki-latest-pages-articles
Database 'enwiki-latest-pages-articles' opened in 778.49 ms.
...
info database
Database Properties
 Name: enwiki-latest-pages-articles
 Size: 23356 MB
 Nodes: 228090153
 Height: 6
Database Creation
 Path: /mnt/hgfs/C/tmp/enwiki-latest-pages-articles.xml
 Time Stamp: 03.04.2011 12:29:15
 Input Size: 30025 MB
 Encoding: UTF-8
 Documents: 1
 Whitespace Chopping: ON
 Entity Parsing: OFF
Indexes
 Up-to-date: true
 Path Summary: ON
 Text Index: ON
 Attribute Index: ON
 Full-Text Index: OFF
...
Timing info:
Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/";
Compiling:

pre-evaluating fn:doc("enwiki-latest-pages-articles")
optimizing descendant-or-self step(s)
binding static variable $d
removing variable $d
simplifying flwor expression

Result: element siteinfo { ... }
Timing:

Parsing:  1.4 ms
Compiling:  52599.0 ms
Evaluating:  0.28 ms
Printing:  0.62 ms
Total Time:  52601.32 ms

Query plan:
<DBNode name="enwiki-latest-pages-articles" pre="5"/>
Result of query:
<siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance">
  <sitename>Wikipedia</sitename>
  <base>http://en.wikipedia.org/wiki/Main_Page</base>
  <generator>MediaWiki 1.17wmf1</generator>
  <case>first-letter</case>
  <namespaces>
    <namespace key="-2" case="first-letter">Media</namespace>
    <namespace key="-1" case="first-letter">Special</namespace>
    <namespace key="0" case="first-letter"/>
    <namespace key="1" case="first-letter">Talk</namespace>
    <namespace key="2" case="first-letter">User</namespace>
    <namespace key="3" case="first-letter">User talk</namespace>
    <namespace key="4" case="first-letter">Wikipedia</namespace>
    <namespace key="5" case="first-letter">Wikipedia talk</namespace>
    <namespace key="6" case="first-letter">File</namespace>
    <namespace key="7" case="first-letter">File talk</namespace>
    <namespace key="8" case="first-letter">MediaWiki</namespace>
    <namespace key="9" case="first-letter">MediaWiki talk</namespace>
    <namespace key="10" case="first-letter">Template</namespace>
    <namespace key="11" case="first-letter">Template talk</namespace>
    <namespace key="12" case="first-letter">Help</namespace>
    <namespace key="13" case="first-letter">Help talk</namespace>
    <namespace key="14" case="first-letter">Category</namespace>
    <namespace key="15" case="first-letter">Category talk</namespace>
    <namespace key="100" case="first-letter">Portal</namespace>
    <namespace key="101" case="first-letter">Portal talk</namespace>
    <namespace key="108" case="first-letter">Book</namespace>
    <namespace key="109" case="first-letter">Book talk</namespace>
  </namespaces>
</siteinfo>
Thanks
Erol Akarsu

BaseX-Talk mailing list
BaseX-Talk@mailman.uni-konstanz.de
https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk