May 2011 - BaseX-Talk - mailman.uni-konstanz.de

out of memory with file:write
by Erol Akarsu 11 May '11

11 May '11

I am filtering one big xml db that 800Mb stored into basex db already. When I write filtered data into a file, I am getting out of memory. I know basex is generating result filtered document in memory before writing and facing memory error. Can basex write results block by block? Thanks On Wed, May 4, 2011 at 9:38 AM, Erol Akarsu <eakarsu(a)gmail.com> wrote: > I tried to delete some nodes with "delete nodes .." xquery update command. > It actually did what was requested. > > I checked the database size that is still same as previous one. I know the > nodes deleted has a lot of data and I think database indexes would be > adjusted accordingly. > > Then, I exported database as xml file and recreated db. I can see it has > real size. > > My question is why DB is not adjusting indexes when nodes deleted? > > Thanks > > Erol Akarsu > > > On Fri, Apr 29, 2011 at 9:23 AM, Erol Akarsu <eakarsu(a)gmail.com> wrote: > >> Hi, >> >> Have we thought on clustering basex servers so we can partition xml >> documents? >> >> Here, I am only interested in a global partitioning: >> >> Let's say if we have xml document like this, I would like to partition >> RECORDS content so that each host will have equal number of RECORD elements. >> Then, we need to aggregate results of individual hosts. >> Can we implement this simple clustering framework with Basex? >> >> >> <RECORDS> >> <RECORD> >> ..... >> </RECORD> >> <RECORD> >> ..... >> </RECORD> >> </RECORDS> >> >> On Mon, Apr 11, 2011 at 12:35 PM, Erol Akarsu <eakarsu(a)gmail.com> wrote: >> >>> Ok, >>> >>> I was able to run full text search with another XML Database. >>> >>> I am primarily interested in how Basex will play with big XML file >>> Wikipedia. >>> >>> Actually, Database create of wikipedia is fine. But when I add full tet >>> search and indexes for it, it always throws out of memory exception error. >>> >>> I have changed -Xmx with 6GB that is still not enough to generate indexes >>> for Wikipedia. >>> >>> Can you help me on how to generate indexes with a machine that case 6GB >>> for Basex process? >>> >>> Thanks >>> >>> Erol Akarsu >>> >>> >>> >>> On Sun, Apr 10, 2011 at 4:07 AM, Andreas Weiler < >>> andreas.weiler(a)uni-konstanz.de> wrote: >>> >>>> Hi, >>>> >>>> the following query could work for you: >>>> >>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>> for $i in doc("enwiki-latest-pages-articles")//w:sitename >>>> return $i[. contains text "Wikipedia"]/.. >>>> >>>> -- Andreas >>>> >>>> Am 09.04.2011 um 20:43 schrieb Erol Akarsu: >>>> >>>> Hi >>>> >>>> I am having difficulty in running full text operators. This script gives >>>> siteinfo below >>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>> let $d := fn:doc ("enwiki-latest-pages-articles") >>>> return ($d//w:siteinfo)[1] >>>> >>>> But return $d//w:siteinfo[w:sitename contains text 'Wikipedia'] does >>>> NOT give same node >>>> Why "contains" ft operator behave incorrectly? I remember it was working >>>> fine. I just dropped and recreated database and turn all indexes. Can you >>>> help me? >>>> Query info is here: >>>> >>>> Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>> Compiling: >>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>> - adding text() step >>>> - optimizing descendant-or-self step(s) >>>> - removing path with no index results >>>> - pre-evaluating (())[1] >>>> - binding static variable $res >>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>> - binding static variable $d >>>> - adding text() step >>>> - optimizing descendant-or-self step(s) >>>> - removing path with no index results >>>> - simplifying flwor expression >>>> Result: () >>>> Timing: >>>> - Parsing: 0.46 ms >>>> - Compiling: 0.42 ms >>>> - Evaluating: 0.17 ms >>>> - Printing: 0.1 ms >>>> - Total Time: 1.15 ms >>>> Query plan: >>>> <sequence size="0"/> >>>> >>>> >>>> >>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" xmlns:xsi=" >>>> http://www.w3.org/2001/XMLSchema-instance"> >>>> <sitename>Wikipedia</sitename> >>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >>>> <generator>MediaWiki 1.17wmf1</generator> >>>> <case>first-letter</case> >>>> <namespaces> >>>> <namespace key="-2" case="first-letter">Media</namespace> >>>> <namespace key="-1" case="first-letter">Special</namespace> >>>> <namespace key="0" case="first-letter"/> >>>> <namespace key="1" case="first-letter">Talk</namespace> >>>> <namespace key="2" case="first-letter">User</namespace> >>>> <namespace key="3" case="first-letter">User talk</namespace> >>>> <namespace key="4" case="first-letter">Wikipedia</namespace> >>>> <namespace key="5" case="first-letter">Wikipedia talk</namespace> >>>> <namespace key="6" case="first-letter">File</namespace> >>>> <namespace key="7" case="first-letter">File talk</namespace> >>>> <namespace key="8" case="first-letter">MediaWiki</namespace> >>>> <namespace key="9" case="first-letter">MediaWiki talk</namespace> >>>> <namespace key="10" case="first-letter">Template</namespace> >>>> <namespace key="11" case="first-letter">Template talk</namespace> >>>> <namespace key="12" case="first-letter">Help</namespace> >>>> <namespace key="13" case="first-letter">Help talk</namespace> >>>> <namespace key="14" case="first-letter">Category</namespace> >>>> <namespace key="15" case="first-letter">Category talk</namespace> >>>> <namespace key="100" case="first-letter">Portal</namespace> >>>> <namespace key="101" case="first-letter">Portal talk</namespace> >>>> <namespace key="108" case="first-letter">Book</namespace> >>>> <namespace key="109" case="first-letter">Book talk</namespace> >>>> </namespaces> >>>> </siteinfo> >>>> >>>> On Mon, Apr 4, 2011 at 7:31 AM, Erol Akarsu <eakarsu(a)gmail.com> wrote: >>>> >>>>> I imported wikipedia xml into basex and tried to search it. >>>>> >>>>> But searching it takes longer. >>>>> >>>>> I tried to search one element that is first child of whole document and >>>>> it took 52 sec. >>>>> I know the XML file is very big 31GB. How can I optimize the search? >>>>> >>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>> >>>>> let $d := fn:doc ("enwiki-latest-pages-articles")//w:siteinfo >>>>> return $d >>>>> >>>>> Database info: >>>>> >>>>> > open enwiki-latest-pages-articles >>>>> Database 'enwiki-latest-pages-articles' opened in 778.49 ms. >>>>> > info database >>>>> Database Properties >>>>> Name: enwiki-latest-pages-articles >>>>> Size: 23356 MB >>>>> Nodes: 228090153 >>>>> Height: 6 >>>>> >>>>> Database Creation >>>>> Path: /mnt/hgfs/C/tmp/enwiki-latest-pages-articles.xml >>>>> Time Stamp: 03.04.2011 12:29:15 >>>>> Input Size: 30025 MB >>>>> Encoding: UTF-8 >>>>> Documents: 1 >>>>> Whitespace Chopping: ON >>>>> Entity Parsing: OFF >>>>> >>>>> Indexes >>>>> Up-to-date: true >>>>> Path Summary: ON >>>>> Text Index: ON >>>>> Attribute Index: ON >>>>> Full-Text Index: OFF >>>>> > >>>>> >>>>> >>>>> Timing info: >>>>> >>>>> Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>> Compiling: >>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>>> - optimizing descendant-or-self step(s) >>>>> - binding static variable $d >>>>> - removing variable $d >>>>> - simplifying flwor expression >>>>> Result: element siteinfo { ... } >>>>> Timing: >>>>> - Parsing: 1.4 ms >>>>> - Compiling: 52599.0 ms >>>>> - Evaluating: 0.28 ms >>>>> - Printing: 0.62 ms >>>>> - Total Time: 52601.32 ms >>>>> Query plan: >>>>> <DBNode name="enwiki-latest-pages-articles" pre="5"/> >>>>> >>>>> >>>>> >>>>> Result of query: >>>>> >>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" xmlns:xsi=" >>>>> http://www.w3.org/2001/XMLSchema-instance"> >>>>> <sitename>Wikipedia</sitename> >>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >>>>> <generator>MediaWiki 1.17wmf1</generator> >>>>> <case>first-letter</case> >>>>> <namespaces> >>>>> <namespace key="-2" case="first-letter">Media</namespace> >>>>> <namespace key="-1" case="first-letter">Special</namespace> >>>>> <namespace key="0" case="first-letter"/> >>>>> <namespace key="1" case="first-letter">Talk</namespace> >>>>> <namespace key="2" case="first-letter">User</namespace> >>>>> <namespace key="3" case="first-letter">User talk</namespace> >>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> >>>>> <namespace key="5" case="first-letter">Wikipedia talk</namespace> >>>>> <namespace key="6" case="first-letter">File</namespace> >>>>> <namespace key="7" case="first-letter">File talk</namespace> >>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> >>>>> <namespace key="9" case="first-letter">MediaWiki talk</namespace> >>>>> <namespace key="10" case="first-letter">Template</namespace> >>>>> <namespace key="11" case="first-letter">Template talk</namespace> >>>>> <namespace key="12" case="first-letter">Help</namespace> >>>>> <namespace key="13" case="first-letter">Help talk</namespace> >>>>> <namespace key="14" case="first-letter">Category</namespace> >>>>> <namespace key="15" case="first-letter">Category talk</namespace> >>>>> <namespace key="100" case="first-letter">Portal</namespace> >>>>> <namespace key="101" case="first-letter">Portal talk</namespace> >>>>> <namespace key="108" case="first-letter">Book</namespace> >>>>> <namespace key="109" case="first-letter">Book talk</namespace> >>>>> </namespaces> >>>>> </siteinfo> >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Erol Akarsu >>>>> >>>>> >>>> _______________________________________________ >>>> BaseX-Talk mailing list >>>> BaseX-Talk(a)mailman.uni-konstanz.de >>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>> >>>> >>>> >>> >> >

2 1

Incorrect implementation of XQUF?
by Computer Software Developer 11 May '11

11 May '11

Hi, I have a collection MyData/Test and another collection MyData/Real. I have the following XML occurring about five times in total, 4 times in collection MyData/Test and once in MyData/Real: <APPLICATION> <APPLICATION_ID>1</APPLICATION_ID> </APPLICATION> When I seek to delete this application only from MyData/Test using updating xquery like so: xquery version "1.0"; declare namespace fn="http://www.w3.org/2005/xpath-functions"; let $dataSet := 'Test' let $databaseName := 'MyData' let $applicationID := '1' let $finalURL := fn:concat($databaseName, "/",$dataSet) let $applicationsModified := <APP-DATA-ROOT xmlns:xforms="http://www.w3.org/2002/xforms"> <APPLICATION> <APPLICATION_ID>2</APPLICATION_ID> </APPLICATION> </APP-DATA-ROOT> for $all in fn:collection($finalURL) for $anApp in $all/APPLICATION[APPLICATION_ID=$applicationID] return ( delete nodes $anApp ) The application gets deleted from MyData/Real collection as well! Moreover, an insert statement to insert an APPLICATION element in the return block above, such insert statement placed after the deletion statement above, does not appear to insert anything.

2 4

Expected usage of map:map in BaseX 6.6.2
by Jan Vlčinský (CAD) 09 May '11

09 May '11

Hi Nice to have again new update of BaseX. In past I had some performance problems, when I had to process one document, enriching the elements by data from another database. I have a traffic-feed document, containing elements loc-id, referencing to detailed location description. Detailed location description is located in another document, which we call PredefinedLocationSet. In some cases we have about 11000 locations or even more, total size of open PredefinedLocationSet document is about 8 MB. Traffic feed can in some cases be of similar nature (the same number of "records" and the same size). My task was to return (not to modify) traffic feed with detailed location description embedded. All my xqueries, trying to accomplish something like that, seemed to me rather slow, regardless of my attempts to find any sort of optimization. I wonder, if map:map approach is something, what has a chance to be faster. Any technical background, hints or ideas are welcome. My idea is to start my xquery by building up map from PredefinedLocationSet (using loc-id as the key) and then run through traffic-feed to get loc-id, find related location details and return enriched "record". With best regards Jan Vlcinsky *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky(a)gmail.com http://cz.linkedin.com/in/vlcinsky 2011/5/9 Christian Grün <christian.gruen(a)gmail.com> > Dear all, > > XQuery: > [ADD] map expressions and functions added: > http://docs.basex.org/wiki/Map_Functions >

1 0

BaseX 6.6.2: LinuxTag Release
by Christian Grün 09 May '11

09 May '11

Dear all, once more we are glad to release a new version of BaseX, aligned with the LinuxTag Event in Berlin/Germany, which offers the following features: API: [ADD] JAX-RX API now supports basic user authentication: http://docs.basex.org/wiki/JAX-RX_API [ADD] The COPY creates identical database copies: http://docs.basex.org/wiki/Commands [ADD] The OPTIMIZE ALL command minimizes all database structures: http://docs.basex.org/wiki/Commands XQuery: [ADD] map expressions and functions added: http://docs.basex.org/wiki/Map_Functions [MOD] File module aligned with latest EXPath specification: http://docs.basex.org/wiki/File_Functions [MOD] Speedup of full-text queries with keyword lists. Example: $x contains text { 'a', 'b', 'c', ...} [MOD] XQuery Update optimizations for replacing nodes; tree-aware updates. [MOD] XQuery optimizations to avoid materialization of sequences. GUI: [ADD] Multiple editor tabs added [ADD] Database management: copy databases Core: [ADD] Internal XML parser: HTML entities added [FIX] Glob syntax: support for multiple file suffixes Your feedback is welcome as usual; have fun, Christian BaseX Team

1 0

Possible to store query on the server side, but payload (XML data) in HTTP post, using jax-rx?
by Computer Software Developer 09 May '11

09 May '11

Hi, Using the run parameter in a jax-rx request, one can specify a query contained in a file stored on the server side. Using HTTP Post, one can specify the xquery and the payload in the HTTP request as XML. If the XML is generated on the client side to be stored on the server using HTTP post, can one execute an updating xquery stored on the server side to store this generated XML on the server? In other words, (how) can one run xqueries stored on the server side with an XML payload submitted using the HTTP post? Thanks.

2 10

Insert nodes in large xml structures generates Exception
by augras philippe 09 May '11

09 May '11

Hello , i send you the following bug trace ....and ask you why the xq correctly works with small xml structure and not with largest ! Details here : Small structure is 12Mg, 130Knodes and 5 Heigh ( 2K incoming Objects ) Largest structure is 43Mg, 2000K nodes and 5 Heigh ( 75K incoming Objects ) DB creation trace is : SET PARSER xml PARSER: xml Command: SET PARSEROPT lines=true,format=verbose,header=true,separator=comma PARSEROPT: lines=true,format=verbose,header=true,separator=comma Command: CREATE DB ea_i_co_o_o E:/ENV_Inst/TrioX/ScriptAUP/EA/i/TKE/co.o.o/co_o_o.xml Database 'ea_i_co_o_o' created in 8158.29 ms. Bugs trace is : "Possible bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 6.5.1 Java: Sun Microsystems Inc., 1.6.0_25 OS: Windows XP, x86 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 25864888 org.basex.io.TableDiskAccess.write4(TableDiskAccess.java:156) org.basex.data.Data.size(Data.java:747) org.basex.data.Data.insert(Data.java:716)" Now i explain how it occurs . My project is to migrate xml structure in new xml structure, i use xq to change progressivly the xml schema It is important for me to check that is can handle xml structure with 200 K objects definition , with about 10-100 attributs definition each. then i've setup a set of small 4 xq file in charge to generate progressivly "the compliant xml structure " for everying incoming Object. Simply , On every incoming object ( Object node ) i add a small Product node with his definition in a first xq , I proceed the same with ( Form node ) and ( ProductRevision node ) and their own xml content ( all about 10 node each ) This was working fine in a small structure , update is done in 10 second and it is fine, so how can i do this with larger structure. In addition , i've limited the nb of object in selection of the update ( position < 250 ) and the result is not good , , Should i deduce that inserting nodes is not recommended in my case study ? PS : i've setup the nodes and attributes index, not the full text ( i never use text node ) and i run baseX651. Bests Regards XQ details here : for $ea_i in $i_type_in let $ApplicationRef := local:build_ApplicationRef($ea_i,"<A/>") let $AssociatedForm := local:build_AssociatedForm ($ea_i,"<A/>") let $UserData := local:build_UserData ($ea_i,"<A/>") let $m := ( $ApplicationRef , $UserData, $AssociatedForm ) let $ProductRevision := local:build_ProductRevision($ea_i,$m) let $out := ($ProductRevision) return (delete node$ea_i/ProductRevision, insert node $out into $ea_i[1]) (:return $out :)

2 1

Re: [basex-talk] What happened to the DeepFS menu? Was it removed?
by John D. Mitchell 08 May '11

08 May '11

Howdy, Just upgraded to the latest Base-x v6.6.1 and the create command no longer accepts "fs" as one of the types. I've been using DeepFS and so this is a bit of an issue. :-) Note also that "mab" is listed in the auto-complete list but is also excluded. Thanks, John

5 4

Possible bug in GUI corrupts database
by Thomas Kaltofen 08 May '11

08 May '11

Hi, I think I found a bug in the GUI which corrupts the database. Steps to reproduce: - Create database 'db1' - Add some documents to database 'db1' through server interface (not through GUI) - Now start GUI and open 'db1' - Add some more documents to database 'db1' through server interface (keep GUI open) - If one re-opens 'db1' now in GUI, database gets corrupted and GUI goes crazy Best regards, Thomas

2 3

XML encoding problem
by Mike Cobo 06 May '11

06 May '11

Hi, I have a collection of XML documents in which I want to replace only one attribute per file. The XML documents contain some XML entities for German "Umlaute" characters (eg. ü). After running the XQuery script which replaces the attributes, I use the "Export XML" function of the GUI to export the modified documents to a local folder. Unfortunaltey, the XML entities are now replaced by the corresponding "real" character (eg. ü). I tried the several parsing options of the "Create database" dialog, but already when creating the DB I can see the "real" character in the GUI text output window. As I don't wanna change anything else of the original XML content butthe attribute, I need the output files to have the XML entities instead of the "real" characters (as in the original files). Here an simple example original file: <?xml version="1.0" encoding="UTF-8"?> <Message xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <header messageId="4523251"> </header> <dataSupply> <supplyProperty description="Gültigkeitsbeginn" value="12.12.2010"/> </dataSupply> </Message> And here's what I get as output: <?xml version="1.0" encoding="UTF-8"?> <Message xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <header messageId="000"/> <dataSupply> <supplyProperty description="Gültigkeitsbeginn" value="12.12.2010"/> </dataSupply> </Message> How can I get the export encoding right? best regards, Mike

2 1

Help on BaseX commands
by Christian Grün 06 May '11

06 May '11

Dear all, we have improved our documentation on the BaseX database commands: http://docs.basex.org/wiki/Commands The next version of BaseX will be released in just a few days. Enjoy, Christian ___________________________ Christian Grün BaseX Team Uni KN, Box 188 78457 Konstanz, Germany http://www.inf.uni-konstanz.de/~gruen

1 0