- BaseX-Talk - mailman.uni-konstanz.de

Bulk add xml files
by Kunal Chauhan 19 Aug '14

19 Aug '14

Hi, I am evaluting BaseX for processing large size xml files(in GBs). I tried initially with XML file size of 1.1 GB to add in BaseX. it took 101 Sec. But when I have splitted it into 10K small instances and tried to add it using "ADD source_dir/" through command line, It took around 2.5 hours to add those files into the DB. System configuration is as below. Linux machine having 8 cores and 64 GB RAM I don't have any clue why there is such a big difference between two scenarios ? if any one can give any clue, will be helpful to me. Thanks & Regards, -- *Kunal Chauhan* mail4ck(a)gmail.com [+918655517141]

2 1

Re: [basex-talk] Extra signature for fn:filter shown on wiki page
by Amanda Galtman 18 Aug '14

18 Aug '14

Thanks for repyling, Dirk. I will delete the gray syntax from that wiki page. Cheers, Amanda > Date: Sun, 17 Aug 2014 07:44:16 +0200 > From: Dirk Kirsten <dk(a)basex.org> > To: basex-talk(a)mailman.uni-konstanz.de > Subject: Re: [basex-talk] Extra signature for fn:filter shown on wiki > page > Message-ID: <53F04130.80100(a)basex.org> > Content-Type: text/plain; charset=ISO-8859-1 > > Hello Amanda, > > you are invited to delete that. I am pretty sure that is left from a change to > the spec (and accordingly the change in our function signatures). Some time > ago, the argument order for the HOF functions in the spec was changed, > moving the sequence items to the first argument position. > So the gray version is actually outdated and as we didn't keep in the the > other function signatures I think it is safe to delete it. > > Cheers, > Dirk > > On 17/08/14 03:19, Amanda Galtman wrote: > > Hi, > > > > The documentation page > > http://docs.basex.org/wiki/Higher-Order_Functions#fn:filter > > shows two signatures for the fn:filter function, the second of which is in > gray. Does it belong? It doesn't seem to work or match the spec, but I'm > hesitant to delete it from the wiki in case I am wrong. > > > > fn:filter($seq as item()*, $pred as function(item()) as xs:boolean)) > > as item()* fn:filter($pred as function(item()) as xs:boolean, $seq as > > item()*) as item()* > > > > Thanks, > > Amanda > > > > -- > Dirk Kirsten, BaseX GmbH, http://basex.org > |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > |-- Registergericht Freiburg, HRB: 708285, Gesch?ftsf?hrer: > | Dr. Christian Gr?n, Dr. Alexander Holupirek, Michael Seiferle > `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22 > > > End of BaseX-Talk Digest, Vol 56, Issue 33 > ******************************************

1 0

Paged results
by Paul Swennenhuis 18 Aug '14

18 Aug '14

Hi all, Triggered by a thread in this forum I decided to make a paged() function that adds paging information to a node-set. It does so by wrapping the node-set in an root element - with a name of your choice - with attributes for total number of nodes, start-index requested page-size, actual page-size and end-index. The last function argument lets you select whether or not to output a paged node-set or just wrap the node-set in a root element. It's not very spectacular but it may be of use if you want to use BaseX for paged results or endless scrolling. Comments are appreciated. The module with the paged() function: ------------------------------------- module namespace lib = "My Library"; (: lib:paged adds paging information to a node-set in order to enable paged results for a web-app Will return a node with the name as specified in $nodeName along with total number of nodes ($total), start-index ($start), requested page-size ($num), actual page-size ($displayed) and end-index ($end) as attributes, followed by the subsequence of the node-set Collection MUST be a node-set (i.e. must NOT have a root element) If $doPaged argument is false(), only the root node with the original collection is returned The parameter makes it easier to do a non-conditional call of the function on a node-set :) declare function lib:paged ($collection, $nodeName, $start, $pageSize, $doPaged) { let $total := count($collection) let $min := lib:max(lib:min($pageSize,lib:min($total,$total - $start + 1)),0) return element {$nodeName} { if ($doPaged) then ( attribute total {$total}, attribute start {$start}, attribute pageSize {$pageSize}, attribute actualPageSize {$min}, attribute end {$start + $min - 1}, subsequence($collection, $start, $pageSize) ) else $collection } }; (: Return smallest of two numbers :) declare function lib:min($a,$b) { if ($a < $b) then $a else $b }; (: Return largest of two numbers :) declare function lib:max($a,$b) { if ($a > $b) then $a else $b }; ------------------------------------ Usage example: ----------------------------------- import module namespace lib="My Library" at "library.xqy"; lib:paged(collection("Facts")//city, 'cities', 500, 300, true()) ------------------------------------ Paul

2 1

Extra signature for fn:filter shown on wiki page
by Amanda Galtman 17 Aug '14

17 Aug '14

Hi, The documentation page http://docs.basex.org/wiki/Higher-Order_Functions#fn:filter shows two signatures for the fn:filter function, the second of which is in gray. Does it belong? It doesn't seem to work or match the spec, but I'm hesitant to delete it from the wiki in case I am wrong. fn:filter($seq as item()*, $pred as function(item()) as xs:boolean)) as item()* fn:filter($pred as function(item()) as xs:boolean, $seq as item()*) as item()* Thanks, Amanda

2 1

relative path between two element in xquery
by Rob Stapper 15 Aug '14

15 Aug '14

Hi, How can I construct the relative path between two elements in an xml-file. Most probably overlooking the obvious I ended up with the solution below, which, unfortunately, triggers an runtime-error. My not-working solution: let $xml := <xml> <A> <B/> </A> </xml> let $b := $xml//B let $a := $xml//A return replace( string( path( $b)) , string( path( $a)) , '' ) Two questions: - How to retrieve the relative path between two element within a xml-file? - Why does the solution above trigger an error? Can it be a bug? ( oops, three questions) Thnx in advance for the reply, Rob Stapper --- Dit e-mailbericht bevat geen virussen en malware omdat avast! Antivirus-bescherming actief is. http://www.avast.com

3 4

Re: [basex-talk] UPDINDEX and ever growing index size
by James Ball 15 Aug '14

15 Aug '14

Hi Christian, I’ve been following along with your snapshots and I think there’s a bug still remaining somewhere but I can’t work out what’s causing it which is why I haven’t been back to you. If you use the GUI and the database is open in the GUI then everything works correctly. I can add, replace or a mix of the two with no issues. However if I use the GUI but the database is closed then I can still get AOOB errors. I’ve spent the evening gradually reducing it down so I can reproduce the error using just one element and one attribute and do it with command files. I’ve attached three files: AOOB-succeed.bxs will always work. The database is created and remains open during the replacements. Everything is fine. AOOB-fail.bxs will fail. The database is closed after being created and the XQuery runs on the closed database. The first replacements (plural but fewer than the number of documents in the database - 1) are fine but subsequent ones fail. AOOB-fail2.bxs will also fail. This is the same as AOOB-fail.bxs except that only one replacement is made. This still fails but will a different error (key does not exist). I hope these help track down this bug - it’s working really well otherwise. Regards, James On 6 Aug 2014, at 11:14, Christian Grün <christian.gruen(a)gmail.com> wrote: > Hi James, > > while writing some more tests, I stumbled upon another small bug, > causing AOOB exceptions during the insertion of new entries. The new > snapshot is online. > > Christian > > > On Mon, Aug 4, 2014 at 7:32 PM, Christian Grün > <christian.gruen(a)gmail.com> wrote: >> A last one for today: I have just uploaded another snapshot which >> should speed up index updates. >> >> Looking forward to your feedback, >> Christian >> >> >> >> On Mon, Aug 4, 2014 at 4:01 PM, Christian Grün >> <christian.gruen(a)gmail.com> wrote: >>> The bug was hidden well [1], but it should be fixed now. Could you >>> check out the latest snapshot? >>> Christian >>> >>> [1] https://github.com/BaseXdb/basex/commit/429585ce26fca98d124d78fb88216ad7317… >>> >>> >>> >>> On Mon, Aug 4, 2014 at 2:41 PM, Christian Grün >>> <christian.gruen(a)gmail.com> wrote: >>>> Hi James, >>>> >>>> I've found a little example for the bug (see attached). >>>> >>>> Sorry for the inconvenience; I'm working on a fix. >>>> >>>> Christian >>>> >>>> >>>> >>>> >>>> On Mon, Aug 4, 2014 at 11:30 AM, Christian Grün >>>> <christian.gruen(a)gmail.com> wrote: >>>>> James, >>>>> >>>>> thanks for testing. We have a bunch of test cases that succeeded for >>>>> the rewritten index handling, but as it seems, we definitely need some >>>>> more. I'm pretty sure it's a single bug that causes all the error >>>>> messages (because the code is in itself pretty straightforward), so I >>>>> would be glad if you could compose a little, self-contained example >>>>> that provokes the error. I have attached a little (working) command >>>>> script which you can open in the gui (and execute there) and modify it >>>>> until it raises one of the reported errors. >>>>> >>>>> Thanks, >>>>> Christian >>>>> >>>>> >>>>> >>>>> On Fri, Aug 1, 2014 at 7:14 PM, James Ball <basex-talk(a)jamesball.co.uk> wrote: >>>>>> Hi Christian, >>>>>> >>>>>>> I'm glad to tell you that I have now implemented the projected optimizations >>>>>> >>>>>> Thank you for providing the snapshot. I've downloaded it and begun running some tests. >>>>>> >>>>>> Unfortunately I'm immediately finding some odd behaviour. I'm using the script I provided in my original issue report to the list. >>>>>> >>>>>> I can use replace() to add as many documents to the database as I want as long as the documents are new (no document exists to be replaced). >>>>>> >>>>>> If I use replace() on one document in a transaction ($files set to 1 in my script) then everything works. >>>>>> >>>>>> However if I try to replace more than one file in a transaction ($files set to 2+) I get an error. >>>>>> >>>>>> Error: >>>>>> Improper use? Potential bug? Your feedback is welcome: >>>>>> Contact: basex-talk(a)mailman.uni-konstanz.de >>>>>> Version: BaseX 8.0 beta 3a7d766 >>>>>> Java: Oracle Corporation, 1.7.0_60 >>>>>> OS: Mac OS X, x86_64 >>>>>> Stack Trace: >>>>>> java.lang.RuntimeException: Key does not exist: 'Name' >>>>>> at org.basex.util.Util.notExpected(Util.java:60) >>>>>> at org.basex.index.value.UpdatableDiskValues.delete(UpdatableDiskValues.java:82) >>>>>> at org.basex.data.DiskData.indexDelete(DiskData.java:390) >>>>>> at org.basex.data.DiskData.indexDelete(DiskData.java:452) >>>>>> at org.basex.data.Data.delete(Data.java:632) >>>>>> at org.basex.data.atomic.Delete.apply(Delete.java:39) >>>>>> at org.basex.data.atomic.AtomicUpdateCache.applyUpdates(AtomicUpdateCache.java:298) >>>>>> at org.basex.data.atomic.AtomicUpdateCache.execute(AtomicUpdateCache.java:282) >>>>>> at org.basex.query.up.DataUpdates.apply(DataUpdates.java:161) >>>>>> at org.basex.query.up.ContextModifier.apply(ContextModifier.java:118) >>>>>> at org.basex.query.up.Updates.apply(Updates.java:129) >>>>>> at org.basex.query.QueryContext.iter(QueryContext.java:351) >>>>>> at org.basex.query.QueryContext.execute(QueryContext.java:605) >>>>>> at org.basex.query.QueryProcessor.execute(QueryProcessor.java:100) >>>>>> at org.basex.core.cmd.AQuery.query(AQuery.java:82) >>>>>> at org.basex.core.cmd.XQuery.run(XQuery.java:22) >>>>>> at org.basex.core.Command.run(Command.java:360) >>>>>> at org.basex.core.Command.execute(Command.java:94) >>>>>> at org.basex.gui.GUI.exec(GUI.java:417) >>>>>> at org.basex.gui.GUI.access$500(GUI.java:41) >>>>>> at org.basex.gui.GUI$8.run(GUI.java:361) >>>>>> >>>>>> If I keep running the command then eventually it will work (eventually is related to the number of documents being replace/in the database). Note that this is in the GUI with the database open in the GUI. >>>>>> >>>>>> If I do it in the GUI but with the database closed I get alternating errors between 'Key does not exist' and 'Key should not exist' each time I run. The error never corrects itself. >>>>>> >>>>>> I'm happy to investigate further and provide more details if required but I'm confused as to what might actually be happening to cause this so I'm not sure where to go next. Let me if you need anything from me. >>>>>> >>>>>> Regards, James

2 3

Why does BaseX create empty System-properties
by rune.engh＠fatax.no 15 Aug '14

15 Aug '14

Hi all, I am wondering why BaseX is setting empty System-properties like http.proxyHost on startup? We are experiencing problems with Spring Social. It tries to create a proxy since http.proxyHost exists, but fails due to the empty value. Best Rune

3 2

Positional predicate optimization
by Daniël Knippers 14 Aug '14

14 Aug '14

Good morning, I wondered if there is room for optimization during the compilation of the following query: let $oneBillion := 1000000000 for $x at $p in (1 to $oneBillion ) where $p = 1 return $x I would have expected this to be compiled into something like for $x in (1 to $oneBillion)[1] return $x Or simply (1 to $oneBillion)[1] Judging only from the execution times of the original query versus the optimized ones, it seems the sequence of 1 to one billion is entirely generated, which seems suboptimal in this case. Kind regards, Daniël Knippers

2 2

Too many different namespaces (limit: 256)
by IrisDeveloper development 14 Aug '14

14 Aug '14

Hi , I am evaluating baseX for my xml based application. I am trying to load my schema files in baseX through command line. (ADD command). It throws following error "Too many different namespaces (limit: 256)" Is there any way I can increase this limit or am I doing something wrong. Can somebody help me out..... Thanks Samby

2 3

locking problem
by Lizzi, Vincent 13 Aug '14

13 Aug '14

I've written a process using the Java BaseXClient to load XML documents to one database (A) and save a small amount of data about each document in a second database (B) using a parallel document path. The data in (B) includes a sha1 hash which is used to determine if the source XML document has changed or not to avoid unnecessary reloading. With database (A) the replace command is used to import documents. With database (B) a query is run to get the current sha1 hash (if any) and the db:replace() function is used to create/update a document for each document in (A). This works well for small sets of documents, often loading several documents per second. However after 1,000 - 4,000 documents it eventually crashes with this exception: [bxerr:BXDB0002] Database (B) is being updated, or update was not completed. The exception is thrown when reading the sha1 hash from (B). After this exception, it's not possible to connect to database (B) even with the BaseX GUI unless I manually delete the upd.basex file. I have to delete both databases and try again. No other processes are accessing the databases. It seems like this problem is related to the high number of reads and writes happening on (B). Has anyone encountered a similar problem or have any suggestions? Thanks, Vincent

2 2