BaseX-Talk March 2015

basex-talk@mailman.uni-konstanz.de

36 participants
58 discussions

Fwd: Re: How to process very long node sequences
by Leonard Wörteler 19 Mar '15

19 Mar '15

Wrong mail account... -------- Weitergeleitete Nachricht -------- Betreff: Re: [basex-talk] How to process very long node sequences Datum: Wed, 18 Mar 2015 19:26:00 +0100 Von: Leo Wörteler <lw(a)basex.org> An: Hans-Juergen Rennau <hrennau(a)yahoo.de> Kopie (CC): basex-talk(a)mailman.uni-konstanz.de <basex-talk(a)mailman.uni-konstanz.de> Dear Hans-Jürgen, Am 18.03.2015 um 15:23 schrieb Hans-Juergen Rennau: > I want to persuade the processor to visit, process and forget the nodes > one after the other, rather than to attempt loading all nodes into > memory before proceeding to process them. BaseX already uses iterative processing as default mode. It only falls back to caching if it has to, e.g. for sorting, reversing or duplicate elimination. Calls to user-defined functions are currently also blocking, but we are currently investigating if that can be changed. > Schematically: > > declare function f:processNode($node as node()) as empty-sequence() {...}; > > for $node in doc('huge-doc')/a/b/c > return f:processNode($node) The example you gave will always be evaluated without caching the node sequence. The `for` clause requests the result of its argument iteratively, and the XPath expression you used only contains child steps, which can be evaluated in document order without duplicates. > Well, in some cases it works, in others in doesn't. Is there any safe > way how to enforce sequential visit-process-forget processing? When the XPath expression becomes more complex, it is not as easy to predict if it uses caching internally. BaseX tries quite hard to detect paths that do not need it, the algorithm can be seen in [1]. If you see a `CachedPath` in the Info View of the GUI, you can try to reformulate the query. > I call this problem general because it is a critical aspect of dealing > with huge documents. XQuery does not have a dedicated *streaming mode* like that of XSLT 3.0 [2] (yet), but it would definitely be possible to check if some part of a query (e.g. marked by a pragma) is evaluated without caching. It would however be quite some work. Hope that helps, Leo [1] https://github.com/BaseXdb/basex/blob/0b828a8/basex-core/src/main/java/org/… [2] http://www.w3.org/TR/xslt-30/#dt-guaranteed-streamable

4 4

How to process very long node sequences
by Hans-Juergen Rennau 18 Mar '15

18 Mar '15

Dear BaseX team, can you help me with the following, very general problem? I have a huge document in a database and want to iterate over many nodes which it contains, performing for each one an action which does not produce a value - for example, storing it in a SQL database. I want to persuade the processor to visit, process and forget the nodes one after the other, rather than to attempt loading all nodes into memory before proceeding to process them. Schematically: declare function f:processNode($node as node()) as empty-sequence() {...}; for $node in doc('huge-doc')/a/b/creturn f:processNode($node) ~ ~ ~ Well, in some cases it works, in others in doesn't. Is there any safe way how to enforce sequential visit-process-forget processing? I call this problem general because it is a critical aspect of dealing with huge documents. Cheers,Hans-Jürgen

1 0

Unexpected format-integer result
by Neven Jovanović 16 Mar '15

16 Mar '15

Hello, when BaseX 8.0 performs the following XQuery expression: format-integer(556, '#,##1') I expect to see just: 556 -- but I get the following: ,556 Am I misreading something in the format-integer() chapter of XPath and XQuery Functions and Operators 3.0[1], or is this a bug? Best, Neven [1] <http://www.w3.org/TR/xpath-functions-30/#func-format-integer>

2 2

Large Document Upload Performance
by Jonathan Clarke 16 Mar '15

16 Mar '15

Hi, I hope you can help me. I am using BaseX to underpin a complex distributed system, which also requires storage of xml document in soft real-time. At the moment, I'm getting storage times for a 4Mb XML file of about 500ms. Can you advise how I might be able to bring that down, please, by at least 75%? I have tried tweaking the systems for BaseX 8, including turning off autoflush, and each setting that I try actually seems to increase the processing time. I also tried to use AddCache, and that just crashed the latest production release of the server. Many thanks for your help in advance, Dr Jonathan Clarke.

3 9

DITA project
by France Baril 14 Mar '15

14 Mar '15

Hi, I'm working on a new DITA project. The DITA standard is used for technical documentation. It creates XML models with inheritance by using @class="ancestor parent child". If the child should behave like it's ancestor the XSL would say: <template match="contains(@class, 'ancestor')"> ... </template> The advantage is that the model can evolve and when new elements are added, you only need to code transformations for the differences. I am trying to figure out if I can use type switching with contains in attribute. Search gets me no syntax for something like this: typeswitch ($node) case attribute(contains(class, 'ancestor')) Maybe I should register to the xquery group to get an answer, but since I'm already here, I though I should ask, and maybe there is a BaseX specific option. I mean other that an unmanageable number of 'if then else' statements. Thanks! -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

3 5

Add document performances with UPDINDEX = true
by Simon Chatelain 13 Mar '15

13 Mar '15

Hello, First let me give you the context: I have a never ending stream of XML element coming in that I want to store and then make available through a REST interface. Thus BaseX seems to be a well suited candidate. To be on the safe side I must be able to sustain an insertion rate of about 200 elements per second. The XML elements I have to store are of the type: <notification ts=”2015-03-13T10.44.25.123” nid=”type-of-data”> <name-1>value1</ name-1> <name-2>value2</ name-2> <name-3>value3</ name-3> <name-4>value4</ name-4> …. </notification> So quite simple and small. I will mainly retrieve data by selecting notifications of a specific @nid between two @ts values, thus I need an attribute index. I am using for now an embedded BaseX DB, to test the insertion of elements. Here is how I configure my DB: Context m_Context = new Context(); new Set(MainOptions.AUTOFLUSH, false).execute(m_Context); new Set(MainOptions.ADDCACHE, false).execute(m_Context); new Set(MainOptions.INTPARSE, true).execute(m_Context); new Set(MainOptions.STRIPNS, true).execute(m_Context); new Set(MainOptions.UPDINDEX, true).execute(m_Context); new Set(MainOptions.TEXTINDEX, false).execute(m_Context); new Set(MainOptions.ATTRINDEX, true).execute(m_Context); new CreateDB(_SourceId).execute(m_Context); And this is how I insert the elements: try { String l_XmlRepresentation = _Notification.getXmlRepresentation(); if (l_XmlRepresentation.isEmpty()) { return; } ByteArrayInputStream l_InputStream = new ByteArrayInputStream(l_XmlRepresentation.getBytes(m_Charset)); Add add = new Add(_Notification.getSourceId()); add.setInput(l_InputStream); add.execute(m_Context); if (_CurrentNotification % 10000 == 0) { // flush every 10000 notifications new Flush().execute(m_Context); } } catch (BaseXException ex) { s_Logger.log(Level.SEVERE, null, ex); } The performances I get are as follows Size 10'000, Speed: 1'292 Size 20'000, Speed: 625 Size 30'000, Speed: 361 Size 40'000, Speed: 248 Size 50'000, Speed: 184 Size 60'000, Speed: 148 Size 70'000, Speed: 123 Size 80'000, Speed: 104 Size 90'000, Speed: 91 Size 100'000, Speed: 77 Size 110'000, Speed: 69 Size 120'000, Speed: 61 Size 130'000, Speed: 56 Size 140'000, Speed: 46 Where “Size” is the number of elements in the collection and “Speed” is average speed of insertion [in element per second] of the last 10000 elements. My question is: do those performances seem normal or am I doing something wrong, knowing that with UPDINDEX = false, I have a steady insertion rate of 10000 elements per second. Thanks a lot Simon

2 3

Query / LocalQuery and number of hits
by Ketill Fenne 13 Mar '15

13 Mar '15

Hi! I am using Query / LocalQuery to loop through the results from a query. For several reasons it would be nice to know the number of hits before/without looping through the results. Is this possible? Kind regards Ketill Fenne

3 2

Can you prevent database modification via GUI?
by Amanda Galtman 13 Mar '15

13 Mar '15

Hi, Is there a way to prevent users of the BaseX GUI from modifying databases? It seems that at least some of the database files need to be writable or else you can't open the database. It also seems that the GUI gives you admin permission[1], making it possible for you to drop a database. I've been showing BaseX to more of my colleagues, and I want to protect against inadvertent modification of the databases that we are all sharing. Most of us use the GUI on Windows, and a few of us execute queries in standalone mode on various platforms. A related question is whether it is problematic for one user of the GUI to have a database open (or to be actively running a query) while another user of the GUI is trying to run a query on the same database. Maybe a third user is executing a query in standalone mode at the same time. Assume all the queries are non-updating. The doc about transaction management[2] seems to suggest that you can have an arbitrary number of read transactions on the same database, which I take to mean that the different users will not interfere with each other. Is that correct and, if so, does it depend on a particular BaseX version? Thanks, Amanda [1] http://docs.basex.org/wiki/User_Management [2] http://docs.basex.org/wiki/Transaction_Management

2 2

BaseX can't add to database and query in different test
by Jason Ricles 13 Mar '15

13 Mar '15

I am currently using BaseX for an XML database. Each XML file currently has a uuid. So, to avoid adding multiple same files to the database, have made a query that will check to see if the uuid from the file being added is already in the database. However I am not able to query the files that already exist in the database until I either open up the database in the BaseX GUI or have the query statement executed at the same time the file is added. So for example if I ran a test of adding a file to the database and then ran a test to query this file it would be unable to query this file. However If I ran the test to add the file to the database and query in the same test the query would work or if I ran the test to add the file to the database and then ran the BaseX GUI and then ran the query test the query will work correctly. Also if I already add the file to basex don't open up the DB then run to add the file again and check before I add the file the query will not work before but only after the file is added a second time. I am writing my code in java. Does anyone know why this is happening and if there is a solution?

2 1

Merging baseX indexes
by Marios Bartzokas 13 Mar '15

13 Mar '15

Hi, I am trying to achieve the following; Have a number of baseX servers each working on a replica of an indexed document. One baseX server will handle concurrent client read queries The rest of the servers will handle concurrent client write queries on their copy of the index. In the end I would like to merge all the indexes. Is there a way to achieve that? Best Regards, Marios

2 7

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk March 2015