January 2017 - BaseX-Talk - mailman.uni-konstanz.de

Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
by Bularca, Lucian 18 Jan '17

18 Jan '17

Guten Tag, im Rahmen einer Performance-Evaluierung der Persistierung von XML Datenstrukturen in einer BaseX Datenbank, haben wir stetig absinkende Persistierungs-Raten umgekehrt proportional zu der Datenbank-Größe festgestellt. Dieses Verhalten ist erklährbar und wäre auch einnehmbar, wenn nicht die Dauer der Persistierung einer ~ 160 KB großen XML Datenstruktur, von Anfang ~ 10 ms auf ~ 2500 ms kommne würde, nach ~ 50.000 Persistierungs-Vorgänge. Dabei versuchen wir 100.000 unterschiedliche, ungefähr 160 KB große XML Datenstrukturen mittels der Java API in eine BaseX Datenbank zu speichern um dabei die Gesammt-Dauer bzw. die Dauer der einzelnen Persistierungs-Vorgänge zu messen. Die BaseX Datenbank wurde im HTTP Modus (basexhttp) mit -Xmx 4048m gestartert. Die oben genannte Messwerte blieben gleich, unabhängig davon, ob alle XM Datenstrukturen in eine einzige Session gespeichert wurden, oder wenn alle 500 Persistierungs-Vorgänge der Socket (DB-Anbindung) geschlossen und erneut geöffnet wurde. Eine Indizierung der Datenbank (mittels der GUI "Optimize All", bzw. "Create Text Index") zwischendurch konnte die Persistierungs-Raten nicht beeinflussen bzw. optimieren. Ein Beispiel der Test-Klassen (nur exemplarisch, nicht kompilierbar !) die wir dazu benutzt haben, sind im Anhang BaseXClient.java.zip zu dieser E-Mail zu finden. Sind generell, Persistierungs-Raten von mehr als 160 KB / 2500 ms bei mehr als 30.000 vorhandene Einträge in der BaseX zu erwarten, oder können wir diese Persistierungs-Zeiten drastisch optimieren (und wenn ja, wie)? Mit freundlichen Grüßen, Lucian Bularca

4 11

How to optimally and reliably go about sequentially applying updates with long running transactions?
by Mustard Seed 14 Jan '17

14 Jan '17

I'm new to BaseX and XQuery, so I apologize if there's some glaring thing I've missed. What I have are three updating functions that I'm wanting to have executed sequentially, successfully, to BaseX. When I simply place them in my RestXQ function, delimited by commas, they all execute, but since at times certain ones take longer than others, and since their results can overwrite/negate the results of each other, I can't manage to get them to consistently execute in the same order. It seems to me that since it seemingly is running in parallel that whatever happens to end last determines what my final state is. I imagine there's got to be some blatantly obvious way to do this, but I'm just not finding it, I went through the BaseX manual a second time and have searched every possible combination of things I can think of to try and figure out a way around this. One function does a mass delete of all dbs with a given prefix, another repopulates from a sql query, and a third updates the repopulated dbs based on yet another sql query. That's the order I want it to execute in, I'm just about at my wits end trying to figure out how to do this. I was even getting into trying to pull off a hackish attempt at making jobs which I'd schedule based on a rough guess of the longest expected execution times for the functions. But even that wasn't working. Any help will be greatly appreciated as insuring the order of execution is rather vital for the overall project I'm trying to do, so I need to not only find a solution for the present use case, but I need to understand how to insure such for any future composite functions. Thanks, --Nicolai

2 2

format-number rounding
by George Sofianos 14 Jan '17

14 Jan '17

Hi there, I was wondering if there is a similar function to format-number, but without rounding, so I don't have to create a custom one that involves string manipulation. For example I have two values: let $x := <value>15.224134</value> let $y := <value>15.2249348734</value> The following command will create different output for the two values (result for $x will be 15.224, result for $y will be 15.225) format-number(xs:decimal($x), "0.000")) Thanks, George

2 1

Speeding up fn:id()
by Sebastian Zimmer 12 Jan '17

12 Jan '17

Hi all, I am surprised that fn:id() does take quite a long time in contrast to (almost) equivalent XPaths. An example: > XQUERY collection('zk/')//id('zkI_108-5_R') <div xmlns="http://www.tei-c.org/ns/1.0" xml:id="zkI_108-5_R"> ... </div> Query executed in 5956.43 ms. > XQUERY collection('zk/')//*[@xml:id='zkI_108-5_R'] <div xmlns="http://www.tei-c.org/ns/1.0" xml:id="zkI_108-5_R"> ... </div> Query executed in 9.43 ms. Couldn't the first XQuery be optimized into the second? If not, maybe an xml:id index would be a good feature. eXist-db does have one, particularly dedicated to speed up fn:id() [1]. Best regards, Sebastian Zimmer [1] http://exist-db.org/exist/apps/doc/indexing.xml -- <---------- Sebastian Zimmer sebastian.zimmer(a)uni-koeln.de Cologne Center for eHumanities (CCeH) - Universität zu Köln - http://www.cceh.uni-koeln.de ------------------------------------>

2 1

BaseX
by Hans-Juergen Rennau 11 Jan '17

11 Jan '17

Dear BaseX team - no bug, no question, no feature request - only the urge to thank you for your work. About a year ago I embarked on a piece of research work which has kept me busy and excited ever since. This work is important to me because so much seems to go into it, of what I have come to think and be convinced of in the course of several years. At every step, BaseX has been my tool to turn ideas into reality. It has not failed me once, in spite of heavy use of advanced features like function items and partial function invocations. And several times I was so glad to find that BaseX offered exactly the extensions I needed. I think BaseX has achieved an admirable combination of standard conformance and reliability, on the one hand, and bold and generous extension on the other hand. So, summing up - thank you very much. Hans-Jürgen Rennau

1 0

fn:doc weirdness with imported files
by Marc van Grootel 11 Jan '17

11 Jan '17

Hi all, Happy new year btw! I got bitten by something and can't explain this behaviour. So the situation is that I have an XML file on the file system in F:/tmp/foo.xml I also have this file imported into a database named "foo" (and the path "/foo.xml". Now in the GUI (and on 8.5.3) I have the following. When I have the database closed in the GUI base-uri(doc('F:/tmp/foo.xml')) => file:///F:/tmp/foo.xml And when I open the database "foo" from the GUI base-uri(doc('F:/tmp/foo.xml')) => /foo/foo.xml Is that right? Also db:* functions also get a database node from the latter and return a node-id for example. When the foo.xml is moved to another filesystem directory then this doesn't happen so it appears as if this happens only when I try to read the file from the original import location. Shouldn't these two situations have the same result? Cheers, --Marc

2 4

Async unit testing
by Andy Bunce 05 Jan '17

05 Jan '17

Hi, Around version 7.8 there was a function that allowed unit tests to be invoked from XQuery: unit:test-uris($uris as xs:string*) as element(testsuites) Later this was removed, but ability to run updating tests was added. Which was great. I wonder how easy it would be, using the jobs module, to implement a function as below? jobs:test-uris($uris as xs:string*, $options as map(xs:string, xs:string)) as xs:string Asynchronously runs all functions in the specified modules that have unit annotations.A test report is generated and optionally cached, which resembles the format returned by other xUnit testing frameworks, such as the Maven Surefire Plugin. returns a job id /Andy

2 4

XML Prague Meeting... Postponed
by Christian Grün 04 Jan '17

04 Jan '17

Dear all, This year, there will be no BaseX Preconference Meeting in XML Prague. Sorry for that! The reason is simple: We were too busy in order to prepare ourselves for the event, look for speakers, etc. At least we are happy to announce that BaseX 8.6 will finally be released this month. Stay tuned! All the best, Christian

2 1

archive:entries - no representation of empty folders
by Hans-Juergen Rennau 03 Jan '17

03 Jan '17

Dear BaseX team, a question concerning the reading of archive files (e.g. .zip) via archive:entries. If I don't overlook something, the read function archive:entries returns one <archive:entry> per file contained in the archive. Folders containing files (directly or recursively) can be inferred, but empty folders cannot be inferred, and therefore it is not possibly to extract the contents of archives in a reliably complete way: any empty folders are ignored. Is that correct? If so, are there plans to add support for the retrieval of empty folders? Personally, I think the ability to unpack archives completely is important, for example because subsequent operations may fail in the absence of the expected folder. One conceivable solution would be to return for each empty folder an entry with an additional attribute: <archive:entry isFolder="true" .../>. Cheers,Hans-Jürgen

1 0

converting xq to java classes
by Tushar Deshmukh 03 Jan '17

03 Jan '17

Namaste, I have a xq files which is working well. But now I want to convert it to java class. Is there any easy way to do so ? Please help. Regards, Tushar Deshmukh

2 1