BaseX-Talk September 2017

basex-talk@mailman.uni-konstanz.de

30 participants
38 discussions

retrieving the name of the archive?
by Graydon Saunders 18 Sep '17

18 Sep '17

Hello -- BaseX will happily consume zip archives; this is just splendid for loading up a bunch of docx files. Now I find myself wanting the name of the docx file -- the original name of the archive -- and I don't know how to retrieve that. (or if it can be!) But I think it must be there somewhere because db:path repeats the standard OOXML file paths: [Content_Types].xml word/document.xml word/footnotes.xml word/footer1.xml word/endnotes.xml word/theme/theme1.xml word/settings.xml docProps/custom.xml customXml/itemProps2.xml docProps/app.xml customXml/item2.xml customXml/itemProps1.xml word/fontTable.xml customXml/item1.xml customXml/item3.xml customXml/itemProps3.xml customXml/item4.xml customXml/itemProps4.xml word/numbering.xml word/styles.xml word/webSettings.xml docProps/core.xml word/people.xml over and over; if they were all going to exactly that there'd be one copy, and all several hundred docx files are there by content. (db:list-details tells me about > 4000 individual xml files.) If I can get the name of the original archive, how do I do that? Thanks! Graydon

2 3

Basex Inner Workings
by Anastasiou A. 18 Sep '17

18 Sep '17

Hello everyone Quick question: Is there any document / URL where I could find out more about how does Basex access the disk during its operation? For example, are there any reads to be expected during executing a query? Through iotop, I can see 3-4 processes reading during startup, then another 2, very briefly firing when opening the database and then during querying there are periodic writes (?) but of very brief duration. I was wondering if there is anything that could be done from the point of view of the hardware to speed up queries (?) (except a more powerful machine at the moment) All the best Athanasios Anastasiou

5 16

Is there any documentation on the narrow limits of XQuery index optimization in BaseX?
by Omar Siam 18 Sep '17

18 Sep '17

Hi list! Recently I started to wonder why functions in my XQuery modules make no use of indexes unless I force them to by using the respective function like db:text(). Now I just did some I think minimal changes to the example for text index at: http://docs.basex.org/wiki/Indexes#Text_Index. If I just change it like below I loose almost all index optimization. The only optimizations left are: * Using db:open or collection with a string but not one in a variable of a for FLOWR expression or the simple map operator. * Using xquery:eval and passing a context with db:open or collection Is there any chance that this will change any time soon? Is this a fundamental restriction? Best regards Omar xquery version "3.1"; declare namespace _ = "urn:local:namespace:_"; import module namespace functx = "http://www.functx.com"; declare function _:_1st_example($ctx as document-node()) { $ctx//*[text() = 'Germany'] }; declare function _:_2nd_example($file as xs:string) { doc($file)//name[. = 'Germany'] }; declare function _:_3rd_example($dbname as xs:string+) { ( for $c in ($dbname!collection(.))//country where $c//city/name = 'Hanoi' return $c/name, $dbname!xquery:eval("//*[text() = 'Vietnam']", map {'': db:open(.)})) }; ( (: 1st example :) _:_1st_example(.), (: 2nd example :) _:_2nd_example('factbook.xml'), (: 3rd example :) _:_3rd_example(('factbook', 'factbook')), xs:string(_:_1st_example) ) Optimized Query: (let $ctx_314 := . return $ctx_314/descendant::*[(text() = "Germany")], db:open-pre("factbook",0)/descendant::name[(. = "Germany")], (for $c_317 in (("factbook", "factbook") ! collection(.))/descendant::country[(descendant::city/name = "Hanoi")] return $c_317/name, (("factbook", "factbook") ! xquery:eval("//*[text() = 'Vietnam']", map { "":db:open(.) }))), _:_1st_example cast as xs:string?) Compiling: [...] - RUNTIME: pre-evaluate root() to document-node() - RUNTIME: rewrite descendant-or-self step(s) - RUNTIME: apply text index for "Vietnam" - RUNTIME: pre-evaluate root() to document-node() - RUNTIME: rewrite descendant-or-self step(s) - RUNTIME: apply text index for "Vietnam"

2 3

Re: [basex-talk] Simple Map Operator in oXygen XML (XQJ)
by Omar Siam 18 Sep '17

18 Sep '17

Ok. Interesting. Please someone note this in the Wiki so there are no surprises. For the ! syntax I can just stick with for in return but missing map {} is a reason to switch to dba. Best regards Omar Am 04.09.2017 um 17:14 schrieb Charles Foster: > Hi Michael, > > BaseX XQJ supports XQuery 3.0 syntax I believe, but not XQuery 3.1 > specific syntax. > > It would be possible to update the XQJ driver to support XQuery 3.1 > syntax, but it would mean upgrading the internal XQuery parser and > re-writer. > > Omar, as a work around you could potentially use equivalent functions > in the standard F&O library (fn). > > Kind Regards, > > Charles > > >> On 4 Sep 2017, at 22:02, Michael Seiferle <ms(a)basex.org >> <mailto:ms@basex.org>> wrote: >> >> Hi Omar, >> /cc Hi @Charles, >> >> you are right, this has nothing to with oXygen in particular! >> Yet: I am not even sure if XQJ is meant to support anything but >> XQuery 1.0. >> Charles might correct me if I am wrong, but I don’t think there’s an >> easy way to add 3.1 features to the current XQJ implementation. >> >> Best from Konstanz >> >> Michael >> >> >> >> >> >>> Am 04.09.2017 um 16:42 schrieb Omar Siam <Omar.Siam(a)oeaw.ac.at >>> <mailto:Omar.Siam@oeaw.ac.at>>: >>> >>> Good idea but I was pretty sure that this comes from within BaseX >>> not oXygen. You can configure the parser and the executor of your >>> XQuery in oXygen und I used BaseX for both. >>> >>> So I found in basex-examples a XQJ tutorial. And just by changing >>> Part1.java a little and filling in the example query from the Wiki I >>> also get >>> >>> XQJQS001 - Invalid XQuery syntax, syntax does not pass static >>> validation. >>> Root Cause: >>> net.xqj.basex.bin.bB: Lexical error at line 1, column 12. >>> Encountered: " " (32), after : "!" >>> at >>> org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34) >>> at >>> org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34) >>> >>> So most probably BaseX'es XQJ parser does not understand the simple >>> map operator. >>> >>> XQSequence xqjs = xqje.executeQuery("xquery version \"3.1\"; (1 to 10) ! element node { . }"); >>> >>> Am 04.09.2017 um 15:37 schrieb Michael Seiferle: >>>> Hi Omar, >>>> >>>> Looks like that error message is generated by oXygen; you might want to crosspost to their list. >>>> >>>> Best >>>> Michael >>>> >>>>> Am 04.09.2017 um 14:31 schrieb Omar Siam<Omar.Siam(a)oeaw.ac.at>: >>>>> >>>>> Hi! >>>>> >>>>> I just tried to use ther Simple Map Operator while writing an XQuery in oXygen XML and executing the query using a client/server BaseX 8.6.6 data source. I configured it as described here:http://docs.basex.org/wiki/Integrating_oXygen >>>>> >>>>> When I use the Simple Map Operator (!) I only get "Invalid XQuery syntax, syntax does not pass static validation.". For example using "(1 to 10) ! element node { . }" >>>>> >>>>> If I run that XQuery in DBA for example it works. >>>>> >>>>> Any ideas? >>>>> >>>>> Best regards >>>>> >>>>> Omar Siam >>>>> >>> >> >

2 1

json-to-xml and xpath
by Günter Dunz-Wolff 18 Sep '17

18 Sep '17

Dear all, I’m trying to get data out of json via xquery. This is my scenario: 1. Fetching data with fetch:text() calling a REST-API 2. Transforming the data with json-to-xml, so I’ll get something like <array xmlns="http://www.w3.org/2005/xpath-functions“> <map> <string key=„word1“>gehen</string> ... </map> ... </array> 3. In my xqm-file I have: let $json_xml := json-to-xml(my_fetched_json) 4. I try to get „gehen“ with $json_xml//string[@key='word1‘], but it yields nothing, also $json_xml//string yields nothing. Only $json_xml//text() yields all string-values into a long string. What am I doing wrong? Or is there another way, to work with JSON-Data to get values of a given key? Thanks in advance, Günter

9 14

Startup hooks or persisting jobs
by Kendall Shaw 18 Sep '17

18 Sep '17

Am I missing an existing way to run xquery at startup (basex web service running under tomcat)? I have jobs that I schedule, but I have to schedule them again if basex is shutdown. I can test for basex being started outside of basex and then execute queries, but if there is already a way to do this within basex, I would rather do that. Kendall

3 3

upgrade of BaseX from 8.6.5 to 8.6.6 how to migrate the data?
by Stefania Axo 18 Sep '17

18 Sep '17

hi I want to upgrade from 8.6.5 to 8.6.6. How do i migrate the data? do i just copy over the Basex\data folder? thanks Stefania

3 2

Possible Bug in BaseX 8.2.3 when importing XML (Was RE: A few general questions about BaseX)
by Anastasiou A. 14 Sep '17

14 Sep '17

Hello everyone Many thanks to Alexander, Bridger, Fabrice, Michael for getting back to me with very detailed responses, these have been really helpful. A few notes: 1) The name is Athanasios :D. Sorry, just couldn’t help it, it seemed incredibly formal to be addressed via the surname in our communications. Our mail server advertises the “Surname. Initial” pattern, so I can see where the confusion came from. 2) I think that there is scope for adding some sort of “logging” to all actions of the server in general because I think I may have hit a bug but I cannot provide any more illuminating comments. Here is what is happening: a. During import, I get an error that file somethingsomething140.xml has an incredibly long element that cannot be imported at line (blahblah). The whole process just dies there. b. This is a bug, because if I simply imported JUST the offending file itself, a single file database is created without any problems and I can query it and all. So, maybe, the error is caused because of the previous file OR because of the way the files are loaded. But I have absolutely no way of knowing the “load history” of the files or the exception that was caught or anything else. In fact, once you press “OK” in the error dialog box, any database files that have been created are lost. In addition to this, the XML files to import are enumerated in a random order. So, I had to run the import again and stay there looking at each one of the files loading, to witness that the system “breaks” after 254 files (which is suspiciously close to 256). None of the files around the vicinity of the offending file caused any problems, so this may be a more difficult to catch bug (but it is thrown with both the internal and external parsers). Following this, I created smaller databases with 250 XML files and then got “predictable” errors on running out of memory and not creating indexes which I can solve more easily. 3) It’s good to know that I don’t need the original files because that’s a lot of space I can get rid of. Thank you. 4) Seems like the ADDCACHE would have saved me some trouble here, many thanks for that, but of course, if you don’t know the file enumeration order, you are still stuck in not knowing which files have already been imported. 5) Michael, logging won’t help with the internal import procedure, except of course if you were implying writing a quick script to do the import “manually”? 6) Michael, the fork-join and “client connect” are really interesting and worth a try before I start connecting things together via Hadoop. Are these modules already available to BaseX? Do I simply import their namespace or is it not even needed? Many thanks again. All the best From: Bridger Dyson-Smith [mailto:bdysonsmith@gmail.com] Sent: 12 September 2017 16:53 To: Anastasiou A. Cc: basex-talk(a)mailman.uni-konstanz.de Subject: Re: [basex-talk] A few general questions about BaseX Hi Anastasiou, Hopefully some of these answers are somewhat helpful. On Tue, Sep 12, 2017 at 4:54 AM, Anastasiou A. <a.anastasiou(a)swansea.ac.uk<mailto:a.anastasiou@swansea.ac.uk>> wrote: Hello everyone I am trying to load BaseX with a large number of XML files (~500), each one a few hundreds of MBs big. BaseX fails with a message along the lines “This is too big for one database”. Can I please ask: 1) Are there any logs, beyond the DB logs? If yes, where can I find them? a. The reason I am asking is because once basexgui gives the message, there is no indication about the error. Ideally, I would like to know if this is a limitation on memory amount or number of items (?). I'm not sure how to enable more verbose logging with the GUI -- hopefully one of the devs or power users can weigh in on this. 2) The parser options include reading XML files from archives, which is very convenient, but once the file has been parsed, does BaseX require the “originals” for queries / returning results? AFAIK, no it does not. BaseX will query and return results from the internal database(s). 3) Is it possible to do federation with BaseX? In other words, let’s say I split a database in two large parts (as per #1), is it possible to launch two baseX servers and then have them talk to each other so that ultimately I just query one of them and get back unified results? AFAIK, the preferred method is to split your files across many databases, then query multiple databases from a single expression[1]. Others will be able to speak to this better, but I don't think there's a straightforward way to run multiple BaseX servers in a single JVM. All the best Best, Bridger [1] http://docs.basex.org/wiki/Databases

2 3

Re: [basex-talk] BaseX 7.9 unstable
by Alexander Holupirek 13 Sep '17

13 Sep '17

> On 13. Sep 2017, at 11:28, Steinhauer, Dorothea <Dorothea.Steinhauer(a)proalpha.de> wrote: > >>> On 13. Sep 2017, at 10:06, Steinhauer, Dorothea <Dorothea.Steinhauer(a)proalpha.de> wrote: >>> >>> Hi, >>> >>> we use BaseX 7.9 and we have the problem that in some (non-replicable) situations no more data is written to the database. >>> The database is still readable. >>> Logfiles are currently not available. >>> >>> If the problem is known, can changes to properties help? >>> >>> For us, it is not an option to switch to a newer version of the baseX-DB because API changes would be a huge effort. >>> >>> >>> Thanks, >>> Dorothea >> >> I think, you would have to detect the pattern that leads to the unwanted behaviour. >> Maybe the documentation on Transaction Management [1] can help to reason about the problem? >> >> By the way, what do you mean with 'no more data is written to the database'. Never ever again after some situation or during a phase (and later on it possible again)? >> >> Thanks, >> Alex >> >> [1] http://docs.basex.org/wiki/Transaction_Management#Concurrency_Control > > > Hi, > > we use the database in production, so we can't wait if anything changed. You wanted to know with the question whether the database is permanently locked? I have problems to understand what 'no more data is written to the database' actually means. - Is it there a permanent write-lock on the database? - Or is the database corrupt after a write operation? - Or can't you write at some point in time and then, at later point, once again write to the database? > When the database is "write-locking", is it possible to back up our data? When a write operation alters the database you can not read (or write) the database [1], so a regular backup is not possible. What you nevertheless can do, is, on the OS-level and only for debugging and failure analysis, copy the database directory to another location and, for instance, inspect [2] the state of the database. Perhaps you can elaborate a bit on your whole scenario. How does your application operate? What do you do, if you encounter a situation in which you can not write to the database? Can you make a guess what is happening in your application when the situation occurs? Cheers, Alex [1] http://docs.basex.org/wiki/Transaction_Management#Concurrency_Control [2] http://docs.basex.org/wiki/Commands#INSPECT

1 0

BaseX 7.9 unstable
by Steinhauer, Dorothea 13 Sep '17

13 Sep '17

Hi, we use BaseX 7.9 and we have the problem that in some (non-replicable) situations no more data is written to the database. The database is still readable. Logfiles are currently not available. If the problem is known, can changes to properties help? For us, it is not an option to switch to a newer version of the baseX-DB because API changes would be a huge effort. Thanks, Dorothea

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk September 2017