BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5250 discussions

Is there any documentation on the narrow limits of XQuery index optimization in BaseX?
by Omar Siam 18 Sep '17

18 Sep '17

Hi list! Recently I started to wonder why functions in my XQuery modules make no use of indexes unless I force them to by using the respective function like db:text(). Now I just did some I think minimal changes to the example for text index at: http://docs.basex.org/wiki/Indexes#Text_Index. If I just change it like below I loose almost all index optimization. The only optimizations left are: * Using db:open or collection with a string but not one in a variable of a for FLOWR expression or the simple map operator. * Using xquery:eval and passing a context with db:open or collection Is there any chance that this will change any time soon? Is this a fundamental restriction? Best regards Omar xquery version "3.1"; declare namespace _ = "urn:local:namespace:_"; import module namespace functx = "http://www.functx.com"; declare function _:_1st_example($ctx as document-node()) { $ctx//*[text() = 'Germany'] }; declare function _:_2nd_example($file as xs:string) { doc($file)//name[. = 'Germany'] }; declare function _:_3rd_example($dbname as xs:string+) { ( for $c in ($dbname!collection(.))//country where $c//city/name = 'Hanoi' return $c/name, $dbname!xquery:eval("//*[text() = 'Vietnam']", map {'': db:open(.)})) }; ( (: 1st example :) _:_1st_example(.), (: 2nd example :) _:_2nd_example('factbook.xml'), (: 3rd example :) _:_3rd_example(('factbook', 'factbook')), xs:string(_:_1st_example) ) Optimized Query: (let $ctx_314 := . return $ctx_314/descendant::*[(text() = "Germany")], db:open-pre("factbook",0)/descendant::name[(. = "Germany")], (for $c_317 in (("factbook", "factbook") ! collection(.))/descendant::country[(descendant::city/name = "Hanoi")] return $c_317/name, (("factbook", "factbook") ! xquery:eval("//*[text() = 'Vietnam']", map { "":db:open(.) }))), _:_1st_example cast as xs:string?) Compiling: [...] - RUNTIME: pre-evaluate root() to document-node() - RUNTIME: rewrite descendant-or-self step(s) - RUNTIME: apply text index for "Vietnam" - RUNTIME: pre-evaluate root() to document-node() - RUNTIME: rewrite descendant-or-self step(s) - RUNTIME: apply text index for "Vietnam"

2 3

Re: [basex-talk] Simple Map Operator in oXygen XML (XQJ)
by Omar Siam 18 Sep '17

18 Sep '17

Ok. Interesting. Please someone note this in the Wiki so there are no surprises. For the ! syntax I can just stick with for in return but missing map {} is a reason to switch to dba. Best regards Omar Am 04.09.2017 um 17:14 schrieb Charles Foster: > Hi Michael, > > BaseX XQJ supports XQuery 3.0 syntax I believe, but not XQuery 3.1 > specific syntax. > > It would be possible to update the XQJ driver to support XQuery 3.1 > syntax, but it would mean upgrading the internal XQuery parser and > re-writer. > > Omar, as a work around you could potentially use equivalent functions > in the standard F&O library (fn). > > Kind Regards, > > Charles > > >> On 4 Sep 2017, at 22:02, Michael Seiferle <ms(a)basex.org >> <mailto:ms@basex.org>> wrote: >> >> Hi Omar, >> /cc Hi @Charles, >> >> you are right, this has nothing to with oXygen in particular! >> Yet: I am not even sure if XQJ is meant to support anything but >> XQuery 1.0. >> Charles might correct me if I am wrong, but I don’t think there’s an >> easy way to add 3.1 features to the current XQJ implementation. >> >> Best from Konstanz >> >> Michael >> >> >> >> >> >>> Am 04.09.2017 um 16:42 schrieb Omar Siam <Omar.Siam(a)oeaw.ac.at >>> <mailto:Omar.Siam@oeaw.ac.at>>: >>> >>> Good idea but I was pretty sure that this comes from within BaseX >>> not oXygen. You can configure the parser and the executor of your >>> XQuery in oXygen und I used BaseX for both. >>> >>> So I found in basex-examples a XQJ tutorial. And just by changing >>> Part1.java a little and filling in the example query from the Wiki I >>> also get >>> >>> XQJQS001 - Invalid XQuery syntax, syntax does not pass static >>> validation. >>> Root Cause: >>> net.xqj.basex.bin.bB: Lexical error at line 1, column 12. >>> Encountered: " " (32), after : "!" >>> at >>> org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34) >>> at >>> org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34) >>> >>> So most probably BaseX'es XQJ parser does not understand the simple >>> map operator. >>> >>> XQSequence xqjs = xqje.executeQuery("xquery version \"3.1\"; (1 to 10) ! element node { . }"); >>> >>> Am 04.09.2017 um 15:37 schrieb Michael Seiferle: >>>> Hi Omar, >>>> >>>> Looks like that error message is generated by oXygen; you might want to crosspost to their list. >>>> >>>> Best >>>> Michael >>>> >>>>> Am 04.09.2017 um 14:31 schrieb Omar Siam<Omar.Siam(a)oeaw.ac.at>: >>>>> >>>>> Hi! >>>>> >>>>> I just tried to use ther Simple Map Operator while writing an XQuery in oXygen XML and executing the query using a client/server BaseX 8.6.6 data source. I configured it as described here:http://docs.basex.org/wiki/Integrating_oXygen >>>>> >>>>> When I use the Simple Map Operator (!) I only get "Invalid XQuery syntax, syntax does not pass static validation.". For example using "(1 to 10) ! element node { . }" >>>>> >>>>> If I run that XQuery in DBA for example it works. >>>>> >>>>> Any ideas? >>>>> >>>>> Best regards >>>>> >>>>> Omar Siam >>>>> >>> >> >

2 1

json-to-xml and xpath
by Günter Dunz-Wolff 18 Sep '17

18 Sep '17

Dear all, I’m trying to get data out of json via xquery. This is my scenario: 1. Fetching data with fetch:text() calling a REST-API 2. Transforming the data with json-to-xml, so I’ll get something like <array xmlns="http://www.w3.org/2005/xpath-functions“> <map> <string key=„word1“>gehen</string> ... </map> ... </array> 3. In my xqm-file I have: let $json_xml := json-to-xml(my_fetched_json) 4. I try to get „gehen“ with $json_xml//string[@key='word1‘], but it yields nothing, also $json_xml//string yields nothing. Only $json_xml//text() yields all string-values into a long string. What am I doing wrong? Or is there another way, to work with JSON-Data to get values of a given key? Thanks in advance, Günter

9 14

Startup hooks or persisting jobs
by Kendall Shaw 18 Sep '17

18 Sep '17

Am I missing an existing way to run xquery at startup (basex web service running under tomcat)? I have jobs that I schedule, but I have to schedule them again if basex is shutdown. I can test for basex being started outside of basex and then execute queries, but if there is already a way to do this within basex, I would rather do that. Kendall

3 3

upgrade of BaseX from 8.6.5 to 8.6.6 how to migrate the data?
by Stefania Axo 18 Sep '17

18 Sep '17

hi I want to upgrade from 8.6.5 to 8.6.6. How do i migrate the data? do i just copy over the Basex\data folder? thanks Stefania

3 2

Possible Bug in BaseX 8.2.3 when importing XML (Was RE: A few general questions about BaseX)
by Anastasiou A. 14 Sep '17

14 Sep '17

Hello everyone Many thanks to Alexander, Bridger, Fabrice, Michael for getting back to me with very detailed responses, these have been really helpful. A few notes: 1) The name is Athanasios :D. Sorry, just couldn’t help it, it seemed incredibly formal to be addressed via the surname in our communications. Our mail server advertises the “Surname. Initial” pattern, so I can see where the confusion came from. 2) I think that there is scope for adding some sort of “logging” to all actions of the server in general because I think I may have hit a bug but I cannot provide any more illuminating comments. Here is what is happening: a. During import, I get an error that file somethingsomething140.xml has an incredibly long element that cannot be imported at line (blahblah). The whole process just dies there. b. This is a bug, because if I simply imported JUST the offending file itself, a single file database is created without any problems and I can query it and all. So, maybe, the error is caused because of the previous file OR because of the way the files are loaded. But I have absolutely no way of knowing the “load history” of the files or the exception that was caught or anything else. In fact, once you press “OK” in the error dialog box, any database files that have been created are lost. In addition to this, the XML files to import are enumerated in a random order. So, I had to run the import again and stay there looking at each one of the files loading, to witness that the system “breaks” after 254 files (which is suspiciously close to 256). None of the files around the vicinity of the offending file caused any problems, so this may be a more difficult to catch bug (but it is thrown with both the internal and external parsers). Following this, I created smaller databases with 250 XML files and then got “predictable” errors on running out of memory and not creating indexes which I can solve more easily. 3) It’s good to know that I don’t need the original files because that’s a lot of space I can get rid of. Thank you. 4) Seems like the ADDCACHE would have saved me some trouble here, many thanks for that, but of course, if you don’t know the file enumeration order, you are still stuck in not knowing which files have already been imported. 5) Michael, logging won’t help with the internal import procedure, except of course if you were implying writing a quick script to do the import “manually”? 6) Michael, the fork-join and “client connect” are really interesting and worth a try before I start connecting things together via Hadoop. Are these modules already available to BaseX? Do I simply import their namespace or is it not even needed? Many thanks again. All the best From: Bridger Dyson-Smith [mailto:bdysonsmith@gmail.com] Sent: 12 September 2017 16:53 To: Anastasiou A. Cc: basex-talk(a)mailman.uni-konstanz.de Subject: Re: [basex-talk] A few general questions about BaseX Hi Anastasiou, Hopefully some of these answers are somewhat helpful. On Tue, Sep 12, 2017 at 4:54 AM, Anastasiou A. <a.anastasiou(a)swansea.ac.uk<mailto:a.anastasiou@swansea.ac.uk>> wrote: Hello everyone I am trying to load BaseX with a large number of XML files (~500), each one a few hundreds of MBs big. BaseX fails with a message along the lines “This is too big for one database”. Can I please ask: 1) Are there any logs, beyond the DB logs? If yes, where can I find them? a. The reason I am asking is because once basexgui gives the message, there is no indication about the error. Ideally, I would like to know if this is a limitation on memory amount or number of items (?). I'm not sure how to enable more verbose logging with the GUI -- hopefully one of the devs or power users can weigh in on this. 2) The parser options include reading XML files from archives, which is very convenient, but once the file has been parsed, does BaseX require the “originals” for queries / returning results? AFAIK, no it does not. BaseX will query and return results from the internal database(s). 3) Is it possible to do federation with BaseX? In other words, let’s say I split a database in two large parts (as per #1), is it possible to launch two baseX servers and then have them talk to each other so that ultimately I just query one of them and get back unified results? AFAIK, the preferred method is to split your files across many databases, then query multiple databases from a single expression[1]. Others will be able to speak to this better, but I don't think there's a straightforward way to run multiple BaseX servers in a single JVM. All the best Best, Bridger [1] http://docs.basex.org/wiki/Databases

2 3

Re: [basex-talk] BaseX 7.9 unstable
by Alexander Holupirek 13 Sep '17

13 Sep '17

> On 13. Sep 2017, at 11:28, Steinhauer, Dorothea <Dorothea.Steinhauer(a)proalpha.de> wrote: > >>> On 13. Sep 2017, at 10:06, Steinhauer, Dorothea <Dorothea.Steinhauer(a)proalpha.de> wrote: >>> >>> Hi, >>> >>> we use BaseX 7.9 and we have the problem that in some (non-replicable) situations no more data is written to the database. >>> The database is still readable. >>> Logfiles are currently not available. >>> >>> If the problem is known, can changes to properties help? >>> >>> For us, it is not an option to switch to a newer version of the baseX-DB because API changes would be a huge effort. >>> >>> >>> Thanks, >>> Dorothea >> >> I think, you would have to detect the pattern that leads to the unwanted behaviour. >> Maybe the documentation on Transaction Management [1] can help to reason about the problem? >> >> By the way, what do you mean with 'no more data is written to the database'. Never ever again after some situation or during a phase (and later on it possible again)? >> >> Thanks, >> Alex >> >> [1] http://docs.basex.org/wiki/Transaction_Management#Concurrency_Control > > > Hi, > > we use the database in production, so we can't wait if anything changed. You wanted to know with the question whether the database is permanently locked? I have problems to understand what 'no more data is written to the database' actually means. - Is it there a permanent write-lock on the database? - Or is the database corrupt after a write operation? - Or can't you write at some point in time and then, at later point, once again write to the database? > When the database is "write-locking", is it possible to back up our data? When a write operation alters the database you can not read (or write) the database [1], so a regular backup is not possible. What you nevertheless can do, is, on the OS-level and only for debugging and failure analysis, copy the database directory to another location and, for instance, inspect [2] the state of the database. Perhaps you can elaborate a bit on your whole scenario. How does your application operate? What do you do, if you encounter a situation in which you can not write to the database? Can you make a guess what is happening in your application when the situation occurs? Cheers, Alex [1] http://docs.basex.org/wiki/Transaction_Management#Concurrency_Control [2] http://docs.basex.org/wiki/Commands#INSPECT

1 0

BaseX 7.9 unstable
by Steinhauer, Dorothea 13 Sep '17

13 Sep '17

Hi, we use BaseX 7.9 and we have the problem that in some (non-replicable) situations no more data is written to the database. The database is still readable. Logfiles are currently not available. If the problem is known, can changes to properties help? For us, it is not an option to switch to a newer version of the baseX-DB because API changes would be a huge effort. Thanks, Dorothea

2 1

How do create XML for a one byte hex field that is used as a flag byte
by Dave Day 07 Sep '17

07 Sep '17

Greetings list, I don't know if I am asking this question correctly. I'm new to the XML world. I can google this, get some hits, but when I read the posts, it is not really clear to me. Any help is appreciated. On an IBM mainframe, I've got a one byte hex field defined as follows: PCDTFLAG DS XL1 FLAG TO INDICATE ORIGIN OF INFO DTFLENTP EQU B'10000000' INFO CAME FROM MODULE ENTRY POINT DTFLMEPS EQU B'01000000' DB2 MEPL LIST HAS BEEN SEARCHED DTFLMEPL EQU B'00100000' INFO CAME FROM DB2 MEPL LIST DTFLDB2S EQU B'00010000' DSNMODS HAS BEEN SEARHCED DTFLDSNM EQU B'00001000' INFO CAME FROM DSNMODS SEARCH DTFLSYST EQU B'00000100' INFO CAME FROM SYSTEM TABLE DTFLSYSS EQU B'00000010' SYSTEM TABLE HAS BEEN SEARCHED DTFLCMNC EQU B'00000001' COMMON AND NUC HAS BEEN SEARCHED I need to be able to create XML for the field PCDTFLAG. OK, define an element as hexBinary. But then how do I define the bits within the byte. On the mainframe side, with the equated values, I can OR the flag byte with the equated value, and this will set the correct bit on in the byte. OI PCDTFLAG,DTFLENTP will turn on bit 0 in the 1 byte field if it is not on already. I can test for the bit on or off. TM PCDTFLAG,DTFLENTP JZ some-other-area-of-code will branch if the bit is off JO another-area-of-code will branch if the bit is on So, I guess I am asking two questions. One, how to give definition to bits within a byte, and two, how to manipulate and test those defined bits. Regards, -- Dave Day

3 2

XML import issue (header related)
by Ron Katriel 06 Sep '17

06 Sep '17

Hi, I ran into a peculiar issue with importing an XML file from DrugBank into BaseX. It involves the file header, which looks like this: <?xml version="1.0" encoding="UTF-8"?> <drugbank xmlns="http://www.drugbank.ca" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.drugbank.ca http://www.drugbank.ca/docs/drugbank.xsd" version="5.0" exported-on="2017-08-31"> The XML file (which is quite large ~3GB) loads in cleanly but queries executed against it produce no output (no errors reported). When I change the header to the following and re-import the file everything works fine: <?xml version="1.0" encoding="UTF-8"?> <drugbank> I had this issue for a while (go thought this every month) so thought it might be good to share. I am currently using BaseX 8.6.4. Any idea why this is happening? Attached is the console trace for both cases (they show subtle differences in the query plans). Thanks, Ron Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions 350 Hudson Street, 7th Floor, New York, NY 10014 rkatriel(a)mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800

5 14

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk