BaseX-Talk October 2017

basex-talk@mailman.uni-konstanz.de

26 participants
36 discussions

Re: [basex-talk] BaseXdb/basex] Packaging: Java and XQuery code (#1413)
by Tim Thompson 26 Oct '17

26 Oct '17

Christian, all, The new "combined" packaging option ( https://github.com/BaseXdb/basex/issues/1413) seems like an interesting enhancement. I have been working on my first BaseX repository module/package (though unfortunately I am not yet fluent in Java). It is working perfectly so far, but I wanted to be sure I was going about it correctly because it still seems a little like magic to me. I am also wondering whether this new feature is something I might want to take advantage of. The module (https://github.com/timathom/basex-rdf) uses Günther Rademacher's REx Parser Generator[1] to create a Java parser for the RDF TriG/Turtle syntax[2], using the BaseX extension function option. Following the BaseX documentation[3], I have created a XAR package that includes the JAR file and an XQuery wrapper. The wrapper calls the function exposed by the Java module (import module namespace trig = "org.basex.modules.rdf.TriG";) and then processes the raw parse tree, overlaying some additional abstractions (I confess that I cheated and used XSLT for most of the processing functionality). Because of the built-in BaseX extension function, this approach seems different from a straightforward "combined" module approach, but, again, I wanted to ask whether the way I'm doing this seems correct. Thanks in advance. Tim [1] http://www.bottlecaps.de/rex/ [2] https://www.w3.org/TR/trig/ [3] docs.basex.org/wiki/Repository -- Tim A. Thompson Discovery Metadata Librarian Yale University Library

3 9

xquery:eval Issue with bindings
by Shaun Flynn 21 Oct '17

21 Oct '17

Hello there, I am trying to use xquery:eval method supplying bindings as map, and following the example on the website where the bindings are supplied as a map. However, when I run the following query, I get the error "Items of type map(*) cannot be atomized". What am I doing wrong? (:Filter out the required diagrams from $diagrams:) declare variable $depotConstraint external := "OC"; declare variable $resConstraint external := "NIL"; declare variable $lowerDiagNoConstraint external := "NIL"; declare variable $upperDiagNoConstraint external := "NIL"; (:Bindings for diagramSelectionCoR.xq:) declare variable $bindings := map { xs:QName("depotConstraint") : $depotConstraint, xs:QName("resConstraint") : $resConstraint, xs:QName("loweDiagNoConstraint") : $lowerDiagNoConstraint, xs:QName("upperDiagNoConstraint") : $upperDiagNoConstraint}; declare variable $diagramSelection := xquery:eval(file:read-text("D:\\XQuery\\VPExchange Queires\\diagramSelectionCoR.xq", $bindings)); $diagramSelection For completeness, the query being loaded in the $diagramExchange "xquery:eval" is: declare default element namespace "http://www.scfy.com/TTVP_Interface"; declare variable $date := "2017-07-10"; declare variable $daysRun := 3; declare variable $diagrams := diagramExchange/unitDiagramList/unitDiagram[@startdate<=$date][@enddate>=$date][substring(@daysrun,$daysRun,1)="Y"]; declare variable $depotConstraint external := "NIL"; declare variable $resConstraint external := "NIL"; declare variable $lowerDiagNoConstraint external := "NIL"; declare variable $upperDiagNoConstraint external := "NIL"; declare variable $depots := tokenize($depotConstraint," "); declare variable $resources := tokenize($resConstraint," "); declare variable $lowerDiagNo := tokenize($lowerDiagNoConstraint," "); declare variable $upperDiagNo := tokenize($upperDiagNoConstraint," "); declare function local:begin($diagrams, $index) { local:applyDiagStartFilter($diagrams,$index) }; (: Chain of responsibility pattern used here : once one function exits, passes to the next:) declare function local:applyDiagStartFilter($diagrams, $index) { let $diagNo := $lowerDiagNo[$index] return if($diagNo = "NIL") then (local:applyDiagEndFilter($diagrams, $index)) else (local:applyDiagEndFilter($diagrams[@id >= $diagNo cast as xs:integer], $index)) }; declare function local:applyDiagEndFilter($diagrams, $index) { let $diagNo := $upperDiagNo[$index] return if($diagNo= "NIL") then (local:applyResFilter($diagrams, $index)) else (local:applyResFilter($diagrams[@id <= $diagNo cast as xs:integer], $index)) }; declare function local:applyResFilter($diagrams, $index) { let $res := $resources[$index] return if($res = "NIL") then (local:applyDepotFilter($diagrams, $index)) else (local:applyDepotFilter($diagrams[res/@id = $res], $index)) }; declare function local:applyDepotFilter($diagrams, $index) { let $depot := $depots[$index] return if($depot = "NIL") then ($diagrams) else ($diagrams[depot/@id = $depot]) }; for $depot in $depots count $count return local:begin($diagrams, $count)

2 1

Re: [basex-talk] Re : FLWOR expression order is lost
by Christian Grün 20 Oct '17

20 Oct '17

> So I should not rely on the order of a sequence (of XML trees) if I do > a path expression on it? The simple map operator (!) is always the best choice if you want to preserve the node order. It would have been more consistent to first define a basic map operator in the language, and extend it with distinct document order semantics later on. I assume that hardly anyone sensed 18 years ago (when XPath 1.0 was finalized) that XPath and XQuery would be used for much more complex things than navigating through single XML documents... > Le vendredi 20 octobre 2017 à 19:02 +0200, Christian Grün a écrit : >> Hi Iwan, >> >> The nodes of a path expression will always be returned in document >> order. See the following query as example: >> >> let $xml := <xml><a>2</a><a>1</a></xml> >> return ( >> '[A]', sort($xml/a), >> '[B]', sort($xml/a) / self::node() >> ) >> >> The result in BaseX is: >> >> [A] >> <a>1</a> >> <a>2</a> >> [B] >> <a>2</a> >> <a>1</a> >> >> While the results of the first sort function will be in the order you >> would expect, the nodes of the second result were reordered. >> >> The order of nodes that occur in distinct XML tree (such as in your >> example) is implementation-defined, so it could happen that another >> query processor returns "1" before "2". However, the reason would not >> be that your sorting was preserved. Instead, it would reflect another >> internal ordering of the elements that you assigned to $data. More >> information on the document order can be found in the spec [1]. >> >> You can use the map operator to get rid of the implicit reordering >> step. In addition, it will be more efficient: >> >> local:nodes() ! @a ! data() >> >> Hope this helps, >> Christian >> >> [1] https://www.w3.org/TR/xquery-31/#id-document-order >> >> >> >> > I noticed that the order of my FLWOR expression is not kept when I >> > do an >> > xpath on the node list: >> > >> > > basex 'declare function local:nodes() {let $data := (<test >> > > a="2"/>, <test >> > > a="1"/>) let $nodes := trace(for $n in $data order by >> > > xs:integer($n/@a/data()) return $n) return $nodes }; >> > > local:nodes()/@a/data()' >> > >> > <test a="1"/> >> > >> > <test a="2"/> >> > >> > 2 >> > >> > 1 >> > >> > >> > I would expect it to return it in order. >> > Is it a bug or or some optimization allowed by XQuery? >> > >> > I tried it on the latest snapshot (BaseX90-20171020.102055.zip). >> > >> > Best regards, >> > Iwan

1 0

FLWOR expression order is lost
by Iwan Briquemont 20 Oct '17

20 Oct '17

Hello, I noticed that the order of my FLWOR expression is not kept when I do an xpath on the node list: > basex 'declare function local:nodes() {let $data := (<test a="2"/>, <test a="1"/>) let $nodes := trace(for $n in $data order by xs:integer($n/@a/data()) return $n) return $nodes }; local:nodes()/@a/data()' <test a="1"/> <test a="2"/> 2 1 I would expect it to return it in order. Is it a bug or or some optimization allowed by XQuery? I tried it on the latest snapshot (BaseX90-20171020.102055.zip). Best regards, Iwan

2 2

Performance issues and query optimizer
by France Baril 20 Oct '17

20 Oct '17

Hi, We are working hard right now on performance issues. I just read this about an upcoming release: Enforce Rewritings In various cases, existing index structures will not be utilized by the query optimizer. This is usually the case if the name of the database is not a static string (e.g., because it is bound to a variable or passed on as argument of a function call). Furthermore, several candidates for index rewritings may exist, and the query optimizer may decide for a rewriting that turns out to be suboptimal. I'm not sure I read it properly so I would like to confirm my understanding: In our code, all our db accesses are in a format like this: db:open($lang)/*[@id='...']. Does the fact that the db name is $lang, a variable, mean that we have been working without the optimizer all this time and therefore that this new feature is there for people like us? Thanks! -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

3 5

ZIP module and ISO-8859-1 filenames
by Andre Bovy 20 Oct '17

20 Oct '17

<DEAR>BaseX</DEAR> I am using the ZIP module for different purpose and until now I didn’t have any problem. Yesterday I received a ZIP file generated by an application and it was impossible to open the file in BaseX. I tried to write a little class opening it in Java. Same behaviour. After different tries, I noticed that the filenames were containing special characters encoded in ISO-8859-1 In my class, I used the java.util.zip.ZipFile method with a charset parameter. It works perfectly. Is there a way to do that with the present ZIP module ? If not, could you please modify the ZIP module to allow the process of ZIP file or ZIP entry with encoded filenames different from UTF-8 Thank you in advance André

2 1

TR: Options for creating database...
by Fabrice ETANCHAUD 20 Oct '17

20 Oct '17

De : Fabrice ETANCHAUD Envoyé : vendredi 20 octobre 2017 11:46 À : 'France Baril' Objet : RE: [basex-talk] Options for creating database... The XQuery will succeed if you remove the INDENT option (serialization), and lower case all your option names. You are right, you may not have a memory overflow issue on creation. But I had the problem with db:add with several huge files, before I switched to BXS. Best regards, Fabrice De : France Baril [mailto:france.baril@architextus.com] Envoyé : vendredi 20 octobre 2017 11:36 À : Fabrice ETANCHAUD Cc : BaseX Objet : Re: [basex-talk] Options for creating database... In db:create I don't think the issue is memory. I get: [bxerr:BASX0002] Unknown database option 'UPDINDEX'. My function is: let $options := map:merge(( map:entry("CHOP", false()), map:entry('INDENT', false()), map:entry('STRIPNS', false()), map:entry('INTPARSE', true()), map:entry('DTD', false()), map:entry('XINCLUDE', false()), map:entry('UPDINDEX', true()), map:entry('AUTOOPTIMIZE', true()) )) return ( db:create($db-name, $src-folder, (), $options), db:output(<div>done</div>) ) On Fri, Oct 20, 2017 at 11:29 AM, Fabrice ETANCHAUD <fetanchaud(a)pch.cerfrance.fr<mailto:fetanchaud@pch.cerfrance.fr>> wrote: Bonjour France, Didn’t you find Updindex and autooptimize options in the ‘Options’ panel of the db creation window ? Db:create last parameter is the place to put all your options : http://docs.basex.org/wiki/Database_Module#db:create But the Pending update list may overflow memory during db creation/update. You should definitively have a look at the BaseX Scripts (BXS). You can declare a batch of commands in XML, and ask BaseX to run it : http://docs.basex.org/wiki/Commands#Command_Scripts This is the way to set options before invoking the CREATE-DB command : http://docs.basex.org/wiki/Commands#SET I usually write XQuery to generate a BXS that will do the job. Cordialement, Fabrice CERFrance Poitou-Charentes De : basex-talk-bounces(a)mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de> [mailto:basex-talk-bounces@mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de>] De la part de France Baril Envoyé : vendredi 20 octobre 2017 10:44 À : BaseX Objet : [basex-talk] Options for creating database... Hi, I usually create dbs using the gui. I'm now trying to create asb without it to be able to set the updindex and autooptimize options, which I can't find in the GUI. I couldn't figure how to set them out using db:create either, so I switch to command line. Now I'm feeling dumb, I can't find how to set up parsing/indexing options for the command line. I'm looking at: http://docs.basex.org/wiki/Command-Line_Options. I found -sindent and -wchop only. What am I missing? Here are all the options that I'd want to set: let $options := map:merge(( map:entry("CHOP", false()), map:entry('INDENT', false()), map:entry('STRIPNS', false()), map:entry('INTPARSE', true()), map:entry('DTD', false()), map:entry('XINCLUDE', false()), map:entry('UPDINDEX', true()), map:entry('AUTOOPTIMIZE', true()) )) I don't want to set UPDINDEX and AUTOOPTIMIZE in .basex because I only want them to be true() on one of my DBs. Other DBs should remain as is. -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com<mailto:france.baril@architextus.com> -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com<mailto:france.baril@architextus.com>

2 3

Options for creating database...
by France Baril 20 Oct '17

20 Oct '17

Hi, I usually create dbs using the gui. I'm now trying to create asb without it to be able to set the updindex and autooptimize options, which I can't find in the GUI. I couldn't figure how to set them out using db:create either, so I switch to command line. Now I'm feeling dumb, I can't find how to set up parsing/indexing options for the command line. I'm looking at: http://docs.basex.org/wiki/Command-Line_Options. I found -sindent and -wchop only. What am I missing? Here are all the options that I'd want to set: *let **$options* := *map:merge*(( *map:entry*("CHOP", *false*()), *map:entry*('INDENT', *false*()), *map:entry*('STRIPNS', *false*()), *map:entry*('INTPARSE', *true*()), *map:entry*('DTD', *false*()), *map:entry*('XINCLUDE', *false*()), *map:entry*('UPDINDEX', *true*()), *map:entry*('AUTOOPTIMIZE', *true*()) )) I don't want to set UPDINDEX and AUTOOPTIMIZE in .basex because I only want them to be true() on one of my DBs. Other DBs should remain as is. -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

Jobs, locking and doing large updates
by Omar Siam 18 Oct '17

18 Oct '17

Hi list! I was experimenting with the jobs module for the last weeks to speed up updates and to make them fit into less than 6 GB of memory. It does not work the way I expected. * Updating jobs don't seem to run in parallel even if they don't work with and lock the same database. Or do they start much later than I expected? * It seems to me there is an upper limit of jobs that can be queued (about 100?). I automatically started all of the update jobs but some updates did not run. The upside is: I can run my updates with just about 1 GB of memory which is much better for me. I work with dictionary like XML documents. They look like <root> <entry>Contents with further tags</entry> <entry>Contents with further tags</entry> <entry>Contents with further tags</entry> ... a few thousand more ... <entry>Contents with further tags</entry> <entry>Contents with further tags</entry> </root> I add or change larger parts of them and I also need to keep track of changes. So in a separate database there are old versions of entries with time stamps (@dt) like <hist> <entry dt="">Contents with further tags</entry> <entrydt="">Contents with further tags</entry> <entrydt="">Contents with further tags</entry> ... a few thousand more ... <entrydt="">Contents with further tags</entry> <entrydt="">Contents with further tags</entry> </hist> I tried to do everything at once and when I need to update most of my entries (about 30000) I exhaust my memory. So I use the jobs module to do this 100 at a time so the updates list does not grow beyond any reasonable size. Having those 100 as one transaction is good enough for my needs. Then I thought maybe I should use jobs to seprate the two tasks. One async job saves a memory (update) copy of the old entry, the other writes the new one. After some transformations to my XQuery BaseX told me they don't lock the same database or global. I expected the jobs running on different databases to work in parallel. They don't. I don't quite understand why. Also I have a much larger dataset split into a number of databases where it would be quite useful to execute updates in parallel. Am I missing something? Is this perhaps the wrong way to tackle this scaling problem? Best regards Omar Siam

2 2

AUTHMETHOD = Custom
by Dave Day 18 Oct '17

18 Oct '17

Greetings list, In the Basex configuration file, I added AUTHMETHOD = Custom After starting the server, the 1st response I get from the server when I connect to it is the realm:nonce values. In the Basex manual, it states for AUTHMETHOD _Specifies the default authentication method, which will be used by the HTTP server for negotiating__ __credentials. Allowed values are Basic, Digest, and Custom:_ _ With custom authentication, the server will not do any authentication._ Is it possible to connect to Basex server without going thru any authentication? Regards, -- Dave Day

3 7

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk October 2017