BaseX-Talk

basex-talk@mailman.uni-konstanz.de

9 participants
5256 discussions

BaseX 9.2: The Spring Edition
by Christian Grün 17 Apr '19

17 Apr '19

Dear all, We are more than pleased to announce version 9.2 of BaseX, our XML framework, database system and XQuery 3.1 processor: http://basex.org/ This time, we have put a special focus on performance improvements: XQUERY PERFORMANCE - comparisons will more often be rewritten to hash joins - optimized access to subsequences and single sequence items - consistent propagation of static types to evaluation results - improved static typing of function items (maps, arrays) - more aggressive pre-evaluation at compile time STORAGE PERFORMANCE - large databases will be opened faster - regular storage offset will be computed at runtime GENERAL XQUERY FEATURES - no need to import basex-api modules anymore! - Archive Module: archive:create-from, new options - Database Module: directory-based access via db:dir - Profiling Module: new functions (prof:gc, prof:runtime) - Validation Module: assign XSD validation features - WebSocket Module: query scheduling via ws:eval - Utility Module: various new helper functions - XSLT Module: support for XML Catalogs (thank you Liam) NEW OPTIONS - RESTXQERRORS: Suppress parsing errors at runtime - FULLPLAN: comprehensive query plan information FULLTEXT FEATURES - stemmer for Arabic texts added USER INTERFACES - GUI: better support for latest JDK versions (incl. OpenJDK) - DBA: revised search features in log panel For a more comprehensive list of added and updated features. please check out our documentation: http://docs.basex.org/. We are looking forward to your feedback. Have fun with the new release, Your BaseX Team

5 5

Func def & performance: element()* vs item()*
by Chuck Bearden 12 Apr '19

12 Apr '19

BaseX is a great tool for analyzing & characterizing large amounts of XML data. I have used it both at work and on personal projects. I hope the following observation is useful. When I define a function that recurs over a sequence of elements in order to build a map of element name counts, I find that when I specify the type of the element sequence as 'element()*', the function runs so slowly that I give up after 5 minutes or so. But when I specify the type as 'item()*', it finishes in 40 seconds or less. Here's an example: -----begin code snippet----- declare namespace local="w00fw00f"; declare function local:count($elems as element()*, $elem_counts as map(*)) as map(*) { let $elem := head($elems), $elem_name := $elem/name(), $elems_new := tail($elems), $elem_name_count := if (map:contains($elem_counts, $elem_name)) then map:get($elem_counts, $elem_name) + 1 else 1, $elem_counts_new := map:put($elem_counts, $elem_name, $elem_name_count) return if (count($elems_new) = 0) then $elem_counts_new else local:count($elems_new, $elem_counts_new) }; let $coll := collection('pure_20190402'), $elems := $coll/result/items/*, $elem_names_map := local:count($elems, map {}) return json:serialize($elem_names_map, map {'format' : 'xquery'}) -----end code snippet----- In the function declaration, changing "$elems as element()*" to "$elems as item()*" makes the difference in performance. Replacing the JSON serialization with a standard XML one does not change the performance. I am running BaseX 9.1.2 under Ubuntu 16.04.6. All the best, Chuck Bearden

4 6

Nooby/#CitizenScientist REQUEST for HELP: Python implementation of this XQuery
by Jim Salmons 11 Apr '19

11 Apr '19

To the BaseX Community, I am a 68-year-old #PayItForward cancer-survivor independent #CitizenScientist doing applied research in #DigitalHumanities and #MachineLearning and I need your help, please. And please forgive me for posting such a TL;DR post. I am not lazy nor a dilettante, I am simply under tight time pressure to get some development done on my Python-based metadata discovery and curation toolkit which my fellow cancer-surviving wife, Timlynn, and I will be showcasing via our poster presentation accepted for next month's #DATeCH2019 conference in Brussels (http://datech.digitisation.eu/). Our poster is entitled "#MAGAZINEgts and #dhSegment: Using a Metamodel Subgraph to Generate Synthetic Data of Under-Sampled Complex Document Structures for Machine-Learning" (ResearchGate preprint: https://is.gd/factminers_datech2019_poster). #MAGAZINEgts is the XML-based ground-truth storage format Timlynn and I are developing based on an ontological "stack" of #cidocCRM/FRBRoo/PRESSoo using a metamodel subgraph design pattern. The goal of our design is to support an integrated complex document structure and content depiction model for digital collections of print magazines and newspapers. (For more, see our #DATeCH2017 poster: https://is.gd/factminers_datech2017_poster) We are evolving a reference implementation of the #MAGAZINEgts format for the collection of Softalk magazine at the Internet Archive. The collection is here: https://archive.org/details/softalkapple?&sort=date, and the MAGAZINEgts file (~10+ MB) is linked from the About page of the collection but is provided here as a shortened link: https://is.gd/softalk_magazinegts_xml_file. MY IMMEDIATE GOAL: Rather than keep with the awkward workflow of generating intermediary JSON metadata files and, in batches, converting to XML and copy-pasting into appropriate positions in the master publication file, I want to incorporate direct incremental updating of fine-grained #MAGAZINEgts metamodels, metadata, and their associated source-document-specific datasets via integrating BaseX into the FactMiners Toolkit (fmtk). We would _really_ like to be showing this significant enhancement of our toolkit at the DATeCH conference (8-10 May). MY CURRENT CONTEXT: I have the latest BaseX installed, working well, and I have done as much "fast track" learning as I can to come up to "toddler" speed on BaseX and its Python-based API. I have the Python API extension installed and working within my PyCharm IDE, and I am on Windows 10. MY CURRENT NEED: I have used the BaseX GUI to develop a sample XQuery for updating/adding a machine-learning data spec for curating the bounding boxes of advertisements on a page within a magazine. The query below is not parameterized for programmatic dynamic execution. It is simply a hard-coded test of my evolving understanding of doing BaseX interactions. So the dataset name ("all_ads"), the issue-page filename (softalk_v2n02pg002.png) and the various dimension numbers, etc. are explicit rather than variable, etc., within this sample query. When I run this query and do an export of the MAGgts master file, the update is there and looks great. Even though my knowledge and skills with BaseX are small but growing, I feel I have enough grip on things to forge ahead to at least get BaseX integrated for the #ML image training dataset feature that we will showcase at #DATeCH2019. (BTW, I stripped the #MAGgts schema's XML namespace during BaseX database creation to make things easier during learning. I expect to simply restore it in the header after exporting and before uploading a new release of Softalk's reference implementation of this ground-truth format. Either that, or I will tweak the eventual Python-implementation of the queries to include the namespace and just leave it intact when importing into the local BaseX database.) HERE IS THE SAMPLE QUERY: === declare option db:WRITEBACK 'true'; declare variable $new_spec := <ML_training_img_spec file_name="softalk_v2n02pg002.png"> <ML_image_dim width="940" height="1280"/> <ML_label_bbox label="ad" status="predicted" left="500" top="680" width="444" height="580"/> <ML_label_bbox label="ad" status="actual" left="490" top="620" width="440" height="575"/> </ML_training_img_spec>; update:output("Update successful."), insert node $new_spec as last into doc("MAGgts") //Metadata//ML_maxpixel_datasets[@max_pixels = "1000000"] /ML_dataset[@name="all_ads"]//ML_training_img_specs === MY REQUEST FOR ASSISTANCE: It would be _extraordinarily_ helpful, and Timlynn and I would be most grateful, if someone within the BaseX community that has familiarity with doing Python-based BaseX integration could provide a brief implementation -- similar to the examples supplied in the Client integration samples -- that would show me how to take a BaseX GUI-developed query and convert it to a usable state in a Python program. BY WAY OF THANK YOU: If anyone can help us short-circuit my path to integrating BaseX into our #DATeCH2019 poster presentation, we will gladly cite you and your assistance in the acknowledgements of the poster and its 2-page companion handout. Again, I am sorry for hitting folks with such a long and detailed request for nooby assistance. In advance, thank you for any help you Good Folks may provide. I look forward to significantly improving the functionality of the FactMiners Toolkit by incorporating BaseX into its core platform. Happy-Healthy Vibes from Colorado USA, -: Jim Salmons :- https://www.researchgate.net/profile/Jim_Salmons

2 3

Add & replace documents
by Johannes Bauer 11 Apr '19

11 Apr '19

Hi, this may seem like a dumb question, but how can I add & replace a number of documents in a database? I have a directory with some files that I want to import. Some of the documents may already exist in the database. If I use the db:replace('db', '', '/path/to/my/files') function it adds the missing docs and overwrites the existing ones. But it also deletes all other documents from the database. I cannot work with db:add() because it does not overwrite existing docs. Any suggestions? Best regards Johannes

2 4

concurrent performance issues?
by Michael Perkonigg 08 Apr '19

08 Apr '19

Hallo, ich versuche gerade, BaseX hier für eine zentrale XML-DB zu evaluieren. Die DB ist read-only. Wir haben also keinerlei Updates. Wir haben verschiedene XQuery-Queries(?), die abgesetzt werden, dabei ist aufgefallen, dass wenn ich mehrere Queries parallel starte die Antwortzeiten unverhätnismässig steigen. Einzeln: Query1: 500ms Query2: 1200ms Parallel brauchen beide zusammen dann zwischen 3 und 5 Sekunden. Erstaunlicherweise sind die Antwortzeiten mit der Option PARALLEL=1 am geringsten, d.h. bei der Defaulteinstellung PARALLEL=8 (und auch schon bei PARALLEL=2) steigen die Antwortzeiten. Ich frage mich nun, ob BaseX als Datenbank für eine verteilte Applikation mit zentraler DB überhaupt geeignet ist und ob ich vielleicht mit irgendwelchen Optionen BaseX für meine Zwecke optimieren kann. Ganz theoretisch könnte die DB ja mit hundert parallelen Anfragen belastet werden. -- mit freundlichen Grüßen

3 2

Two questions on use of the serialize function and character maps
by Martin Honnen 08 Apr '19

08 Apr '19

I am struggling to use the "serialize" XPath 3.1 function as specified by the W3C with BaseX 9.1.2: 1) Is there any way to provide the "use-character-maps" parameter supplied the XPath 3.1 way with a second argument to "serialize" as a map: serialize("a,b", map { 'method' : 'text', 'use-character-maps' : map { "'" : "&apos;" }}) I get an error saying [SEPM0017] Invalid 'use-character-maps' value 'map { "'": "&apos;" }'; must be a string. 2) When I try the XPath 3.0 way of providing the serialization parameters as an XML element I also get an error: When I run declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method "text"; serialize( ( <text>'</text>, "'", map { "Ä":"\'C4", "Ö":"\'D6", "Ü":"\'DC", "ß":"\'DF", "ä":"\'E4", "ö":"\'F6", "ü":"\'FC" } ), <output:serialization-parameters xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization"> <output:method value="adaptive"/> <output:use-character-maps> <output:character-map character="'" map-string="&apos;"/> </output:use-character-maps> </output:serialization-parameters> ) in BaseX it tells me [SEPM0017] Character map 'character-map=,' is not defined. Saxon 9.9 and Altova 2018 run that example without giving an error (although not with the same serialization result, but that is a different issue).

2 7

BigInteger
by Andre Bovy 05 Apr '19

05 Apr '19

Hi Chistian, I am trying to use a very Big Integer in BaseX (9.1.2) With this value : declare namespace BigInteger = "java:java.math.BigInteger"; BigInteger:new("12345678901234567890”) I received : Cannot convert xs:string to xs:integer: "12345678901234567890”. The java method that I used returns a BigInteger, but I just need the value as a String. I tried : BigInteger:toString(BigInteger:new("12345678901234567890”)) without success (same error) Is there a workaround to this behaviour ? without writing a Java class. Thank you in advance André

3 5

jobs:eval as a service, "no variables allowed"
by Zimmel, Daniel 03 Apr '19

03 Apr '19

Hi, when I try to register jobs:eval as a service, BaseX 9.2 says: "no variables allowed". Is it not possible to register a service with a variable? Not working: jobs:eval("declare variable $xml external; $xml", map { 'xml': 'XML' }, map { 'id':'TEST', 'start':'11:31:50', 'interval':'P1D', 'service': true() }) working: jobs:eval("declare variable $xml external; $xml", map { 'xml': 'XML' }, map { 'id':'TEST', 'start':'11:31:50', 'interval':'P1D', 'service': false() }) http://docs.basex.org/wiki/Jobs_Module#jobs:eval Thanks, Daniel

2 2

Tumbling window
by Giuseppe G. A. Celano 03 Apr '19

03 Apr '19

I have written the following code: for tumbling window $s in ("this", "is", "an", "example", "." , "this", "is", "another", "[", "example", ".", "]", "Another", "example", ".") start $a when fn:true() end $b next $r when ($b = "." and $r = "]") or $b = "." return <s>{$s}</s> which returns: <s>this is an example .</s> <s>this is another [ example .</s> <s>] Another example .</s> but I am looking for <s>this is an example .</s> <s>this is another [ example . ]</s> <s>Another example .</s> I am playing around with if-clauses, but I am wondering whether there is a more direct way to force the end of a sentence at "]" when it is preceded by "." or at "." when there is no following "]". Thanks. Ciao, Giuseppe

2 2

Hitting the Tail Recursion wall
by Andreas Mixich 02 Apr '19

02 Apr '19

Hi, I am trying to implement the algorithm outlined at https://tools.ietf.org/html/rfc3986#section-5.2.4 (Two other implementations are at [1] and [2]) The following function is my work in progress. It seems, all goes fine, till the last `else`, which is commented with `(: E :)`. I hit the tail recursion barrier and I do not see why. The function may not (yet) make sense, since I am still experimenting. `local:remove-dot-segments("/a/b/c/./../../g",[])` ``` (:~ Remove dot segments from a URI path component according to RFC 3986 Section 5.2.4 @param $path the path component of an URL @param $out an empty (!) array @return the path component with the relative part (dot segments) removed and resolved to the absolute path. @see https://tools.ietf.org/html/rfc3986#section-5.2.4 :) declare function local:remove-dot-segments( $path as xs:string, $out as array(*)) as item()* { concat("in: ", $path, " out: ", string-join(array:flatten($out))), if (not(contains($path, "."))) then $path else if (string-length($path) > 0) then (: A :) if (substring($path,1,3) = "../") then local:remove-dot-segments(substring($path, 4), $out) else if (substring($path, 1, 2) = "./") then local:remove-dot-segments(substring($path, 3), $out) else (: B :) if (substring($path,1,3) = "/./") then local:remove-dot-segments(concat("/", substring($path, 4)), $out) else if (substring($path, 1, 2) = "/.") then local:remove-dot-segments(concat("/", substring($path, 3)), $out) else (: C :) if (substring($path, 1, 4) = "/../") then let $ret := array:remove($out, array:size($out)) return local:remove-dot-segments(concat("/", substring($path, 5)), $ret) else if (substring($path, 1, 3) = "/..") then let $ret := array:remove($out, array:size($out)) return local:remove-dot-segments(concat("/", substring($path, 4)), $ret) else (: D :) if ($path = ".." or $path = ".") then local:remove-dot-segments(concat(substring((), 4), ""), $out) else (: E :) let $ret := if (starts-with($path, "/")) then array:append($out, concat("/", substring-before(substring($path,2), "/"))) else array:append($out, substring-before(substring($path,1), "/")) return local:remove-dot-segments($path, $ret) else $path }; ``` Sorry for throwing such a big function at you, but I do not find the place, where the stack overflow gets triggered. It does not happen, if I return an empty sequence at `else (: E :)`. If I replace that part (after `else (: E :)`) with `local:remove-dot-segments($path, $out)` I get the stack overflow as well, which is, what confuses me the most. [1]https://github.com/ariutta/remove-dot-segments/blob/master/index.js [2] https://gist.github.com/rdlowrey/5f56cc540099de9d5006 -- Goody Bye, Minden jót, Mit freundlichen Grüßen, Andreas Mixich

2 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk