BaseX-Talk April 2019

basex-talk@mailman.uni-konstanz.de

26 participants
26 discussions

my first baseX db
by Robert Kirkpatrick 18 Apr '19

18 Apr '19

Hello, Novice, newbie request: GUI snapshot: Command: CREATE DB science C:/Users/rober/Google Drive/science/ Error: "Configurations2/accelerator/current.xml" (Line 1): Premature end of file. TIA, Robert.

2 2

BaseX/GUI v9.1.2 memory use
by Rick Graham 18 Apr '19

18 Apr '19

Hello, Thanks again, as always, for a great product. I just installed BaseX v9.1.2 (upgrading from a previous v9.1.2 snapshot), launched the GUI and then got interrupted. When I returned, almost all of the JVM's memory was being used. I hit "GC" several times but it didn't seem to help. I had no database loaded/open. Seems like some memory isn't getting freed properly. Here's my "INFO" General Information: > Version: 9.1.2 > Used Memory: 1593 MB > Global options: > AUTHMETHOD: Basic > CACHETIMEOUT: 3600 > DBPATH: /usr/local/src/basex/data > DEBUG: false > FAIRLOCK: false > HOST: localhost > HTTPLOCAL: false > IGNORECERT: false > IGNOREHOSTNAME: false > KEEPALIVE: 600 > LANG: English > LANGKEYS: false > LOG: true > LOGMSGMAXLEN: 1000 > LOGPATH: .logs > NONPROXYHOSTS: > PARALLEL: 8 > PARSERESTXQ: 3 > PASSWORD: > PORT: 1984 > PROXYHOST: > PROXYPORT: 0 > REPOPATH: /usr/local/src/basex/repo > RESTPATH: > RESTXQPATH: > SERVERHOST: > SERVERPORT: 1984 > STOPPORT: 8985 > TIMEOUT: 30 > USER: > WEBPATH: /usr/local/src/basex/webapp > Local options > ADDARCHIVES: true > ADDCACHE: false > ADDRAW: false > ARCHIVENAME: false > ATTRINCLUDE: > ATTRINDEX: true > AUTOFLUSH: true > AUTOOPTIMIZE: false > BINDINGS: > CASESENS: false > CATFILE: > CHECKSTRINGS: true > CHOP: true > COMPPLAN: true > COPYNODE: true > CREATEFILTER: *.xml > CREATEONLY: false > CSVPARSER: > DEFAULTDB: false > DIACRITICS: false > DOTCOMPACT: false > DOTPLAN: false > DTD: false > ENFORCEINDEX: false > EXPORTER: > FORCECREATE: false > FTINCLUDE: > FTINDEX: false > HTMLPARSER: > INLINELIMIT: 100 > INTPARSE: false > JSONPARSER: > LANGUAGE: en > LSERROR: 0 > MAINMEM: false > MAXCATS: 100 > MAXLEN: 96 > MAXSTAT: 30 > MIXUPDATES: false > PARSER: xml > QUERYINFO: true > RUNQUERY: true > RUNS: 1 > SERIALIZE: true > SERIALIZER: > SKIPCORRUPT: false > SPLITSIZE: 0 > STEMMING: false > STOPWORDS: > STRIPNS: false > TAILCALLS: 256 > TEXTINCLUDE: > TEXTINDEX: true > TEXTPARSER: > TOKENINCLUDE: > TOKENINDEX: false > UPDINDEX: false > WRITEBACK: false > XINCLUDE: true > XMLPLAN: true Best regards, Richard

6 15

name condition in predicates
by Alexander Witzigmann 18 Apr '19

18 Apr '19

the following query > for $hit in //*[./name() ="p"] > > return > <out >{$hit}</out> leads to > Improper use? Potential bug? Your feedback is welcome: > Contact: basex-talk(a)mailman.uni-konstanz.de > Version: BaseX 9.2 > Java: Oracle Corporation, 10 > OS: Windows 7, amd64 > Stack Trace: > java.lang.NullPointerException > at org.basex.query.expr.SimpleMap.get(SimpleMap.java:43) > at org.basex.query.expr.path.Path.optimize(Path.java:202) > at org.basex.query.expr.path.Path.compile(Path.java:153) > at org.basex.query.expr.Arr.compile(Arr.java:42) > at org.basex.query.expr.Preds.compile(Preds.java:46) > at org.basex.query.expr.path.Step.compile(Step.java:95) > at org.basex.query.expr.path.Path.compile(Path.java:142) > at org.basex.query.expr.gflwor.ForLet.compile(ForLet.java:43) > at org.basex.query.expr.gflwor.GFLWOR.compile(GFLWOR.java:88) > at org.basex.query.scope.MainModule.comp(MainModule.java:81) > at org.basex.query.QueryCompiler.compile(QueryCompiler.java:114) > at org.basex.query.QueryCompiler.compile(QueryCompiler.java:105) > at org.basex.query.QueryContext.compile(QueryContext.java:312) > at org.basex.query.QueryProcessor.compile(QueryProcessor.java:79) > at org.basex.core.cmd.AQuery.query(AQuery.java:93) > at org.basex.core.cmd.XQuery.run(XQuery.java:22) > at org.basex.core.Command.run(Command.java:257) > at org.basex.core.Command.execute(Command.java:93) > at org.basex.gui.GUI.exec(GUI.java:422) > at org.basex.gui.GUI.lambda$execute$4(GUI.java:365) > at java.base/java.lang.Thread.run(Unknown Source) > Compiling: > - pre-evaluate root to document-node() sequence: root() -> > (db:open-pre("publication-toc", 0), ...) > Optimized Query: > for $hit_0 in (db:open-pre("publication-toc", 0), > ...)/descendant-or-self::node()/*[(name() = "p")] return element out { > ($hit_0) } > Query: > for $hit in //*[./name() ="p"] return <out >{$hit}</out> might be some side effect from latest improvements? Alex

2 1

FTP client module
by Fabrice ETANCHAUD 17 Apr '19

17 Apr '19

Hi all ! I am happy to get back to basics, after a lot of SQL ;-) On november 2017, Christian proposed to broadcast a FTP client module to the list, I must have missed the link, I cannot find the module. Is it still possible to get it ? Thank you, Best regards, Fabrice

3 2

BaseX 9.2: The Spring Edition
by Christian Grün 17 Apr '19

17 Apr '19

Dear all, We are more than pleased to announce version 9.2 of BaseX, our XML framework, database system and XQuery 3.1 processor: http://basex.org/ This time, we have put a special focus on performance improvements: XQUERY PERFORMANCE - comparisons will more often be rewritten to hash joins - optimized access to subsequences and single sequence items - consistent propagation of static types to evaluation results - improved static typing of function items (maps, arrays) - more aggressive pre-evaluation at compile time STORAGE PERFORMANCE - large databases will be opened faster - regular storage offset will be computed at runtime GENERAL XQUERY FEATURES - no need to import basex-api modules anymore! - Archive Module: archive:create-from, new options - Database Module: directory-based access via db:dir - Profiling Module: new functions (prof:gc, prof:runtime) - Validation Module: assign XSD validation features - WebSocket Module: query scheduling via ws:eval - Utility Module: various new helper functions - XSLT Module: support for XML Catalogs (thank you Liam) NEW OPTIONS - RESTXQERRORS: Suppress parsing errors at runtime - FULLPLAN: comprehensive query plan information FULLTEXT FEATURES - stemmer for Arabic texts added USER INTERFACES - GUI: better support for latest JDK versions (incl. OpenJDK) - DBA: revised search features in log panel For a more comprehensive list of added and updated features. please check out our documentation: http://docs.basex.org/. We are looking forward to your feedback. Have fun with the new release, Your BaseX Team

5 5

Func def & performance: element()* vs item()*
by Chuck Bearden 12 Apr '19

12 Apr '19

BaseX is a great tool for analyzing & characterizing large amounts of XML data. I have used it both at work and on personal projects. I hope the following observation is useful. When I define a function that recurs over a sequence of elements in order to build a map of element name counts, I find that when I specify the type of the element sequence as 'element()*', the function runs so slowly that I give up after 5 minutes or so. But when I specify the type as 'item()*', it finishes in 40 seconds or less. Here's an example: -----begin code snippet----- declare namespace local="w00fw00f"; declare function local:count($elems as element()*, $elem_counts as map(*)) as map(*) { let $elem := head($elems), $elem_name := $elem/name(), $elems_new := tail($elems), $elem_name_count := if (map:contains($elem_counts, $elem_name)) then map:get($elem_counts, $elem_name) + 1 else 1, $elem_counts_new := map:put($elem_counts, $elem_name, $elem_name_count) return if (count($elems_new) = 0) then $elem_counts_new else local:count($elems_new, $elem_counts_new) }; let $coll := collection('pure_20190402'), $elems := $coll/result/items/*, $elem_names_map := local:count($elems, map {}) return json:serialize($elem_names_map, map {'format' : 'xquery'}) -----end code snippet----- In the function declaration, changing "$elems as element()*" to "$elems as item()*" makes the difference in performance. Replacing the JSON serialization with a standard XML one does not change the performance. I am running BaseX 9.1.2 under Ubuntu 16.04.6. All the best, Chuck Bearden

4 6

Nooby/#CitizenScientist REQUEST for HELP: Python implementation of this XQuery
by Jim Salmons 11 Apr '19

11 Apr '19

To the BaseX Community, I am a 68-year-old #PayItForward cancer-survivor independent #CitizenScientist doing applied research in #DigitalHumanities and #MachineLearning and I need your help, please. And please forgive me for posting such a TL;DR post. I am not lazy nor a dilettante, I am simply under tight time pressure to get some development done on my Python-based metadata discovery and curation toolkit which my fellow cancer-surviving wife, Timlynn, and I will be showcasing via our poster presentation accepted for next month's #DATeCH2019 conference in Brussels (http://datech.digitisation.eu/). Our poster is entitled "#MAGAZINEgts and #dhSegment: Using a Metamodel Subgraph to Generate Synthetic Data of Under-Sampled Complex Document Structures for Machine-Learning" (ResearchGate preprint: https://is.gd/factminers_datech2019_poster). #MAGAZINEgts is the XML-based ground-truth storage format Timlynn and I are developing based on an ontological "stack" of #cidocCRM/FRBRoo/PRESSoo using a metamodel subgraph design pattern. The goal of our design is to support an integrated complex document structure and content depiction model for digital collections of print magazines and newspapers. (For more, see our #DATeCH2017 poster: https://is.gd/factminers_datech2017_poster) We are evolving a reference implementation of the #MAGAZINEgts format for the collection of Softalk magazine at the Internet Archive. The collection is here: https://archive.org/details/softalkapple?&sort=date, and the MAGAZINEgts file (~10+ MB) is linked from the About page of the collection but is provided here as a shortened link: https://is.gd/softalk_magazinegts_xml_file. MY IMMEDIATE GOAL: Rather than keep with the awkward workflow of generating intermediary JSON metadata files and, in batches, converting to XML and copy-pasting into appropriate positions in the master publication file, I want to incorporate direct incremental updating of fine-grained #MAGAZINEgts metamodels, metadata, and their associated source-document-specific datasets via integrating BaseX into the FactMiners Toolkit (fmtk). We would _really_ like to be showing this significant enhancement of our toolkit at the DATeCH conference (8-10 May). MY CURRENT CONTEXT: I have the latest BaseX installed, working well, and I have done as much "fast track" learning as I can to come up to "toddler" speed on BaseX and its Python-based API. I have the Python API extension installed and working within my PyCharm IDE, and I am on Windows 10. MY CURRENT NEED: I have used the BaseX GUI to develop a sample XQuery for updating/adding a machine-learning data spec for curating the bounding boxes of advertisements on a page within a magazine. The query below is not parameterized for programmatic dynamic execution. It is simply a hard-coded test of my evolving understanding of doing BaseX interactions. So the dataset name ("all_ads"), the issue-page filename (softalk_v2n02pg002.png) and the various dimension numbers, etc. are explicit rather than variable, etc., within this sample query. When I run this query and do an export of the MAGgts master file, the update is there and looks great. Even though my knowledge and skills with BaseX are small but growing, I feel I have enough grip on things to forge ahead to at least get BaseX integrated for the #ML image training dataset feature that we will showcase at #DATeCH2019. (BTW, I stripped the #MAGgts schema's XML namespace during BaseX database creation to make things easier during learning. I expect to simply restore it in the header after exporting and before uploading a new release of Softalk's reference implementation of this ground-truth format. Either that, or I will tweak the eventual Python-implementation of the queries to include the namespace and just leave it intact when importing into the local BaseX database.) HERE IS THE SAMPLE QUERY: === declare option db:WRITEBACK 'true'; declare variable $new_spec := <ML_training_img_spec file_name="softalk_v2n02pg002.png"> <ML_image_dim width="940" height="1280"/> <ML_label_bbox label="ad" status="predicted" left="500" top="680" width="444" height="580"/> <ML_label_bbox label="ad" status="actual" left="490" top="620" width="440" height="575"/> </ML_training_img_spec>; update:output("Update successful."), insert node $new_spec as last into doc("MAGgts") //Metadata//ML_maxpixel_datasets[@max_pixels = "1000000"] /ML_dataset[@name="all_ads"]//ML_training_img_specs === MY REQUEST FOR ASSISTANCE: It would be _extraordinarily_ helpful, and Timlynn and I would be most grateful, if someone within the BaseX community that has familiarity with doing Python-based BaseX integration could provide a brief implementation -- similar to the examples supplied in the Client integration samples -- that would show me how to take a BaseX GUI-developed query and convert it to a usable state in a Python program. BY WAY OF THANK YOU: If anyone can help us short-circuit my path to integrating BaseX into our #DATeCH2019 poster presentation, we will gladly cite you and your assistance in the acknowledgements of the poster and its 2-page companion handout. Again, I am sorry for hitting folks with such a long and detailed request for nooby assistance. In advance, thank you for any help you Good Folks may provide. I look forward to significantly improving the functionality of the FactMiners Toolkit by incorporating BaseX into its core platform. Happy-Healthy Vibes from Colorado USA, -: Jim Salmons :- https://www.researchgate.net/profile/Jim_Salmons

2 3

Add & replace documents
by Johannes Bauer 11 Apr '19

11 Apr '19

Hi, this may seem like a dumb question, but how can I add & replace a number of documents in a database? I have a directory with some files that I want to import. Some of the documents may already exist in the database. If I use the db:replace('db', '', '/path/to/my/files') function it adds the missing docs and overwrites the existing ones. But it also deletes all other documents from the database. I cannot work with db:add() because it does not overwrite existing docs. Any suggestions? Best regards Johannes

2 4

concurrent performance issues?
by Michael Perkonigg 08 Apr '19

08 Apr '19

Hallo, ich versuche gerade, BaseX hier für eine zentrale XML-DB zu evaluieren. Die DB ist read-only. Wir haben also keinerlei Updates. Wir haben verschiedene XQuery-Queries(?), die abgesetzt werden, dabei ist aufgefallen, dass wenn ich mehrere Queries parallel starte die Antwortzeiten unverhätnismässig steigen. Einzeln: Query1: 500ms Query2: 1200ms Parallel brauchen beide zusammen dann zwischen 3 und 5 Sekunden. Erstaunlicherweise sind die Antwortzeiten mit der Option PARALLEL=1 am geringsten, d.h. bei der Defaulteinstellung PARALLEL=8 (und auch schon bei PARALLEL=2) steigen die Antwortzeiten. Ich frage mich nun, ob BaseX als Datenbank für eine verteilte Applikation mit zentraler DB überhaupt geeignet ist und ob ich vielleicht mit irgendwelchen Optionen BaseX für meine Zwecke optimieren kann. Ganz theoretisch könnte die DB ja mit hundert parallelen Anfragen belastet werden. -- mit freundlichen Grüßen

3 2

Two questions on use of the serialize function and character maps
by Martin Honnen 08 Apr '19

08 Apr '19

I am struggling to use the "serialize" XPath 3.1 function as specified by the W3C with BaseX 9.1.2: 1) Is there any way to provide the "use-character-maps" parameter supplied the XPath 3.1 way with a second argument to "serialize" as a map: serialize("a,b", map { 'method' : 'text', 'use-character-maps' : map { "'" : "&apos;" }}) I get an error saying [SEPM0017] Invalid 'use-character-maps' value 'map { "'": "&apos;" }'; must be a string. 2) When I try the XPath 3.0 way of providing the serialization parameters as an XML element I also get an error: When I run declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method "text"; serialize( ( <text>'</text>, "'", map { "Ä":"\'C4", "Ö":"\'D6", "Ü":"\'DC", "ß":"\'DF", "ä":"\'E4", "ö":"\'F6", "ü":"\'FC" } ), <output:serialization-parameters xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization"> <output:method value="adaptive"/> <output:use-character-maps> <output:character-map character="'" map-string="&apos;"/> </output:use-character-maps> </output:serialization-parameters> ) in BaseX it tells me [SEPM0017] Character map 'character-map=,' is not defined. Saxon 9.9 and Altova 2018 run that example without giving an error (although not with the same serialization result, but that is a different issue).

2 7

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk April 2019