BaseX-Talk

basex-talk@mailman.uni-konstanz.de

13 participants
5244 discussions

Happy New Year! -- A #DigitialHumanities/#MachineLearning #CitizenScientist needs you help... :-)
by Jim Salmons 04 Jan '20

04 Jan '20

Hello Christian & the BaseX Community, Happy New Year and Decade! I will get to my specific BaseX/XQuery question below, but first some long but mostly necessary context... I am a 69-year-old post-cancer #PayItForward indie #CitizenScientist working at the intersection of #DigitalHumanities and #AI/#MachineLearning. I am developing the #MAGAZINEgts ground-truth storage format for serial publications (primarily print magazines and newspapers). My format is based on a ontological stack of CIDOC-CRM, FRBRoo, and PRESSoo, and utilizes a metamodel subgraph design pattern to support integrated complex document structure and content depiction models. In addition, my fellow cancer-surviving wife and I are developing a reference implementation of the #MAGAZINEgts format for the 48-issue run of Softalk magazine published between 1980 and 1984 during the dawn of the Microcomputer and Digital Age. Post-cancer and with determination to reinvent ourselves as Citizen Scientists in the Digital Humanities, we funded digitization of our magazine collection into the Internet Archive as a gift to each other for our 25th wedding anniversary. Initially, I did all our exploratory #MAGAZINEgts prototyping development in Python using import/export of data via JSON format with one-off scripts to format batches of information into XML format to fold into the published #MAGAZINEgts file for the Softalk magazine reference implementation at the Archive. Then I found BaseX!!! :D Upon discovering BaseX, I began using it for direct #MAGAZINEgts XML access and development. The FactMiners Toolkit is, at this stage, the old pre-BaseX stuff plus the new features related to the generation of balanced training datasets for #MachineLearning model training. I am in the process now of revisiting my early development and refactoring it to incorporate BaseX direct editing. I recently completed the generation of a "seed"/master dataset for all the advertisements in Softalk magazine (https://github.com/SoftalkAppleProject/datasets_ml_all_ads_1M). This seed dataset can be used in conjunction with the PRESSoo Issuing Rules of the Advertising Model in the Metamodel partition of the #MAGAZINEgts file. This "seed" dataset and the metamodel can be used to generate any "slice and dice" dataset for #ML model training that is aware of the interrelationships of advertisements' size, shape, and allowable positions on variously formatted multi-column pages. (If this domain of research interests anyone, please do not hesitate to contact me. I can point you to our #DATeCH papers/posters and Neo4j #GraphGists, etc. for more information. You will also find relevant links to our research on the GitHub repo cited above.) Now on to my specific question/request-for-information... To generate #MachineLearning training datasets, I need to not only slice-and-dice the "actual" data of the advertisement dataset, I also need to incorporate page examples which DO NOT include the document structure being trained to be recognized. In other words, the "non-label" instances of a #ML training dataset are the members of the set-theoretic complement of pages that DO have advertisements. Fortunately and by design, the #MAGAZINEgts format incorporates all the information to do this dataset generation... the problem is that I have only ever known enough XML-related technologies to "git'er done." And even whatever I may have known is long-since gone following my cancer battle. Now, given the OCD-ness of my basic personality structure, I thought it would be best to develop a solution to present before asking for community help... and here I am. The digital collection of Softalk magazine is in the Internet Archive here: https://archive.org/details/softalkapple?sort=date The #MAGAZINEgts file for Softalk magazine is here: https://archive.org/download/softalkapple/softalkapple_publication.xml For simplicity's sake during my exploratory/learning experience, I import the softalkapple_publication.xml file into BaseX with the "strip namespaces" option checked and rename the imported database as 'MAGgts'. My problem is not that I cannot generate the non-label dataset, but rather that this query takes roughly around 30 seconds to execute!? Honestly, the fact that this query works is all I need. But given the world of modern near-instantaneous responses to virtually all data inquiries, I am wondering if there is a far better solution to my problem. (Also note, the somewhat strange format of the returned results -- 'NoAdPg' elements wrapped within a 'Result' element is for the convenience of bringing this data into Python to be parsed by the 'xmltodict' module.) If it turns out that this 30+-second run is, in fact, justified due to the unseen actual computations being done, no problem. An explanation of why that much processing power is needed is one possible response to this inquiry. On the other hand, there is a vast domain of XQuery know-how that I am blind to and would value any explanation and optimization that can be suggested to speed this operation up. Here is my current 30+-second query: ====== (: Round up a list by issue of the leaf images of Softalk magazine that DO NOT have an advertisement on them and that excludes Cover 1, inserts (e.g. 'blow-in' cards), and missing pages... :) (: The softalkapple_publication.xml file from the Internet Archive's Softalk collection is imported as 'MAGgts' into BaseX XML database, with stripped namespaces for interactive development/learning convenience. :) (: This NS declaration is not needed when namespaces are stripped. declare default element namespace "http://www.factminers.org/MAGAZINE/gts/2019-01-18"; :) declare variable $ad_index := doc('MAGgts')//AdIndex/*; (: Gather all issues in the Leaf2ppg map (page image to print page number map) :) declare variable $all_issues := doc('MAGgts')//Metadata//Leaf2ppg_map[1]/*; if (count($all_issues) > 0) then (<Result>{for $issue in $all_issues return (for $leaf_spec in $issue/* let $leaf_id := $leaf_spec/Issue_id let $leaf_pgnum := $leaf_spec/PageNum let $leaf_type := $leaf_spec/PgType where ($leaf_pgnum != 'Cover1' and $leaf_pgnum != 'Insert Stub' and $leaf_pgnum != 'Insert Content' and $leaf_type != 'MissingPgs') return (if (exists($ad_index[PageNum = $leaf_pgnum and Issue_id = $leaf_id])) (: We're not interested in pages w/ ads, only those WITHOUT ads :) then () else <NoAdPg> <LeafNum>{data($leaf_spec/@leafnum)}</LeafNum> {$leaf_spec/(Issue_id | PgType | PageNum) }</NoAdPg> ))}</Result>) else "Nothing found." ====== Again, Happy New Year/Decade. Any explanation or optimization greatly appreciated. Happy-Healthy Vibes from Colorado, -: Jim :-

2 2

Syntax to use profiling functions on updating functions
by France Baril 31 Dec '19

31 Dec '19

Hi, I can't seem to use the profiling functions on updating functions. I was wondering if it's a known limit or if I just need to use a different syntax in this context. For example in an updating function that returns 2 updates, I have something like this declare updating function > translation:import-and-process-returned-translations($package-number as > xs:string) { > .... some code .... > return ( > translation:save-space-issues($package-name, $space-issues) => > prof:time('Translate save space issues: '), > translation:save-source-mismatches($package-name, > $source-mismatches) => prof:time('Translate save source mismatches: '), > translation:save-zhtw-brackets-issues($menu-id, $package-name) => > prof:time('Translate check zhtw brackets, not saved?: ') > ) > }; This works with the prof:time, but if I try to profile, I get: [XUST0001] Expressions must all be updating or return an empty sequence. Thanks! P.S. By the way, I'm very excited about the improvements I am able to implement now that I can use the performance functions in this module. I'm one version late to say this, but awesome addition! -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

Re: [basex-talk] HTTPServer + gzip compression
by Stefan Koch 30 Dec '19

30 Dec '19

3 2

Parsing MusicXML files
by Joe Curtis 29 Dec '19

29 Dec '19

I am trying to solve a problem transferring musical information between two high end scoring programs using the industry standard MusicXML language. Although the majority (say 95%) of information is transferred correctly the remainder does not because the originating program gives a general classification of <words> in the file whilst the receiving program has a more sophisticated breakdown of the information. Rather than classifying it incorrectly it doesn't translate the information at all. The information occurs randomly throughout the score rather than in a regular pattern. What I am trying to do is read the MusicXML file and list the <words> where they occur together with the part identification and measure (bar) number to enable me to add the information manually at a later stage. I have come across BaseX while researching how to parse the files but am struggling to master the logic of xquery to achieve my aim. The structure of the file at my interest level is:-  <measure number="27" width="224"> <direction> <direction-type> <words default-x="9" default-y="20" justify="left" valign="middle" font-family="Times New Roman" font-style="normal" font-size="5.9683" font-weight="normal">tutti</words> and my xquery entries are:- let $score := doc('score')/score-partwise(:/part/measure/direction/direction-type/words:) for $word in ($score//words/text()) let $p := $score/part/@id let $m := $score/part/measure/@number return $word, $p, $m If I return just $word I get the relevant <words>entries but if I add $p and $m I get an error message 'undeclared variable $p' I realise that this is more of an xquery question than BaseX but I thought I would come here first. Any help would be greatly appreciated. Joe Curtis .

2 1

Update to a single document changes @modified-date on all DB documents in db:list-details()
by France Baril 29 Dec '19

29 Dec '19

Hi, I'm running a few tests after upgrading a web application to 9.3.1. I have this issue where I am trying to get the documents modified since a certain date/time. However, even is a single document is modified db:list-details returns the same modified-date for all listed documents. To reproduce; 1. Run list-details() on a DB to see dates. 2. Thought WebDAV, modify one of the document in the DB 3. Run list-details() again on that DB. Expected result: the @modifed-date should be different on the document modified in #2 Actual result: all documents have a new @modified-date. -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

Lost prof:time in terminal after upgrade to 9.3.1
by France Baril 28 Dec '19

28 Dec '19

Hi, After upgrading to 9.3.1, I no longer get my prof:time info in the terminal. DEBUG is set to true in .basex and I don't remember changing any option on in older installations to enable prof:time. Documentation of profiling module doesn't mention any required settings and I'm not seeing a note in the CHANGELOG either. What am I missing? -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 1

Re: [basex-talk] HTTPServer + gzip compression
by Stefan Koch 23 Dec '19

23 Dec '19

1 0

R package
by Ben Engbers 23 Dec '19

23 Dec '19

Hi, Some years ago, I have written a R-client for BaseX. That client used a R-file that could be sourced into your R-code (https://github.com/BaseXdb/basex/tree/master/basex-api/src/main/r/RbaseXCli…). Since not all functions from the server protocol were implemented, I decided to rewrite the client and build a package. This package has now been accepted by Cran (https://cran.r-project.org/web/packages/RBaseX/index.html). I am still working on some improvements but that will take some time. In the meantime, enjoy it. Ben Engbers

1 0

Improper use or potential bug
by France Baril 23 Dec '19

23 Dec '19

This .xq works the first time when returning db:create. However, once the DB are created, if I run it again, files are note replaced. Instead, I get java.lang.ArrayIndexOutOfBoundsException. I can send the DB in private if necessary for debugging. The result is not exactly an index, it's just flattening a few levels of indirection to make future queries faster. I was testing this on 9.2.3. Also a side question, is there a way to rewrite this query so not all DBs are locked? I get lock: global. I would like to lock only DBs that start with 'index-prompt-' I tried hard coding the prefix in the db:create and db:replace fonctions but that didn't help. Since I will have this running on a loop (cannot catch updated on src DB), I don't want to lock the whole system each time this query is running. It does take a good 12 seconds to run. (: USAGE: for $prompt in db:attribute('index-prompt-en-us', 'setup_as_new', 'name')/.. return $prompt => prof:time() :) let $langs := for $lang in db:list()[string-length(.) = 5][substring(., 3, 1) = '-'] return $lang return for $lang in $langs let $index-db := 'index-prompt-' || $lang let $prompt-file-name := 'prompts-' || $lang || '.xml' let $index := <index>{ for $string-group in db:open('global-content')//*[name()='string-group'] let $prompt-this-lang := $string-group/child::*[@xml:lang=$lang][normalize-space(.)!=''] let $prompt-base := if (exists($prompt-this-lang)) then $prompt-this-lang else $string-group/child::*[@xml:lang='en-us'] let $p2 := db:attribute('global-content', $string-group/data(@key), 'key')/..[name()='prompt'][ancestor::*[last()]/name()='prompt-refs'][@package=$string-group/data(@package)][@pkg-version=$string-group/data(@pkg-version)] let $prompts := ( for $prompt-match in $p2 return copy $copy := $prompt-base modify ( for $attr in $prompt-match/@*[name()!='package' and name()!='pkg-version' and name()!='key'] return insert node $attr into $copy ) return $copy ) return $prompts }</index> return if (db:exists($index-db)) then db:replace($index-db, $prompt-file-name, $index) else db:create($index-db, $index, $prompt-file-name , map { 'updindex': true() }) -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

UPDINDEX on custom index structures not working... where is my missing link?
by France Baril 21 Dec '19

21 Dec '19

Hi, I'm trying to build something that is similar to custom indices but that actually returns resolved nodes (nodes created by manipulating and merging nodes from the original DB). 1- I created a script that creates new index DBs. UPDINDEX and AUTOOPTIMIZED are set on both the 'index' DB and the DB from which the new DBs were created 2- I can create a query on any of the new 'index' DBs to extract the nodes I want. 3- After I change the content in the source DB, the response shows that the nodes in the 'index' DBs are not updated. I get the old result. I feel that I am missing a link between the source and the index DBs to make the update mechanism work. Since I do not want to get the original node, but one that I create, I did not use db:node-id or db:open-id shown in the documentation example. Is that where the link is created? CODE THAT CREATES THE INDEX DBs let $langs := for $lang in ('en-us', 'es-es') (:db:list()[string-length(.) = 5][substring(., 3, 1) = '-']:) return $lang return for $lang in $langs let $index := <index>{ for $string-group in db:open('global-content')//*[name()='string-group'] let $prompt-this-lang := $string-group/child::*[@xml:lang=$lang][normalize-space(.)!=''] let $prompt-base := if (exists($prompt-this-lang)) then $prompt-this-lang else $string-group/child::*[@xml:lang='en-us'] let $prompts := for $prompt-match in db:open('global-content')/*[name()='prompt-refs']/*[name()='prompt'][@package=$string-group/data(@package)][@pkg-version=$string-group/data(@pkg-version)][@key=$string-group/data(@key)] return copy $copy := $prompt-base modify ( for $attr in $prompt-match/@*[name()!='package' and name()!='pkg-version' and name()!='key'] return insert node $attr into $copy ) return $copy return $prompts }</index> return db:create('index-prompt-' || $lang, $index, 'prompts-' || $lang || '.xml', map { 'updindex': true(), 'autooptimize': true() }) CODE THAT QUERIES THE 'INDEX' DBS (# db:enforceindex #) { for $prompt in db:attribute('index-prompt-en-us', 'setup_as_new', 'name')/.. return $prompt => prof:time() } -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 7

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk