October 2023 - BaseX-Talk - mailman.uni-konstanz.de

Advice on accepting UTF-16 XML (and others) from RESTXQ?
by Clark, Ash 20 Oct '23

20 Oct '23

Hello! In brief, I’m looking for any advice folks have on accepting XML of various encodings into a single database. Are there any considerations I should take into account with, say, RESTXQ form parameters? Or with storage, indexing, querying, etc.? For context, I’m currently (re)building a RESTXQ API for a service that publishes TEI. The site, TAPAS, is open to anyone with TEI that might need a place to store it and show it off online. We do some minimal testing on uploaded files [1], but up to this point we haven’t applied any limitations on the character encoding that folks use. While I expect that many people use UTF-8 encoding for TEI, I would like to try to ensure that folks using other encodings can use the service as well. The previous version of the TAPAS-xq API was not tested in this regard (and in other ways as well). I’m trying to do better in writing this new version. One problem that I’m running into is simply retrieving UTF-16 XML from a multipart form parameter.[2] I think BaseX is serializing the file as a string, but when I try to parse it as XML, the file is flagged as ill-formed (“Content is not allowed in prolog”). Is there a way to be flexible about uploaded XML while giving BaseX any parsing/serialization hints it might need? I’m about at the limit of my ability to figure out how different encodings might be working within BaseX; I appreciate any and all information or advice! Warmly, Ash [1]: Our minimal testing: Is this well-formed XML? Is it in the TEI namespace? Is the outermost element <TEI> rather than <teiCorpus> or anything else? We also try to make sure the XML doesn’t have Javascript that might make it into a reader’s browser. [2]: I first discovered this might be a problem when creating a unit test<https://github.com/NEU-DSG/tapas-xq/blob/d66066e65a661a4b909c21ae852d6479f0…> for this function<https://github.com/NEU-DSG/tapas-xq/blob/d66066e65a661a4b909c21ae852d6479f0…> — the UTF-16 file-as-string could not be parsed as XML. I then tested a UTF-16 file against the RESTXQ endpoint with curl, and found the same problem (though I do not know if the string has been read as UTF-8 or -16). In contrast, when I tried to create a standalone test module<https://gist.github.com/amclark42/9f8d8135a30e4659774a673627a263ac>, a UTF-16 file-as-string could be parsed as XML, but not when encoded as binary and then decoded again. Ash Clark (my pronouns are e/em/eir) XML Applications Developer Digital Scholarship Group Northeastern University Libraries as.clark(a)northeastern.edu (617) 373-5983

2 2

Trying to get process details on macOS
by Eliot Kimber 20 Oct '23

20 Oct '23

In the context of a 10.6 server running as a web app, I’m trying to implement a function that uses the process module to run commands to get information about the running BaseX servers, in particular, the PID and port each server is on. I start a number of “worker” BaseXHTTP instances to handle long-running processes. On Linux this code works: let $cmdResult := proc:execute(‘ps’, ('-o', 'pid,%cpu,command')) >From which I can parse out the PID and port for BaseX processes: let $lines as xs:string* := ($cmdResult/output/text() => tokenize('
')) return map{ 'data' : let $servers as map(*) := map:merge( for $line at $p in $lines[contains(., 'BaseXHTTP')] let $pid as xs:string := tokenize($line, '\s+')[1] let $port as xs:string := tokenize(substring-after($line, ' -p'),'\s+')[1] return map{ $port : map{ 'port' : $port, 'pid' : $pid } } ) return $servers } However, on macOS there’s not an exact equivalent of the linux version of ps, but there is the “pgrep” command, which, from the command line, lets me get the data I need: pgrep -lf basex However, when I try to run this from within BaseX, I get “sysmon request failed with error: sysmond service not found pgrep: Cannot get process list” I assume this is a function of how the HTTP servers are running on macOS. I’m starting them like so: "${basexBinDir}/basexhttp" ${debugFlag} -S -p${clientPort} -h${httpPort} -s${stopPort} & So as a service Any ideas how I might make this work on mapOS? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 3

More options for http-client?
by Andreas Hengsbach | nexoma 15 Oct '23

15 Oct '23

Hi Christian, are there any plans for the http client to become even more configurable? I have seen in the BaseX sources that the HttpClient.Builder (org.basex.io.IOUrl:173) is mostly served with standards. (except sslContext, connectTimeout, followRedirects). For example, I have a special case where I need to force the version to HTTP/1.1. Tried as a workaround to create my own instance via HttpClient.Builder using the XQuery Java binding, but failed on the interfaces. Thanks a lot Andreas

2 1

use-character-maps vs parameter-document with db:export()
by Zimmel, Daniel 09 Oct '23

09 Oct '23

Hi, when using db:export, the serialization parameter use-character-maps works just fine. But can someone explain why it is not possible to use parameter-document? At least, it does not work here with 10.3. As in: db:export($dbname,$path, map{ 'indent':'no', 'omit-xml-declaration':'no', 'parameter-document':'map.xml' }) OK I can see that there is no parameter-document option in the table I can find here defined for fn:serialize(): https://www.w3.org/TR/xpath-functions-31/#func-serialize When not using db:export(), all is well as described: https://docs.basex.org/wiki/Serialization#Character_mappings So in short, I think the answer is the spec. Is it? Have a nice weekend, Daniel

2 3

Strange "group by" behavior
by Alexander Krasnogolowy 07 Oct '23

07 Oct '23

Given the following XQuery: let $input := <root> <item><key>a</key><value>1</value></item> <item><key>b</key><value>2</value></item> <item><key>b</key><value>3</value></item> </root> let $map := map:merge(( map:entry(1, "A"), map:entry(2, "B"), map:entry(3, "C") )) for $result in $input/* let $key := $result/key/text() let $value := $result/value/text() group by $key return map:get($map, $key) When trying to execute that query, I get the error "Item expected, sequence found: (map {1:"A",2:"B",3:"C"}, map {1:"A",2:"B",3:"C"}). But why does it group the map? It isn't part of that for-loop but rather some kind of global variable. Or do I misunderstand the "group by" clause? I only expect the $value to be grouped which results in a sequence. Thanks, Alex

3 3

BaseX-10.7: error selecting CSV parser when creating a new database in the GUI
by ron.vdbranden＠gmail.com 03 Oct '23

03 Oct '23

Hi all, I think I'm experiencing a regression in the BaseX-10.7 GUI w.r.t. the CSV parser, when creating a new database. When I select CSV in the "Input format" dropdown on the "General" tab, nothing seems to be selected: * the dropdown stays open * no file mask is changed in "File patterns" * the "Parsing" tab doesn't show the CSV parsing options Additionally, I'm seeing logs starting with: Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -1 is negative at java.base/java.lang.System.arraycopy(Native Method) at org.basex.util.Array.copy(Array.java:124) This looks like an error to me; worked perfectly in 10.6. Best, Ron

2 2