January 2016 - BaseX-Talk - mailman.uni-konstanz.de

Fulltext Stemming
by kleist 12 Jan '16

12 Jan '16

Hi all, with query options: contains text "grün" all using diacritics sensitive using stemming using language 'German' I'll get: "grüne", "grüner", "Grüns" etc. But I also get "Gruß", "Grüße", "grüßen" or something like "Gruner" (Eigenname) Does the stemming have problems with german umlaute and "ß"? with query options: contains text "grün" all using stemming using language 'German' same results, but less. Why is that? The indexes of the database are the following: Indexes Up-to-date: true TEXTINDEX: true ATTRINDEX: true FTINDEX: true TEXTINCLUDE: ATTRINCLUDE: FTINCLUDE: LANGUAGE: German STEMMING: true CASESENS: true DIACRITICS: true STOPWORDS: UPDINDEX: false AUTOOPTIMIZE: false MAXCATS: 100 MAXLEN: 96 INDEXSPLITSIZE: 0 FTINDEXSPLITSIZE: 0 Optimized Query is the following: Optimized Query: let $hits_137 := ft:mark((db:open-pre("kleist-searchindex",147544), ...)/(((descendant::tei:s union descendant::tei:l)))[descendant::text() contains text "grün" all using diacritics sensitive using stemming using language 'German']) return element div { (element p { ("grün") }, element p { (count($hits_137)) }, element ul { (for $hit_139 in $hits_137 return element li { ($hit_139/descendant-or-self::node()/ancestor::tei:TEI/descendant-or-self::node()/tei:classCode[position() = 1]/string(), $hit_139) }) }) } Best, Günter

2 1

group-by behaviour for clustering XML fragments
by Hondros, Constantine (ELS-AMS) 11 Jan '16

11 Jan '16

Hello all, I'm using BaseX to cluster a set of millions of small XML fragments which look something like this: <affiliation> <organization>Institut für Organische Chemie der Universität Heidelberg</organization> <country iso-code="DEU"/> </affiliation> I need to cluster based on fragment similarity - so taking into account elements, attributes and text nodes. If I use the entire XML fragment as a grouping key, something like this: for $a at $c in db:open('DB')/item/*/affiliation group by $val := $a ... then will the grouping be equivalent to the functionality of the deep-equal function? First results seem to suggest this, but I want to make sure that grouping is not done on text node value alone or anything like that. Incidentally, BaseX is simply unbelievably fast at executing this - a million fragments clustered and written out to another DB in 16 seconds on a laptop. My congratulations on an amazing product. Regards, Constantine ________________________________ Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.

3 3

Strange behaviour on db:add inside restxq
by Marco Lettere 11 Jan '16

11 Jan '16

Hi all, We've run into a strange issue that we never experienced beforehand even if it's something we should have done thousands of time. Consider storing a resource sent through a RestXQ POST directly to a database. The restxq module module namespace o = "test"; (:sendMessage: Send a new message:) declare %rest:path('test/savemsg') %rest:POST('{$body}') %rest:consumes("application/xml") function o:save-message($body as document-node()) { db:add('hereiam', $body, 'saved.xml') }; The client request: http:send-request( <http:request method="POST" href="http://localhost:8984/test/savemsg"> <http:body media-type="application/xml"> <a> <b/> </a> </http:body> </http:request> ) The resource gets stored into the DB with all the extra text nodes containing white spaces (codepoints 10 and 32) something like: <a> <b/> </a> That's pretty strange because if I PUT the resource through Basex REST interface it doesn't create the spaces and the same if I directly db:add the resource. We noticed this oddity on BaseX 8.3.1 and could verify it on a Basex 8.2 and 8.1.1 as well. It works neatly on an older 7.9 version we still have around The applies same if the resource is sent through curl or Chrome Postman. What are we doing wrong? Thanks. Marco.

2 6

Lookups and arrows
by Tim Thompson 10 Jan '16

10 Jan '16

Hello, I'm testing some XQuery 3.1 features against a JSON-LD[1] document and had a few questions. In the JSON-LD format, the "@" symbol has special semantics in key names, but seems to cause problems with the 3.1 lookup operator. For example: json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld")("@context") works as expected, but json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld")?@context throws an error: [XPST0003] No specifier after lookup operator: '@'. Also, when using the "=>" operator, should it be possible to perform a lookup on the last expression in a chain, if that expression returns a map? For example: json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld")("@context") evaluates to "http://iiif.io/api/presentation/2/context.json" and json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld")("@context") => json-doc() returns another map object. So, how would one achieve this: json-doc(json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld ")("@context"))("@context") using the arrow operator? Thanks, Tim [1] http://www.w3.org/TR/json-ld/ -- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library

3 5

Mimicking RDBMS uniqueness constraints
by E. Wray Johnson 09 Jan '16

09 Jan '16

What is the best practice in BaseX to implement a uniqueness constraint similar to a RDBMS? Given a database, a path to the documents, an optional RegEx expression on files in the path, and one or more XPaths to the constraint elements/attributes. Can I create an index of this nature? Can the index not allow duplicates?

2 3

Database folders with numbers after
by James Ball 09 Jan '16

09 Jan '16

Hello, I’m seeing regular occurrences where I end up with extra folders in my database location with numbers after, for example: ODF-PUSH-T-JU ODF-PUSH-T-JU_1079560404 These have some but not all of the files from the original database with the same name, crucially don’t have the inf.basex file and so prevent the /dba from listing databases. What creates these folders? * I have a job that runs each night to optimise all the databases that have changed that day, I think these are coming about due to a failure to optimise but I’m not sure, I’m trying to understand what might creates them to be created so I can look deeper into the cause. Running 8.4 beta 5687c30 on Linux. Many thanks for your help, James

2 5

problems with BaseX under OpenShift (connection failed on port 15007)
by C. M. Sperberg-McQueen 08 Jan '16

08 Jan '16

I'm running into a problem that perhaps others here can help with. It appears to be almost identical to the problem reported by "P.C." on 5 March 2014 under the subject line 'BaseX server.close() "Connection refused" in Openshift' [1]; the mail record in the basex-talk archive doesn't show a resolution to P.C.'s problem. The main difference I see is that P.C. was deploying BaseX under Tomcat 7 on OpenShift, and I'm using Andy Bunce's quickstart method. [1] https://www.mail-archive.com/basex-talk%40mailman.uni-konstanz.de/msg03861.… I'm experimenting with Andy Bunce's BaseX quick-start for OpenShift, which I like a lot, and I am experiencing the following sequence of events: 1 Following the instructions at [2], I use rhc to make a new app, add the Quodatum repository as a remote, merge it into the application source, and push the result to OpenShift. (Note: no changes made between the commands "git pull -s recursive -X theirs upstream master" and "git push origin master". The first time I tried it, I did change the admin password, but I thought that that might have something to do with the problem, so I omitted that step for this test.) [2] https://github.com/Quodatum/openshift-basex-quick-start/ The OpenShift app is now available as promised at the URI indicated in [1], namely http://TestAPP-CMSMcQ.rhcloud.com/ (except that TestAPP is not really the name of the app, and CMSMcQ is not the name of the domain). It's a very nice looking console app. 2 I make some change -- any change -- to a file in the repository. To keep things simple, I'll add a comment line reading (: Kilroy was here :) to the file $app-name/basex/repo/quodatum/basex/env.xqm. I'm pretty sure I didn't break anything in the env.xqm file in making the change, and I didn't change anything in any other file. 3 I check the change in to the local git repository: # confirm that we have not touched anything else git status # commit the change git add basex/repo/quodatum/basex/env.xqm git commit -m "Vacuous change to env.xqm, just to have something to check in" 4 I attempt to push the change to the deployed app. The response is shown below (in the extract below, 'Salerno' is the local name of the machine, 'TestAPP' stands in for the name of the test application, and "CMSMcQ" for the OpenShift domain -- I've changed them here, because as can be seen from the log below, the app is currently open to anyone who knows the default admin userid and password). Salerno:TestAPP cmsmcq$ git push origin master Counting objects: 7, done. Delta compression using up to 8 threads. Compressing objects: 100% (5/5), done. Writing objects: 100% (7/7), 649 bytes | 0 bytes/s, done. Total 7 (delta 2), reused 0 (delta 0) remote: Stopping DIY cartridge remote: HOST: 127.12.13.129 remote: PORT: 15005 remote: DEBUG: true remote: USER: admin remote: SERVERHOST: 127.12.13.129 remote: STOPPORT: 15007 remote: SERVERPORT: 15005 remote: PASSWORD: admin remote: java.io.IOException: Connection failed (port: 15007). remote: at org.basex.BaseXHTTP.stop(BaseXHTTP.java:367) remote: at org.basex.BaseXHTTP.stop(BaseXHTTP.java:180) remote: at org.basex.BaseXHTTP.<init>(BaseXHTTP.java:94) remote: at org.basex.BaseXHTTP.main(BaseXHTTP.java:50) remote: Connection failed (port: 15007). remote: An error occurred executing 'gear prereceive' (exit code: 1) remote: Error message: CLIENT_ERROR: Failed to execute: 'control stop' for /var/lib/openshift/568d73d67628e13ca2000004/diy remote: remote: For more details about the problem, try running the command again with the '--trace' option. To ssh://568d73d67628e13ca2000004@TestAPP-CMSMcQ.rhcloud.com/~/git/TestAPP.git/ ! [remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to 'ssh://568d73d67628e13ca2000004@TestAPP-CMSMcQ.rhcloud.com/~/git/TestAPP.git/' Salerno:TestAPP cmsmcq$ I've logged in to the application using ssh and run the basexhttp stop command manually, and what one gets then is essentially the same. I think the root of the matter here is the line reading "remote: java.io.IOException: Connection failed (port: 15007)", but I'm having trouble thinking of a fix. One could imagine that OpenShift doesn't like application code using a port like 15007. But since it seems to have no trouble with BaseX using 15005, for the SERVERPORT, that doesn't seem to make sense. One could suppose that BaseX isn't actually listening on port 15007. The debugging output produced by -d, however does seems to show that BaseX knows that port 15007 is the stop port. Or does it? Perhaps it means only that BaseX has successfully parsed the -s option on the 'stop' command, which reads: ${OPENSHIFT_DATA_DIR}/basex/binbasexhttp -d \ -n ${OPENSHIFT_DIY_IP} -h ${OPENSHIFT_DIY_PORT} \ -p ${PORT_DATABASE} -s ${PORT_STOP} \ -U ${BASEX_USER} -P ${BASEX_PASSWORD} \ stop which in turn expands to .../basex/binbasexhttp -d \ -n 127.12.13.129 -h 8080 \ -p 15005 -s 15007 \ -U admin -P admin \ stop The startup command uses the same values for the -p and -s options, so it seems plausible that the start command is successfully setting the port. Several questions arise in connection with this problem, apart from the obvious and most important one, namely Q1 Does anyone reading this list understand where the problem might lie? The others that occur to me are: Q2 Is there a server command or an admin query that will show what ports a running instance of BaseX is listening to? Or would that be too broad a security violation? (It would be nice to confirm that BaseX is actually listening for a stop signal on port 15007. Q3 A search through the file system of the application shows that logs are being written to $APPLICATION_HOME/app-root/data/basex/data/.logs/ -- is there any way to find out from a running server what directory it's writing the logs to, to save the trouble of sshing to the application server and running find . -print0 | xargs -0 grep "User was logged in" or the equivalent? The admin:logs() function was helpful, but it would have been even more helpful if it had helped me find the logs directory. Of course, I see now, looking at the documentation, that logs are always (? or only by default?) written to $DATABASE_DIRECTORY/.logs -- if I had been confident of knowing what directory was the database directory in this (still unfamiliar) setup, I would have known where to look. Q4 Is there a command or admin-module query that indicates what directories BaseX regards as the base directory and the database directory? It could be very helpful, if it's not regarded as too great a security risk, for users trying to set things up in a new configuration. Q5 OpenShift's documentation claims that it really wants application logs to go into $APPLICATION-HOME/app-root/logs/ -- is there any way to induce BaseX to write its logs there? The main advantage I can currently see is that it then becomes possible to examine them with the OpenShift rhc tools. Any light anyone can shed on these issues would be very welcome. Thanks! Michael Sperberg-McQueen -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

3 13

Re: [basex-talk] XQuery Update: remove-elements-not-contents
by kleist-digital 08 Jan '16

08 Jan '16

Christian, I tried the proposed way, but it doesn’t work as expected. I’ll give a shortened example: Input is: <s> <w>u.<reg>und</reg> </w> <w>für</w> <w>die</w> <w>Folge</w> <w>da</w> <w>werden</w> <w>Sie</w>, <persName> <w>Gnädigste<reg>gnädigste</reg></w> <lb/> <w>Tante<c>,</c></w> </persName> <w>schon</w> <w>sorgen</w>. </s> Output should be: <s>u. für die Folge da werden Sie, Gnädigste Tante, schon sorgen.</s> With db:create('kleist_index', for $item in db:open('kleist-data') return $item update delete node .//(tei:note|tei:rdg|tei:lb|tei:del|tei:reg|tei:sic), db:open('kleist-data')/db:path(.)) I’ll get <s> <w>u.</w> <w>für</w> <w>die</w> <w>Folge</w> <w>da</w> <w>werden</w> <w>Sie</w>, <persName> <w>Gnädigste</w> <w>Tante<c>,</c></w> </persName> <w>schon</w> <w>sorgen</w>. </s> That’s what I expected. With db:create('kleist_index', for $item in db:open('kleist-data') return $item update (delete node .//(tei:note|tei:rdg|tei:lb|tei:del|tei:reg|tei:sic), for $n in .//tei:w return replace value of node $n with string($n)), db:open('kleist-data')/db:path(.)) I’ll get <s> <w>u.und</w> <w>für</w> <w>die</w> <w>Folge</w> <w>da</w> <w>werden</w> <w>Sie</w>, <persName> <w>Gnädigstegnädigste </w> <w>Tante,</w> </persName> <w>schon</w> <w>sorgen</w>. </s> The w-element isn’t removed and the reg-element isn’t deleted, but the value of w-element is replaced with string of w and reg. Any idea, how to reach <s>u. für die Folge da werden Sie, Gnädigste Tante, schon sorgen.</s>>? Best regards, Günter

2 5

Notes on running BaseX under OpenShift (using Andy Bunce's BaseX Quick Start for OpenShift)
by C. M. Sperberg-McQueen 08 Jan '16

08 Jan '16

For the benefit of any readers of this list who have been wondering whether to try Andy Bunce's Openshift quick start for BaseX, I have written up some notes on the subject [1] which are a little less terse than Andy's README file and perhaps slightly less forbidding for people (like me) who haven't used OpenShift before. The short version is: yes, if you have any interest in XQuery-driven Web applications, this is definitely worth trying: very quick to install, very straightforward. I've also updated my earlier notes on installing BaseX to run under Tomcat in a commercial Java hosting environment [2] to cover Tomcat 7 and BaseX 8.3.1. As always, my thanks to the BaseX team for the great tool, and to Andy Bunce for the quick-start, without which I would certainly not have been able to understand OpenShift well enough to get BaseX running. -C. M. Sperberg-McQueen [1] http://cmsmcq.com/mib/?p=1395 [2] http://cmsmcq.com/mib/?p=1368 -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

1 0

best approach to add dtd file into baseX
by Wang, Yvonne 07 Jan '16

07 Jan '16

Hi, I have sample dtd file such as below, I want to add the dtd file as is in baseX. Since the dtd file is not in xml format, there are currently two option I can thought about 1. Use TEXT as input format When I add it using baseX client UI, I got following output. Since we just need store the dtd file and later get it back as is. So I can not use doc function to retrieve it back as is. <text> <line><dtd></line> <line></line> </text> 2. Format it to xml using <dtd> <!-original data --> </dtd> when add, and remove those dtd and comment tag when download using doc() function. And there is extra handling required for nested xml comment. Is there any other better approach for adding dtd file which can be retrieved as its original content? Thanks Yvonne =========================================================================================================================== Sample.dtd <?xml encoding="UTF-8"?>  <!ELEMENT first (#PCDATA)> <!ELEMENT last (#PCDATA)> <!ELEMENT user (first,last)> <!ATTLIST user xmlns:xsi CDATA #FIXED 'http://www.w3.org/2001/XMLSchema-instance' xsi:schemaLocation CDATA #FIXED 'http://ariba.com.com/Core_after_xsd.xsd'>

3 3