From scfy12 at googlemail.com Tue Jan 1 13:26:30 2019 From: scfy12 at googlemail.com (Shaun Flynn) Date: Tue, 1 Jan 2019 12:26:30 +0000 Subject: [basex-talk] Serving Static Content from Tomcat Message-ID: Hello there, I am using BaseX 9.1.1 inside a Tomcat server trying to build a webapp using RESTXQ. I have studied the DBA xqm files, and see that there is static content delivered to the generated login page, located in dba/static folder. When I try to replicate this in my app, I get 404 errors saying the resource can not be found in the Developer Tools of Opera. Below is my .xqm file. The resources in question are in bold and underlined. What am I doing wrong? module namespace scheduleGui = "www.tt.co.uk/schedule"; import module namespace path = "www.tt.co.uk/path" at "DFSPath.xqm"; (: Produces a simple HTML form for entering origin and destination to produce a path :) declare %output:method("html") %rest:path("/rail/a2b") function scheduleGui:welcome() { * * * *

Paths

Origin
Destination
}; declare %rest:POST %rest:form-param("origin","{$origin}", "") %rest:form-param("destination","{$destination}", "") %output:method("html") %rest:path("/rail/a2bp") function scheduleGui:a2b($origin, $destination) { if($origin != "") then (

Find path from {$origin} to {$destination}

), }; -------------- next part -------------- An HTML attachment was scrubbed... URL: From scfy12 at googlemail.com Tue Jan 1 14:18:33 2019 From: scfy12 at googlemail.com (Shaun Flynn) Date: Tue, 1 Jan 2019 13:18:33 +0000 Subject: [basex-talk] Serving Static Content from Tomcat In-Reply-To: References: Message-ID: Hello there, Further to the below, I have found the common.xqm module which has the resource function to do this. Happy New Year :) Shaun On Tue, 1 Jan 2019, 12:26 Shaun Flynn Hello there, > > I am using BaseX 9.1.1 inside a Tomcat server trying to build a webapp > using RESTXQ. > > I have studied the DBA xqm files, and see that there is static content > delivered to the generated login page, located in dba/static folder. > > When I try to replicate this in my app, I get 404 errors saying the > resource can not be found in the Developer Tools of Opera. > > Below is my .xqm file. The resources in question are in bold and > underlined. What am I doing wrong? > > module namespace scheduleGui = "www.tt.co.uk/schedule"; > import module namespace path = "www.tt.co.uk/path" at "DFSPath.xqm"; > > (: > Produces a simple HTML form for entering origin and destination to produce > a path > :) > > > > declare > %output:method("html") > %rest:path("/rail/a2b") > function scheduleGui:welcome() { > > > * * > * * > > >

Paths

> > > > > > > > > > > > > >
Origin
Destination
> >
> >
> > > }; > > declare > %rest:POST > %rest:form-param("origin","{$origin}", "") > %rest:form-param("destination","{$destination}", "") > %output:method("html") > %rest:path("/rail/a2bp") > function scheduleGui:a2b($origin, $destination) { > if($origin != "") then (

Find path from {$origin} to {$destination} >

), >
    { > for $path in path:DFSSpec($origin, $destination, "up") > return >
  • {$path}
    • > }
> }; > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at zopyx.com Tue Jan 1 16:21:30 2019 From: lists at zopyx.com (Andreas Jung) Date: Tue, 01 Jan 2019 16:21:30 +0100 Subject: [basex-talk] Bulk import (moving from eXist to BaseX) Message-ID: Hi there, consider moving my XML stuff for a particular project from eXist to BaseX. I have a full dump from eXist with all related content and I want to import it 1:1 into BaseX?is there a bulk loader or what is the preferred way to import an existing folder structure? Of course I can create my own script what would import the data through WebDAV but the dump has about 50.000 files. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: From hawat.thufir at gmail.com Wed Jan 2 00:59:55 2019 From: hawat.thufir at gmail.com (thufir) Date: Tue, 1 Jan 2019 15:59:55 -0800 Subject: [basex-talk] basic xquery Message-ID: <4ae27c79-7674-0cb7-7270-10ae3908a040@gmail.com> How can I run this xquery: fetch:xml("http://books.toscrape.com/", map { 'parser': 'html', 'htmlparser': map { 'html': false(), 'nodefaults': true() } }) from basex using Java? I can run that xquery from the GUI. thanks, Thufir From fetanchaud at pch.cerfrance.fr Wed Jan 2 09:18:41 2019 From: fetanchaud at pch.cerfrance.fr (Fabrice ETANCHAUD) Date: Wed, 2 Jan 2019 08:18:41 +0000 Subject: [basex-talk] Bulk import (moving from eXist to BaseX) In-Reply-To: References: Message-ID: Hi Andreas, The CREATE-DB command can create a collection from a directory's structure [1] Best regards, Fabrice ETANCHAUD [1] : http://docs.basex.org/wiki/Commands#CREATE_DB -----Message d'origine----- De?: BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Andreas Jung Envoy??: mardi 1 janvier 2019 16:22 ??: BaseX Objet?: [basex-talk] Bulk import (moving from eXist to BaseX) Hi there, consider moving my XML stuff for a particular project from eXist to BaseX. I have a full dump from eXist with all related content and I want to import it 1:1 into BaseX?is there a bulk loader or what is the preferred way to import an existing folder structure? Of course I can create my own script what would import the data through WebDAV but the dump has about 50.000 files. Andreas From lists at zopyx.com Wed Jan 2 15:13:34 2019 From: lists at zopyx.com (Andreas Jung) Date: Wed, 2 Jan 2019 15:13:34 +0100 Subject: [basex-talk] JSON serialization/compatibility with exist-db JSON? Message-ID: Hi there, I have an issue with the migration of an XQuery script from eXist to Basex: The link https://gist.github.com/zopyx/467a01264d53bdaa48ad27cd553cfda8 shows the script and the expected (status quo) output for XML and JSON. Calling the script using /rest/foo/?run=foo.xql&method=json generates an HTTP response with Content-Type: application/json;charset=utf-8 but the output is a quoted XML string like: "\/foo\/de\/ayapedia\/addendums\/ernaehrung_9861fc6a-ba68-11e8-a8a1-001c4210b7a0\/current-20180920-111838.bak\/xml\/index.xml<\/uri>Ern?hrung<\/title><description\/><language>de<\/language><status>current<\/status><area>ayapedia<\/area><path\/><date>2018-09<\/date><url\/><atc_code\/><guideline_id>bef53078-2d8d-42d1-af8c-b3d387dfa52e<\/guideline_id><plone_uuid>464520ec154a42839b9f5d2635caccaf<\/plone_uuid><old_authors_additional\/><authors_additional>f?r den Arbeitskreis Ern?hrung, Stoffwechsel und Bewegung der DGHO<\/authors_additional><documenttype>ayapedia<\/documenttype><societies xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">dgho<\/societies><guideline_topics xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">183<\/guideline_topics><authors xmlns:json=\"http:\/\/ basex.org\/modules\/json\" json:array=\"true\">9211<\/authors><authors xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">5577<\/authors><authors xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">13728<\/authors><authors xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">373215<\/authors><\/item>" Calling /rest/foo/?run=foo.xql&method=json&json=format=jsonml" finally generates JSON but not in the related format. Isn't the JSON serialization supposed being standardized and compatible? Or how can I generate a JSON result that is compatible with the existing JSON response from eXist-dB? Andreas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190102/48d5fb33/attachment.htm> From christian.gruen at gmail.com Wed Jan 2 16:46:38 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 2 Jan 2019 16:46:38 +0100 Subject: [basex-talk] Scaling and Jsoniq In-Reply-To: <CAPB8cd0E6vdJk6K9C2U3EU_WTT_2nEkvZXwTB9Wz6oRdVtXvAA@mail.gmail.com> References: <CAPB8cd0E6vdJk6K9C2U3EU_WTT_2nEkvZXwTB9Wz6oRdVtXvAA@mail.gmail.com> Message-ID: <CAP94bnPyhiWUc7x81fX+gPfLJEbU_sJ2QWJ=ZuuUnm3-xP-mSw@mail.gmail.com> Hi Ben, > First, are there any support plans for scaling outwards? I know MarkLogic has splits clusters into multiple enodes and dnodes. But the idea is that there isn't one central bottleneck for requests. BaseX as open source project provides no built-in features for horizontal scaling. We have built individual solutions for our commercial customers; feel free to write to info at basexgmbh.de if you are interested. > Second, I read about Jsoniq and was intrigued by it. It seemed like it supported JSON without requiring an XML intermediary. Having good JSON support would really help with frameworks like Angular or React. Yes, Jsoniq is cool. We had some thoughts on implementing some years ago. In the end, we decided not to, because it would have introduced incompatibilities with features of the official XQuery 3.1 recommendation. Nowadays, you have two choices in BaseX: ? If you are processing JSON documents in main-memory, you can use the standard XQuery functions to convert JSON strings to XQuery data structures [1]. Maps and arrays in XQuery are very efficient, and often a better choice than XML if you have to repeatedly process the same data. ? If you want to store your data in a database, and benefit from indexing, updates, etc., you can convert JSON to an XML structure that suits you best [2]. You will notice that the time required for converting JSON to XML and backwards is hardly noticeable. In practice, it is frequently used to store and query millions of JSON documents in a database. Hope this helps, Christian [1] http://docs.basex.org/wiki/XQuery_3.1#JSON_Functions [2] http://docs.basex.org/wiki/Json_Module From christian.gruen at gmail.com Wed Jan 2 16:59:45 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 2 Jan 2019 16:59:45 +0100 Subject: [basex-talk] update:output() update:cache() In-Reply-To: <01BB7566-A8EF-47F2-9620-C7806155BA44@icloud.com> References: <01BB7566-A8EF-47F2-9620-C7806155BA44@icloud.com> Message-ID: <CAP94bnNn0kOVDVaQm6FxCEwTyNW535w4jqmnjAr=4a+semOKOw@mail.gmail.com> Dear Dieter, > I do not exactly understand, for which purpose update:output() is meant. > Does anybody have an example, where one can see, how it works? (There is no example in the docs.) By default, no result can be returned with XQuery Update 1.0 if an update is performed. The update:output function can be used to circumvent this restriction. You will find two examples if you look at [1,2]. The first one looks as follows: update:output("Prices have been deleted."), delete node //price In the currently opened database, all price elements will be deleted. In addition, the string "Prices have been deleted." will be returned. If you enable the MIXUPDATES option, the update:output function can be omitted. > At present, i am helping me with a copy/modify/return-solution and am giving back the altered result, before I actually write back the result-node of the previous to the db. But I am searching for a more intuitive way to get this result). If you want to return the updated database contents, this is one option indeed. update:output alone won?t help you here, because it will always return the original nodes. This is in conformance with the other XQuery Update expressions, which are all based on the original database contents. > (I am searching for a way, to run a update-expression on a database and give the altered nodes as result back to the restq-user in one query. If you use RESTXQ, another alternative is to call a second function via web:redirect [3], which will return the modified result to a user. If you follow the link, you will find a little example how this can look like. You will find a lot of other usages in our RESTXQ DBA code. > The update:output() or update:cache() - expression sounded like they do something similar? The update:cache function returns all items that will eventually be returned to the user. Please note that there are no serious use cases for this function in productive code (at least I?m not aware of such cases); I have only used them in some of our XQUnit tests. Hope this helps, Christian [1] http://docs.basex.org/wiki/Update_Module#update:output [2] http://docs.basex.org/wiki/XQuery_Update#Returning_Results [3] http://docs.basex.org/wiki/Web_Module#web:redirect From christian.gruen at gmail.com Wed Jan 2 17:01:25 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 2 Jan 2019 17:01:25 +0100 Subject: [basex-talk] basic xquery In-Reply-To: <4ae27c79-7674-0cb7-7270-10ae3908a040@gmail.com> References: <4ae27c79-7674-0cb7-7270-10ae3908a040@gmail.com> Message-ID: <CAP94bnMvY5Z=Sr94RK+S-3=YvNvH5PMsavBF_P3w=_Dqhr847A@mail.gmail.com> Hi Thufir, You can save this query on disk (e.g. as fetch.xq) and run it "basex fetch.xq". See the Startup section in our documentation [1] to learn how BaseX is used on the command-line. Best, Christian [1] http://docs.basex.org/wiki/Startup On Wed, Jan 2, 2019 at 1:00 AM thufir <hawat.thufir at gmail.com> wrote: > > How can I run this xquery: > > > fetch:xml("http://books.toscrape.com/", map { > 'parser': 'html', > 'htmlparser': map { 'html': false(), 'nodefaults': true() } > }) > > > from basex using Java? I can run that xquery from the GUI. > > > > thanks, > > Thufir From hawat.thufir at gmail.com Wed Jan 2 17:01:35 2019 From: hawat.thufir at gmail.com (thufir) Date: Wed, 2 Jan 2019 08:01:35 -0800 Subject: [basex-talk] console usage Message-ID: <801bc2db-ac66-a5fc-e1b2-d5b40cd7c22b@gmail.com> How can I run this script? thufir at dur:~/basex$ thufir at dur:~/basex$ basex [warning] /usr/bin/basex: Unable to locate /usr/share/java/jing.jar in /usr/share/java BaseX 9.0.1 [Standalone] Try 'help' to get more information. > > RUN fetch.note.text.xq Resource "/home/thufir/fetch.note.text.xq" not found. > > exit Have fun. thufir at dur:~/basex$ thufir at dur:~/basex$ cat fetch.note.text.xq fetch:xml("https://www.w3schools.com/xml/note.xml", map { 'chop': true() }) thufir at dur:~/basex$ The script seems to work fine from the GUI. thanks, Thufir From jonathan.robie at gmail.com Wed Jan 2 17:02:16 2019 From: jonathan.robie at gmail.com (Jonathan Robie) Date: Wed, 2 Jan 2019 11:02:16 -0500 Subject: [basex-talk] Scaling and Jsoniq In-Reply-To: <CAP94bnPyhiWUc7x81fX+gPfLJEbU_sJ2QWJ=ZuuUnm3-xP-mSw@mail.gmail.com> References: <CAPB8cd0E6vdJk6K9C2U3EU_WTT_2nEkvZXwTB9Wz6oRdVtXvAA@mail.gmail.com> <CAP94bnPyhiWUc7x81fX+gPfLJEbU_sJ2QWJ=ZuuUnm3-xP-mSw@mail.gmail.com> Message-ID: <CAOkFc+sdX1XTtV3XV6FskfJM=GWQKUuga3BVyu8oqyt1Lzm7hg@mail.gmail.com> As an author of both JSONiq and XQuery 3.1, I generally think of XQuery 3.1 as the standard that implemented what JSONiq did. Some authors of JSONiq see this very differently, preferring the JSONiq way of doing things. Is there anything JSONiq can do that XQuery 3.1 can't? Jonathan On Wed, Jan 2, 2019 at 10:47 AM Christian Gr?n <christian.gruen at gmail.com> wrote: > Hi Ben, > > > First, are there any support plans for scaling outwards? I know > MarkLogic has splits clusters into multiple enodes and dnodes. But the idea > is that there isn't one central bottleneck for requests. > > BaseX as open source project provides no built-in features for > horizontal scaling. We have built individual solutions for our > commercial customers; feel free to write to info at basexgmbh.de if you > are interested. > > > Second, I read about Jsoniq and was intrigued by it. It seemed like it > supported JSON without requiring an XML intermediary. Having good JSON > support would really help with frameworks like Angular or React. > > Yes, Jsoniq is cool. We had some thoughts on implementing some years > ago. In the end, we decided not to, because it would have introduced > incompatibilities with features of the official XQuery 3.1 > recommendation. > > Nowadays, you have two choices in BaseX: > > ? If you are processing JSON documents in main-memory, you can use the > standard XQuery functions to convert JSON strings to XQuery data > structures [1]. Maps and arrays in XQuery are very efficient, and > often a better choice than XML if you have to repeatedly process the > same data. > ? If you want to store your data in a database, and benefit from > indexing, updates, etc., you can convert JSON to an XML structure that > suits you best [2]. You will notice that the time required for > converting JSON to XML and backwards is hardly noticeable. In > practice, it is frequently used to store and query millions of JSON > documents in a database. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/XQuery_3.1#JSON_Functions > [2] http://docs.basex.org/wiki/Json_Module > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190102/b71078cf/attachment.htm> From christian.gruen at gmail.com Wed Jan 2 17:05:25 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 2 Jan 2019 17:05:25 +0100 Subject: [basex-talk] Bulk import (moving from eXist to BaseX) In-Reply-To: <B574D9D2-8DAC-48EA-BDD9-C84733AD7E9A@zopyx.com> References: <B574D9D2-8DAC-48EA-BDD9-C84733AD7E9A@zopyx.com> Message-ID: <CAP94bnPP3ObSO6tD8kQLeb3j6Fo8x37+9+i3RSrSc+Hkh8kdrw@mail.gmail.com> Hi Andreas, In BaseX, you can use the EXPORT command or the db:export function to write all database contents to disk [1,2]. I know too little about eXist-db; maybe they offer a similar feature? After that, as Fabrice indicated, you can use the CREATE DB command to create a database by pointing to that directory. If the database contains non-XML resources, you may need to write a little XQuery script or a BaseX command script that imports the data in the format you prefer. Hope this helps Christian [1] http://docs.basex.org/wiki/Commands#EXPORT [2] http://docs.basex.org/wiki/Database_Module#db:export On Tue, Jan 1, 2019 at 4:21 PM Andreas Jung <lists at zopyx.com> wrote: > > Hi there, > > consider moving my XML stuff for a particular project from eXist to BaseX. > > I have a full dump from eXist with all related content and I want to import it 1:1 > into BaseX?is there a bulk loader or what is the preferred way to import an existing folder structure? > Of course I can create my own script what would import the data through WebDAV but the dump has about > 50.000 files. > > Andreas From christian.gruen at gmail.com Wed Jan 2 17:07:57 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 2 Jan 2019 17:07:57 +0100 Subject: [basex-talk] console usage In-Reply-To: <801bc2db-ac66-a5fc-e1b2-d5b40cd7c22b@gmail.com> References: <801bc2db-ac66-a5fc-e1b2-d5b40cd7c22b@gmail.com> Message-ID: <CAP94bnO6GQrRtc3EyN7mtQkXj2qwSBYFLegu+e=A++H7QECf3Q@mail.gmail.com> > > RUN fetch.note.text.xq > Resource "/home/thufir/fetch.note.text.xq" not found. Is the script located at this path? If not, you will have to change to this directory and start BaseX from there, or you will need to specify the full path. From fetanchaud at pch.cerfrance.fr Wed Jan 2 17:13:21 2019 From: fetanchaud at pch.cerfrance.fr (Fabrice ETANCHAUD) Date: Wed, 2 Jan 2019 16:13:21 +0000 Subject: [basex-talk] Bulk import (moving from eXist to BaseX) In-Reply-To: <CAP94bnPP3ObSO6tD8kQLeb3j6Fo8x37+9+i3RSrSc+Hkh8kdrw@mail.gmail.com> References: <B574D9D2-8DAC-48EA-BDD9-C84733AD7E9A@zopyx.com> <CAP94bnPP3ObSO6tD8kQLeb3j6Fo8x37+9+i3RSrSc+Hkh8kdrw@mail.gmail.com> Message-ID: <AM5P18901MB014871FC5DD7EC4217B9B0A3FF8C0@AM5P18901MB0148.EURP189.PROD.OUTLOOK.COM> My mistake Andreas, I completely forgot raw content... By the way, Happy new year to all happy BaseX users, the BaseX team, and to you, Christian ! Best regards, Fabrice -----Message d'origine----- De?: BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Christian Gr?n Envoy??: mercredi 2 janvier 2019 17:05 ??: Andreas Jung Cc?: BaseX Objet?: Re: [basex-talk] Bulk import (moving from eXist to BaseX) Hi Andreas, In BaseX, you can use the EXPORT command or the db:export function to write all database contents to disk [1,2]. I know too little about eXist-db; maybe they offer a similar feature? After that, as Fabrice indicated, you can use the CREATE DB command to create a database by pointing to that directory. If the database contains non-XML resources, you may need to write a little XQuery script or a BaseX command script that imports the data in the format you prefer. Hope this helps Christian [1] http://docs.basex.org/wiki/Commands#EXPORT [2] http://docs.basex.org/wiki/Database_Module#db:export On Tue, Jan 1, 2019 at 4:21 PM Andreas Jung <lists at zopyx.com> wrote: > > Hi there, > > consider moving my XML stuff for a particular project from eXist to BaseX. > > I have a full dump from eXist with all related content and I want to import it 1:1 > into BaseX?is there a bulk loader or what is the preferred way to import an existing folder structure? > Of course I can create my own script what would import the data through WebDAV but the dump has about > 50.000 files. > > Andreas From christian.gruen at gmail.com Wed Jan 2 17:14:54 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 2 Jan 2019 17:14:54 +0100 Subject: [basex-talk] JSON serialization/compatibility with exist-db JSON? In-Reply-To: <CAL3mDbyaQd8jwsfb31wXxGF7N10n+anhVzR7Bia7AaRzrpfV2w@mail.gmail.com> References: <CAL3mDbyaQd8jwsfb31wXxGF7N10n+anhVzR7Bia7AaRzrpfV2w@mail.gmail.com> Message-ID: <CAP94bnND7Eiz133pGaLf5cZ6j=LzoeP7jHEgTgLpA9AaLMckSA@mail.gmail.com> Hi Andreas, Once again I must confess that I don?t know enough about eXist-db. What I can say is that only some JSON features were standardized, and not necessarily the ones that you find in eXist-db, BaseX and other processors (which were implemented much earlier). I have just referenced some of our JSON features in another mail some minutes ago [1,2]. They might give you an impression what is possible with BaseX, and how it differs from eXist-db. > Or how can I generate a JSON result that is compatible with the existing JSON response from eXist-dB? It may be best to export the eXist-db documents as JSON and reimport back (i.e., convert them to our JSON XML representation) in BaseX. Out of interest: Is the eXist-db JSON format fully bidirectional? If yes, we could think about adding the conversion rules to the next version of BaseX. Best, Christian [1] http://docs.basex.org/wiki/XQuery_3.1#JSON_Functions [2] http://docs.basex.org/wiki/Json_Module > /rest/foo/?run=foo.xql&method=json > > generates an HTTP response with > > Content-Type: application/json;charset=utf-8 > > but the output is a quoted XML string like: > > "<item><uri>\/foo\/de\/ayapedia\/addendums\/ernaehrung_9861fc6a-ba68-11e8-a8a1-001c4210b7a0\/current-20180920-111838.bak\/xml\/index.xml<\/uri><title>Ern?hrung<\/title><description\/><language>de<\/language><status>current<\/status><area>ayapedia<\/area><path\/><date>2018-09<\/date><url\/><atc_code\/><guideline_id>bef53078-2d8d-42d1-af8c-b3d387dfa52e<\/guideline_id><plone_uuid>464520ec154a42839b9f5d2635caccaf<\/plone_uuid><old_authors_additional\/><authors_additional>f?r den Arbeitskreis Ern?hrung, Stoffwechsel und Bewegung der DGHO<\/authors_additional><documenttype>ayapedia<\/documenttype><societies xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">dgho<\/societies><guideline_topics xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">183<\/guideline_topics><authors xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">9211<\/authors><authors xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">5577<\/authors><authors xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">13728<\/authors><authors xmlns:json=\"http:\/\/basex.org\/modules\/json\" json:array=\"true\">373215<\/authors><\/item>" > > Calling > > /rest/foo/?run=foo.xql&method=json&json=format=jsonml" > > finally generates JSON but not in the related format. > > Isn't the JSON serialization supposed being standardized and compatible? > Or how can I generate a JSON result that is compatible with the existing JSON response from eXist-dB? > > Andreas > > From hawat.thufir at gmail.com Wed Jan 2 17:35:09 2019 From: hawat.thufir at gmail.com (thufir) Date: Wed, 2 Jan 2019 08:35:09 -0800 Subject: [basex-talk] console usage In-Reply-To: <CAP94bnO6GQrRtc3EyN7mtQkXj2qwSBYFLegu+e=A++H7QECf3Q@mail.gmail.com> References: <801bc2db-ac66-a5fc-e1b2-d5b40cd7c22b@gmail.com> <CAP94bnO6GQrRtc3EyN7mtQkXj2qwSBYFLegu+e=A++H7QECf3Q@mail.gmail.com> Message-ID: <4f9f4e11-5410-1d4e-269d-568d48e6ba53@gmail.com> On 2019-01-02 8:07 a.m., Christian Gr?n wrote: >> > RUN fetch.note.text.xq >> Resource "/home/thufir/fetch.note.text.xq" not found. > > Is the script located at this path? If not, you will have to change to > this directory and start BaseX from there, or you will need to specify > the full path. > Yes, the script is there: thufir at dur:~/basex$ thufir at dur:~/basex$ cat fetch.note.text.xq fetch:xml("https://www.w3schools.com/xml/note.xml", map { 'chop': true() }) thufir at dur:~/basex$ thufir at dur:~/basex$ ls -al fetch.note.text.xq -rw-r--r-- 1 thufir thufir 79 Jan 2 07:54 fetch.note.text.xq thufir at dur:~/basex$ thufir at dur:~/basex$ pwd /home/thufir/basex thufir at dur:~/basex$ Yet my attempts to execute from the console give an error. Thanks for confirming that the error, as it appears, is that the file cannot be found. Makes no sense to me. The script itself works from the GUI: https://stackoverflow.com/questions/54009208/ thanks, Thufir ps: I missed your earlier reply but have fixed my gmail filters. From hawat.thufir at gmail.com Wed Jan 2 17:36:33 2019 From: hawat.thufir at gmail.com (thufir) Date: Wed, 2 Jan 2019 08:36:33 -0800 Subject: [basex-talk] console usage In-Reply-To: <801bc2db-ac66-a5fc-e1b2-d5b40cd7c22b@gmail.com> References: <801bc2db-ac66-a5fc-e1b2-d5b40cd7c22b@gmail.com> Message-ID: <9c7e45dc-46c0-cd7c-8e5c-645cfcb461b2@gmail.com> On 2019-01-02 8:01 a.m., thufir wrote: > Resource "/home/thufir/fetch.note.text.xq" not found. Ohhhh My pwd is /home/thufir/basex which is where the script is.. From hawat.thufir at gmail.com Wed Jan 2 17:38:45 2019 From: hawat.thufir at gmail.com (thufir) Date: Wed, 2 Jan 2019 08:38:45 -0800 Subject: [basex-talk] console usage In-Reply-To: <CAP94bnO6GQrRtc3EyN7mtQkXj2qwSBYFLegu+e=A++H7QECf3Q@mail.gmail.com> References: <801bc2db-ac66-a5fc-e1b2-d5b40cd7c22b@gmail.com> <CAP94bnO6GQrRtc3EyN7mtQkXj2qwSBYFLegu+e=A++H7QECf3Q@mail.gmail.com> Message-ID: <e2e4ccaf-f0b4-d1d0-7ff7-be24ceaab340@gmail.com> On 2019-01-02 8:07 a.m., Christian Gr?n wrote: >> > RUN fetch.note.text.xq >> Resource "/home/thufir/fetch.note.text.xq" not found. > > Is the script located at this path? If not, you will have to change to > this directory and start BaseX from there, or you will need to specify > the full path. > You were spot on: thufir at dur:~/basex$ thufir at dur:~/basex$ basex [warning] /usr/bin/basex: Unable to locate /usr/share/java/jing.jar in /usr/share/java BaseX 9.0.1 [Standalone] Try 'help' to get more information. > > RUN fetch.note.text.xq <note> <to>Tove</to> <from>Jani</from> <heading>Reminder</heading> <body>Don't forget me this weekend!</body> </note> Query "fetch.note.text.xq" executed in 1433.71 ms. > > exit Enjoy life. thufir at dur:~/basex$ Thanks. And, enjoy life :) From hawat.thufir at gmail.com Wed Jan 2 18:09:28 2019 From: hawat.thufir at gmail.com (thufir) Date: Wed, 2 Jan 2019 09:09:28 -0800 Subject: [basex-talk] Java XQuery Message-ID: <2c7ba44f-39fc-002c-063f-a0eb1f81f34f@gmail.com> After creating and populating a database: try { new DropDB(databaseName).execute(context); } catch (BaseXException ex) { Logger.getLogger(Database.class.getName()).log(Level.SEVERE, null, ex); LOG.fine("no databases to drop"); } how would I query it? The query: db:open("note")//note thanks, Thufir From hawat.thufir at gmail.com Wed Jan 2 22:39:01 2019 From: hawat.thufir at gmail.com (thufir) Date: Wed, 2 Jan 2019 13:39:01 -0800 Subject: [basex-talk] upper limits on storage; database admin Message-ID: <f3841f34-62f3-1016-987e-65cf92520b7e@gmail.com> What are the upper bounds to basex in size? Assuming it's just text xml, gigabytes is quite a bit to my thinking. At a certain, it's "big data" -- but how do you know when you're approaching that point? Or, is the bottleneck more read/write and consistency problems? What little I know of RDBMS is that master/slave can alleviate some bottlenecks. To put this another way: I'm so enthusiastic about basex that I'm having trouble finding a place it doesn't fit. As you approach terabytes and beyond what dbadmin approaches are employed? -Thufir From hawat.thufir at gmail.com Thu Jan 3 02:50:13 2019 From: hawat.thufir at gmail.com (thufir) Date: Wed, 2 Jan 2019 17:50:13 -0800 Subject: [basex-talk] XQuery from Java Message-ID: <4d482e37-6f1c-b8fc-9481-86b6543c8d08@gmail.com> I'm a tad more proficient with BaseX, but, how do I actually build an XQuery? Yes, I can write an XQuery and save it in a text file, then use BaseX for execution. But it would be far more flexible to write the actual XQuery from within Java itself. But how? Oracle gives an example: OXQDataSource ds = new OXQDataSource(); XQConnection con = ds.getConnection(); String query = "<hello-world>{1 + 1}</hello-world>"; XQPreparedExpression expr = con.prepareExpression(query); XQSequence result = expr.executeQuery(); using their datasource. Cannot BaseX do something similar? see also: https://stackoverflow.com/q/53996575/262852 thanks, Thufir From lists at zopyx.com Thu Jan 3 07:28:34 2019 From: lists at zopyx.com (Andreas Jung) Date: Thu, 03 Jan 2019 07:28:34 +0100 Subject: [basex-talk] Bulk import (moving from eXist to BaseX) In-Reply-To: <B574D9D2-8DAC-48EA-BDD9-C84733AD7E9A@zopyx.com> References: <B574D9D2-8DAC-48EA-BDD9-C84733AD7E9A@zopyx.com> Message-ID: <4886B19C-EBDB-4190-BE2B-560895775DCB@zopyx.com> I decided to implement a direkt exist2basex copy script that allows me to copy over the content through WebDAV from one database to another other with just some Python lines with the benefit to taking care of arbitrary data and not only markup content: https://gist.github.com/zopyx/770a4a0e0b34de5250d0ce54a4e509ec -aj On 1 Jan 2019, at 16:21, Andreas Jung wrote: > Hi there, > > consider moving my XML stuff for a particular project from eXist to BaseX. > > I have a full dump from eXist with all related content and I want to import it 1:1 > into BaseX?is there a bulk loader or what is the preferred way to import an existing folder structure? > Of course I can create my own script what would import the data through WebDAV but the dump has about > 50.000 files. > > Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190103/5b808f1f/attachment.sig> From lists at zopyx.com Thu Jan 3 09:34:40 2019 From: lists at zopyx.com (Andreas Jung) Date: Thu, 03 Jan 2019 09:34:40 +0100 Subject: [basex-talk] Setting modification timestamp (for content migrated over WebDAV) over REST? Message-ID: <AF311E9B-2F3C-46AB-BBB0-8E02FADF7A28@zopyx.com> As part of a migration we move data into BaseX over WebDAV. It would be neat if we could preserver the original timestamp. I am aware that it is not possible to set the timestamp over WebDAV. Would it be possible to set the modified timestamp for particular resource over the resource API or is this timestamp tied to storage internals of BaseX? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190103/5c8e327d/attachment.sig> From fetanchaud at pch.cerfrance.fr Thu Jan 3 10:12:59 2019 From: fetanchaud at pch.cerfrance.fr (Fabrice ETANCHAUD) Date: Thu, 3 Jan 2019 09:12:59 +0000 Subject: [basex-talk] upper limits on storage; database admin In-Reply-To: <f3841f34-62f3-1016-987e-65cf92520b7e@gmail.com> References: <f3841f34-62f3-1016-987e-65cf92520b7e@gmail.com> Message-ID: <DB6P18901MB0149932E4934504A5A2C511CFF8D0@DB6P18901MB0149.EURP189.PROD.OUTLOOK.COM> Hi Thufir, You will find here [1] the BaseX current limitations. The only limit I found was the maximum number of XML nodes in a collection. But I never handled more than 100 Gb of content. You can safely add documents by enabling the ADDCACHE option [2]; May the collection overflow, the operation will fail gracefully without corrupting it. As indexes are mono-collection structures, it is all about dispatching your data wisely in collections. And you are right, IMHO BaseX can fit in many places, and shines in prototyping ! Best regards, [1] http://docs.basex.org/wiki/Statistics [2] http://docs.basex.org/wiki/Options#ADDCACHE -----Message d'origine----- De?: BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de thufir Envoy??: mercredi 2 janvier 2019 22:39 ??: BaseX Objet?: [basex-talk] upper limits on storage; database admin What are the upper bounds to basex in size? Assuming it's just text xml, gigabytes is quite a bit to my thinking. At a certain, it's "big data" -- but how do you know when you're approaching that point? Or, is the bottleneck more read/write and consistency problems? What little I know of RDBMS is that master/slave can alleviate some bottlenecks. To put this another way: I'm so enthusiastic about basex that I'm having trouble finding a place it doesn't fit. As you approach terabytes and beyond what dbadmin approaches are employed? -Thufir From christian.gruen at gmail.com Thu Jan 3 15:00:52 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Thu, 3 Jan 2019 15:00:52 +0100 Subject: [basex-talk] Java XQuery In-Reply-To: <2c7ba44f-39fc-002c-063f-a0eb1f81f34f@gmail.com> References: <2c7ba44f-39fc-002c-063f-a0eb1f81f34f@gmail.com> Message-ID: <CAP94bnNzA+0WDTx=F2Onj02Ea=31EH0QZhyQYcpR8hgOzPb8xg@mail.gmail.com> There are various open issues in your question: > After creating and populating a database: It seems that you are actualling dropping an existing database in your Java example. > how would I query it? If you use Java, there is quite a variety on running queries. Maybe you could give us some insight into your use case first? For example, what do you want to do with the result? Thanks in advance Christian From christian.gruen at gmail.com Thu Jan 3 15:04:07 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Thu, 3 Jan 2019 15:04:07 +0100 Subject: [basex-talk] XQuery from Java In-Reply-To: <4d482e37-6f1c-b8fc-9481-86b6543c8d08@gmail.com> References: <4d482e37-6f1c-b8fc-9481-86b6543c8d08@gmail.com> Message-ID: <CAP94bnMm7OO3Vr1HynN10caAu7ACRFRuD976ZkuTuGQp9JaksQ@mail.gmail.com> > Oracle gives an example: This example is based on the XQJ API. Have a look at [1] to see how it is used in BaseX (and what other APIs exist). [1] http://docs.basex.org/wiki/Java_Examples > > OXQDataSource ds = new OXQDataSource(); > XQConnection con = ds.getConnection(); > String query = "<hello-world>{1 + 1}</hello-world>"; > XQPreparedExpression expr = con.prepareExpression(query); > XQSequence result = expr.executeQuery(); > > using their datasource. Cannot BaseX do something similar? > > > see also: > > https://stackoverflow.com/q/53996575/262852 > > > thanks, > > Thufir From christian.gruen at gmail.com Thu Jan 3 15:13:32 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Thu, 3 Jan 2019 15:13:32 +0100 Subject: [basex-talk] Setting modification timestamp (for content migrated over WebDAV) over REST? In-Reply-To: <AF311E9B-2F3C-46AB-BBB0-8E02FADF7A28@zopyx.com> References: <AF311E9B-2F3C-46AB-BBB0-8E02FADF7A28@zopyx.com> Message-ID: <CAP94bnNCt6DG1bp3pfxNeL8WeBsWdd3uyFZpG4_=xv5bQ6htTA@mail.gmail.com> Hi Andreas, You may be disappointed to hear that timestamps of documents are not stored as resource meta data in BaseX (unless your resource is binary). If timestamps are relevant, they are usually kept in project-specific meta databases. This will give you all kind of freedom you are looking for; but obviously it would often be simpler if there were database functions for requesting the timestamp of a file. There is a related GitHub issue on storing meta data in BaseX [1]. 2019 might be a good year to resume work on this. Reflecting on that, the beginning of a new year is actually always a good incentive to do everything you postponed the years before ;) Best, Christian [1] https://github.com/BaseXdb/basex/issues/988 On Thu, Jan 3, 2019 at 9:34 AM Andreas Jung <lists at zopyx.com> wrote: > > As part of a migration we move data into BaseX over WebDAV. > > It would be neat if we could preserver the original timestamp. > > I am aware that it is not possible to set the timestamp over WebDAV. > > Would it be possible to set the modified timestamp for particular resource > over the resource API or is this timestamp tied to storage internals of BaseX? > > Andreas From hawat.thufir at gmail.com Thu Jan 3 15:44:07 2019 From: hawat.thufir at gmail.com (thufir) Date: Thu, 3 Jan 2019 06:44:07 -0800 Subject: [basex-talk] Java XQuery In-Reply-To: <CAP94bnNzA+0WDTx=F2Onj02Ea=31EH0QZhyQYcpR8hgOzPb8xg@mail.gmail.com> References: <2c7ba44f-39fc-002c-063f-a0eb1f81f34f@gmail.com> <CAP94bnNzA+0WDTx=F2Onj02Ea=31EH0QZhyQYcpR8hgOzPb8xg@mail.gmail.com> Message-ID: <576991cd-664b-32e7-2c53-ffa17c18f46b@gmail.com> On 2019-01-03 6:00 a.m., Christian Gr?n wrote: > If you use Java, there is quite a variety on running queries. Maybe > you could give us some insight into your use case first? For example, > what do you want to do with the result? Yes, bit spaghetti-ish, pardon. The notion is to first drop the database, then populate, then query. For grabbing xml from w3schools, popping in a database, running an xquery, that works fine. Moving to html, it then sortof works. The db is dropped, a db is created and then populated. Browsing in the GUI I can see, for example, a list of book categories -- so there's data to work from. (Which tagsoup has fixed so that basex can parse it.) That's really the end goal: just running XQuery against html. The only query I can get working against the html is for the query string to be "text()" or perhaps "/text()" which then returns all the html. Rather, I'd want to traverse to pick out specific parts. It's related, to a degree, with Selenium efforts. --- The upshot being that the way tagsoup fixes malformed html either causes (me) problems with running xquery queries, or, more likely, I'm not understanding how to run xpath and xquery against the db properly. The GUI is very interesting in this respect because it allows me to visualize the raw data, it's "clickable", and I can run type xpath queries right in the GUI. However, the *only* xpath query I can get results on is "text()". Not so with "raw" xml from w3schools. With that xml I can drill down to varying degrees as expected. ------- Either tagsoup is mashing the html too extremely, or it's my lack of knowledge. Hey, I appreciate the input. Hope I made sense. -Thufir From hawat.thufir at gmail.com Thu Jan 3 15:47:57 2019 From: hawat.thufir at gmail.com (thufir) Date: Thu, 3 Jan 2019 06:47:57 -0800 Subject: [basex-talk] Java XQuery In-Reply-To: <576991cd-664b-32e7-2c53-ffa17c18f46b@gmail.com> References: <2c7ba44f-39fc-002c-063f-a0eb1f81f34f@gmail.com> <CAP94bnNzA+0WDTx=F2Onj02Ea=31EH0QZhyQYcpR8hgOzPb8xg@mail.gmail.com> <576991cd-664b-32e7-2c53-ffa17c18f46b@gmail.com> Message-ID: <72b2a078-ff5d-3a37-dc4e-9b484558a9ac@gmail.com> The project is: https://github.com/THUFIR/helloWorldBaseX While I've been through many of the Java examples it's also entirely possible that I overlooked something. Learning XQuery and Xpath syntax and capabilities. -thufir From hawat.thufir at gmail.com Thu Jan 3 20:03:57 2019 From: hawat.thufir at gmail.com (thufir) Date: Thu, 3 Jan 2019 11:03:57 -0800 Subject: [basex-talk] console versus gui Message-ID: <0c1f19bc-f88c-617f-c683-8496a52c85ad@gmail.com> Are the console and GUI in "sync"? By which I mean, can a database in the GUI not list from the console, and vice versa? I'm wondering if this perhaps relates to using BaseX as db server. I notice that my Java application can create and delete databases fine, but, it seems that that only the GUI can view those databases. Perhaps there's a way to migrate them from the GUI to the console? I'd like it consistent across access through the Java hooks, GUI and console. Or, perhaps I'm just writing bad code. I asked on SO: https://stackoverflow.com/q/54028203/262852 thanks, Thufir From hawat.thufir at gmail.com Thu Jan 3 21:30:19 2019 From: hawat.thufir at gmail.com (thufir) Date: Thu, 3 Jan 2019 12:30:19 -0800 Subject: [basex-talk] console versus gui In-Reply-To: <0c1f19bc-f88c-617f-c683-8496a52c85ad@gmail.com> References: <0c1f19bc-f88c-617f-c683-8496a52c85ad@gmail.com> Message-ID: <ef55dfb7-1ff3-b968-2fef-7e5fffe1c3dd@gmail.com> I'll take a look at the database directory: http://docs.basex.org/wiki/Configuration#Database_Directory On 2019-01-03 11:03 a.m., thufir wrote: > > I'd like it consistent across access through the Java hooks, GUI and > console. From bdysonsmith at gmail.com Thu Jan 3 21:41:56 2019 From: bdysonsmith at gmail.com (Bridger Dyson-Smith) Date: Thu, 3 Jan 2019 15:41:56 -0500 Subject: [basex-talk] console versus gui In-Reply-To: <ef55dfb7-1ff3-b968-2fef-7e5fffe1c3dd@gmail.com> References: <0c1f19bc-f88c-617f-c683-8496a52c85ad@gmail.com> <ef55dfb7-1ff3-b968-2fef-7e5fffe1c3dd@gmail.com> Message-ID: <CAD-FYmRG7xOBCz+E6t4zx+QJuBjFiREttZHFshfnRxdEkijvZg@mail.gmail.com> Hi Thufir, AFAIK, the GUI and console (client or stand-alone mode) are *not* in sync. There is some wording on this in the wiki but I am unable to find it at the moment. On Thu, Jan 3, 2019 at 3:30 PM thufir <hawat.thufir at gmail.com> wrote: > I'll take a look at the database directory: > > http://docs.basex.org/wiki/Configuration#Database_Directory > > > > On 2019-01-03 11:03 a.m., thufir wrote: > > > > I'd like it consistent across access through the Java hooks, GUI and > > console. > Best, Bridger -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190103/06e0e6f1/attachment.htm> From hawat.thufir at gmail.com Thu Jan 3 22:49:48 2019 From: hawat.thufir at gmail.com (thufir) Date: Thu, 3 Jan 2019 13:49:48 -0800 Subject: [basex-talk] console versus gui In-Reply-To: <CAD-FYmRG7xOBCz+E6t4zx+QJuBjFiREttZHFshfnRxdEkijvZg@mail.gmail.com> References: <0c1f19bc-f88c-617f-c683-8496a52c85ad@gmail.com> <ef55dfb7-1ff3-b968-2fef-7e5fffe1c3dd@gmail.com> <CAD-FYmRG7xOBCz+E6t4zx+QJuBjFiREttZHFshfnRxdEkijvZg@mail.gmail.com> Message-ID: <0e00c5e3-e55f-2b00-cc7e-e4bb5b9da455@gmail.com> I thought I was losing my mind -- thanks for at least confirming your understanding. I'll be futzing with it, bit burnt out at the moment. thanks, Thufir On 2019-01-03 12:41 p.m., Bridger Dyson-Smith wrote: > Hi Thufir, > > AFAIK, the GUI and console (client or stand-alone mode) are *not* in > sync. There is some wording on this in the wiki but I am unable to find > it at the moment. > > On Thu, Jan 3, 2019 at 3:30 PM thufir <hawat.thufir at gmail.com > <mailto:hawat.thufir at gmail.com>> wrote: > > I'll take a look at the database directory: > > http://docs.basex.org/wiki/Configuration#Database_Directory > > > > On 2019-01-03 11:03 a.m., thufir wrote: > > > > I'd like it consistent across access through the Java hooks, GUI and > > console. > > > Best, > Bridger From lists at zopyx.com Fri Jan 4 07:34:45 2019 From: lists at zopyx.com (Andreas Jung) Date: Fri, 04 Jan 2019 07:34:45 +0100 Subject: [basex-talk] Concurrency WebDAV vs. REST API Message-ID: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> For development purposes I am running basehttp in foreground. The WebDAV endpoint seems to block while I run a larger import for REST with lots of ADD and STORE commands within one <commands>..</commands> section. The WebDAV connection times out?what is the explanation for this? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190104/0bb84147/attachment.sig> From lists at zopyx.com Fri Jan 4 11:37:03 2019 From: lists at zopyx.com (Andreas Jung) Date: Fri, 4 Jan 2019 11:37:03 +0100 Subject: [basex-talk] Slow WebDAV operation Message-ID: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> I have an optimized database with 25K objects, 8 GB of data running on 9.1.1. WebDAV operations are very slow (client running and connecting locally). The response times are by a magnitude slower compared to exist-db. Is there some something missing in the configuration in order to speed this up significantly? Andreas 11:28:16.899 127.0.0.1:45822 admin OK Database 'onkopedia' was optimized in 29379.65 ms. 29380.24 ms 11:28:24.689 0:0:0:0:0:0:0:1:34258 admin REQUEST [HEAD] http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html 11:28:25.382 0:0:0:0:0:0:0:1:34258 admin 200 694.07 ms 11:28:25.389 0:0:0:0:0:0:0:1:34260 admin REQUEST [PROPFIND] http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html 11:28:26.065 0:0:0:0:0:0:0:1:34260 admin 200 675.31 ms 11:28:26.078 0:0:0:0:0:0:0:1:34262 admin REQUEST [HEAD] http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html 11:28:26.770 0:0:0:0:0:0:0:1:34262 admin 200 691.53 ms 11:28:26.789 0:0:0:0:0:0:0:1:34264 admin REQUEST [PROPFIND] http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/ 11:28:29.231 0:0:0:0:0:0:0:1:34264 admin 200 2444.93 ms 11:28:29.240 0:0:0:0:0:0:0:1:34266 admin REQUEST [HEAD] http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html 11:28:29.856 0:0:0:0:0:0:0:1:34266 admin 200 616.41 ms 11:28:29.864 0:0:0:0:0:0:0:1:34268 admin REQUEST [PROPFIND] http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/ 11:28:32.310 0:0:0:0:0:0:0:1:34268 admin 200 2445.99 ms 11:28:32.317 0:0:0:0:0:0:0:1:34270 admin REQUEST [HEAD] http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html 11:28:33.083 0:0:0:0:0:0:0:1:34270 admin 200 766.14 ms 11:28:33.103 0:0:0:0:0:0:0:1:34272 admin REQUEST [GET] http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html 11:28:33.719 0:0:0:0:0:0:0:1:34272 admin 200 615.38 ms -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190104/a0d0e7af/attachment.htm> From lists at zopyx.com Fri Jan 4 12:13:01 2019 From: lists at zopyx.com (Andreas Jung) Date: Fri, 04 Jan 2019 12:13:01 +0100 Subject: [basex-talk] Slow WebDAV operation In-Reply-To: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> References: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> Message-ID: <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> The original tests below have been made on a virtualized machine with 8 cores, 64 GB RAM. My first idea was about the shared storage, filesystem. So I tried with the same dataset on my local machine and the result is even worse: 1700ms for PROPFIND on the root of the database containing only on object in the root: 12:10:26.294 127.0.0.1:53202 admin REQUEST OPTIMIZE ALL 2.37 ms 12:10:41.396 127.0.0.1:53202 admin OK Database 'onkopedia' was optimized in 15102.34 ms. 15102.77 ms 12:10:42.646 127.0.0.1:35556 admin REQUEST [PROPFIND] http://localhost:8984/webdav/onkopedia/ 12:10:44.346 127.0.0.1:35556 admin 200 1700.02 ms Andreas On 4 Jan 2019, at 11:37, Andreas Jung wrote: > I have an optimized database with 25K objects, 8 GB of data running on > 9.1.1. > > WebDAV operations are very slow (client running and connecting > locally). > The response times are by a magnitude slower compared to exist-db. > Is there some something missing in the configuration in order to speed > this > up significantly? > > Andreas > > 11:28:16.899 127.0.0.1:45822 admin OK Database 'onkopedia' was > optimized in > 29379.65 ms. 29380.24 ms > 11:28:24.689 0:0:0:0:0:0:0:1:34258 admin REQUEST [HEAD] > http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html > 11:28:25.382 0:0:0:0:0:0:0:1:34258 admin 200 694.07 ms > 11:28:25.389 0:0:0:0:0:0:0:1:34260 admin REQUEST [PROPFIND] > http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html > 11:28:26.065 0:0:0:0:0:0:0:1:34260 admin 200 675.31 ms > 11:28:26.078 0:0:0:0:0:0:0:1:34262 admin REQUEST [HEAD] > http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html > 11:28:26.770 0:0:0:0:0:0:0:1:34262 admin 200 691.53 ms > 11:28:26.789 0:0:0:0:0:0:0:1:34264 admin REQUEST [PROPFIND] > http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/ > 11:28:29.231 0:0:0:0:0:0:0:1:34264 admin 200 2444.93 ms > 11:28:29.240 0:0:0:0:0:0:0:1:34266 admin REQUEST [HEAD] > http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html > 11:28:29.856 0:0:0:0:0:0:0:1:34266 admin 200 616.41 ms > 11:28:29.864 0:0:0:0:0:0:0:1:34268 admin REQUEST [PROPFIND] > http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/ > 11:28:32.310 0:0:0:0:0:0:0:1:34268 admin 200 2445.99 ms > 11:28:32.317 0:0:0:0:0:0:0:1:34270 admin REQUEST [HEAD] > http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html > 11:28:33.083 0:0:0:0:0:0:0:1:34270 admin 200 766.14 ms > 11:28:33.103 0:0:0:0:0:0:0:1:34272 admin REQUEST [GET] > http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html > 11:28:33.719 0:0:0:0:0:0:0:1:34272 admin 200 615.38 ms -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190104/97ad0e4b/attachment.htm> From lists at zopyx.com Fri Jan 4 12:48:24 2019 From: lists at zopyx.com (Andreas Jung) Date: Fri, 04 Jan 2019 12:48:24 +0100 Subject: [basex-talk] Slow WebDAV operation In-Reply-To: <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> References: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> Message-ID: <E6A14B98-B268-475C-A276-802CBAFE6450@zopyx.com> I re-verified the behavior on a third (fast) machine with SDD and the result as still bad with an average of 900-1000ms per request. Andreas On 4 Jan 2019, at 12:13, Andreas Jung wrote: > The original tests below have been made on a virtualized machine with > 8 cores, 64 GB RAM. > My first idea was about the shared storage, filesystem. So I tried > with the same dataset on my local > machine and the result is even worse: 1700ms for PROPFIND on the root > of the database containing only > on object in the root: > > 12:10:26.294 127.0.0.1:53202 admin REQUEST OPTIMIZE ALL 2.37 ms > 12:10:41.396 127.0.0.1:53202 admin OK Database 'onkopedia' was > optimized in 15102.34 ms. 15102.77 ms > 12:10:42.646 127.0.0.1:35556 admin REQUEST [PROPFIND] > http://localhost:8984/webdav/onkopedia/ > 12:10:44.346 127.0.0.1:35556 admin 200 1700.02 ms > > Andreas > > On 4 Jan 2019, at 11:37, Andreas Jung wrote: > >> I have an optimized database with 25K objects, 8 GB of data running >> on >> 9.1.1. >> >> WebDAV operations are very slow (client running and connecting >> locally). >> The response times are by a magnitude slower compared to exist-db. >> Is there some something missing in the configuration in order to >> speed this >> up significantly? >> >> Andreas >> >> 11:28:16.899 127.0.0.1:45822 admin OK Database 'onkopedia' was >> optimized in >> 29379.65 ms. 29380.24 ms >> 11:28:24.689 0:0:0:0:0:0:0:1:34258 admin REQUEST [HEAD] >> http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html >> 11:28:25.382 0:0:0:0:0:0:0:1:34258 admin 200 694.07 ms >> 11:28:25.389 0:0:0:0:0:0:0:1:34260 admin REQUEST [PROPFIND] >> http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html >> 11:28:26.065 0:0:0:0:0:0:0:1:34260 admin 200 675.31 ms >> 11:28:26.078 0:0:0:0:0:0:0:1:34262 admin REQUEST [HEAD] >> http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html >> 11:28:26.770 0:0:0:0:0:0:0:1:34262 admin 200 691.53 ms >> 11:28:26.789 0:0:0:0:0:0:0:1:34264 admin REQUEST [PROPFIND] >> http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/ >> 11:28:29.231 0:0:0:0:0:0:0:1:34264 admin 200 2444.93 ms >> 11:28:29.240 0:0:0:0:0:0:0:1:34266 admin REQUEST [HEAD] >> http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html >> 11:28:29.856 0:0:0:0:0:0:0:1:34266 admin 200 616.41 ms >> 11:28:29.864 0:0:0:0:0:0:0:1:34268 admin REQUEST [PROPFIND] >> http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/ >> 11:28:32.310 0:0:0:0:0:0:0:1:34268 admin 200 2445.99 ms >> 11:28:32.317 0:0:0:0:0:0:0:1:34270 admin REQUEST [HEAD] >> http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html >> 11:28:33.083 0:0:0:0:0:0:0:1:34270 admin 200 766.14 ms >> 11:28:33.103 0:0:0:0:0:0:0:1:34272 admin REQUEST [GET] >> http://localhost:8984/webdav/onkopedia/onkopedia/de/onkopedia/guidelines/mammakarzinom-der-frau_f7b16ba6-1826-11e6-97a4-001c4210b7a0/draft/html/index.html >> 11:28:33.719 0:0:0:0:0:0:0:1:34272 admin 200 615.38 ms -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190104/07d3cb0e/attachment.htm> From adamlaw at adamlaw.email Sat Jan 5 08:31:52 2019 From: adamlaw at adamlaw.email (Adam Law) Date: Sat, 5 Jan 2019 15:31:52 +0800 Subject: [basex-talk] [DELIVER]JETTY SSL & Cheeky Reload Message-ID: <026401d4a4c8$bc59e750$350db5f0$@adamlaw.email> Is it possible to configure BASEX's jetty to respond over HTTPS using a keystore (eg port 8985). I have tried editing jetty.xml to no avail and have read https://stackoverflow.com/questions/32734920/using-jetty-9-and-jetty-xml-w ith-basex. Ideally it would be great if the keystore could be uploaded without having to restart baseX - (possibly with a script) I know this is cheeky. Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190105/ea097a0a/attachment.htm> From christian.gruen at gmail.com Mon Jan 7 11:11:15 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Mon, 7 Jan 2019 11:11:15 +0100 Subject: [basex-talk] Java XQuery In-Reply-To: <576991cd-664b-32e7-2c53-ffa17c18f46b@gmail.com> References: <2c7ba44f-39fc-002c-063f-a0eb1f81f34f@gmail.com> <CAP94bnNzA+0WDTx=F2Onj02Ea=31EH0QZhyQYcpR8hgOzPb8xg@mail.gmail.com> <576991cd-664b-32e7-2c53-ffa17c18f46b@gmail.com> Message-ID: <CAP94bnPC-gMtbzngmi7gHJD+RbuXUQH19+7qFJBgHy2BpzuvzQ@mail.gmail.com> Hi Thufir, > That's really the end goal: just running XQuery against html. In that case, you can simply use "new XQuery(...).execute(ctx)". Any query that works in the GUI can be supplied as argument. > The only query I can get working against the html is for the query > string to be "text()" or perhaps "/text()" which then returns all the > html. Rather, I'd want to traverse to pick out specific parts. If you want to process the single results of your query, you should have a look at the examples [1], specifically [2]. Best, Christian [1] http://docs.basex.org/wiki/Java_Examples [2] https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/local/RunQueries.java From christian.gruen at gmail.com Mon Jan 7 11:14:28 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Mon, 7 Jan 2019 11:14:28 +0100 Subject: [basex-talk] Concurrency WebDAV vs. REST API In-Reply-To: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> References: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> Message-ID: <CAP94bnNo6vKC2NAax402ZOA5TCgc8VgxgTmt6OXL2tEH9=CCxQ@mail.gmail.com> Hi Andreas, If you send multiple commands via a single REST operation, they will be treated as a single transaction. In order to preserve the ACID principles, no other operations will run in parallel. If you want to run concurrent operations, you should be fine if you send your commands one by one via multiple REST requests. Best Christian On Fri, Jan 4, 2019 at 7:35 AM Andreas Jung <lists at zopyx.com> wrote: > > For development purposes I am running basehttp in foreground. > The WebDAV endpoint seems to block while I run a larger import for REST with lots of ADD and STORE commands within one > <commands>..</commands> section. The WebDAV connection times out?what is the explanation for this? > > Andreas From christian.gruen at gmail.com Mon Jan 7 11:20:23 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Mon, 7 Jan 2019 11:20:23 +0100 Subject: [basex-talk] Slow WebDAV operation In-Reply-To: <E6A14B98-B268-475C-A276-802CBAFE6450@zopyx.com> References: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> <E6A14B98-B268-475C-A276-802CBAFE6450@zopyx.com> Message-ID: <CAP94bnNZUanrePtDPYf234stB7e7GHBEx9pkbLUDK4mxjJLhiw@mail.gmail.com> The WebDAV implementation is definitely not the one that provides best performance in BaseX. Nevertheless, I think I?ll have a look at the use case you describe. Some questions in return: > I re-verified the behavior on a third (fast) machine with SDD and the result as still bad with an average of 900-1000ms per request. 1. What does "request" mean in that context? Do you request directory entries, do you request documents, or do you run updates? 2. Does the eXist-db WebDAV implementation list multiple databases in the WebDAV root directory (as BaseX does), or do all operations work on a specific database that is chosen at startup time? From christian.gruen at gmail.com Mon Jan 7 11:24:45 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Mon, 7 Jan 2019 11:24:45 +0100 Subject: [basex-talk] [DELIVER]JETTY SSL & Cheeky Reload In-Reply-To: <026401d4a4c8$bc59e750$350db5f0$@adamlaw.email> References: <026401d4a4c8$bc59e750$350db5f0$@adamlaw.email> Message-ID: <CAP94bnP23GzrpmS6qWKfPj5tyX6nrxR6=5PxzrSpWVj-WSoVKQ@mail.gmail.com> Dear Adam, I remember that Andy Bunce has given an answer on this in a previous post [1]. I haven?t tried this by myself (in most scenarios I got my hands on, additional proxies are used to handle https). Feel free to report back to us if this works for you; we might include your observations in our documentation. Best, Christian [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg10375.html On Sat, Jan 5, 2019 at 8:32 AM Adam Law <adamlaw at adamlaw.email> wrote: > > Is it possible to configure BASEX?s jetty to respond over HTTPS using a keystore (eg port 8985). I have tried editing jetty.xml to no avail and have read https://stackoverflow.com/questions/32734920/using-jetty-9-and-jetty-xml-with-basex. Ideally it would be great if the keystore could be uploaded without having to restart baseX ? (possibly with a script) I know this is cheeky. Adam > > From lists at zopyx.com Mon Jan 7 18:29:33 2019 From: lists at zopyx.com (Andreas Jung) Date: Mon, 07 Jan 2019 18:29:33 +0100 Subject: [basex-talk] Concurrency WebDAV vs. REST API In-Reply-To: <CAP94bnNo6vKC2NAax402ZOA5TCgc8VgxgTmt6OXL2tEH9=CCxQ@mail.gmail.com> References: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> <CAP94bnNo6vKC2NAax402ZOA5TCgc8VgxgTmt6OXL2tEH9=CCxQ@mail.gmail.com> Message-ID: <134A0719-5BC0-466E-9FF0-CD808EEF5FA0@zopyx.com> Are you saying that a write transaction blocks pending READ requests until the transaction has been committed? On 7 Jan 2019, at 11:14, Christian Gr?n wrote: > Hi Andreas, > > If you send multiple commands via a single REST operation, they will > be treated as a single transaction. In order to preserve the ACID > principles, no other operations will run in parallel. If you want to > run concurrent operations, you should be fine if you send your > commands one by one via multiple REST requests. > > Best > Christian > > > On Fri, Jan 4, 2019 at 7:35 AM Andreas Jung <lists at zopyx.com> wrote: >> >> For development purposes I am running basehttp in foreground. >> The WebDAV endpoint seems to block while I run a larger import for REST with lots of ADD and STORE commands within one >> <commands>..</commands> section. The WebDAV connection times out?what is the explanation for this? >> >> Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190107/1f77318d/attachment.sig> From lists at zopyx.com Mon Jan 7 18:34:13 2019 From: lists at zopyx.com (Andreas Jung) Date: Mon, 07 Jan 2019 18:34:13 +0100 Subject: [basex-talk] Slow WebDAV operation In-Reply-To: <CAP94bnNZUanrePtDPYf234stB7e7GHBEx9pkbLUDK4mxjJLhiw@mail.gmail.com> References: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> <E6A14B98-B268-475C-A276-802CBAFE6450@zopyx.com> <CAP94bnNZUanrePtDPYf234stB7e7GHBEx9pkbLUDK4mxjJLhiw@mail.gmail.com> Message-ID: <A455D88A-A346-4212-A80F-F28EF3A86530@zopyx.com> On 7 Jan 2019, at 11:20, Christian Gr?n wrote: > The WebDAV implementation is definitely not the one that provides best > performance in BaseX. Nevertheless, I think I?ll have a look at the > use case you describe. Some questions in return: > >> I re-verified the behavior on a third (fast) machine with SDD and the result as still bad with an average of 900-1000ms per request. > > 1. What does "request" mean in that context? Do you request directory > entries, do you request documents, or do you run updates? Please check the log entries..there are only HEAD/PROFIND/GET requests on the WebDAV level against a single XML resource. > 2. Does the eXist-db WebDAV implementation list multiple databases in > the WebDAV root directory (as BaseX does), or do all operations work > on a specific database that is chosen at startup time? eXist basically serves only one database per database server instance. But how is eXist relevant here? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190107/6ec2b40d/attachment.sig> From graydonish at gmail.com Wed Jan 9 18:35:10 2019 From: graydonish at gmail.com (Graydon Saunders) Date: Wed, 9 Jan 2019 12:35:10 -0500 Subject: [basex-talk] replacing text nodes Message-ID: <CAO3ciQF3vddTPrz__xkcreBjAkCL=fHsiVGqLrb3xukA1U0D_g@mail.gmail.com> Hello! I'm using BaseX 9.1.1 on Linux. So I need to go through a whole bunch of documents and emit them in an obfusticated form so the folks doing publication development can have them without the client's security people becoming upset. I don't want to obfusticate the db contents; I just want to be able to write out an obfusticated version. Minimal example: declare function local:gib($in as text()) as text() { text { 'ABCD' } }; declare function local:obfusc($in as document-node()) as document-node() { copy $c := $in modify ( for $x in $c/descendant::text()[normalize-space] return replace node $x with local:gib($x) ) return $c }; for $doc in db:open('out')[contains(document-uri(),'/scr_')][1] let $blurred := local:obfusc($doc) return $blurred If I run this, I get $doc back unaltered. If I look at the optimized query, it looks like it ought to be changing things. What am I doing wrong? Thanks! Graydon -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190109/0a5dd159/attachment.htm> From christian.gruen at gmail.com Wed Jan 9 18:41:51 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 9 Jan 2019 18:41:51 +0100 Subject: [basex-talk] replacing text nodes In-Reply-To: <CAO3ciQF3vddTPrz__xkcreBjAkCL=fHsiVGqLrb3xukA1U0D_g@mail.gmail.com> References: <CAO3ciQF3vddTPrz__xkcreBjAkCL=fHsiVGqLrb3xukA1U0D_g@mail.gmail.com> Message-ID: <CAP94bnNmN9Jf+-Uh00OCvVLZG-FkTKe__3kL9eYUWEJ42Bm11A@mail.gmail.com> Hi Graydon, $c/descendant::text()[normalize-space] Maybe it should be as follows? $c/descendant::text()[normalize-space()] Cheers, Christian Am Mi., 9. Jan. 2019, 18:35 hat Graydon Saunders <graydonish at gmail.com> geschrieben: > Hello! > > I'm using BaseX 9.1.1 on Linux. > > So I need to go through a whole bunch of documents and emit them in an > obfusticated form so the folks doing publication development can have them > without the client's security people becoming upset. I don't want to > obfusticate the db contents; I just want to be able to write out an > obfusticated version. > > Minimal example: > > declare function local:gib($in as text()) as text() { > text { 'ABCD' } > }; > > declare function local:obfusc($in as document-node()) as document-node() { > copy $c := $in > modify ( > for $x in $c/descendant::text()[normalize-space] > return replace node $x with local:gib($x) > ) > return $c > }; > > for $doc in db:open('out')[contains(document-uri(),'/scr_')][1] > let $blurred := local:obfusc($doc) > return $blurred > > If I run this, I get $doc back unaltered. If I look at the optimized > query, it looks like it ought to be changing things. > > What am I doing wrong? > > Thanks! > Graydon > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190109/d8c9d1db/attachment.htm> From graydonish at gmail.com Wed Jan 9 18:45:09 2019 From: graydonish at gmail.com (Graydon Saunders) Date: Wed, 9 Jan 2019 12:45:09 -0500 Subject: [basex-talk] replacing text nodes In-Reply-To: <CAP94bnNmN9Jf+-Uh00OCvVLZG-FkTKe__3kL9eYUWEJ42Bm11A@mail.gmail.com> References: <CAO3ciQF3vddTPrz__xkcreBjAkCL=fHsiVGqLrb3xukA1U0D_g@mail.gmail.com> <CAP94bnNmN9Jf+-Uh00OCvVLZG-FkTKe__3kL9eYUWEJ42Bm11A@mail.gmail.com> Message-ID: <CAO3ciQGOpGKCtQsDXE6+snG=8kg23fzBfOPjs=BqnbuXBDMmtw@mail.gmail.com> All die, and O! the embarrassment! It should indeed! Thank you, Christian. On Wed, Jan 9, 2019 at 12:42 PM Christian Gr?n <christian.gruen at gmail.com> wrote: > Hi Graydon, > > $c/descendant::text()[normalize-space] > > Maybe it should be as follows? > > $c/descendant::text()[normalize-space()] > > Cheers, > Christian > > > > > Am Mi., 9. Jan. 2019, 18:35 hat Graydon Saunders <graydonish at gmail.com> > geschrieben: > >> Hello! >> >> I'm using BaseX 9.1.1 on Linux. >> >> So I need to go through a whole bunch of documents and emit them in an >> obfusticated form so the folks doing publication development can have them >> without the client's security people becoming upset. I don't want to >> obfusticate the db contents; I just want to be able to write out an >> obfusticated version. >> >> Minimal example: >> >> declare function local:gib($in as text()) as text() { >> text { 'ABCD' } >> }; >> >> declare function local:obfusc($in as document-node()) as document-node() { >> copy $c := $in >> modify ( >> for $x in $c/descendant::text()[normalize-space] >> return replace node $x with local:gib($x) >> ) >> return $c >> }; >> >> for $doc in db:open('out')[contains(document-uri(),'/scr_')][1] >> let $blurred := local:obfusc($doc) >> return $blurred >> >> If I run this, I get $doc back unaltered. If I look at the optimized >> query, it looks like it ought to be changing things. >> >> What am I doing wrong? >> >> Thanks! >> Graydon >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190109/52e61c1e/attachment.htm> From christian.gruen at gmail.com Wed Jan 9 20:39:04 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 9 Jan 2019 20:39:04 +0100 Subject: [basex-talk] Slow WebDAV operation In-Reply-To: <A455D88A-A346-4212-A80F-F28EF3A86530@zopyx.com> References: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> <E6A14B98-B268-475C-A276-802CBAFE6450@zopyx.com> <CAP94bnNZUanrePtDPYf234stB7e7GHBEx9pkbLUDK4mxjJLhiw@mail.gmail.com> <A455D88A-A346-4212-A80F-F28EF3A86530@zopyx.com> Message-ID: <CAP94bnPudWfp+BEV8z2TwsC-Y8c+W51WSOeFOraPdLsfPh3QMg@mail.gmail.com> > > Please check the log entries..there are only HEAD/PROFIND/GET requests > on the WebDAV level against a single XML resource. > I just created a collection with 120,000 documents. The time for retrieving a single resource was about 100 ms; but it took much longer indeed to request entries of the root directory. I?ll see if I can so something about this. In general, you?ll definitely get better performance when using one of our other APIs. eXist basically serves only one database per database server instance. > But how is eXist relevant here? > It my be relevant because both eXist-db and BaseX are based on the same Milton library (but maybe a different version?). I experienced bottlenecks with this library in the past, so we once thought about writing our own WebDAV library (because the protocol is using XML anyway. It?s interesting to hear that you are getting better performance with eXist. One reason might be that eXist keeps an index of all files of the chosen database in main-memory while BaseX needs to scan the corresponding database every time it is accessed. If you are only working with a single database, you can try to open this database at startup time: basexhhtp -c "open your-db" However, I will check our WebDAV implementation and give you some more feedback. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190109/d59962e2/attachment.htm> From christian.gruen at gmail.com Wed Jan 9 20:48:17 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 9 Jan 2019 20:48:17 +0100 Subject: [basex-talk] Concurrency WebDAV vs. REST API In-Reply-To: <134A0719-5BC0-466E-9FF0-CD808EEF5FA0@zopyx.com> References: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> <CAP94bnNo6vKC2NAax402ZOA5TCgc8VgxgTmt6OXL2tEH9=CCxQ@mail.gmail.com> <134A0719-5BC0-466E-9FF0-CD808EEF5FA0@zopyx.com> Message-ID: <CAP94bnP=BbfP8sY4JGo=_mTBypPZJotXMpOAprU=L5BACxD8EA@mail.gmail.com> > Are you saying that a write transaction blocks pending READ requests until the transaction has been committed? No. What I was trying to say is that we deliberately designed REST requests to be atomic transactions. If you perform multiple updates in a single REST operation, you can be sure that a concurrent read operation will see the state before or after the first operation, but no state in-between. But it?s correct that, for the very same reason, we are using a multiple reader and single writer transaction model (on database level). This goes hand in hand with the Pending Update List semantics of the W3 XQuery Update Facility. Check out our documentation to learn more about transactions and updates in BaseX [1,2]. [1] http://docs.basex.org/wiki/Transactions [2] http://docs.basex.org/wiki/Update Am Mo., 7. Jan. 2019, 18:29 hat Andreas Jung <lists at zopyx.com> geschrieben: > Are you saying that a write transaction blocks pending READ requests until > the transaction > has been committed? > > > On 7 Jan 2019, at 11:14, Christian Gr?n wrote: > > > Hi Andreas, > > > > If you send multiple commands via a single REST operation, they will > > be treated as a single transaction. In order to preserve the ACID > > principles, no other operations will run in parallel. If you want to > > run concurrent operations, you should be fine if you send your > > commands one by one via multiple REST requests. > > > > Best > > Christian > > > > > > On Fri, Jan 4, 2019 at 7:35 AM Andreas Jung <lists at zopyx.com> wrote: > >> > >> For development purposes I am running basehttp in foreground. > >> The WebDAV endpoint seems to block while I run a larger import for REST > with lots of ADD and STORE commands within one > >> <commands>..</commands> section. The WebDAV connection times out?what > is the explanation for this? > >> > >> Andreas > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190109/0edd9ff7/attachment.htm> From lists at zopyx.com Thu Jan 10 06:28:19 2019 From: lists at zopyx.com (Andreas Jung) Date: Thu, 10 Jan 2019 06:28:19 +0100 Subject: [basex-talk] Slow WebDAV operation In-Reply-To: <CAP94bnPudWfp+BEV8z2TwsC-Y8c+W51WSOeFOraPdLsfPh3QMg@mail.gmail.com> References: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> <E6A14B98-B268-475C-A276-802CBAFE6450@zopyx.com> <CAP94bnNZUanrePtDPYf234stB7e7GHBEx9pkbLUDK4mxjJLhiw@mail.gmail.com> <A455D88A-A346-4212-A80F-F28EF3A86530@zopyx.com> <CAP94bnPudWfp+BEV8z2TwsC-Y8c+W51WSOeFOraPdLsfPh3QMg@mail.gmail.com> Message-ID: <8B109602-6196-4C34-99DA-F311F17883B6@zopyx.com> On 9 Jan 2019, at 20:39, Christian Gr?n wrote: >> >> Please check the log entries..there are only HEAD/PROFIND/GET requests >> on the WebDAV level against a single XML resource. >> > > I just created a collection with 120,000 documents. The time for retrieving > a single resource was about 100 ms; but it took much longer indeed to > request entries of the root directory. I?ll see if I can so something about > this. > > In general, you?ll definitely get better performance when using one of our > other APIs. See my first mail?also requests to other arbitrary documents are slow (in the range of course 700ms and higher). And there is only one test db involved here. We did not choose a vendor specific API because we want to have data portability and a unified API. By doing this we were able to move from eXist to Marklogic to BaseX to Filesystem storage without changing the access layer of our applications. Only a few specific scripts for querying the database had to be adjusted. > > > eXist basically serves only one database per database server instance. >> But how is eXist relevant here? >> > > It my be relevant because both eXist-db and BaseX are based on the same > Milton library (but maybe a different version?). I experienced bottlenecks > with this library in the past, so we once thought about writing our own > WebDAV library (because the protocol is using XML anyway. It?s interesting > to hear that you are getting better performance with eXist. One reason > might be that eXist keeps an index of all files of the chosen database in > main-memory while BaseX needs to scan the corresponding database every time > it is accessed. > > If you are only working with a single database, you can try to open this > database at startup time: > > basexhhtp -c "open your-db" > > However, I will check our WebDAV implementation and give you some more > feedback. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190110/fb6b8ef7/attachment.sig> From lists at zopyx.com Thu Jan 10 06:36:55 2019 From: lists at zopyx.com (Andreas Jung) Date: Thu, 10 Jan 2019 06:36:55 +0100 Subject: [basex-talk] Concurrency WebDAV vs. REST API In-Reply-To: <CAP94bnP=BbfP8sY4JGo=_mTBypPZJotXMpOAprU=L5BACxD8EA@mail.gmail.com> References: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> <CAP94bnNo6vKC2NAax402ZOA5TCgc8VgxgTmt6OXL2tEH9=CCxQ@mail.gmail.com> <134A0719-5BC0-466E-9FF0-CD808EEF5FA0@zopyx.com> <CAP94bnP=BbfP8sY4JGo=_mTBypPZJotXMpOAprU=L5BACxD8EA@mail.gmail.com> Message-ID: <8FDCF910-D542-4153-95DF-C9D04506D379@zopyx.com> On 9 Jan 2019, at 20:48, Christian Gr?n wrote: >> Are you saying that a write transaction blocks pending READ requests > until the transaction > has been committed? > > No. What I was trying to say is that we deliberately designed REST requests > to be atomic transactions. If you perform multiple updates in a single REST > operation, you can be sure that a concurrent read operation will see the > state before or after the first operation, but no state in-between. This is not true. A long running REST operation (executing a bunch of ADD and STORE commands) blocks parallel reads. A long running REST operation as in my case has serious impact on reading operations in parallel over WebDAV?the WebDAV connection timed out in my case. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190110/36e30153/attachment.sig> From christian.gruen at gmail.com Thu Jan 10 06:53:29 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Thu, 10 Jan 2019 06:53:29 +0100 Subject: [basex-talk] Concurrency WebDAV vs. REST API In-Reply-To: <8FDCF910-D542-4153-95DF-C9D04506D379@zopyx.com> References: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> <CAP94bnNo6vKC2NAax402ZOA5TCgc8VgxgTmt6OXL2tEH9=CCxQ@mail.gmail.com> <134A0719-5BC0-466E-9FF0-CD808EEF5FA0@zopyx.com> <CAP94bnP=BbfP8sY4JGo=_mTBypPZJotXMpOAprU=L5BACxD8EA@mail.gmail.com> <8FDCF910-D542-4153-95DF-C9D04506D379@zopyx.com> Message-ID: <CAP94bnOR2SJEZLTwu0CxRMB1DbU5uwr6csiJ3vhibJ6sssm9_w@mail.gmail.com> > > This is not true. Sounds like a communication problem. My impression is that we are both right. Did you read our documentation? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190110/07d1c4da/attachment.htm> From christian.gruen at gmail.com Thu Jan 10 07:19:28 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Thu, 10 Jan 2019 07:19:28 +0100 Subject: [basex-talk] Slow WebDAV operation In-Reply-To: <8B109602-6196-4C34-99DA-F311F17883B6@zopyx.com> References: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> <E6A14B98-B268-475C-A276-802CBAFE6450@zopyx.com> <CAP94bnNZUanrePtDPYf234stB7e7GHBEx9pkbLUDK4mxjJLhiw@mail.gmail.com> <A455D88A-A346-4212-A80F-F28EF3A86530@zopyx.com> <CAP94bnPudWfp+BEV8z2TwsC-Y8c+W51WSOeFOraPdLsfPh3QMg@mail.gmail.com> <8B109602-6196-4C34-99DA-F311F17883B6@zopyx.com> Message-ID: <CAP94bnMXz8zY9wO_Yrgh7qrHoU9NzkmurLefKpLLsv=QH9guzw@mail.gmail.com> > > See my first mail?also requests to other arbitrary documents are slow (in > the range > of course 700ms and higher). Well. That?s why I said I?ll have another look at this. Feel free to send us a link to your data, this might simplify the analysis. We did not choose > a vendor specific API because we want to have data portability and a > unified API. Sounds completely reasonable. Please note that developers regularly experience a lot of surprises and challenges when switching from one vendor to another, as XML databases have much more idiosyncrasies than relational databases (performance of APIs is just one of them). We try to minimize the challenge (e.g. by developing joint EXPath and EXQuery modules with other vendors), but of course this is often difficult if vendors have already chosen a specific solution. Would you like to share your experiences with MarkLogic and WebDAV with us? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190110/e391a9f8/attachment.htm> From johannes.bauer at tanner.de Thu Jan 10 07:42:01 2019 From: johannes.bauer at tanner.de (Johannes Bauer) Date: Thu, 10 Jan 2019 07:42:01 +0100 Subject: [basex-talk] web:response-header() Message-ID: <7ea12852-98a2-9845-4f33-cfcb6f278e69@tanner.de> Hello, I've a question to the response-header() function of the web module. In my RESTXQ function I try to return a response with HTTP Status 404: module namespace basex ='basex'; declare %rest:GET %rest:path("/db/{$db}/doc/{$doc}") %rest:produces("application/xml") %output:method("xml") function basex:doc($dbas xs:string, $docas xs:string)as item()* { web:response-header(map {},map {},map {"status" :404}),<notfound/> }; When I call this function I get the error: [basex:restxq] Unexpected node: status="404". If I call the function outside of the RESTXQ context the following response is generated: <rest:response xmlns:rest="http://exquery.org/ns/restxq" *status="404"*> <http:response xmlns:http="http://expath.org/ns/http-client"/> <output:serialization-parameters xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization"/> </rest:response> <notfound/> Shouldn't the status attribute be put on the http:response element instead? Best regards Johannes -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190110/0a5ed8da/attachment.htm> From christian.gruen at gmail.com Thu Jan 10 07:56:17 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Thu, 10 Jan 2019 07:56:17 +0100 Subject: [basex-talk] web:response-header() In-Reply-To: <7ea12852-98a2-9845-4f33-cfcb6f278e69@tanner.de> References: <7ea12852-98a2-9845-4f33-cfcb6f278e69@tanner.de> Message-ID: <CAP94bnPUe6zEUhSE8bxS4KZbb8JAMMrYmzEKdcj5zjm2RjjOow@mail.gmail.com> Dear Johannes, Thanks for the observation. The third argument was added to the function just recently, and it was done wrong. I have just fixed this, a new stable snapshot is available [1]. The next patch release (BaseX 9.1.2) will be released in the near future. Best, Christian [1] http://files.basex.org/releases/latest/ On Thu, Jan 10, 2019 at 7:42 AM Johannes Bauer <johannes.bauer at tanner.de> wrote: > > Hello, > > I've a question to the response-header() function of the web module. > > In my RESTXQ function I try to return a response with HTTP Status 404: > > module namespace basex = 'basex'; > > declare > %rest:GET > %rest:path("/db/{$db}/doc/{$doc}") > %rest:produces("application/xml") > %output:method("xml") > function basex:doc($db as xs:string, $doc as xs:string) as item()* > { > web:response-header(map {}, map {}, map {"status" : 404}), <notfound/> > }; > > > When I call this function I get the error: > > [basex:restxq] Unexpected node: status="404". > > > If I call the function outside of the RESTXQ context the following response is generated: > > <rest:response xmlns:rest="http://exquery.org/ns/restxq" status="404"> > <http:response xmlns:http="http://expath.org/ns/http-client"/> > <output:serialization-parameters xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization"/> > </rest:response> > <notfound/> > > Shouldn't the status attribute be put on the http:response element instead? > > Best regards > Johannes From christian.gruen at gmail.com Thu Jan 10 14:38:06 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Thu, 10 Jan 2019 14:38:06 +0100 Subject: [basex-talk] Slow WebDAV operation In-Reply-To: <CAP94bnMXz8zY9wO_Yrgh7qrHoU9NzkmurLefKpLLsv=QH9guzw@mail.gmail.com> References: <CAL3mDbyX_WKtTSCBXFZhc1SjVqW-UthCJPi=dRhDTKzY53cCRQ@mail.gmail.com> <7B226238-7304-46EB-BB96-AE63CC0C596E@zopyx.com> <E6A14B98-B268-475C-A276-802CBAFE6450@zopyx.com> <CAP94bnNZUanrePtDPYf234stB7e7GHBEx9pkbLUDK4mxjJLhiw@mail.gmail.com> <A455D88A-A346-4212-A80F-F28EF3A86530@zopyx.com> <CAP94bnPudWfp+BEV8z2TwsC-Y8c+W51WSOeFOraPdLsfPh3QMg@mail.gmail.com> <8B109602-6196-4C34-99DA-F311F17883B6@zopyx.com> <CAP94bnMXz8zY9wO_Yrgh7qrHoU9NzkmurLefKpLLsv=QH9guzw@mail.gmail.com> Message-ID: <CAP94bnOXLbjFc2myqgd+9pQ5=K_REnJ8nhoBk3MVhNbjsK8C+A@mail.gmail.com> > Well. That?s why I said I?ll have another look at this. Feel free to send us a link to your data, this might simplify the analysis. I have done some tests with larger databases (with up to 1 mio documents), and I have found a way to speed up the directory traversal via WebDAV. It MAY also affect the targeted retrieval of single resources. A new snapshot is available [1]. I?d appreciate if you could give me some feedback on the questions in my last mail; this might help us to help you (I?m still not sure which operations in your database instance are slow and which are fast, as I couldn?t reproduce this on my test database instances). @everyone: The new convenience function db:dir was added [1]. It is utilized by our WebDAV implementation, and it can be used to list all resources and sub folders of a database directory. Database directories in BaseX are all virtual: They implicitly result from the paths of stored resources. Cheers, Christian [1] http://files.basex.org/releases/latest/ [2] http://docs.basex.org/wiki/Database_Module#db:dir On Thu, Jan 10, 2019 at 7:19 AM Christian Gr?n <christian.gruen at gmail.com> wrote: >> >> See my first mail?also requests to other arbitrary documents are slow (in the range >> of course 700ms and higher). > > > Well. That?s why I said I?ll have another look at this. Feel free to send us a link to your data, this might simplify the analysis. > >> We did not choose >> a vendor specific API because we want to have data portability and a unified API. > > > Sounds completely reasonable. > > Please note that developers regularly experience a lot of surprises and challenges when switching from one vendor to another, as XML databases have much more idiosyncrasies than relational databases (performance of APIs is just one of them). We try to minimize the challenge (e.g. by developing joint EXPath and EXQuery modules with other vendors), but of course this is often difficult if vendors have already chosen a specific solution. > > Would you like to share your experiences with MarkLogic and WebDAV with us? From lists at zopyx.com Mon Jan 14 08:10:56 2019 From: lists at zopyx.com (Andreas Jung) Date: Mon, 14 Jan 2019 08:10:56 +0100 Subject: [basex-talk] Concurrency WebDAV vs. REST API In-Reply-To: <CAP94bnOR2SJEZLTwu0CxRMB1DbU5uwr6csiJ3vhibJ6sssm9_w@mail.gmail.com> References: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> <CAP94bnNo6vKC2NAax402ZOA5TCgc8VgxgTmt6OXL2tEH9=CCxQ@mail.gmail.com> <134A0719-5BC0-466E-9FF0-CD808EEF5FA0@zopyx.com> <CAP94bnP=BbfP8sY4JGo=_mTBypPZJotXMpOAprU=L5BACxD8EA@mail.gmail.com> <8FDCF910-D542-4153-95DF-C9D04506D379@zopyx.com> <CAP94bnOR2SJEZLTwu0CxRMB1DbU5uwr6csiJ3vhibJ6sssm9_w@mail.gmail.com> Message-ID: <9C85B0D4-EBB3-4C89-914F-4062A0030450@zopyx.com> On 10 Jan 2019, at 6:53, Christian Gr?n wrote: >> >> This is not true. > > > Sounds like a communication problem. My impression is that we are both > right. Did you read our documentation? The documentation perhaps does not explain the behavior properly. but anyway? A write operation in any database should not block parallel read operations. Read-last-committed as isolation is what one expect but not blocking of read operations until the end of the write transaction. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 858 bytes Desc: OpenPGP digital signature URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190114/298ff305/attachment.sig> From gsf.greece at gmail.com Mon Jan 14 13:18:33 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Mon, 14 Jan 2019 14:18:33 +0200 Subject: [basex-talk] prof:variables() and inlining Message-ID: <e68fbb24-b7c9-d0d8-0913-7521dde9e362@gmail.com> Hello, just something I noticed. I was trying to check the variables of one of my functions, but it only returned an empty string. Reading [1] <http://docs.basex.org/wiki/Profiling_Module#prof:variables> gave me the impression that disabling inlining (locally or globally) would return these values. But it doesn't work for me in version 9.1.1 or latest snapshot. I tested it on 8.4.4 and it works, so either it is a regression or a changed behaviour. I usually open an issue on Github directly but I think you now prefer it in the mail list first. Regards, George -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190114/edbc586c/attachment.htm> From christian.gruen at gmail.com Mon Jan 14 14:58:37 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Mon, 14 Jan 2019 14:58:37 +0100 Subject: [basex-talk] prof:variables() and inlining In-Reply-To: <e68fbb24-b7c9-d0d8-0913-7521dde9e362@gmail.com> References: <e68fbb24-b7c9-d0d8-0913-7521dde9e362@gmail.com> Message-ID: <CAP94bnP6L0WfVbpMgH2n3C35vena2Eo2LEX121xQUzPEG8-kQw@mail.gmail.com> Hi George, > I usually open an issue on Github directly but I think you now prefer it in the mail list first. Exactly (too many issues were created in the past that were hidden questions, or were based on misunderstandings). > I was trying to check the variables of one of my functions, but it only returned an empty string. Reading [1] gave me the impression that disabling inlining (locally or globally) would return these values. But it doesn't work for me in version 9.1.1 or latest snapshot. Do you have a self-contained example that shows the behavior? Thanks in advance, Christian From gsf.greece at gmail.com Mon Jan 14 15:03:23 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Mon, 14 Jan 2019 16:03:23 +0200 Subject: [basex-talk] prof:variables() and inlining In-Reply-To: <CAP94bnP6L0WfVbpMgH2n3C35vena2Eo2LEX121xQUzPEG8-kQw@mail.gmail.com> References: <e68fbb24-b7c9-d0d8-0913-7521dde9e362@gmail.com> <CAP94bnP6L0WfVbpMgH2n3C35vena2Eo2LEX121xQUzPEG8-kQw@mail.gmail.com> Message-ID: <3a98788a-37d6-0b32-5e67-4d1c832e2b19@gmail.com> > Do you have a self-contained example that shows the behavior? > > Thanks in advance, > Christian Sorry for not providing one, I thought it was easy to reproduce. Trying the example from the wiki:|for $x in 1 to 2 return prof:variables()| |Will return empty strings in 9.1.1 while it returns valid result (1,2) in version 8.4.4 - So I guess it's not an issue with inlining but with prof:variables() itself. | |George. | -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190114/8d6ce3f5/attachment.htm> From gsf.greece at gmail.com Mon Jan 14 15:20:35 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Mon, 14 Jan 2019 16:20:35 +0200 Subject: [basex-talk] prof:variables() and inlining In-Reply-To: <CAP94bnP6L0WfVbpMgH2n3C35vena2Eo2LEX121xQUzPEG8-kQw@mail.gmail.com> References: <e68fbb24-b7c9-d0d8-0913-7521dde9e362@gmail.com> <CAP94bnP6L0WfVbpMgH2n3C35vena2Eo2LEX121xQUzPEG8-kQw@mail.gmail.com> Message-ID: <d352d7dc-8d88-8f43-7299-1bdaf10037b4@gmail.com> > Do you have a self-contained example that shows the behavior? > > Thanks in advance, > Christian Actually looking at the compilation information it looks like something more is going on there. \ This example returns the following compilation information and optimized query: declare %basex:inline(0) function local:test() { ? for $x in 1 to 200 ? return 1 }; local:test() Compiling: - pre-evaluate range expression to range sequence: (1 to 200) - pre-evaluate range sequence to singleton xs:string sequence: (1 to 200) -> util:replicate("", 200) - pre-evaluate util:replicate(items,count) to singleton xs:integer sequence: util:replicate(1, 200) - pre-evaluate FLWOR expression to singleton xs:integer sequence: for $x_0 in util:replicate("", 200) retu... -> util:replicate(1, 200) - inline local:test#0 Optimized Query: util:replicate(1, 200) ----- But trying to return prof:variables() returns the following compilation information and optimized query: for $x in 1 to 2 return prof:variables() Compiling: - pre-evaluate range expression to range sequence: (1 to 2) - pre-evaluate range sequence to singleton xs:string sequence: (1 to 2) -> util:replicate("", 2) Optimized Query: for $x_0 in util:replicate("", 2) return prof:variables() It looks like there is an issue both with prof:variables(), and maybe with inlining. Why is the compilation for the first example tries to inline the function even if I have defined it to 0? in BaseX 8.4.4 that doesn't happen. Thanks again, George -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190114/34fa213b/attachment.htm> From christian.gruen at gmail.com Mon Jan 14 15:22:03 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Mon, 14 Jan 2019 15:22:03 +0100 Subject: [basex-talk] prof:variables() and inlining In-Reply-To: <3a98788a-37d6-0b32-5e67-4d1c832e2b19@gmail.com> References: <e68fbb24-b7c9-d0d8-0913-7521dde9e362@gmail.com> <CAP94bnP6L0WfVbpMgH2n3C35vena2Eo2LEX121xQUzPEG8-kQw@mail.gmail.com> <3a98788a-37d6-0b32-5e67-4d1c832e2b19@gmail.com> Message-ID: <CAP94bnNN33f-eLmyb=f-RR_sfug=Fhu3oh2MMCW4409+oUW9=w@mail.gmail.com> > Sorry for not providing one, I thought it was easy to reproduce. Trying the example from the wiki: for $x in 1 to 2 return prof:variables() Thanks, I can reproduce this now. The reason why the log output has changed is that some new optimizations were added in BaseX. As the variable $x will never be referenced in the subsequent expressions, the query is rewritten to: for $x in util:replicate("", 2) return prof:variables() The empty string is just a dummy here (internally, it?s a static object that requires no repeated instantiations of objects that will never be referenced). I have updated the example in the documentation: for $x in 1 to 2 return ($x, prof:variables()) Hope this helps, Christian From christian.gruen at gmail.com Mon Jan 14 16:08:05 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Mon, 14 Jan 2019 16:08:05 +0100 Subject: [basex-talk] prof:variables() and inlining In-Reply-To: <d352d7dc-8d88-8f43-7299-1bdaf10037b4@gmail.com> References: <e68fbb24-b7c9-d0d8-0913-7521dde9e362@gmail.com> <CAP94bnP6L0WfVbpMgH2n3C35vena2Eo2LEX121xQUzPEG8-kQw@mail.gmail.com> <d352d7dc-8d88-8f43-7299-1bdaf10037b4@gmail.com> Message-ID: <CAP94bnNx9dixFJvno_VxOpmhWYJvbbOV9PAc7B8+kJhUaQaizw@mail.gmail.com> Oh yes, many things are happening here; I am always surprised by myself. Here is some background information: ? Up to now, the body of a function was always inlined if it is a static value, no matter which value is currently assigned as inline limit. ? BaseX 9 pre-evaluates your function body to a value (a so-called "singleton sequence", to be exact; it is represented as ?SingletonSeq? in the query plan). ? As a result, the body was inlined with BaseX 9, whereas it was not in previous versions. The decision to ignore the inline limit if the function body is a value has been taken 6 years ago, so I cannot recollect what was the reason behind that. I decided to change this optimization rules, though (in practice, hardly anyone will notice this, as it?s only observable if the function inlining limit is changed). The behavior of the latest snapshot may be easier to understand now [1]. Hope this helps Christian [1] http://files.basex.org/releases/latest/ From gsf.greece at gmail.com Mon Jan 14 16:36:11 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Mon, 14 Jan 2019 17:36:11 +0200 Subject: [basex-talk] prof:variables() and inlining In-Reply-To: <CAP94bnNx9dixFJvno_VxOpmhWYJvbbOV9PAc7B8+kJhUaQaizw@mail.gmail.com> References: <e68fbb24-b7c9-d0d8-0913-7521dde9e362@gmail.com> <CAP94bnP6L0WfVbpMgH2n3C35vena2Eo2LEX121xQUzPEG8-kQw@mail.gmail.com> <d352d7dc-8d88-8f43-7299-1bdaf10037b4@gmail.com> <CAP94bnNx9dixFJvno_VxOpmhWYJvbbOV9PAc7B8+kJhUaQaizw@mail.gmail.com> Message-ID: <a3c35eee-52be-472c-bb16-dba5a3f301ca@gmail.com> On 1/14/19 5:08 PM, Christian Gr?n wrote: > Oh yes, many things are happening here; I am always surprised by > myself. Here is some background information: > > ? Up to now, the body of a function was always inlined if it is a > static value, no matter which value is currently assigned as inline > limit. > ? BaseX 9 pre-evaluates your function body to a value (a so-called > "singleton sequence", to be exact; it is represented as ?SingletonSeq? > in the query plan). > ? As a result, the body was inlined with BaseX 9, whereas it was not > in previous versions. > > The decision to ignore the inline limit if the function body is a > value has been taken 6 years ago, so I cannot recollect what was the > reason behind that. I decided to change this optimization rules, > though (in practice, hardly anyone will notice this, as it?s only > observable if the function inlining limit is changed). The behavior of > the latest snapshot may be easier to understand now [1]. > > Hope this helps > Christian > > [1] http://files.basex.org/releases/latest/ Thanks Christian, I will use this snapshot, always happy to help testing :) From gsf.greece at gmail.com Tue Jan 15 11:21:20 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Tue, 15 Jan 2019 12:21:20 +0200 Subject: [basex-talk] file:read-text-lines performance Message-ID: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> Hello, I'm trying to read a 4GB text file with 5 million lines and parse its contents. I'm using file:read-text-lines function <http://docs.basex.org/wiki/File_Module#file:read-text-lines> to do that. I managed to use fork-join and use 16 CPU threads to read the whole file by reading 10000 lines in each iteration, but it still takes 500 seconds for parsing / analyzing the data. Using a profiler I can see that most of the time is wasted reading each line - method readline <https://github.com/BaseXdb/basex/blob/0ef57de84659263c565ec41fff666ba5fa4f07dd/basex-core/src/main/java/org/basex/io/in/NewlineInput.java>. I plan to make some changes on the code tonight and see if I can find a way to read it faster, but I thought I should also post it here in case you have any tips. I'm also very inexperienced with using profilers so I hope I read the output correctly :) Regards, George -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190115/8f4a67d3/attachment.htm> From m.lettere at gmail.com Tue Jan 15 11:35:47 2019 From: m.lettere at gmail.com (Marco Lettere) Date: Tue, 15 Jan 2019 11:35:47 +0100 Subject: [basex-talk] Serialization issue with HTTP response Message-ID: <1dd0f533-5986-518f-a691-79ef75987857@gmail.com> Hi all, we have to deal with a third-party REST service which in case of error conditions returns this mime type: /*application/problem+json; charset=utf-8*/. I wrote this RestXQ [1] to mock it. Just copy it into restxq.xqm... When I call it like [2] I get /*[XPTY0004] Cannot convert xs:base64Binary to xs:string: "IntcImFcIjpcImJcIn0i"*/. This is because the response is seen as a binary and this can be double checked calling with [3]. I felt (maybe wrongly) that specializations of the mimetype had been faced in BaseX implementation of http:request somewhere back in time but I couldn't find anything in the mail archives. The strange thing however is that I get the same error when calling like [4] with override-mime-type set to text/json or application/json which instead I was expecting to work. Is the behaviour in this scenario expected or does this hide some inconsistency? Thanks, as usual, for the support. Regards, Marco. [1] declare%rest:path("mc/dummy") %rest:GET %output:method("json") function page:dummy() { ? <rest:response> ??? <http:response status="400"> ????? <http:header name="Content-Type" value="application/problem+json; charset=utf-8"/> ??? </http:response> ? </rest:response>, ? '{"a":"b"}' }; [2] json:parse( ? http:send-request(<http:request method="get"/>,"http://localhost:8984/mc/dummy")[2] ) [3] json:parse( ? convert:binary-to-string( ??? http:send-request(<http:request method="get"/>,"http://localhost:8984/mc/dummy")[2] ? ) ) [4] json:parse( ? http:send-request(<http:request method="get" override-media-type="application/json"/>,"http://localhost:8984/mc/dummy")[2] ) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190115/f2107cc2/attachment.htm> From christian.gruen at gmail.com Tue Jan 15 11:43:44 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Tue, 15 Jan 2019 11:43:44 +0100 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> Message-ID: <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> Hi George, an interesting use case. Reading lines of a text file feels like a natural candidate for iterative processing. As we need to ensure that the accessed file will eventually be closed, it is completely parsed before its contents can be accessed (all this happens in [1]), In future, we could possibly avoid this by registering file handles in the global query context and closing files that remained opened after query execution. What are your experiences with using a single thread? If memory consumption is too exhaustive, you could play with the window clause of the FLWOR expression [2,3]. It takes some time to explore the full magic of this XQuery 3.0 extension (the syntax is somewhat verbose), but it?s often a good alternative to complex functional code. Feel free to keep us updated, Christian [1] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/query/func/file/FileReadTextLines.java [2] http://docs.basex.org/wiki/XQuery_3.0#window [3] https://www.w3.org/TR/xquery-30/#id-windows On Tue, Jan 15, 2019 at 11:21 AM George Sofianos <gsf.greece at gmail.com> wrote: > > Hello, > > I'm trying to read a 4GB text file with 5 million lines and parse its contents. I'm using file:read-text-lines function to do that. I managed to use fork-join and use 16 CPU threads to read the whole file by reading 10000 lines in each iteration, but it still takes 500 seconds for parsing / analyzing the data. Using a profiler I can see that most of the time is wasted reading each line - method readline. I plan to make some changes on the code tonight and see if I can find a way to read it faster, but I thought I should also post it here in case you have any tips. I'm also very inexperienced with using profilers so I hope I read the output correctly :) > > Regards, > > George From gsf.greece at gmail.com Tue Jan 15 12:16:33 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Tue, 15 Jan 2019 13:16:33 +0200 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> Message-ID: <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> Hi Christian, On 1/15/19 12:43 PM, Christian Gr?n wrote: > What are your experiences with using a single thread? If memory > consumption is too exhaustive, you could play with the window clause > of the FLWOR expression [2,3]. It takes some time to explore the full > magic of this XQuery 3.0 extension (the syntax is somewhat verbose), > but it?s often a good alternative to complex functional code. Using a single thread looks to be OK too, about 10k lines per second, and I'm not sure reading the same file with 16 threads (on SSD) is the way to go from an I/O point of view. Searching on stackoverflow there are many suggestions on how to read a file with one or multiple threads e.g [1] I immediately return the data I need for each line (a small string for example) so the memory consumption is low, I have provided 12GB but I never see over 2-3GB of memory usage. My initial thoughts were that maybe garbage collection was causing delays but after profiling BaseX I don't think this is an issue. It's interesting to know about the window function though, I will certainly find a use for it. While I know most of these functions exist, I can always learn much more about a language. Only yesterday I managed to use fork-join successfully and I think it will save me a lot of time and effort for my use cases. I will post again if I have any updates, thanks again, George. [1]: https://stackoverflow.com/questions/40412008/how-to-read-a-file-using-multiple-threads-in-java-when-a-high-throughput3gb-s From christian.gruen at gmail.com Tue Jan 15 12:48:34 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Tue, 15 Jan 2019 12:48:34 +0100 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> Message-ID: <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> Hi George, I?m glad to announce that files are now processed in an iterative manner [1,2]. That?s something I wanted to try a while ago, and your mail was another motivation to get it done. It works pretty fine: I reduced the JVM memory to a tiny maximum of 4mb, and I managed to count the line numbers of a file with several gigabytes: count(file:read-text-lines('huge.txt')) I?d be interested to hear if your code runs faster with the latest snapshot. Christian [1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/commit/cfb7a7965de85139ec9595a6e79a45d873da7c25 On Tue, Jan 15, 2019 at 12:16 PM George Sofianos <gsf.greece at gmail.com> wrote: > > Hi Christian, > > On 1/15/19 12:43 PM, Christian Gr?n wrote: > > What are your experiences with using a single thread? If memory > > consumption is too exhaustive, you could play with the window clause > > of the FLWOR expression [2,3]. It takes some time to explore the full > > magic of this XQuery 3.0 extension (the syntax is somewhat verbose), > > but it?s often a good alternative to complex functional code. > > Using a single thread looks to be OK too, about 10k lines per second, > and I'm not sure reading the same file with 16 threads (on SSD) is the > way to go from an I/O point of view. Searching on stackoverflow there > are many suggestions on how to read a file with one or multiple threads > e.g [1] > > I immediately return the data I need for each line (a small string for > example) so the memory consumption is low, I have provided 12GB but I > never see over 2-3GB of memory usage. My initial thoughts were that > maybe garbage collection was causing delays but after profiling BaseX I > don't think this is an issue. It's interesting to know about the window > function though, I will certainly find a use for it. While I know most > of these functions exist, I can always learn much more about a language. > Only yesterday I managed to use fork-join successfully and I think it > will save me a lot of time and effort for my use cases. I will post > again if I have any updates, thanks again, > > George. > > [1]: > https://stackoverflow.com/questions/40412008/how-to-read-a-file-using-multiple-threads-in-java-when-a-high-throughput3gb-s > From gsf.greece at gmail.com Tue Jan 15 12:57:52 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Tue, 15 Jan 2019 13:57:52 +0200 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> Message-ID: <4db78e97-2a33-8810-5456-f6a71e1b8b13@gmail.com> Wow, thanks for your fast response! I will give it a try tonight, George. On 1/15/19 1:48 PM, Christian Gr?n wrote: > Hi George, > > I?m glad to announce that files are now processed in an iterative > manner [1,2]. That?s something I wanted to try a while ago, and your > mail was another motivation to get it done. > > It works pretty fine: I reduced the JVM memory to a tiny maximum of > 4mb, and I managed to count the line numbers of a file with several > gigabytes: > > count(file:read-text-lines('huge.txt')) > > I?d be interested to hear if your code runs faster with the latest snapshot. > Christian > > [1] http://files.basex.org/releases/latest/ > [2] https://github.com/BaseXdb/basex/commit/cfb7a7965de85139ec9595a6e79a45d873da7c25 > From christian.gruen at gmail.com Tue Jan 15 13:15:49 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Tue, 15 Jan 2019 13:15:49 +0100 Subject: [basex-talk] Concurrency WebDAV vs. REST API In-Reply-To: <9C85B0D4-EBB3-4C89-914F-4062A0030450@zopyx.com> References: <6C6B03E2-64D3-46DE-A6A6-B6459E32F41F@zopyx.com> <CAP94bnNo6vKC2NAax402ZOA5TCgc8VgxgTmt6OXL2tEH9=CCxQ@mail.gmail.com> <134A0719-5BC0-466E-9FF0-CD808EEF5FA0@zopyx.com> <CAP94bnP=BbfP8sY4JGo=_mTBypPZJotXMpOAprU=L5BACxD8EA@mail.gmail.com> <8FDCF910-D542-4153-95DF-C9D04506D379@zopyx.com> <CAP94bnOR2SJEZLTwu0CxRMB1DbU5uwr6csiJ3vhibJ6sssm9_w@mail.gmail.com> <9C85B0D4-EBB3-4C89-914F-4062A0030450@zopyx.com> Message-ID: <CAP94bnNwYStMejcWAqgMZjw1vYWR7+HvjtTrMEhRW3-gqrho2w@mail.gmail.com> > The documentation perhaps does not explain the behavior properly. but anyway? > A write operation in any database should not block parallel read operations. > Read-last-committed as isolation is what one expect but not blocking of read operations > until the end of the write transaction. It seems we can?t make you happy with BaseX, Andreas. Sorry for that. In order to allow concurrent reads during updates and at the same time preserve all ACID guarantees and other guarantees that we can give to our users, we might switch to an MVCC architecture one day in future. It might get reality if we find enough sponsors out there. The current architecture works fine in practice, though; it is applied in commercial use cases in which millions of datasets are stored and thousands of concurrent updates and reads are performed in realtime by hundreds of users. However, building such applications requires some knowledge and experience with BaseX. One crucial issue is, for example, that your update operations will be fast. From m.lettere at gmail.com Tue Jan 15 15:04:28 2019 From: m.lettere at gmail.com (Marco Lettere) Date: Tue, 15 Jan 2019 15:04:28 +0100 Subject: [basex-talk] Documentation suggestion Message-ID: <0d912a53-81f7-c496-d751-5f12654a36e4@gmail.com> Hi Christian, I suggest to add an explicit statement of which is the default value of the combine option of map:merge. Currently the only reference is in the Change notes at the bottom of the page. Thanks! Marco. From liam at fromoldbooks.org Wed Jan 16 00:51:46 2019 From: liam at fromoldbooks.org (Liam R. E. Quin) Date: Tue, 15 Jan 2019 18:51:46 -0500 Subject: [basex-talk] many distinct namespaces In-Reply-To: <CAP94bnNdWRD4CEBSGsy9bYK2oQ52XTeMf7-+rs2ZUGeXS3MMTw@mail.gmail.com> References: <1541022422.882950422@f403.i.mail.ru> <39e0d303-f8eb-3180-a6ec-c9aacb3a44a5@le-tex.de> <CAP94bnNdWRD4CEBSGsy9bYK2oQ52XTeMf7-+rs2ZUGeXS3MMTw@mail.gmail.com> Message-ID: <57bdf06bc00d50ea6a8f004e4af2bf7b3204b6d4.camel@fromoldbooks.org> On Fri, 2018-11-02 at 17:25 +0100, Christian Gr?n wrote: did this ever happen? > > Some more details: The current storage layout per node has been fixed > to 16 bytes. One byte (8 bits) is reserved for the namespace > reference. Here are a couple of hacky appraches in the spirit of brainstorming ;) * reserve 7 bits for the namespace, and 1 bit for "uses extended namespace". In the top-bit-set objects only, add an extra byte. * Use the first 2 bytes of the element name :) Liam -- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Web slave for vintage clipart http://www.fromoldbooks.org/ From gsf.greece at gmail.com Wed Jan 16 01:36:11 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Wed, 16 Jan 2019 02:36:11 +0200 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> Message-ID: <03cbaa55-327f-12fb-1459-aedbbd5c519c@gmail.com> Hi Christian, what I failed to mention last time was that I was using the offset / limit mode of the file:read-text-lines. I never tried to load the whole file into memory with the previous version, because I thought it would be inefficient. I just tried now with the latest snapshot using a single core and while the whole file is being loaded into memory (4GB+), the process completes in about 120 seconds, which is fine for me. Using the offset mode looks to still be more memory efficient (stays around 1-1,3GB), but is very slow (both single core and multi core). One issue, I can't make the non offset version work with fork-join. It fills the whole memory quickly, so I guess it reads the whole file into memory for each thread(?) - I tried up to 12GB. I've also noticed that in both versions (old and new snapshot), interrupting the fork-join mode will keep the threads running until I manually kill the BaseX process. Maybe I'm doing something wrong, or maybe I'm asking too much from fork-join :) I will try with the window clause tomorrow, maybe it will help. I'm posting an example of my code to help explain better my use case. For now, it is fine because I'm only reading a 4GB file, but potentially I might have to read up to 200GB files so having multi-core capabilities will help. let $data := file:read-text-lines($file, "UTF-8", false()) let $count := count($data) let $all := xquery:fork-join( ? for $i in $data return function() { ? parse-json($i)?('object1')?*?('object2')?('object3') ? } ) return distinct-values($all) Regards, George On 1/15/19 1:48 PM, Christian Gr?n wrote: > Hi George, > > I?m glad to announce that files are now processed in an iterative > manner [1,2]. That?s something I wanted to try a while ago, and your > mail was another motivation to get it done. > > It works pretty fine: I reduced the JVM memory to a tiny maximum of > 4mb, and I managed to count the line numbers of a file with several > gigabytes: > > count(file:read-text-lines('huge.txt')) > > I?d be interested to hear if your code runs faster with the latest snapshot. > Christian > > [1] http://files.basex.org/releases/latest/ > [2] https://github.com/BaseXdb/basex/commit/cfb7a7965de85139ec9595a6e79a45d873da7c25 > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190116/b05dcd53/attachment.htm> From gsf.greece at gmail.com Wed Jan 16 02:16:49 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Wed, 16 Jan 2019 03:16:49 +0200 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> Message-ID: <d33401d5-4906-46e6-a443-1805b8b28723@gmail.com> There also looks to be a difference on how the read-text-lines is used. The following similar queries produce different Query paths, and have different memory usage. This is probably why I can't benefit from the update on more complex queries. 1) return count(file:read-text-lines($file, "UTF-8", false())) Memory usage - about 20 megabytes Query path: <QueryPlan compiled="true" updating="false"> ? <FnCount name="count(items)" type="xs:integer" size="1"> ??? <FileReadTextLines name="read-text-lines(path[,encoding[,fallback[,offset[,length]]]])" type="xs:string*"> ????? <Str type="xs:string">/home/lumiel/eworx/betmechs/bme/webservice/samples/betfair/September-2015/output.json</Str> ????? <Str type="xs:string">UTF-8</Str> ????? <Bln type="xs:boolean">false</Bln> ??? </FileReadTextLines> ? </FnCount> </QueryPlan> 2) let $data := file:read-text-lines($file, "UTF-8", false()) ??? return count($data) Memory? usage: 4.5GB Query path: <QueryPlan compiled="true" updating="false"> ? <GFLWOR type="xs:integer" size="1"> ??? <Let type="xs:string*"> ????? <Var name="$data" id="1" type="xs:string*"/> ????? <FileReadTextLines name="read-text-lines(path[,encoding[,fallback[,offset[,length]]]])" type="xs:string*"> ??????? <Str type="xs:string">/full/path/file.txt</Str> ??????? <Str type="xs:string">UTF-8</Str> ??????? <Bln type="xs:boolean">false</Bln> ????? </FileReadTextLines> ??? </Let> ??? <FnCount name="count(items)" type="xs:integer" size="1"> ????? <VarRef type="xs:string*"> ??????? <Var name="$data" id="1" type="xs:string*"/> ????? </VarRef> ??? </FnCount> ? </GFLWOR> </QueryPlan> On 1/15/19 1:48 PM, Christian Gr?n wrote: > Hi George, > > I?m glad to announce that files are now processed in an iterative > manner [1,2]. That?s something I wanted to try a while ago, and your > mail was another motivation to get it done. > > It works pretty fine: I reduced the JVM memory to a tiny maximum of > 4mb, and I managed to count the line numbers of a file with several > gigabytes: > > count(file:read-text-lines('huge.txt')) > > I?d be interested to hear if your code runs faster with the latest snapshot. > Christian > > [1] http://files.basex.org/releases/latest/ > [2] https://github.com/BaseXdb/basex/commit/cfb7a7965de85139ec9595a6e79a45d873da7c25 > From christian.gruen at gmail.com Wed Jan 16 12:37:56 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 16 Jan 2019 12:37:56 +0100 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <03cbaa55-327f-12fb-1459-aedbbd5c519c@gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> <03cbaa55-327f-12fb-1459-aedbbd5c519c@gmail.com> Message-ID: <CAP94bnNapPuCW6_GZO=8h6Pn16uztEJEVSF8uq5mpHF-R+NQew@mail.gmail.com> > Using the offset mode looks to still be more memory efficient (stays around 1-1,3GB), but is very slow (both single core and multi core). A general note on xquery:fork-join (you may be aware of that anyway): While it may sound enticing to use the function for as many jobs as possible, it is often slower than clever single-core processing. The reason is that (in your case) a file will be accessed by several competing threads (which leads to random access I/O patterns, which are difficult to schedule for the OS, even with SSDs). Even in Java programming, code is often faster that doesn?t use the Java 8 streaming features and, instead, relies on the internal JVM optimizations for distributing atomic operations to multiple cores. > let $data := file:read-text-lines($file, "UTF-8", false()) > let $count := count($data) > > let $all := > xquery:fork-join( > for $i in $data return function() { > parse-json($i)?('object1')?*?('object2')?('object3') > } > ) This code will potentially create thousands or millions of Java threads. Maybe you are getting better results by splitting your input into 4 or 8 parts, and process each part in a dedicated function. I would indeed assume that the following code? distinct-values( for $line in file:read-text-lines($file, "UTF-8", false()) return parse-json($line)?('object1')?*?('object2')?('object3') ) ?will be most efficient, even if you process files of 100 GB or more (especially with the new, iterative approach). From christian.gruen at gmail.com Wed Jan 16 12:41:55 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 16 Jan 2019 12:41:55 +0100 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <d33401d5-4906-46e6-a443-1805b8b28723@gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> <d33401d5-4906-46e6-a443-1805b8b28723@gmail.com> Message-ID: <CAP94bnN1wyQUdtoxdmmU_WCaBJgWjG0NxJLj+Ph1gvHVdxOTWg@mail.gmail.com> The reason for that: file:read-text-lines is a non-deterministic function. Each invocation might yield different results (as the file contents may change in the background). This is different with non-deterministic function calls, such as fn:doc('abc.xml'). If you call such a function repreatedly, it will always access the same document, which has been opened and parsed by the first call of this function. > 1) return count(file:read-text-lines($file, "UTF-8", false())) Here, file processing will be iterative. > 2) let $data := file:read-text-lines($file, "UTF-8", false()) > return count($data) The file contents will be bound to $data, and counted in a second step. If the expression of your let clause was deterministic, the variable would be inlined, and the resulting query plan would be identical to the one of your first query. From gsf.greece at gmail.com Wed Jan 16 13:07:15 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Wed, 16 Jan 2019 14:07:15 +0200 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <CAP94bnN1wyQUdtoxdmmU_WCaBJgWjG0NxJLj+Ph1gvHVdxOTWg@mail.gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> <d33401d5-4906-46e6-a443-1805b8b28723@gmail.com> <CAP94bnN1wyQUdtoxdmmU_WCaBJgWjG0NxJLj+Ph1gvHVdxOTWg@mail.gmail.com> Message-ID: <19b27b4b-e42c-acd7-fefe-303e20c6c19d@gmail.com> Thanks Christian, I will check the code examples you posted tonight, your explanation makes it easier to understand. I can see there is a list with the deterministic functions in the specs [1] but not so sure about the BaseX specific functions. Is it possible to know if a function is deterministic or not? I tried file:read-text-lines("/path.txt")? is file:read-text-lines("/path.txt") but it doesn't work. George. [1] - https://www.w3.org/TR/xpath-functions-31/#dt-deterministic On 1/16/19 1:41 PM, Christian Gr?n wrote: > The reason for that: file:read-text-lines is a non-deterministic > function. Each invocation might yield different results (as the file > contents may change in the background). This is different with > non-deterministic function calls, such as fn:doc('abc.xml'). If you > call such a function repreatedly, it will always access the same > document, which has been opened and parsed by the first call of this > function. > >> 1) return count(file:read-text-lines($file, "UTF-8", false())) > Here, file processing will be iterative. > >> 2) let $data := file:read-text-lines($file, "UTF-8", false()) >> return count($data) > The file contents will be bound to $data, and counted in a second > step. If the expression of your let clause was deterministic, the > variable would be inlined, and the resulting query plan would be > identical to the one of your first query. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190116/4218b944/attachment.htm> From christian.gruen at gmail.com Wed Jan 16 13:11:03 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 16 Jan 2019 13:11:03 +0100 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <19b27b4b-e42c-acd7-fefe-303e20c6c19d@gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> <d33401d5-4906-46e6-a443-1805b8b28723@gmail.com> <CAP94bnN1wyQUdtoxdmmU_WCaBJgWjG0NxJLj+Ph1gvHVdxOTWg@mail.gmail.com> <19b27b4b-e42c-acd7-fefe-303e20c6c19d@gmail.com> Message-ID: <CAP94bnM2ZnWmUshwpKX4X_DL417jneOuMR8SSpu9qCzk0PnRqA@mail.gmail.com> > I can see there is a list with the deterministic functions in the specs [1] but not so sure about the BaseX specific functions. Is it possible to know if a function is deterministic or not? You can have a look into the appropriate Java class, and check for functions tagged with NDT: https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/query/func/Function.java If we have some more time, we might include this information in our Wiki. > I tried file:read-text-lines("/path.txt") is file:read-text-lines("/path.txt") but it doesn't work. The "is" operator operates only on nodes. From christian.gruen at gmail.com Wed Jan 16 15:31:39 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 16 Jan 2019 15:31:39 +0100 Subject: [basex-talk] many distinct namespaces In-Reply-To: <57bdf06bc00d50ea6a8f004e4af2bf7b3204b6d4.camel@fromoldbooks.org> References: <1541022422.882950422@f403.i.mail.ru> <39e0d303-f8eb-3180-a6ec-c9aacb3a44a5@le-tex.de> <CAP94bnNdWRD4CEBSGsy9bYK2oQ52XTeMf7-+rs2ZUGeXS3MMTw@mail.gmail.com> <57bdf06bc00d50ea6a8f004e4af2bf7b3204b6d4.camel@fromoldbooks.org> Message-ID: <CAP94bnNCJ4Ru1HPNkCs1T8jz6Tc+vrysyQ8cO-VTPhR8T__zGg@mail.gmail.com> Hi Liam, > did this ever happen? What exactly? ;) > * reserve 7 bits for the namespace, and 1 bit for "uses extended > namespace". In the top-bit-set objects only, add an extra byte. > * Use the first 2 bytes of the element name :) There are a few online resources with details on our chosen storage layout [1,2,3]. I think that we pretty much use every available bit for storing references and flags (except e.g. for text nodes, which have less metadata than elements or attributes), but maybe there?s indeed something specific that we could optimize? More suggestions are welcome. Cheers Christian [1] http://files.basex.org/publications/Gruen%20[2010],%20Storing%20and%20Querying%20Large%20XML%20Instances.pdf [2] http://docs.basex.org/wiki/Storage_Layout [3] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/Data.java From christian.gruen at gmail.com Wed Jan 16 16:25:29 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 16 Jan 2019 16:25:29 +0100 Subject: [basex-talk] Documentation suggestion In-Reply-To: <0d912a53-81f7-c496-d751-5f12654a36e4@gmail.com> References: <0d912a53-81f7-c496-d751-5f12654a36e4@gmail.com> Message-ID: <CAP94bnNy18iQ1vadegP9V-OTnAFmiCsQqZ6ZCj0uUW-Lo=u4Ag@mail.gmail.com> Hi Marco, > I suggest to add an explicit statement of which is the default value of > the combine option of map:merge. Done. If you are interested, feel free to get a little account for our Wiki! Christian From christian.gruen at gmail.com Wed Jan 16 16:27:56 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 16 Jan 2019 16:27:56 +0100 Subject: [basex-talk] Serialization issue with HTTP response In-Reply-To: <1dd0f533-5986-518f-a691-79ef75987857@gmail.com> References: <1dd0f533-5986-518f-a691-79ef75987857@gmail.com> Message-ID: <CAP94bnNGa-Z5DG+-e0p8cdFsQVxfTQggXg4Q=QkyESt2d=nvkw@mail.gmail.com> I hope I?ll be able to test this tomorrow. And your mail is yet another motivation to get the upcoming version 2 of the HTTP Client finished (http://expath.github.io/expath-cg/specs/http-client-2). On Tue, Jan 15, 2019 at 11:35 AM Marco Lettere <m.lettere at gmail.com> wrote: > > Hi all, > we have to deal with a third-party REST service which in case of error conditions returns this mime type: application/problem+json; charset=utf-8. > > I wrote this RestXQ [1] to mock it. Just copy it into restxq.xqm... > > When I call it like [2] I get [XPTY0004] Cannot convert xs:base64Binary to xs:string: "IntcImFcIjpcImJcIn0i". This is because the response is seen as a binary and this can be double checked calling with [3]. > I felt (maybe wrongly) that specializations of the mimetype had been faced in BaseX implementation of http:request somewhere back in time but I couldn't find anything in the mail archives. > > The strange thing however is that I get the same error when calling like [4] with override-mime-type set to text/json or application/json which instead I was expecting to work. > > Is the behaviour in this scenario expected or does this hide some inconsistency? > > Thanks, as usual, for the support. > > Regards, > > Marco. > > [1] > declare%rest:path("mc/dummy") > %rest:GET > %output:method("json") > function page:dummy() { > <rest:response> > <http:response status="400"> > <http:header name="Content-Type" value="application/problem+json; charset=utf-8"/> > </http:response> > </rest:response>, > '{"a":"b"}' > }; > > [2] > json:parse( > http:send-request(<http:request method="get"/>,"http://localhost:8984/mc/dummy")[2] > ) > > [3] > json:parse( > convert:binary-to-string( > http:send-request(<http:request method="get"/>,"http://localhost:8984/mc/dummy")[2] > ) > ) > > [4] > json:parse( > http:send-request(<http:request method="get" override-media-type="application/json"/>,"http://localhost:8984/mc/dummy")[2] > ) From m.lettere at gmail.com Wed Jan 16 16:36:17 2019 From: m.lettere at gmail.com (Marco Lettere) Date: Wed, 16 Jan 2019 16:36:17 +0100 Subject: [basex-talk] Serialization issue with HTTP response In-Reply-To: <CAP94bnNGa-Z5DG+-e0p8cdFsQVxfTQggXg4Q=QkyESt2d=nvkw@mail.gmail.com> References: <1dd0f533-5986-518f-a691-79ef75987857@gmail.com> <CAP94bnNGa-Z5DG+-e0p8cdFsQVxfTQggXg4Q=QkyESt2d=nvkw@mail.gmail.com> Message-ID: <539a788c-e22d-f3a7-bc46-c8fa876fa7ac@gmail.com> Just WoW! Looks great! M. On 16/01/19 16:27, Christian Gr?n wrote: > I hope I?ll be able to test this tomorrow. And your mail is yet > another motivation to get the upcoming version 2 of the HTTP Client > finished (http://expath.github.io/expath-cg/specs/http-client-2). > > > > On Tue, Jan 15, 2019 at 11:35 AM Marco Lettere <m.lettere at gmail.com> wrote: >> Hi all, >> we have to deal with a third-party REST service which in case of error conditions returns this mime type: application/problem+json; charset=utf-8. >> >> I wrote this RestXQ [1] to mock it. Just copy it into restxq.xqm... >> >> When I call it like [2] I get [XPTY0004] Cannot convert xs:base64Binary to xs:string: "IntcImFcIjpcImJcIn0i". This is because the response is seen as a binary and this can be double checked calling with [3]. >> I felt (maybe wrongly) that specializations of the mimetype had been faced in BaseX implementation of http:request somewhere back in time but I couldn't find anything in the mail archives. >> >> The strange thing however is that I get the same error when calling like [4] with override-mime-type set to text/json or application/json which instead I was expecting to work. >> >> Is the behaviour in this scenario expected or does this hide some inconsistency? >> >> Thanks, as usual, for the support. >> >> Regards, >> >> Marco. >> >> [1] >> declare%rest:path("mc/dummy") >> %rest:GET >> %output:method("json") >> function page:dummy() { >> <rest:response> >> <http:response status="400"> >> <http:header name="Content-Type" value="application/problem+json; charset=utf-8"/> >> </http:response> >> </rest:response>, >> '{"a":"b"}' >> }; >> >> [2] >> json:parse( >> http:send-request(<http:request method="get"/>,"http://localhost:8984/mc/dummy")[2] >> ) >> >> [3] >> json:parse( >> convert:binary-to-string( >> http:send-request(<http:request method="get"/>,"http://localhost:8984/mc/dummy")[2] >> ) >> ) >> >> [4] >> json:parse( >> http:send-request(<http:request method="get" override-media-type="application/json"/>,"http://localhost:8984/mc/dummy")[2] >> ) From m.lettere at gmail.com Wed Jan 16 16:37:00 2019 From: m.lettere at gmail.com (Marco Lettere) Date: Wed, 16 Jan 2019 16:37:00 +0100 Subject: [basex-talk] Documentation suggestion In-Reply-To: <CAP94bnNy18iQ1vadegP9V-OTnAFmiCsQqZ6ZCj0uUW-Lo=u4Ag@mail.gmail.com> References: <0d912a53-81f7-c496-d751-5f12654a36e4@gmail.com> <CAP94bnNy18iQ1vadegP9V-OTnAFmiCsQqZ6ZCj0uUW-Lo=u4Ag@mail.gmail.com> Message-ID: <febabd3b-20ed-bfc1-cef4-7c9b7df8c66d@gmail.com> Thanks. I had one but lost it like 99% of my account credentials ... :-( M. On 16/01/19 16:25, Christian Gr?n wrote: > Hi Marco, > >> I suggest to add an explicit statement of which is the default value of >> the combine option of map:merge. > Done. If you are interested, feel free to get a little account for our Wiki! > > Christian From christian.gruen at gmail.com Wed Jan 16 16:42:29 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 16 Jan 2019 16:42:29 +0100 Subject: [basex-talk] Documentation suggestion In-Reply-To: <febabd3b-20ed-bfc1-cef4-7c9b7df8c66d@gmail.com> References: <0d912a53-81f7-c496-d751-5f12654a36e4@gmail.com> <CAP94bnNy18iQ1vadegP9V-OTnAFmiCsQqZ6ZCj0uUW-Lo=u4Ag@mail.gmail.com> <febabd3b-20ed-bfc1-cef4-7c9b7df8c66d@gmail.com> Message-ID: <CAP94bnOE12Xt26K1nf8m3SCHd3Kx3BCiMP3+P2R=1fPAYyaHog@mail.gmail.com> > I had one but lost it like 99% of my account credentials ... :-( Sorry to hear that. ? I?ll be happy to enable another one for you. > M. > > On 16/01/19 16:25, Christian Gr?n wrote: > > Hi Marco, > > > >> I suggest to add an explicit statement of which is the default value of > >> the combine option of map:merge. > > Done. If you are interested, feel free to get a little account for our Wiki! > > > > Christian > > From joewiz at gmail.com Wed Jan 16 19:51:02 2019 From: joewiz at gmail.com (Joe Wicentowski) Date: Wed, 16 Jan 2019 13:51:02 -0500 Subject: [basex-talk] BaseX-specific serialization parameters Message-ID: <CAHwerk2w=rWPGHEsmkaMcOCJYtjuQL=b08rRfo-JFaRcsi_XQw@mail.gmail.com> Hi all, As I read the serialization spec ( https://www.w3.org/TR/xslt-xquery-serialization-31/#serparam), additional values for the "method" parameter and additional serialization parameters should use a non-null namespace URI (see quotes below). I believe BaseX's "csv" method or "tabulator" parameter ( http://docs.basex.org/wiki/Serialization) would fall into this category. Without such a non-null namespace URI, users may mistake BaseX-specific serialization features for those supported in the spec. The solution that the spec seems to suggest would be to use "basex:csv" or "basex:tabulator". I bet such a change would be unwelcome to those with code that uses the existing supported values, but aside from a disruptive cycle of deprecating existing values, etc., an alternative would be for the BaseX documentation to note this non-compliance with the spec. I realize this is nit-picky, but I thought it might be worth raising given the potential for confusion mentioned above, and should this discussion be of interest to anyone. Best, Joe Regarding values for the "method" parameter: > The value of the method parameter is an expanded QName. If the value has a null namespace URI, then the local name identifies a method specified in this document and MUST be one of xml, html, xhtml, text, json , or adaptive; in this case, the output method specified MUST be used for serializing. If the namespace URI is non-null, then it identifies an implementation-defined output method; the behavior in this case is not specified by this document. Regarding additional serialization parameters: > Implementations MAY define additional serialization parameters, and MAY allow users to do so. For this purpose, the name of a serialization parameter is considered to be a QName; the parameters listed above are QNames whose expanded-QName has a null namespace URI, while any additional serialization parameters that are either implementation-defined or defined by the host language MUST have names that are namespace-qualified. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190116/b33a6050/attachment.htm> From gsf.greece at gmail.com Wed Jan 16 22:34:16 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Wed, 16 Jan 2019 23:34:16 +0200 Subject: [basex-talk] file:read-text-lines performance In-Reply-To: <CAP94bnNapPuCW6_GZO=8h6Pn16uztEJEVSF8uq5mpHF-R+NQew@mail.gmail.com> References: <64ceb2f8-c073-17c9-b075-0244528f173e@gmail.com> <CAP94bnNsr8wZiD6Q8MeU8+UcyeA0RVBZBGtqnhOLZkuXpcs6jA@mail.gmail.com> <fe3f426e-a1c9-1a91-edf3-0a3f88e9cb2e@gmail.com> <CAP94bnPX++34BKABD4h8i7wEhaWweBYzuQZBua-qObGuspzSpg@mail.gmail.com> <03cbaa55-327f-12fb-1459-aedbbd5c519c@gmail.com> <CAP94bnNapPuCW6_GZO=8h6Pn16uztEJEVSF8uq5mpHF-R+NQew@mail.gmail.com> Message-ID: <ef370e12-36f0-9d7e-d772-314e3fe0b519@gmail.com> Just posting to say I'm having a lot of fun with the updated read-text-lines function. On 1/16/19 1:37 PM, Christian Gr?n wrote: > This code will potentially create thousands or millions of Java > threads. Maybe you are getting better results by splitting your input > into 4 or 8 parts, and process each part in a dedicated function. I refactored the code to the following, and it completes in 60 seconds, of which 20 are for counting the lines and only 40 seconds for parsing and returning the correct data!!! So I get a 3x improvement from multiple threads. I have no idea if it stresses the SSD at all. let $file := "/path/to/large.txt" let $count := prof:time(count(file:read-text-lines($file, "UTF-8", false())), "COUNTING: ") let $cpus := 15 let $parts := ($count div $cpus) => xs:integer() => trace("PER CORE: ") let $all := xquery:fork-join( for $cpu in 0 to $cpus return function() { let $offset := $cpu * $parts let $length := $parts for $line in file:read-text-lines($file, "UTF-8", false(), $offset, $length) return parse-json($line)?('obj1')?*?('obj2')?('obj3') }) => prof:time("CALCULATING: ") return distinct-values($all) > I would indeed assume that the following code? > > distinct-values( > for $line in file:read-text-lines($file, "UTF-8", false()) > return parse-json($line)?('object1')?*?('object2')?('object3') > ) > > ?will be most efficient, even if you process files of 100 GB or more > (especially with the new, iterative approach). Indeed, it is also using tiny amounts of memory and completes in the same time (120 seconds) with loading the whole file into memory? on a single core :) George. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190116/0c5a2bed/attachment.htm> From christian.gruen at gmail.com Thu Jan 17 10:22:36 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Thu, 17 Jan 2019 10:22:36 +0100 Subject: [basex-talk] BaseX-specific serialization parameters In-Reply-To: <CAHwerk2w=rWPGHEsmkaMcOCJYtjuQL=b08rRfo-JFaRcsi_XQw@mail.gmail.com> References: <CAHwerk2w=rWPGHEsmkaMcOCJYtjuQL=b08rRfo-JFaRcsi_XQw@mail.gmail.com> Message-ID: <CAP94bnMb6id3Wr8n9xr58f8_7rNKUorzXzyPBFUgB-OaZ_Z8ug@mail.gmail.com> Hi Joe, You are completely right; we ignored this specific rule of the specification in the past. One of the reasons is that there are many ways in BaseX how parameters can be supplied, and in some cases, no namespace prefixes can be declared. I have just updated our serialization [1]; I hope that the new wording is helpful. As you indicated, a change would require various changes in existing XQuery libraries. I?ll have some more thoughts on this; maybe we could tackle this with version 10 of BaseX. Thanks for your observation! Christian [1] http://docs.basex.org/wiki/Serialization > As I read the serialization spec (https://www.w3.org/TR/xslt-xquery-serialization-31/#serparam), additional values for the "method" parameter and additional serialization parameters should use a non-null namespace URI (see quotes below). > > I believe BaseX's "csv" method or "tabulator" parameter (http://docs.basex.org/wiki/Serialization) would fall into this category. Without such a non-null namespace URI, users may mistake BaseX-specific serialization features for those supported in the spec. The solution that the spec seems to suggest would be to use "basex:csv" or "basex:tabulator". I bet such a change would be unwelcome to those with code that uses the existing supported values, but aside from a disruptive cycle of deprecating existing values, etc., an alternative would be for the BaseX documentation to note this non-compliance with the spec. > > I realize this is nit-picky, but I thought it might be worth raising given the potential for confusion mentioned above, and should this discussion be of interest to anyone. > > Best, > Joe > > Regarding values for the "method" parameter: > > > The value of the method parameter is an expanded QName. If the value has a null namespace URI, then the local name identifies a method specified in this document and MUST be one of xml, html, xhtml, text, json , or adaptive; in this case, the output method specified MUST be used for serializing. If the namespace URI is non-null, then it identifies an implementation-defined output method; the behavior in this case is not specified by this document. > > Regarding additional serialization parameters: > > > Implementations MAY define additional serialization parameters, and MAY allow users to do so. For this purpose, the name of a serialization parameter is considered to be a QName; the parameters listed above are QNames whose expanded-QName has a null namespace URI, while any additional serialization parameters that are either implementation-defined or defined by the host language MUST have names that are namespace-qualified. From m.lettere at gmail.com Thu Jan 17 12:12:29 2019 From: m.lettere at gmail.com (Marco Lettere) Date: Thu, 17 Jan 2019 12:12:29 +0100 Subject: [basex-talk] Documentation suggestion In-Reply-To: <CAP94bnOE12Xt26K1nf8m3SCHd3Kx3BCiMP3+P2R=1fPAYyaHog@mail.gmail.com> References: <0d912a53-81f7-c496-d751-5f12654a36e4@gmail.com> <CAP94bnNy18iQ1vadegP9V-OTnAFmiCsQqZ6ZCj0uUW-Lo=u4Ag@mail.gmail.com> <febabd3b-20ed-bfc1-cef4-7c9b7df8c66d@gmail.com> <CAP94bnOE12Xt26K1nf8m3SCHd3Kx3BCiMP3+P2R=1fPAYyaHog@mail.gmail.com> Message-ID: <264ef72b-718f-d13a-cfac-e15e3eb7939d@gmail.com> One more ... the signature of hmac function at [1] looks messed up... [1] http://docs.basex.org/wiki/Cryptographic_Module#crypto:hmac On 16/01/19 16:42, Christian Gr?n wrote: >> I had one but lost it like 99% of my account credentials ... :-( > Sorry to hear that. ? I?ll be happy to enable another one for you. > > >> M. >> >> On 16/01/19 16:25, Christian Gr?n wrote: >>> Hi Marco, >>> >>>> I suggest to add an explicit statement of which is the default value of >>>> the combine option of map:merge. >>> Done. If you are interested, feel free to get a little account for our Wiki! >>> >>> Christian >> From wapiez at wendellpiez.com Fri Jan 18 22:10:28 2019 From: wapiez at wendellpiez.com (Wendell Piez) Date: Fri, 18 Jan 2019 16:10:28 -0500 Subject: [basex-talk] BaseX setup under Docker Message-ID: <CAAO_-xxkKF1RbHVxG1+YHSmqjX6E3=0dzK=WdSHRbyHJ8KG7kA@mail.gmail.com> Hello BaseX friends, So it happens that I am trying to discover how to set up BaseX to run in Docker. By no means am I skilled in system administration. There are some instructions at http://docs.basex.org/wiki/Docker which are very helpful, not least in confirming that I have a reasonable idea. The last example given on that page, however, gives me pause. It appears to install git into the Docker image, but the commentary suggests it's something else including a modification to CLASSPATH. I wonder if the commentary is not out of sync with the example. Yet at the same time I just need to know more -- inasmuch as I would like my image to have SaxonHE available, the most recent version freely available from maven or best available means. What would the syntax be for that. and how do I determine what that is (presumably for apk)? As well as potentially amending the CLASSPATH, if need be, to add the saxon.jar ... Thanks for any hints. relevant tips or links to Docker help :-) Regards, Wendell -- Wendell Piez | wendell -dot- piez -at- nist -dot- gov | http://www.wendellpiez.com pellucidliterature.org | github.com/wendellpiez | gitlab.coko.foundation/wendell - pausepress.org From alex at holupirek.de Fri Jan 18 22:18:50 2019 From: alex at holupirek.de (Alexander Holupirek) Date: Fri, 18 Jan 2019 22:18:50 +0100 Subject: [basex-talk] BaseX setup under Docker In-Reply-To: <CAAO_-xxkKF1RbHVxG1+YHSmqjX6E3=0dzK=WdSHRbyHJ8KG7kA@mail.gmail.com> References: <CAAO_-xxkKF1RbHVxG1+YHSmqjX6E3=0dzK=WdSHRbyHJ8KG7kA@mail.gmail.com> Message-ID: <857DB92C-013B-4BFA-971D-593A86C2D7EC@holupirek.de> > On 18. Jan 2019, at 22:10, Wendell Piez <wapiez at wendellpiez.com> wrote: > > Hello BaseX friends, > > So it happens that I am trying to discover how to set up BaseX to run > in Docker. By no means am I skilled in system administration. > > There are some instructions at http://docs.basex.org/wiki/Docker which > are very helpful, not least in confirming that I have a reasonable > idea. > > The last example given on that page, however, gives me pause. It > appears to install git into the Docker image, but the commentary > suggests it's something else including a modification to CLASSPATH. I > wonder if the commentary is not out of sync with the example. > > Yet at the same time I just need to know more -- inasmuch as I would > like my image to have SaxonHE available, the most recent version > freely available from maven or best available means. What would the > syntax be for that. and how do I determine what that is (presumably > for apk)? As well as potentially amending the CLASSPATH, if need be, > to add the saxon.jar ... > > Thanks for any hints. relevant tips or links to Docker help :-) > > Regards, Wendell > > -- > Wendell Piez | wendell -dot- piez -at- nist -dot- gov | > http://www.wendellpiez.com > pellucidliterature.org | github.com/wendellpiez | > gitlab.coko.foundation/wendell - pausepress.org Hi Wendell, you're right, the documentation is out of sync. Comparing the history of Wiki edits explains the mistakes: http://docs.basex.org/index.php?title=Docker&diff=prev&oldid=13817 It is not Debian anymore, it is Alpine Linux. And it is not a lib for image manipulation that is installed, but git. And there is no CLASSPATH update. We'll have a look at it and revise the paragraph. All the best Alex From wapiez at wendellpiez.com Sat Jan 19 21:10:20 2019 From: wapiez at wendellpiez.com (Wendell Piez) Date: Sat, 19 Jan 2019 15:10:20 -0500 Subject: [basex-talk] BaseX setup under Docker In-Reply-To: <857DB92C-013B-4BFA-971D-593A86C2D7EC@holupirek.de> References: <CAAO_-xxkKF1RbHVxG1+YHSmqjX6E3=0dzK=WdSHRbyHJ8KG7kA@mail.gmail.com> <857DB92C-013B-4BFA-971D-593A86C2D7EC@holupirek.de> Message-ID: <CAAO_-xw_cBGK9-odjeg7asMLgjVJV1d+c2Krs3o4Y7L7SCnYqA@mail.gmail.com> Alex, thanks for confirming, that is awesome, I'll keep my eye open for the edits. Thanks, Wendell On Fri, Jan 18, 2019 at 4:18 PM Alexander Holupirek <alex at holupirek.de> wrote: > > > > On 18. Jan 2019, at 22:10, Wendell Piez <wapiez at wendellpiez.com> wrote: > > > > Hello BaseX friends, > > > > So it happens that I am trying to discover how to set up BaseX to run > > in Docker. By no means am I skilled in system administration. > > > > There are some instructions at http://docs.basex.org/wiki/Docker which > > are very helpful, not least in confirming that I have a reasonable > > idea. > > > > The last example given on that page, however, gives me pause. It > > appears to install git into the Docker image, but the commentary > > suggests it's something else including a modification to CLASSPATH. I > > wonder if the commentary is not out of sync with the example. > > > > Yet at the same time I just need to know more -- inasmuch as I would > > like my image to have SaxonHE available, the most recent version > > freely available from maven or best available means. What would the > > syntax be for that. and how do I determine what that is (presumably > > for apk)? As well as potentially amending the CLASSPATH, if need be, > > to add the saxon.jar ... > > > > Thanks for any hints. relevant tips or links to Docker help :-) > > > > Regards, Wendell > > > > -- > > Wendell Piez | wendell -dot- piez -at- nist -dot- gov | > > http://www.wendellpiez.com > > pellucidliterature.org | github.com/wendellpiez | > > gitlab.coko.foundation/wendell - pausepress.org > > Hi Wendell, > > you're right, the documentation is out of sync. > Comparing the history of Wiki edits explains the mistakes: > > http://docs.basex.org/index.php?title=Docker&diff=prev&oldid=13817 > > It is not Debian anymore, it is Alpine Linux. > And it is not a lib for image manipulation that is installed, but git. > And there is no CLASSPATH update. > > We'll have a look at it and revise the paragraph. > > All the best > Alex > > -- Wendell Piez | wendell -dot- piez -at- nist -dot- gov | http://www.wendellpiez.com pellucidliterature.org | github.com/wendellpiez | gitlab.coko.foundation/wendell - pausepress.org From christian.gruen at gmail.com Tue Jan 22 10:06:33 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Tue, 22 Jan 2019 10:06:33 +0100 Subject: [basex-talk] BaseX 9.1.2: Maintenance Release Message-ID: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> Hi all, A new BaseX maintenance release is available. Minor bugs have been fixed, and some performance tweaks have been added. In particular, access to large WebDAV directories should be faster now. As usual, you find the latest version on our homepage basex.org. Maven artifacts have been added, various other distributions will follow soon. Have fun everyone, Christian, BaseX Team From hrennau at yahoo.de Tue Jan 22 15:15:08 2019 From: hrennau at yahoo.de (Hans-Juergen Rennau) Date: Tue, 22 Jan 2019 14:15:08 +0000 (UTC) Subject: [basex-talk] XSD 1.1 - Element not found References: <1392102084.2889486.1548166508668.ref@mail.yahoo.com> Message-ID: <1392102084.2889486.1548166508668@mail.yahoo.com> Dear BaseX team, trying to use XSD 1.1, I encounter a problem. As indicated in the documentation, I added to the lib folder the jar files from the newest Xerxes distribution (Xerces-J-bin.2.12.0-xml-schema-1.1.zip, download from http://xerces.apache.org/mirrors.cgi#binary ). The problem: when using the schema attribute vc:minVersion="1.1", validation fails with the message: 1:41: cvc-elt.1.a: Cannot find the declaration of element 'Foo'. But the element is there, and the namespace matches the target namespace. When changing the attribute vc:minVersion="1.0", validation works, but, of course, cannot cope with 1.1 features. (The same applies to validation after removing the attribute.) I tried both - copying only the four jar files named in the documention, cupv10k-runtime.jar org.eclipse.wst.xml.xpath2.processor_1.2.0.jar xercesImpl.jar xml-apis.jar and then I tried copying all jars: cupv10k-runtime.jar icu4j.jar org.eclipse.wst.xml.xpath2.processor_1.2.0.jar resolver.jar serializer.jar xercesImpl.jar xercesSamples.jar xml-apis.jar Any suggestions would be welcome! With kind regards,Hans-J?rgen PS: I use Java version 1.8.0_181-b13 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/e1e237ab/attachment.htm> From D.Zimmel at ESVmedien.de Tue Jan 22 15:43:21 2019 From: D.Zimmel at ESVmedien.de (Zimmel, Daniel) Date: Tue, 22 Jan 2019 14:43:21 +0000 Subject: [basex-talk] XSD 1.1 - Element not found In-Reply-To: <1392102084.2889486.1548166508668@mail.yahoo.com> References: <1392102084.2889486.1548166508668.ref@mail.yahoo.com> <1392102084.2889486.1548166508668@mail.yahoo.com> Message-ID: <94795773643d4123bd38e2a9c07a57d3@IRIS3.esvmedien.de> With 1.1, I found that validate:xsd() requires you to explicitly give ?1.1? as third parameter. Having vs:minVersion is not being honored by BaseX. Daniel Von: Hans-Juergen Rennau [mailto:hrennau at yahoo.de] Gesendet: Dienstag, 22. Januar 2019 15:15 An: BaseX Support Betreff: [basex-talk] XSD 1.1 - Element not found Dear BaseX team, trying to use XSD 1.1, I encounter a problem. As indicated in the documentation, I added to the lib folder the jar files from the newest Xerxes distribution (Xerces-J-bin.2.12.0-xml-schema-1.1.zip, download from http://xerces.apache.org/mirrors.cgi#binary ). The problem: when using the schema attribute vc:minVersion="1.1", validation fails with the message: 1:41: cvc-elt.1.a: Cannot find the declaration of element 'Foo'. But the element is there, and the namespace matches the target namespace. When changing the attribute vc:minVersion="1.0", validation works, but, of course, cannot cope with 1.1 features. (The same applies to validation after removing the attribute.) I tried both - copying only the four jar files named in the documention, cupv10k-runtime.jar org.eclipse.wst.xml.xpath2.processor_1.2.0.jar xercesImpl.jar xml-apis.jar and then I tried copying all jars: cupv10k-runtime.jar icu4j.jar org.eclipse.wst.xml.xpath2.processor_1.2.0.jar resolver.jar serializer.jar xercesImpl.jar xercesSamples.jar xml-apis.jar Any suggestions would be welcome! With kind regards, Hans-J?rgen PS: I use Java version 1.8.0_181-b13 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/431ab8b3/attachment.htm> From hrennau at yahoo.de Tue Jan 22 17:09:15 2019 From: hrennau at yahoo.de (Hans-Juergen Rennau) Date: Tue, 22 Jan 2019 16:09:15 +0000 (UTC) Subject: [basex-talk] XSD 1.1 - Element not found In-Reply-To: <94795773643d4123bd38e2a9c07a57d3@IRIS3.esvmedien.de> References: <1392102084.2889486.1548166508668.ref@mail.yahoo.com> <1392102084.2889486.1548166508668@mail.yahoo.com> <94795773643d4123bd38e2a9c07a57d3@IRIS3.esvmedien.de> Message-ID: <606343237.3036261.1548173355568@mail.yahoo.com> Many thanks, Daniel! Now it works. Kind regards,Hans-J?rgen Am Dienstag, 22. Januar 2019, 15:43:23 MEZ hat Zimmel, Daniel <D.Zimmel at ESVmedien.de> Folgendes geschrieben: #yiv5615938040 #yiv5615938040 -- _filtered #yiv5615938040 {font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv5615938040 {panose-1:2 2 6 9 4 2 5 8 3 4;} _filtered #yiv5615938040 {panose-1:2 2 6 9 4 2 5 8 3 4;} _filtered #yiv5615938040 {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv5615938040 {panose-1:2 2 6 9 4 2 5 8 3 4;}#yiv5615938040 #yiv5615938040 p.yiv5615938040MsoNormal, #yiv5615938040 li.yiv5615938040MsoNormal, #yiv5615938040 div.yiv5615938040MsoNormal {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;font-family:New;}#yiv5615938040 a:link, #yiv5615938040 span.yiv5615938040MsoHyperlink {color:blue;text-decoration:underline;}#yiv5615938040 a:visited, #yiv5615938040 span.yiv5615938040MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv5615938040 span.yiv5615938040E-MailFormatvorlage17 {color:#1F497D;}#yiv5615938040 .yiv5615938040MsoChpDefault {font-size:10.0pt;} _filtered #yiv5615938040 {margin:70.85pt 70.85pt 2.0cm 70.85pt;}#yiv5615938040 div.yiv5615938040WordSection1 {}#yiv5615938040 With 1.1, I found that validate:xsd() requires you to explicitly give ?1.1? as third parameter. Having vs:minVersion is not being honored by BaseX. ? Daniel ? Von: Hans-Juergen Rennau [mailto:hrennau at yahoo.de] Gesendet: Dienstag, 22. Januar 2019 15:15 An: BaseX Support Betreff: [basex-talk] XSD 1.1 - Element not found ? Dear BaseX team, ? trying to use XSD 1.1, I encounter a problem. ? As indicated in the documentation, I added to the lib folder the jar files from the newest Xerxes distribution (Xerces-J-bin.2.12.0-xml-schema-1.1.zip, download fromhttp://xerces.apache.org/mirrors.cgi#binary ). ? The problem: when using the schema attribute vc:minVersion="1.1", validation fails with the message: ? 1:41: cvc-elt.1.a: Cannot find the declaration of element 'Foo'. ? But the element is there, and the namespace matches the target namespace. When changing the attribute vc:minVersion="1.0", validation works, but, of course, cannot cope with 1.1 features. (The same applies to validation after removing the attribute.) ? I tried both - copying only the four jar files named in the documention, ? cupv10k-runtime.jar org.eclipse.wst.xml.xpath2.processor_1.2.0.jar xercesImpl.jar xml-apis.jar ? and then I tried copying all jars: ? cupv10k-runtime.jar icu4j.jar org.eclipse.wst.xml.xpath2.processor_1.2.0.jar resolver.jar serializer.jar xercesImpl.jar xercesSamples.jar xml-apis.jar ? Any suggestions would be welcome! ? With kind regards, Hans-J?rgen ? PS: I use Java version 1.8.0_181-b13 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/068ead7b/attachment.htm> From rickhg12hs at gmail.com Tue Jan 22 21:51:22 2019 From: rickhg12hs at gmail.com (Rick Graham) Date: Tue, 22 Jan 2019 21:51:22 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use Message-ID: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> Hello, Thanks again, as always, for a great product. I just installed BaseX v9.1.2 (upgrading from a previous v9.1.2 snapshot), launched the GUI and then got interrupted. When I returned, almost all of the JVM's memory was being used. I hit "GC" several times but it didn't seem to help. I had no database loaded/open. Seems like some memory isn't getting freed properly. Here's my "INFO" General Information: > Version: 9.1.2 > Used Memory: 1593 MB > Global options: > AUTHMETHOD: Basic > CACHETIMEOUT: 3600 > DBPATH: /usr/local/src/basex/data > DEBUG: false > FAIRLOCK: false > HOST: localhost > HTTPLOCAL: false > IGNORECERT: false > IGNOREHOSTNAME: false > KEEPALIVE: 600 > LANG: English > LANGKEYS: false > LOG: true > LOGMSGMAXLEN: 1000 > LOGPATH: .logs > NONPROXYHOSTS: > PARALLEL: 8 > PARSERESTXQ: 3 > PASSWORD: > PORT: 1984 > PROXYHOST: > PROXYPORT: 0 > REPOPATH: /usr/local/src/basex/repo > RESTPATH: > RESTXQPATH: > SERVERHOST: > SERVERPORT: 1984 > STOPPORT: 8985 > TIMEOUT: 30 > USER: > WEBPATH: /usr/local/src/basex/webapp > Local options > ADDARCHIVES: true > ADDCACHE: false > ADDRAW: false > ARCHIVENAME: false > ATTRINCLUDE: > ATTRINDEX: true > AUTOFLUSH: true > AUTOOPTIMIZE: false > BINDINGS: > CASESENS: false > CATFILE: > CHECKSTRINGS: true > CHOP: true > COMPPLAN: true > COPYNODE: true > CREATEFILTER: *.xml > CREATEONLY: false > CSVPARSER: > DEFAULTDB: false > DIACRITICS: false > DOTCOMPACT: false > DOTPLAN: false > DTD: false > ENFORCEINDEX: false > EXPORTER: > FORCECREATE: false > FTINCLUDE: > FTINDEX: false > HTMLPARSER: > INLINELIMIT: 100 > INTPARSE: false > JSONPARSER: > LANGUAGE: en > LSERROR: 0 > MAINMEM: false > MAXCATS: 100 > MAXLEN: 96 > MAXSTAT: 30 > MIXUPDATES: false > PARSER: xml > QUERYINFO: true > RUNQUERY: true > RUNS: 1 > SERIALIZE: true > SERIALIZER: > SKIPCORRUPT: false > SPLITSIZE: 0 > STEMMING: false > STOPWORDS: > STRIPNS: false > TAILCALLS: 256 > TEXTINCLUDE: > TEXTINDEX: true > TEXTPARSER: > TOKENINCLUDE: > TOKENINDEX: false > UPDINDEX: false > WRITEBACK: false > XINCLUDE: true > XMLPLAN: true Best regards, Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/6e74899f/attachment.htm> From christian.gruen at gmail.com Tue Jan 22 22:02:53 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Tue, 22 Jan 2019 22:02:53 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> Message-ID: <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> Hm, I couldn?t reproduce this out of the box. Does the problem only occur in your GUI instance? Did you check out the behavior on command-line or in the DBA as well? On Tue, Jan 22, 2019 at 9:51 PM Rick Graham <rickhg12hs at gmail.com> wrote: > > Hello, > > Thanks again, as always, for a great product. > > I just installed BaseX v9.1.2 (upgrading from a previous v9.1.2 snapshot), launched the GUI and then got interrupted. When I returned, almost all of the JVM's memory was being used. I hit "GC" several times but it didn't seem to help. I had no database loaded/open. Seems like some memory isn't getting freed properly. > > Here's my "INFO" > >> General Information: >> Version: 9.1.2 >> Used Memory: 1593 MB >> Global options: >> AUTHMETHOD: Basic >> CACHETIMEOUT: 3600 >> DBPATH: /usr/local/src/basex/data >> DEBUG: false >> FAIRLOCK: false >> HOST: localhost >> HTTPLOCAL: false >> IGNORECERT: false >> IGNOREHOSTNAME: false >> KEEPALIVE: 600 >> LANG: English >> LANGKEYS: false >> LOG: true >> LOGMSGMAXLEN: 1000 >> LOGPATH: .logs >> NONPROXYHOSTS: >> PARALLEL: 8 >> PARSERESTXQ: 3 >> PASSWORD: >> PORT: 1984 >> PROXYHOST: >> PROXYPORT: 0 >> REPOPATH: /usr/local/src/basex/repo >> RESTPATH: >> RESTXQPATH: >> SERVERHOST: >> SERVERPORT: 1984 >> STOPPORT: 8985 >> TIMEOUT: 30 >> USER: >> WEBPATH: /usr/local/src/basex/webapp >> Local options >> ADDARCHIVES: true >> ADDCACHE: false >> ADDRAW: false >> ARCHIVENAME: false >> ATTRINCLUDE: >> ATTRINDEX: true >> AUTOFLUSH: true >> AUTOOPTIMIZE: false >> BINDINGS: >> CASESENS: false >> CATFILE: >> CHECKSTRINGS: true >> CHOP: true >> COMPPLAN: true >> COPYNODE: true >> CREATEFILTER: *.xml >> CREATEONLY: false >> CSVPARSER: >> DEFAULTDB: false >> DIACRITICS: false >> DOTCOMPACT: false >> DOTPLAN: false >> DTD: false >> ENFORCEINDEX: false >> EXPORTER: >> FORCECREATE: false >> FTINCLUDE: >> FTINDEX: false >> HTMLPARSER: >> INLINELIMIT: 100 >> INTPARSE: false >> JSONPARSER: >> LANGUAGE: en >> LSERROR: 0 >> MAINMEM: false >> MAXCATS: 100 >> MAXLEN: 96 >> MAXSTAT: 30 >> MIXUPDATES: false >> PARSER: xml >> QUERYINFO: true >> RUNQUERY: true >> RUNS: 1 >> SERIALIZE: true >> SERIALIZER: >> SKIPCORRUPT: false >> SPLITSIZE: 0 >> STEMMING: false >> STOPWORDS: >> STRIPNS: false >> TAILCALLS: 256 >> TEXTINCLUDE: >> TEXTINDEX: true >> TEXTPARSER: >> TOKENINCLUDE: >> TOKENINDEX: false >> UPDINDEX: false >> WRITEBACK: false >> XINCLUDE: true >> XMLPLAN: true > > > Best regards, > Richard > From rickhg12hs at gmail.com Tue Jan 22 22:08:07 2019 From: rickhg12hs at gmail.com (Rick Graham) Date: Tue, 22 Jan 2019 22:08:07 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> Message-ID: <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> No, I haven't tried the other apps yet. I've only ever used the GUI and the command-line. I'll try that soon. Meanwhile, here's some Java error traces from launching the GUI. $ /usr/local/src/basex/bin/basexgui > java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC > overhead limit exceeded > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at javax.swing.SwingWorker.get(SwingWorker.java:602) > at org.basex.gui.layout.GUIWorker$1.done(GUIWorker.java:40) > at javax.swing.SwingWorker$5.run(SwingWorker.java:737) > at > javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.run(SwingWorker.java:832) > at sun.swing.AccumulativeRunnable.run(AccumulativeRunnable.java:112) > at > javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.actionPerformed(SwingWorker.java:842) > at javax.swing.Timer.fireActionPerformed(Timer.java:313) > at javax.swing.Timer$DoPostEvent.run(Timer.java:245) > at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311) > at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758) > at java.awt.EventQueue.access$500(EventQueue.java:97) > at java.awt.EventQueue$3.run(EventQueue.java:709) > at java.awt.EventQueue$3.run(EventQueue.java:703) > at java.security.AccessController.doPrivileged(Native Method) > at > java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74) > at java.awt.EventQueue.dispatchEvent(EventQueue.java:728) > at > java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205) > at > java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) > at > java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105) > at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) > at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93) > at java.awt.EventDispatchThread.run(EventDispatchThread.java:82) > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > at java.util.Arrays.copyOf(Arrays.java:3332) > at java.lang.String.<init>(String.java:166) > at org.basex.util.Token.string(Token.java:115) > at org.basex.util.TokenBuilder.toString(TokenBuilder.java:405) > at org.basex.io.IOFile.add(IOFile.java:528) > at org.basex.io.IOFile.create(IOFile.java:500) > at org.basex.io.IOFile.<init>(IOFile.java:74) > at org.basex.io.IOFile.<init>(IOFile.java:39) > at org.basex.io.IOFile.children(IOFile.java:226) > at org.basex.io.IOFile.children(IOFile.java:193) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java > heap space > at java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75) > at com.sun.java.swing.plaf.gtk.GTKEngine.finishPainting(GTKEngine.java:591) > at com.sun.java.swing.plaf.gtk.GTKEngine.finishPainting(GTKEngine.java:582) > at > com.sun.java.swing.plaf.gtk.GTKPainter.paintTabbedPaneContentBackground(GTKPainter.java:866) > at > javax.swing.plaf.synth.SynthTabbedPaneUI.paintContentBorder(SynthTabbedPaneUI.java:731) > at > javax.swing.plaf.synth.SynthTabbedPaneUI.paint(SynthTabbedPaneUI.java:486) > at > javax.swing.plaf.synth.SynthTabbedPaneUI.update(SynthTabbedPaneUI.java:376) > at javax.swing.JComponent.paintComponent(JComponent.java:780) > at javax.swing.JComponent.paint(JComponent.java:1056) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at org.basex.gui.view.ViewContainer.paint(ViewContainer.java:221) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) > at javax.swing.JComponent.paintChildren(JComponent.java:889) > at javax.swing.JComponent.paint(JComponent.java:1065) On Tue, Jan 22, 2019 at 10:03 PM Christian Gr?n <christian.gruen at gmail.com> wrote: > Hm, I couldn?t reproduce this out of the box. Does the problem only > occur in your GUI instance? Did you check out the behavior on > command-line or in the DBA as well? > > > On Tue, Jan 22, 2019 at 9:51 PM Rick Graham <rickhg12hs at gmail.com> wrote: > > > > Hello, > > > > Thanks again, as always, for a great product. > > > > I just installed BaseX v9.1.2 (upgrading from a previous v9.1.2 > snapshot), launched the GUI and then got interrupted. When I returned, > almost all of the JVM's memory was being used. I hit "GC" several times > but it didn't seem to help. I had no database loaded/open. Seems like > some memory isn't getting freed properly. > > > > Here's my "INFO" > > > >> General Information: > >> Version: 9.1.2 > >> Used Memory: 1593 MB > >> Global options: > >> AUTHMETHOD: Basic > >> CACHETIMEOUT: 3600 > >> DBPATH: /usr/local/src/basex/data > >> DEBUG: false > >> FAIRLOCK: false > >> HOST: localhost > >> HTTPLOCAL: false > >> IGNORECERT: false > >> IGNOREHOSTNAME: false > >> KEEPALIVE: 600 > >> LANG: English > >> LANGKEYS: false > >> LOG: true > >> LOGMSGMAXLEN: 1000 > >> LOGPATH: .logs > >> NONPROXYHOSTS: > >> PARALLEL: 8 > >> PARSERESTXQ: 3 > >> PASSWORD: > >> PORT: 1984 > >> PROXYHOST: > >> PROXYPORT: 0 > >> REPOPATH: /usr/local/src/basex/repo > >> RESTPATH: > >> RESTXQPATH: > >> SERVERHOST: > >> SERVERPORT: 1984 > >> STOPPORT: 8985 > >> TIMEOUT: 30 > >> USER: > >> WEBPATH: /usr/local/src/basex/webapp > >> Local options > >> ADDARCHIVES: true > >> ADDCACHE: false > >> ADDRAW: false > >> ARCHIVENAME: false > >> ATTRINCLUDE: > >> ATTRINDEX: true > >> AUTOFLUSH: true > >> AUTOOPTIMIZE: false > >> BINDINGS: > >> CASESENS: false > >> CATFILE: > >> CHECKSTRINGS: true > >> CHOP: true > >> COMPPLAN: true > >> COPYNODE: true > >> CREATEFILTER: *.xml > >> CREATEONLY: false > >> CSVPARSER: > >> DEFAULTDB: false > >> DIACRITICS: false > >> DOTCOMPACT: false > >> DOTPLAN: false > >> DTD: false > >> ENFORCEINDEX: false > >> EXPORTER: > >> FORCECREATE: false > >> FTINCLUDE: > >> FTINDEX: false > >> HTMLPARSER: > >> INLINELIMIT: 100 > >> INTPARSE: false > >> JSONPARSER: > >> LANGUAGE: en > >> LSERROR: 0 > >> MAINMEM: false > >> MAXCATS: 100 > >> MAXLEN: 96 > >> MAXSTAT: 30 > >> MIXUPDATES: false > >> PARSER: xml > >> QUERYINFO: true > >> RUNQUERY: true > >> RUNS: 1 > >> SERIALIZE: true > >> SERIALIZER: > >> SKIPCORRUPT: false > >> SPLITSIZE: 0 > >> STEMMING: false > >> STOPWORDS: > >> STRIPNS: false > >> TAILCALLS: 256 > >> TEXTINCLUDE: > >> TEXTINDEX: true > >> TEXTPARSER: > >> TOKENINCLUDE: > >> TOKENINDEX: false > >> UPDINDEX: false > >> WRITEBACK: false > >> XINCLUDE: true > >> XMLPLAN: true > > > > > > Best regards, > > Richard > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/a5d7331c/attachment.htm> From christian.gruen at gmail.com Tue Jan 22 22:21:00 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Tue, 22 Jan 2019 22:21:00 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> Message-ID: <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> > > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >> > Looks like a endless loop that is caused by parsing the files in your project directory. Do you possibly have any symbolic links? Can you reproduce the problem with a completely fresh BaseX zip archive? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/4af6c2bc/attachment.htm> From rickhg12hs at gmail.com Tue Jan 22 22:56:06 2019 From: rickhg12hs at gmail.com (Rick Graham) Date: Tue, 22 Jan 2019 22:56:06 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> Message-ID: <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> The command-line seemed to be operating normally. What exactly is/are my project directories? I attached to the running GUI instance `strace -f -e trace=stat -p 13368` and it has infinite repetitions of: [pid 13436] > stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", > 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) What's going on here? On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n <christian.gruen at gmail.com> wrote: > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>> >> > Looks like a endless loop that is caused by parsing the files in your > project directory. Do you possibly have any symbolic links? > > Can you reproduce the problem with a completely fresh BaseX zip archive? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/b50ca71e/attachment.htm> From bdysonsmith at gmail.com Tue Jan 22 23:10:18 2019 From: bdysonsmith at gmail.com (Bridger Dyson-Smith) Date: Tue, 22 Jan 2019 17:10:18 -0500 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> Message-ID: <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> Hi Rick, et al, I think (but am not 100% sure) that the GUI defaults to looking through your home directory on startup. So, somewhere in `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. I think you might be able to circumvent this problem by finding `.basexgui` - it would probably be close to wherever you started the GUI from on your filesystem. I think you can edit some of the PATHS there and that may help? Again, I'm not sure. HTH! Best, Bridger On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com> wrote: > The command-line seemed to be operating normally. > > What exactly is/are my project directories? > > I attached to the running GUI instance `strace -f -e trace=stat -p 13368` > and it has infinite repetitions of: > > [pid 13436] >> stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", >> 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) > > > What's going on here? > > On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n <christian.gruen at gmail.com> > wrote: > >> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>> >>> >> Looks like a endless loop that is caused by parsing the files in your >> project directory. Do you possibly have any symbolic links? >> >> Can you reproduce the problem with a completely fresh BaseX zip archive? >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/2bbea490/attachment.htm> From rickhg12hs at gmail.com Tue Jan 22 23:17:27 2019 From: rickhg12hs at gmail.com (Rick Graham) Date: Tue, 22 Jan 2019 23:17:27 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> Message-ID: <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> Thanks Bridger! Indeed, I quit basexgui and manually edited .basexgui to set the project directory to a newly created empty directory. basexgui seems normal/stable after that. I rarely, as in almost never, use wine but I didn't have this issue with previous versions of BaseX. Something seems unexpected here. On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith <bdysonsmith at gmail.com> wrote: > Hi Rick, et al, > I think (but am not 100% sure) that the GUI defaults to looking through > your home directory on startup. So, somewhere in > `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. > > I think you might be able to circumvent this problem by finding > `.basexgui` - it would probably be close to wherever you started the GUI > from on your filesystem. I think you can edit some of the PATHS there and > that may help? > > Again, I'm not sure. HTH! > Best, > Bridger > > On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com> wrote: > >> The command-line seemed to be operating normally. >> >> What exactly is/are my project directories? >> >> I attached to the running GUI instance `strace -f -e trace=stat -p 13368` >> and it has infinite repetitions of: >> >> [pid 13436] >>> stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", >>> 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) >> >> >> What's going on here? >> >> On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n < >> christian.gruen at gmail.com> wrote: >> >>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>> >>>> >>> Looks like a endless loop that is caused by parsing the files in your >>> project directory. Do you possibly have any symbolic links? >>> >>> Can you reproduce the problem with a completely fresh BaseX zip archive? >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/9c9f689d/attachment.htm> From bdysonsmith at gmail.com Tue Jan 22 23:22:16 2019 From: bdysonsmith at gmail.com (Bridger Dyson-Smith) Date: Tue, 22 Jan 2019 17:22:16 -0500 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> Message-ID: <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> Glad that helped :) I see this when I start from a fresh install vs expanding the ZIP into the same directory. On Tue, Jan 22, 2019, 5:17 PM Rick Graham <rickhg12hs at gmail.com wrote: > Thanks Bridger! > > Indeed, I quit basexgui and manually edited .basexgui to set the project > directory to a newly created empty directory. basexgui seems normal/stable > after that. > > I rarely, as in almost never, use wine but I didn't have this issue with > previous versions of BaseX. Something seems unexpected here. > > > On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith < > bdysonsmith at gmail.com> wrote: > >> Hi Rick, et al, >> I think (but am not 100% sure) that the GUI defaults to looking through >> your home directory on startup. So, somewhere in >> `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. >> >> I think you might be able to circumvent this problem by finding >> `.basexgui` - it would probably be close to wherever you started the GUI >> from on your filesystem. I think you can edit some of the PATHS there and >> that may help? >> >> Again, I'm not sure. HTH! >> Best, >> Bridger >> >> On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com> wrote: >> >>> The command-line seemed to be operating normally. >>> >>> What exactly is/are my project directories? >>> >>> I attached to the running GUI instance `strace -f -e trace=stat -p >>> 13368` and it has infinite repetitions of: >>> >>> [pid 13436] >>>> stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", >>>> 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) >>> >>> >>> What's going on here? >>> >>> On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n < >>> christian.gruen at gmail.com> wrote: >>> >>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >>>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>> >>>>> >>>> Looks like a endless loop that is caused by parsing the files in your >>>> project directory. Do you possibly have any symbolic links? >>>> >>>> Can you reproduce the problem with a completely fresh BaseX zip archive? >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/15481b70/attachment.htm> From christian.gruen at gmail.com Tue Jan 22 23:36:19 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Tue, 22 Jan 2019 23:36:19 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> Message-ID: <CAP94bnPh_MLZDoFTiBuApjXx=mdvXgYG5vMn_y106iJnhmAzYg@mail.gmail.com> Good to hear that! I can?t recollect that something particular has changed in version 9.1.2, regarding the scanning of project files, but I?ll have some thoughts how we can trace and interrupt such loops (or ignore symbolic links instead). Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith <bdysonsmith at gmail.com> geschrieben: > Glad that helped :) > > I see this when I start from a fresh install vs expanding the ZIP into the > same directory. > > On Tue, Jan 22, 2019, 5:17 PM Rick Graham <rickhg12hs at gmail.com wrote: > >> Thanks Bridger! >> >> Indeed, I quit basexgui and manually edited .basexgui to set the project >> directory to a newly created empty directory. basexgui seems normal/stable >> after that. >> >> I rarely, as in almost never, use wine but I didn't have this issue with >> previous versions of BaseX. Something seems unexpected here. >> >> >> On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith < >> bdysonsmith at gmail.com> wrote: >> >>> Hi Rick, et al, >>> I think (but am not 100% sure) that the GUI defaults to looking through >>> your home directory on startup. So, somewhere in >>> `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. >>> >>> I think you might be able to circumvent this problem by finding >>> `.basexgui` - it would probably be close to wherever you started the GUI >>> from on your filesystem. I think you can edit some of the PATHS there and >>> that may help? >>> >>> Again, I'm not sure. HTH! >>> Best, >>> Bridger >>> >>> On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com> >>> wrote: >>> >>>> The command-line seemed to be operating normally. >>>> >>>> What exactly is/are my project directories? >>>> >>>> I attached to the running GUI instance `strace -f -e trace=stat -p >>>> 13368` and it has infinite repetitions of: >>>> >>>> [pid 13436] >>>>> stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", >>>>> 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) >>>> >>>> >>>> What's going on here? >>>> >>>> On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n < >>>> christian.gruen at gmail.com> wrote: >>>> >>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >>>>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>> >>>>>> >>>>> Looks like a endless loop that is caused by parsing the files in your >>>>> project directory. Do you possibly have any symbolic links? >>>>> >>>>> Can you reproduce the problem with a completely fresh BaseX zip >>>>> archive? >>>>> >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/1739321f/attachment.htm> From graydonish at gmail.com Tue Jan 22 23:50:32 2019 From: graydonish at gmail.com (Graydon Saunders) Date: Tue, 22 Jan 2019 17:50:32 -0500 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAP94bnPh_MLZDoFTiBuApjXx=mdvXgYG5vMn_y106iJnhmAzYg@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> <CAP94bnPh_MLZDoFTiBuApjXx=mdvXgYG5vMn_y106iJnhmAzYg@mail.gmail.com> Message-ID: <CAO3ciQH_irnSHq3ffzSkCxb3BQLsd+imZKCABiy=BL1P8DU3Pg@mail.gmail.com> I've been handling updates by making data/ a symbolic link to a data directory that's a sibling of the basex directory. (Move the old, unpack the new, go into new and replace data/ with a symbolic link up and over.) Would hate to see that stop working. On Tue, Jan 22, 2019, 17:36 Christian Gr?n <christian.gruen at gmail.com> wrote: > Good to hear that! I can?t recollect that something particular has changed > in version 9.1.2, regarding the scanning of project files, but I?ll have > some thoughts how we can trace and interrupt such loops (or ignore symbolic > links instead). > > > > Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith < > bdysonsmith at gmail.com> geschrieben: > >> Glad that helped :) >> >> I see this when I start from a fresh install vs expanding the ZIP into >> the same directory. >> >> On Tue, Jan 22, 2019, 5:17 PM Rick Graham <rickhg12hs at gmail.com wrote: >> >>> Thanks Bridger! >>> >>> Indeed, I quit basexgui and manually edited .basexgui to set the project >>> directory to a newly created empty directory. basexgui seems normal/stable >>> after that. >>> >>> I rarely, as in almost never, use wine but I didn't have this issue with >>> previous versions of BaseX. Something seems unexpected here. >>> >>> >>> On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith < >>> bdysonsmith at gmail.com> wrote: >>> >>>> Hi Rick, et al, >>>> I think (but am not 100% sure) that the GUI defaults to looking through >>>> your home directory on startup. So, somewhere in >>>> `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. >>>> >>>> I think you might be able to circumvent this problem by finding >>>> `.basexgui` - it would probably be close to wherever you started the GUI >>>> from on your filesystem. I think you can edit some of the PATHS there and >>>> that may help? >>>> >>>> Again, I'm not sure. HTH! >>>> Best, >>>> Bridger >>>> >>>> On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com> >>>> wrote: >>>> >>>>> The command-line seemed to be operating normally. >>>>> >>>>> What exactly is/are my project directories? >>>>> >>>>> I attached to the running GUI instance `strace -f -e trace=stat -p >>>>> 13368` and it has infinite repetitions of: >>>>> >>>>> [pid 13436] >>>>>> stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", >>>>>> 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) >>>>> >>>>> >>>>> What's going on here? >>>>> >>>>> On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n < >>>>> christian.gruen at gmail.com> wrote: >>>>> >>>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >>>>>>>> at >>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>> at >>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>> at >>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>> >>>>>>> >>>>>> Looks like a endless loop that is caused by parsing the files in your >>>>>> project directory. Do you possibly have any symbolic links? >>>>>> >>>>>> Can you reproduce the problem with a completely fresh BaseX zip >>>>>> archive? >>>>>> >>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190122/0f8ff287/attachment.htm> From alex at holupirek.de Wed Jan 23 09:09:43 2019 From: alex at holupirek.de (Alexander Holupirek) Date: Wed, 23 Jan 2019 09:09:43 +0100 Subject: [basex-talk] BaseX 9.1.2: Maintenance Release In-Reply-To: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> References: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> Message-ID: <E23E55D4-88BC-4079-8EE7-56F82EDBF7E4@holupirek.de> For all the macOS/homebrew [1] users, BaseX 9.1.2 is now available: $ brew update Updated Homebrew from 80c2ed310 to 550202d00. Updated 2 taps (homebrew/core and homebrew/cask). ==> New Formulae grpcurl ==> Updated Formulae basex ? $ brew upgrade basex ==> Upgrading 1 outdated package: basex 9.1.1 -> 9.1.2 ==> Upgrading basex ==> Downloading http://files.basex.org/releases/9.1.2/BaseX912.zip ? /usr/local/Cellar/basex/9.1.2: 129 files, 10.5MB, built in 2 seconds Cheers, Alex [1] https://brew.sh > On 22. Jan 2019, at 10:06, Christian Gr?n <christian.gruen at gmail.com> wrote: > > Hi all, > > A new BaseX maintenance release is available. Minor bugs have been > fixed, and some performance tweaks have been added. In particular, > access to large WebDAV directories should be faster now. > > As usual, you find the latest version on our homepage basex.org. Maven > artifacts have been added, various other distributions will follow > soon. > > Have fun everyone, > Christian, BaseX Team From m.lettere at gmail.com Wed Jan 23 09:12:30 2019 From: m.lettere at gmail.com (Marco Lettere) Date: Wed, 23 Jan 2019 09:12:30 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAO3ciQH_irnSHq3ffzSkCxb3BQLsd+imZKCABiy=BL1P8DU3Pg@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> <CAP94bnPh_MLZDoFTiBuApjXx=mdvXgYG5vMn_y106iJnhmAzYg@mail.gmail.com> <CAO3ciQH_irnSHq3ffzSkCxb3BQLsd+imZKCABiy=BL1P8DU3Pg@mail.gmail.com> Message-ID: <a0c6e037-dbb0-0b94-4a6e-4fa26b9ba415@gmail.com> Same for us over here. The ability to follow symlinks is a very powerful feature that we use to externalize folders (data, restxq for instance). So please don't remove it altogether! M. On 22/01/19 23:50, Graydon Saunders wrote: > I've been handling updates by making data/ a symbolic link to a data > directory that's a sibling of the basex directory.? (Move the old, > unpack the new, go into new and replace data/ with a symbolic link up > and over.) > > Would hate to see that stop working. > > On Tue, Jan 22, 2019, 17:36 Christian Gr?n <christian.gruen at gmail.com > <mailto:christian.gruen at gmail.com>> wrote: > > Good to hear that! I can?t recollect that something particular has > changed in version 9.1.2, regarding the scanning of project files, > but I?ll have some thoughts how we can trace and interrupt such > loops (or ignore symbolic links instead). > > > > Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith > <bdysonsmith at gmail.com <mailto:bdysonsmith at gmail.com>> geschrieben: > > Glad that helped :) > > I see this when I start from a fresh install vs expanding the > ZIP into the same directory. > > On Tue, Jan 22, 2019, 5:17 PM Rick Graham > <rickhg12hs at gmail.com <mailto:rickhg12hs at gmail.com> wrote: > > Thanks Bridger! > > Indeed, I quit basexgui and manually edited .basexgui to > set the project directory to a newly created empty > directory.? basexgui seems normal/stable after that. > > I rarely, as in almost never, use wine but I didn't have > this issue with previous versions of BaseX.? Something > seems unexpected here. > > > On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith > <bdysonsmith at gmail.com <mailto:bdysonsmith at gmail.com>> wrote: > > Hi Rick, et al, > I think (but am not 100% sure) that the GUI defaults > to looking through your home directory on startup. So, > somewhere in `~/rick/.wine/dosdevices/...` you have > symbolic links that are looped. > > I think you might be able to circumvent this problem > by finding `.basexgui` - it would probably be close to > wherever you started the GUI from on your filesystem. > I think you can edit some of the PATHS there and that > may help? > > Again, I'm not sure. HTH! > Best, > Bridger > > On Tue, Jan 22, 2019 at 4:56 PM Rick Graham > <rickhg12hs at gmail.com <mailto:rickhg12hs at gmail.com>> > wrote: > > The command-line seemed to be operating normally. > > What exactly is/are my project directories? > > I attached to the running GUI instance `strace -f > -e trace=stat -p 13368` and it has infinite > repetitions of: > > [pid 13436] > stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", > 0x7f7beb2796e0) = -1 ELOOP (Too many levels of > symbolic links) > > > What's going on here? > > On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n > <christian.gruen at gmail.com > <mailto:christian.gruen at gmail.com>> wrote: > > at > org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) > at > org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at > org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at > org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > > > Looks like a endless loop that is caused by > parsing the files in your project directory. > Do you possibly have any symbolic links? > > Can you reproduce the problem with a > completely fresh BaseX zip archive? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190123/362947ae/attachment.htm> From gsf.greece at gmail.com Wed Jan 23 11:42:51 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Wed, 23 Jan 2019 12:42:51 +0200 Subject: [basex-talk] BaseX 9.1.2: Maintenance Release In-Reply-To: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> References: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> Message-ID: <55717221-de23-cc44-f46c-97b6c0d2710e@gmail.com> Hi Christian, thanks for the new version. Will it be available in maven soon? Also another question that I hate to make, but I have to, are there any plans to re-introduce QUERYPATH? This has been keeping me from upgrading to a more recent version than 8.4.4, but we have another need that requires 9.1.2 at the moment. Regards, George On 1/22/19 11:06 AM, Christian Gr?n wrote: > Hi all, > > A new BaseX maintenance release is available. Minor bugs have been > fixed, and some performance tweaks have been added. In particular, > access to large WebDAV directories should be faster now. > > As usual, you find the latest version on our homepage basex.org. Maven > artifacts have been added, various other distributions will follow > soon. > > Have fun everyone, > Christian, BaseX Team From gsf.greece at gmail.com Wed Jan 23 11:57:55 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Wed, 23 Jan 2019 12:57:55 +0200 Subject: [basex-talk] BaseX 9.1.2: Maintenance Release In-Reply-To: <55717221-de23-cc44-f46c-97b6c0d2710e@gmail.com> References: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> <55717221-de23-cc44-f46c-97b6c0d2710e@gmail.com> Message-ID: <5f69d10d-10ad-5fd9-3468-f33ec6a5941e@gmail.com> Oops, I just noticed it already is available. I need to stop looking into mvnrepository first. Sorry about that! George On 1/23/19 12:42 PM, George Sofianos wrote: > Hi Christian, > > thanks for the new version. Will it be available in maven soon? > > Also another question that I hate to make, but I have to, are there > any plans to re-introduce QUERYPATH? This has been keeping me from > upgrading to a more recent version than 8.4.4, but we have another > need that requires 9.1.2 at the moment. > > Regards, > > George > From christian.gruen at gmail.com Wed Jan 23 14:04:32 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 23 Jan 2019 14:04:32 +0100 Subject: [basex-talk] BaseX 9.1.2: Maintenance Release In-Reply-To: <55717221-de23-cc44-f46c-97b6c0d2710e@gmail.com> References: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> <55717221-de23-cc44-f46c-97b6c0d2710e@gmail.com> Message-ID: <CAP94bnMdVW2c0Usniquigm=+DR1CYp3g9iMZzUDQdPZkveGAjA@mail.gmail.com> Hi George, Sorry, we have no plans to reintroduce the old option (to much has changed in our architecture). Maybe there are other solutions for your use case in the latest version? Best Christian Am Mi., 23. Jan. 2019, 11:42 hat George Sofianos <gsf.greece at gmail.com> geschrieben: > Hi Christian, > > thanks for the new version. Will it be available in maven soon? > > Also another question that I hate to make, but I have to, are there any > plans to re-introduce QUERYPATH? This has been keeping me from upgrading > to a more recent version than 8.4.4, but we have another need that > requires 9.1.2 at the moment. > > Regards, > > George > > On 1/22/19 11:06 AM, Christian Gr?n wrote: > > Hi all, > > > > A new BaseX maintenance release is available. Minor bugs have been > > fixed, and some performance tweaks have been added. In particular, > > access to large WebDAV directories should be faster now. > > > > As usual, you find the latest version on our homepage basex.org. Maven > > artifacts have been added, various other distributions will follow > > soon. > > > > Have fun everyone, > > Christian, BaseX Team > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190123/2843a794/attachment.htm> From christian.gruen at gmail.com Wed Jan 23 14:09:22 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 23 Jan 2019 14:09:22 +0100 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <a0c6e037-dbb0-0b94-4a6e-4fa26b9ba415@gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> <CAP94bnPh_MLZDoFTiBuApjXx=mdvXgYG5vMn_y106iJnhmAzYg@mail.gmail.com> <CAO3ciQH_irnSHq3ffzSkCxb3BQLsd+imZKCABiy=BL1P8DU3Pg@mail.gmail.com> <a0c6e037-dbb0-0b94-4a6e-4fa26b9ba415@gmail.com> Message-ID: <CAP94bnNaczDGa5oOd9vXx8drCmSwKCZCzdop5GdNKDk3=kRxiw@mail.gmail.com> Thanks for your assessments. So maybe we should ensure that every file and directory will only be parsed once. Did anyone of you have problems with the automatic parsing of project files in the BaseX GUI, or with symbolic links in particular? Am Mi., 23. Jan. 2019, 09:12 hat Marco Lettere <m.lettere at gmail.com> geschrieben: > Same for us over here. The ability to follow symlinks is a very powerful > feature that we use to externalize folders (data, restxq for instance). So > please don't remove it altogether! > M. > > On 22/01/19 23:50, Graydon Saunders wrote: > > I've been handling updates by making data/ a symbolic link to a data > directory that's a sibling of the basex directory. (Move the old, unpack > the new, go into new and replace data/ with a symbolic link up and over.) > > Would hate to see that stop working. > > On Tue, Jan 22, 2019, 17:36 Christian Gr?n <christian.gruen at gmail.com> > wrote: > >> Good to hear that! I can?t recollect that something particular has >> changed in version 9.1.2, regarding the scanning of project files, but I?ll >> have some thoughts how we can trace and interrupt such loops (or ignore >> symbolic links instead). >> >> >> >> Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith < >> bdysonsmith at gmail.com> geschrieben: >> >>> Glad that helped :) >>> >>> I see this when I start from a fresh install vs expanding the ZIP into >>> the same directory. >>> >>> On Tue, Jan 22, 2019, 5:17 PM Rick Graham <rickhg12hs at gmail.com wrote: >>> >>>> Thanks Bridger! >>>> >>>> Indeed, I quit basexgui and manually edited .basexgui to set the >>>> project directory to a newly created empty directory. basexgui seems >>>> normal/stable after that. >>>> >>>> I rarely, as in almost never, use wine but I didn't have this issue >>>> with previous versions of BaseX. Something seems unexpected here. >>>> >>>> >>>> On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith < >>>> bdysonsmith at gmail.com> wrote: >>>> >>>>> Hi Rick, et al, >>>>> I think (but am not 100% sure) that the GUI defaults to looking >>>>> through your home directory on startup. So, somewhere in >>>>> `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. >>>>> >>>>> I think you might be able to circumvent this problem by finding >>>>> `.basexgui` - it would probably be close to wherever you started the GUI >>>>> from on your filesystem. I think you can edit some of the PATHS there and >>>>> that may help? >>>>> >>>>> Again, I'm not sure. HTH! >>>>> Best, >>>>> Bridger >>>>> >>>>> On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com> >>>>> wrote: >>>>> >>>>>> The command-line seemed to be operating normally. >>>>>> >>>>>> What exactly is/are my project directories? >>>>>> >>>>>> I attached to the running GUI instance `strace -f -e trace=stat -p >>>>>> 13368` and it has infinite repetitions of: >>>>>> >>>>>> [pid 13436] >>>>>>> stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", >>>>>>> 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) >>>>>> >>>>>> >>>>>> What's going on here? >>>>>> >>>>>> On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n < >>>>>> christian.gruen at gmail.com> wrote: >>>>>> >>>>>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >>>>>>>>> at >>>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>>> at >>>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>>> at >>>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>>> >>>>>>>> >>>>>>> Looks like a endless loop that is caused by parsing the files in >>>>>>> your project directory. Do you possibly have any symbolic links? >>>>>>> >>>>>>> Can you reproduce the problem with a completely fresh BaseX zip >>>>>>> archive? >>>>>>> >>>>>>> >>>>>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190123/1bafa439/attachment.htm> From graydonish at gmail.com Wed Jan 23 14:18:43 2019 From: graydonish at gmail.com (Graydon Saunders) Date: Wed, 23 Jan 2019 08:18:43 -0500 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAP94bnNaczDGa5oOd9vXx8drCmSwKCZCzdop5GdNKDk3=kRxiw@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> <CAP94bnPh_MLZDoFTiBuApjXx=mdvXgYG5vMn_y106iJnhmAzYg@mail.gmail.com> <CAO3ciQH_irnSHq3ffzSkCxb3BQLsd+imZKCABiy=BL1P8DU3Pg@mail.gmail.com> <a0c6e037-dbb0-0b94-4a6e-4fa26b9ba415@gmail.com> <CAP94bnNaczDGa5oOd9vXx8drCmSwKCZCzdop5GdNKDk3=kRxiw@mail.gmail.com> Message-ID: <CAO3ciQHWnh7tgi3Y-xA9r5oosNYr4fvDUrM3NFuwrrJD6FWKfA@mail.gmail.com> Never had a problem with the GUI parsing project files. No issues with symbolic links. I have generally found basex very reliable but never try to update the installed version; it's always move the old, unpack the new. On Wed, Jan 23, 2019, 08:09 Christian Gr?n <christian.gruen at gmail.com> wrote: > Thanks for your assessments. So maybe we should ensure that every file and > directory will only be parsed once. > > Did anyone of you have problems with the automatic parsing of project > files in the BaseX GUI, or with symbolic links in particular? > > > > > > Am Mi., 23. Jan. 2019, 09:12 hat Marco Lettere <m.lettere at gmail.com> > geschrieben: > >> Same for us over here. The ability to follow symlinks is a very powerful >> feature that we use to externalize folders (data, restxq for instance). So >> please don't remove it altogether! >> M. >> >> On 22/01/19 23:50, Graydon Saunders wrote: >> >> I've been handling updates by making data/ a symbolic link to a data >> directory that's a sibling of the basex directory. (Move the old, unpack >> the new, go into new and replace data/ with a symbolic link up and over.) >> >> Would hate to see that stop working. >> >> On Tue, Jan 22, 2019, 17:36 Christian Gr?n <christian.gruen at gmail.com> >> wrote: >> >>> Good to hear that! I can?t recollect that something particular has >>> changed in version 9.1.2, regarding the scanning of project files, but I?ll >>> have some thoughts how we can trace and interrupt such loops (or ignore >>> symbolic links instead). >>> >>> >>> >>> Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith < >>> bdysonsmith at gmail.com> geschrieben: >>> >>>> Glad that helped :) >>>> >>>> I see this when I start from a fresh install vs expanding the ZIP into >>>> the same directory. >>>> >>>> On Tue, Jan 22, 2019, 5:17 PM Rick Graham <rickhg12hs at gmail.com wrote: >>>> >>>>> Thanks Bridger! >>>>> >>>>> Indeed, I quit basexgui and manually edited .basexgui to set the >>>>> project directory to a newly created empty directory. basexgui seems >>>>> normal/stable after that. >>>>> >>>>> I rarely, as in almost never, use wine but I didn't have this issue >>>>> with previous versions of BaseX. Something seems unexpected here. >>>>> >>>>> >>>>> On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith < >>>>> bdysonsmith at gmail.com> wrote: >>>>> >>>>>> Hi Rick, et al, >>>>>> I think (but am not 100% sure) that the GUI defaults to looking >>>>>> through your home directory on startup. So, somewhere in >>>>>> `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. >>>>>> >>>>>> I think you might be able to circumvent this problem by finding >>>>>> `.basexgui` - it would probably be close to wherever you started the GUI >>>>>> from on your filesystem. I think you can edit some of the PATHS there and >>>>>> that may help? >>>>>> >>>>>> Again, I'm not sure. HTH! >>>>>> Best, >>>>>> Bridger >>>>>> >>>>>> On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com> >>>>>> wrote: >>>>>> >>>>>>> The command-line seemed to be operating normally. >>>>>>> >>>>>>> What exactly is/are my project directories? >>>>>>> >>>>>>> I attached to the running GUI instance `strace -f -e trace=stat -p >>>>>>> 13368` and it has infinite repetitions of: >>>>>>> >>>>>>> [pid 13436] >>>>>>>> stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", >>>>>>>> 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) >>>>>>> >>>>>>> >>>>>>> What's going on here? >>>>>>> >>>>>>> On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n < >>>>>>> christian.gruen at gmail.com> wrote: >>>>>>> >>>>>>>> at >>>>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) >>>>>>>>>> at >>>>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>>>> at >>>>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>>>> at >>>>>>>>>> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) >>>>>>>>>> >>>>>>>>> >>>>>>>> Looks like a endless loop that is caused by parsing the files in >>>>>>>> your project directory. Do you possibly have any symbolic links? >>>>>>>> >>>>>>>> Can you reproduce the problem with a completely fresh BaseX zip >>>>>>>> archive? >>>>>>>> >>>>>>>> >>>>>>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190123/823a546b/attachment.htm> From echterhoff at interactive-instruments.de Wed Jan 23 14:29:54 2019 From: echterhoff at interactive-instruments.de (Johannes Echterhoff) Date: Wed, 23 Jan 2019 13:29:54 +0000 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <CAO3ciQHWnh7tgi3Y-xA9r5oosNYr4fvDUrM3NFuwrrJD6FWKfA@mail.gmail.com> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> <CAP94bnPh_MLZDoFTiBuApjXx=mdvXgYG5vMn_y106iJnhmAzYg@mail.gmail.com> <CAO3ciQH_irnSHq3ffzSkCxb3BQLsd+imZKCABiy=BL1P8DU3Pg@mail.gmail.com> <a0c6e037-dbb0-0b94-4a6e-4fa26b9ba415@gmail.com> <CAP94bnNaczDGa5oOd9vXx8drCmSwKCZCzdop5GdNKDk3=kRxiw@mail.gmail.com> <CAO3ciQHWnh7tgi3Y-xA9r5oosNYr4fvDUrM3NFuwrrJD6FWKfA@mail.gmail.com> Message-ID: <eebf35282cf0477aa7cd462a8121f481@luna.interactive-instruments.de> For what it?s worth: in our developments we had a directory as part of the project files where we stored all queries that were executed during development and testing (these queries were dynamically created by other queries). Saving these queries is just part of our process. However, after a while that resulted in hundreds to thousands of query files. That resulted in a massive memory demand, and we noticed increased CPU usage, both of which we could not explain at first. Eventually we found out that the BaseX GUI automatically parsed all these queries, each time initializing a custom module we wrote, where the initialization of that module created a temporary directory with a number of files. The solution for us was to simply move the directory where we save the executed queries somewhere outside of the project files opened in the BaseX GUI. Best regards, Johannes Von: BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von Graydon Saunders Gesendet: Mittwoch, 23. Januar 2019 14:19 An: Christian Gr?n <christian.gruen at gmail.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] BaseX/GUI v9.1.2 memory use Never had a problem with the GUI parsing project files. No issues with symbolic links. I have generally found basex very reliable but never try to update the installed version; it's always move the old, unpack the new. On Wed, Jan 23, 2019, 08:09 Christian Gr?n <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Thanks for your assessments. So maybe we should ensure that every file and directory will only be parsed once. Did anyone of you have problems with the automatic parsing of project files in the BaseX GUI, or with symbolic links in particular? Am Mi., 23. Jan. 2019, 09:12 hat Marco Lettere <m.lettere at gmail.com<mailto:m.lettere at gmail.com>> geschrieben: Same for us over here. The ability to follow symlinks is a very powerful feature that we use to externalize folders (data, restxq for instance). So please don't remove it altogether! M. On 22/01/19 23:50, Graydon Saunders wrote: I've been handling updates by making data/ a symbolic link to a data directory that's a sibling of the basex directory. (Move the old, unpack the new, go into new and replace data/ with a symbolic link up and over.) Would hate to see that stop working. On Tue, Jan 22, 2019, 17:36 Christian Gr?n <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Good to hear that! I can?t recollect that something particular has changed in version 9.1.2, regarding the scanning of project files, but I?ll have some thoughts how we can trace and interrupt such loops (or ignore symbolic links instead). Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith <bdysonsmith at gmail.com<mailto:bdysonsmith at gmail.com>> geschrieben: Glad that helped :) I see this when I start from a fresh install vs expanding the ZIP into the same directory. On Tue, Jan 22, 2019, 5:17 PM Rick Graham <rickhg12hs at gmail.com<mailto:rickhg12hs at gmail.com> wrote: Thanks Bridger! Indeed, I quit basexgui and manually edited .basexgui to set the project directory to a newly created empty directory. basexgui seems normal/stable after that. I rarely, as in almost never, use wine but I didn't have this issue with previous versions of BaseX. Something seems unexpected here. On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith <bdysonsmith at gmail.com<mailto:bdysonsmith at gmail.com>> wrote: Hi Rick, et al, I think (but am not 100% sure) that the GUI defaults to looking through your home directory on startup. So, somewhere in `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. I think you might be able to circumvent this problem by finding `.basexgui` - it would probably be close to wherever you started the GUI from on your filesystem. I think you can edit some of the PATHS there and that may help? Again, I'm not sure. HTH! Best, Bridger On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com<mailto:rickhg12hs at gmail.com>> wrote: The command-line seemed to be operating normally. What exactly is/are my project directories? I attached to the running GUI instance `strace -f -e trace=stat -p 13368` and it has infinite repetitions of: [pid 13436] stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) What's going on here? On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) Looks like a endless loop that is caused by parsing the files in your project directory. Do you possibly have any symbolic links? Can you reproduce the problem with a completely fresh BaseX zip archive? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190123/fd611f34/attachment.htm> From bdysonsmith at gmail.com Wed Jan 23 16:02:20 2019 From: bdysonsmith at gmail.com (Bridger Dyson-Smith) Date: Wed, 23 Jan 2019 10:02:20 -0500 Subject: [basex-talk] BaseX/GUI v9.1.2 memory use In-Reply-To: <eebf35282cf0477aa7cd462a8121f481@luna.interactive-instruments.de> References: <CAF+zCGTX=B99NisXVOPF7Akr2DsWamYjPK7+knLF8+rh3US_Ag@mail.gmail.com> <CAP94bnNhr_=4=4dgyrhMPwTr2=GnP5UpAVw6RxewKa=iYM3TYg@mail.gmail.com> <CAF+zCGRibjp+NUoPTf5TvO9fauuZAorGOfn9vRxboP1mZCn08A@mail.gmail.com> <CAP94bnMNqNRXM=gwej2o0qTx_XL+9o4Pu1ezjK1ywJPzZQJMKA@mail.gmail.com> <CAF+zCGSfop8qbsn8=P=MRZz8LDjLkhogDk7y0MqFgwHWMzmXUg@mail.gmail.com> <CAD-FYmQricni950MzeNj0ZnMfX0EKdB+7Z6t3XoDTL3GrX5yMg@mail.gmail.com> <CAF+zCGQJQrjNcR53U4E_Vrfr0NnWEjB056HFZ_dsYCnx1bmCwA@mail.gmail.com> <CAD-FYmQGTg93jOVFuNEsEgbXYiPyjvrwLxeL_koXY2b+-VNaHA@mail.gmail.com> <CAP94bnPh_MLZDoFTiBuApjXx=mdvXgYG5vMn_y106iJnhmAzYg@mail.gmail.com> <CAO3ciQH_irnSHq3ffzSkCxb3BQLsd+imZKCABiy=BL1P8DU3Pg@mail.gmail.com> <a0c6e037-dbb0-0b94-4a6e-4fa26b9ba415@gmail.com> <CAP94bnNaczDGa5oOd9vXx8drCmSwKCZCzdop5GdNKDk3=kRxiw@mail.gmail.com> <CAO3ciQHWnh7tgi3Y-xA9r5oosNYr4fvDUrM3NFuwrrJD6FWKfA@mail.gmail.com> <eebf35282cf0477aa7cd462a8121f481@luna.interactive-instruments.de> Message-ID: <CAD-FYmSo+kP1bHB8RTsT3piR+jNTtRot0M0cWLjz+i2U0cN6cg@mail.gmail.com> Mea culpa! I may have steered the train of conversation in a non-productive direction: my home is parsed due to how I launch the GUI. In any event, I haven't had any issues with parsing project files or symbolic links, either. Apologies, Bridger On Wed, Jan 23, 2019 at 8:30 AM Johannes Echterhoff < echterhoff at interactive-instruments.de> wrote: > For what it?s worth: in our developments we had a directory as part of the > project files where we stored all queries that were executed during > development and testing (these queries were dynamically created by other > queries). Saving these queries is just part of our process. However, after > a while that resulted in hundreds to thousands of query files. That > resulted in a massive memory demand, and we noticed increased CPU usage, > both of which we could not explain at first. Eventually we found out that > the BaseX GUI automatically parsed all these queries, each time > initializing a custom module we wrote, where the initialization of that > module created a temporary directory with a number of files. The solution > for us was to simply move the directory where we save the executed queries > somewhere outside of the project files opened in the BaseX GUI. > > > > Best regards, > > Johannes > > > > > > *Von:* BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] *Im > Auftrag von *Graydon Saunders > *Gesendet:* Mittwoch, 23. Januar 2019 14:19 > *An:* Christian Gr?n <christian.gruen at gmail.com> > *Cc:* BaseX <basex-talk at mailman.uni-konstanz.de> > *Betreff:* Re: [basex-talk] BaseX/GUI v9.1.2 memory use > > > > Never had a problem with the GUI parsing project files. No issues with > symbolic links. > > > > I have generally found basex very reliable but never try to update the > installed version; it's always move the old, unpack the new. > > On Wed, Jan 23, 2019, 08:09 Christian Gr?n <christian.gruen at gmail.com> > wrote: > > Thanks for your assessments. So maybe we should ensure that every file and > directory will only be parsed once. > > > > Did anyone of you have problems with the automatic parsing of project > files in the BaseX GUI, or with symbolic links in particular? > > > > > > > > > > > > Am Mi., 23. Jan. 2019, 09:12 hat Marco Lettere <m.lettere at gmail.com> > geschrieben: > > Same for us over here. The ability to follow symlinks is a very powerful > feature that we use to externalize folders (data, restxq for instance). So > please don't remove it altogether! > > M. > > > > On 22/01/19 23:50, Graydon Saunders wrote: > > I've been handling updates by making data/ a symbolic link to a data > directory that's a sibling of the basex directory. (Move the old, unpack > the new, go into new and replace data/ with a symbolic link up and over.) > > > > Would hate to see that stop working. > > On Tue, Jan 22, 2019, 17:36 Christian Gr?n <christian.gruen at gmail.com> > wrote: > > Good to hear that! I can?t recollect that something particular has changed > in version 9.1.2, regarding the scanning of project files, but I?ll have > some thoughts how we can trace and interrupt such loops (or ignore symbolic > links instead). > > > > > > Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith < > bdysonsmith at gmail.com> geschrieben: > > Glad that helped :) > > > > I see this when I start from a fresh install vs expanding the ZIP into the > same directory. > > > > On Tue, Jan 22, 2019, 5:17 PM Rick Graham <rickhg12hs at gmail.com wrote: > > Thanks Bridger! > > > > Indeed, I quit basexgui and manually edited .basexgui to set the project > directory to a newly created empty directory. basexgui seems normal/stable > after that. > > > > I rarely, as in almost never, use wine but I didn't have this issue with > previous versions of BaseX. Something seems unexpected here. > > > > > > On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith < > bdysonsmith at gmail.com> wrote: > > Hi Rick, et al, > > I think (but am not 100% sure) that the GUI defaults to looking through > your home directory on startup. So, somewhere in > `~/rick/.wine/dosdevices/...` you have symbolic links that are looped. > > > > I think you might be able to circumvent this problem by finding > `.basexgui` - it would probably be close to wherever you started the GUI > from on your filesystem. I think you can edit some of the PATHS there and > that may help? > > > > Again, I'm not sure. HTH! > > Best, > > Bridger > > > > On Tue, Jan 22, 2019 at 4:56 PM Rick Graham <rickhg12hs at gmail.com> wrote: > > The command-line seemed to be operating normally. > > > > What exactly is/are my project directories? > > > > I attached to the running GUI instance `strace -f -e trace=stat -p 13368` > and it has infinite repetitions of: > > > > [pid 13436] > stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02", > 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links) > > > > What's going on here? > > > > On Tue, Jan 22, 2019 at 10:21 PM Christian Gr?n <christian.gruen at gmail.com> > wrote: > > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173) > > > > Looks like a endless loop that is caused by parsing the files in your > project directory. Do you possibly have any symbolic links? > > > > Can you reproduce the problem with a completely fresh BaseX zip archive? > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190123/ec18ec06/attachment.htm> From gsf.greece at gmail.com Wed Jan 23 20:54:24 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Wed, 23 Jan 2019 21:54:24 +0200 Subject: [basex-talk] BaseX 9.1.2: Maintenance Release In-Reply-To: <CAP94bnMdVW2c0Usniquigm=+DR1CYp3g9iMZzUDQdPZkveGAjA@mail.gmail.com> References: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> <55717221-de23-cc44-f46c-97b6c0d2710e@gmail.com> <CAP94bnMdVW2c0Usniquigm=+DR1CYp3g9iMZzUDQdPZkveGAjA@mail.gmail.com> Message-ID: <7933eab3-ad2f-0e15-3301-8737af9e5192@gmail.com> Thanks Christian, I think I managed to make it work, if I'm not missing anything. The problem was (the last time I tried it, over a year ago), that I need to work with relative paths, and location uris would not work properly. I managed to set it up today, by using a QueryProcessor constructor that includes a base-uri parameter. The problem is, that when you call the QueryProcessor [1] the base URI should be set to a fake uri or file path, for example "/home/user/application/queries/fake". This way, both location URIs (import module declarations), and document requests to relative urls - doc(concat("../myxml.xml")) work. If base URI is just set as "/home/user/application/queries" - which I thought it would be correct - then only doc() function will look at the correct base-uri directory, while the import module statements, will try to import files from the parent of the base directory. I think it has something to do with how baseURI function works in StaticContext.java [2], but I'm not sure. Maybe a Java example showing the correct usage of the base-uri, or a wiki page will help others in the future. [1]: https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/local/BindVariables.java [2]: https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/query/StaticContext.java I'm happy I got this out of the way, my next email will be about some issues I found with parallel xquery processing :) Regards, George On 1/23/19 3:04 PM, Christian Gr?n wrote: > Hi George, > > Sorry, we have no plans to reintroduce the old option (to much has > changed in our architecture). Maybe there are other solutions for your > use case in the latest version? > > Best > Christian > From liam at fromoldbooks.org Thu Jan 24 10:53:48 2019 From: liam at fromoldbooks.org (Liam R. E. Quin) Date: Thu, 24 Jan 2019 04:53:48 -0500 Subject: [basex-talk] many distinct namespaces In-Reply-To: <CAP94bnNCJ4Ru1HPNkCs1T8jz6Tc+vrysyQ8cO-VTPhR8T__zGg@mail.gmail.com> References: <1541022422.882950422@f403.i.mail.ru> <39e0d303-f8eb-3180-a6ec-c9aacb3a44a5@le-tex.de> <CAP94bnNdWRD4CEBSGsy9bYK2oQ52XTeMf7-+rs2ZUGeXS3MMTw@mail.gmail.com> <57bdf06bc00d50ea6a8f004e4af2bf7b3204b6d4.camel@fromoldbooks.org> <CAP94bnNCJ4Ru1HPNkCs1T8jz6Tc+vrysyQ8cO-VTPhR8T__zGg@mail.gmail.com> Message-ID: <9ac3942a175e5f83a0355e359fb481bdc80c909f.camel@fromoldbooks.org> On Wed, 2019-01-16 at 15:31 +0100, Christian Gr?n wrote: > Hi Liam, > > > did this ever happen? > > What exactly? ;) sorry! support for more than 256 namespaces in one db. -- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Web slave for vintage clipart http://www.fromoldbooks.org/ From xavierlaurent.salvador at gmail.com Thu Jan 24 21:41:18 2019 From: xavierlaurent.salvador at gmail.com (Xavier-Laurent SALVADOR) Date: Thu, 24 Jan 2019 21:41:18 +0100 Subject: [basex-talk] not() and != result Message-ID: <CAP+T2dw5rYQmbDaAAU1oUo3t7MBqanTELB0SRvt-0qq4NS3mCQ@mail.gmail.com> Hi List, I'm seing a little problem I can't understand with a small 27M thesaurus database. I created all indexes. When using the '!=' operator to compare two lists, I get a quick and wrong result: ------- *Case 1:* let $a := /thesaurus/entry/synonym/term let $b := /thesaurus/entry/term return $a[. = $b] ---> First result: "raccourcir" ------- and ------- *Case 2:* let $a := /thesaurus/entry/synonym/term let $b := /thesaurus/entry/term return $a[. != $b] ---> First result: "raccourcir" ------ while ------ *Case 3:* count(//entry[term='raccourcir']) --> result: 1 ------- But when I use the "not()" function as usual: ------ *Case 4:* let $a := /thesaurus/entry/synonym/term let $b := /thesaurus/entry/term return $a[not(.=$b)] ------- It dies out of memory. Is there something I did not see ? tx Xavier-Laurent Salvador -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190124/c46de700/attachment.htm> From graydonish at gmail.com Thu Jan 24 21:52:54 2019 From: graydonish at gmail.com (Graydon) Date: Thu, 24 Jan 2019 15:52:54 -0500 Subject: [basex-talk] not() and != result In-Reply-To: <CAP+T2dw5rYQmbDaAAU1oUo3t7MBqanTELB0SRvt-0qq4NS3mCQ@mail.gmail.com> References: <CAP+T2dw5rYQmbDaAAU1oUo3t7MBqanTELB0SRvt-0qq4NS3mCQ@mail.gmail.com> Message-ID: <20190124205253.GA18706@fenja> On Thu, Jan 24, 2019 at 09:41:18PM +0100, Xavier-Laurent SALVADOR scripsit: > Hi List, > > I'm seing a little problem I can't understand with a small 27M thesaurus > database. I created all indexes. > When using the '!=' operator to compare two lists, I get a quick and wrong > result: > ------- > *Case 1:* > let $a := /thesaurus/entry/synonym/term > let $b := /thesaurus/entry/term > return $a[. = $b] > ---> First result: "raccourcir" The = and != operators compare the sequences. = returns true if _any_ member of the left-hand sequence can be found in the right-hand sequence. ('1','2','3') = ('1','asparagus','guillotine') returns true(). Similarly, != returns true if _any_ member of the left-hand sequence can NOT be found in the right-hand sequence. ('1','2','3') != ('1','asparagus','guillotine') returns true(). != is an exceedingly tricksy operator; (1,2,3) != (1,2,3) is true. > *Case 4:* > let $a := /thesaurus/entry/synonym/term > let $b := /thesaurus/entry/term > return $a[not(.=$b)] > ------- > > It dies out of memory. That's odd. That pattern generally works pretty well for me on much larger datasets than 27 MB. If you're trying to check that all your synonyms are terms, I'd try let $a as xs:string+ := /thesaurus/entry/synonym/term/string() let $b as xs:string+ := /thesaurus/entry/term/string() return $a[not(. = $b)] so you do the "I want the string value of this element" only once, or if for some reason you need the elements later, maybe create maps and then compare the map keys? -- Graydon From xavierlaurent.salvador at gmail.com Thu Jan 24 22:10:58 2019 From: xavierlaurent.salvador at gmail.com (Xavier-Laurent SALVADOR) Date: Thu, 24 Jan 2019 22:10:58 +0100 Subject: [basex-talk] not() and != result In-Reply-To: <20190124205253.GA18706@fenja> References: <CAP+T2dw5rYQmbDaAAU1oUo3t7MBqanTELB0SRvt-0qq4NS3mCQ@mail.gmail.com> <20190124205253.GA18706@fenja> Message-ID: <CAP+T2dz07dqJq=D_PpN8iLiuU0WE+aXc674y2Us+owc9aAGWHg@mail.gmail.com> Awesome, thanks! Le jeu. 24 janv. 2019 ? 21:52, Graydon <graydonish at gmail.com> a ?crit : > On Thu, Jan 24, 2019 at 09:41:18PM +0100, Xavier-Laurent SALVADOR scripsit: > > Hi List, > > > > I'm seing a little problem I can't understand with a small 27M thesaurus > > database. I created all indexes. > > When using the '!=' operator to compare two lists, I get a quick and > wrong > > result: > > ------- > > *Case 1:* > > let $a := /thesaurus/entry/synonym/term > > let $b := /thesaurus/entry/term > > return $a[. = $b] > > ---> First result: "raccourcir" > > The = and != operators compare the sequences. > > = returns true if _any_ member of the left-hand sequence can be found in > the right-hand sequence. > > ('1','2','3') = ('1','asparagus','guillotine') > > returns true(). > > Similarly, != returns true if _any_ member of the left-hand sequence can > NOT be found in the right-hand sequence. > > ('1','2','3') != ('1','asparagus','guillotine') > > returns true(). > > != is an exceedingly tricksy operator; > > (1,2,3) != (1,2,3) > > is true. > > > *Case 4:* > > let $a := /thesaurus/entry/synonym/term > > let $b := /thesaurus/entry/term > > return $a[not(.=$b)] > > ------- > > > > It dies out of memory. > > That's odd. That pattern generally works pretty well for me on much > larger datasets than 27 MB. > > If you're trying to check that all your synonyms are terms, I'd try > > let $a as xs:string+ := /thesaurus/entry/synonym/term/string() > let $b as xs:string+ := /thesaurus/entry/term/string() > > return $a[not(. = $b)] > > so you do the "I want the string value of this element" only once, or if > for some reason you need the elements later, maybe create maps and then > compare the map keys? > > -- Graydon > > > -- Xavier-Laurent Salvador Professeur Agr?g?, Ma?tre de Conf?rence HDR ECC TTN 2018 22868H - ?quipe "Humanit?s Num?riques" Coordinateur du r?seau international HiBHidEM http://ttn.univ-paris13.fr Universit? Paris 13 Sorbonne Paris Cit? 99 avenue Jean-Baptiste Cl?ment 93430 Villetaneuse t?l. : (+33) 06 51.65.84.38 email : xavier-laurent.salvador at univ-paris13.fr site web: http://www.biblehistoriale.fr site web: http://www.humanitesnumeriques.fr ------------------------------------------------------------ Ce message peut contenir des informations r?serv?es exclusivement ? son destinataire. Toute diffusion sans autorisation est interdite. Si vous n'en ?tes pas le destinataire, merci de prendre contact avec l'exp?diteur et de d?truire ce message. *This email may contain material for the sole use of the intended recipient. Any forwarding without express permission is prohibited. If you are not the intended recipient, please contact the sender and delete all copies*. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190124/314bbf70/attachment.htm> From xavierlaurent.salvador at gmail.com Fri Jan 25 09:00:49 2019 From: xavierlaurent.salvador at gmail.com (Xavier-Laurent SALVADOR) Date: Fri, 25 Jan 2019 09:00:49 +0100 Subject: [basex-talk] not() and != result In-Reply-To: <CAP+T2dz07dqJq=D_PpN8iLiuU0WE+aXc674y2Us+owc9aAGWHg@mail.gmail.com> References: <CAP+T2dw5rYQmbDaAAU1oUo3t7MBqanTELB0SRvt-0qq4NS3mCQ@mail.gmail.com> <20190124205253.GA18706@fenja> <CAP+T2dz07dqJq=D_PpN8iLiuU0WE+aXc674y2Us+owc9aAGWHg@mail.gmail.com> Message-ID: <CAP+T2dzxK=fsPUhUH7K-OJKYi_qb=q0-bi6bdZFyAP60XASrLw@mail.gmail.com> Hi, I checked it during night: same symptom. And I realized I just downloaded a new 9.1.2 Basex on my good old and full hard drive while using this param: BASEX_JVM="-Xmx2g $BASEX_JVM"... I moved it to an external drive, and ... shazam ! Hardware problem, so. Thanks a lot ! Xavier-Laurent Salvador Le jeu. 24 janv. 2019 ? 22:10, Xavier-Laurent SALVADOR < xavierlaurent.salvador at gmail.com> a ?crit : > Awesome, thanks! > > Le jeu. 24 janv. 2019 ? 21:52, Graydon <graydonish at gmail.com> a ?crit : > >> On Thu, Jan 24, 2019 at 09:41:18PM +0100, Xavier-Laurent SALVADOR >> scripsit: >> > Hi List, >> > >> > I'm seing a little problem I can't understand with a small 27M thesaurus >> > database. I created all indexes. >> > When using the '!=' operator to compare two lists, I get a quick and >> wrong >> > result: >> > ------- >> > *Case 1:* >> > let $a := /thesaurus/entry/synonym/term >> > let $b := /thesaurus/entry/term >> > return $a[. = $b] >> > ---> First result: "raccourcir" >> >> The = and != operators compare the sequences. >> >> = returns true if _any_ member of the left-hand sequence can be found in >> the right-hand sequence. >> >> ('1','2','3') = ('1','asparagus','guillotine') >> >> returns true(). >> >> Similarly, != returns true if _any_ member of the left-hand sequence can >> NOT be found in the right-hand sequence. >> >> ('1','2','3') != ('1','asparagus','guillotine') >> >> returns true(). >> >> != is an exceedingly tricksy operator; >> >> (1,2,3) != (1,2,3) >> >> is true. >> >> > *Case 4:* >> > let $a := /thesaurus/entry/synonym/term >> > let $b := /thesaurus/entry/term >> > return $a[not(.=$b)] >> > ------- >> > >> > It dies out of memory. >> >> That's odd. That pattern generally works pretty well for me on much >> larger datasets than 27 MB. >> >> If you're trying to check that all your synonyms are terms, I'd try >> >> let $a as xs:string+ := /thesaurus/entry/synonym/term/string() >> let $b as xs:string+ := /thesaurus/entry/term/string() >> >> return $a[not(. = $b)] >> >> so you do the "I want the string value of this element" only once, or if >> for some reason you need the elements later, maybe create maps and then >> compare the map keys? >> >> -- Graydon >> >> >> -- > Xavier-Laurent Salvador > Professeur Agr?g?, Ma?tre de Conf?rence HDR > ECC TTN 2018 22868H - ?quipe "Humanit?s Num?riques" > Coordinateur du r?seau international HiBHidEM > http://ttn.univ-paris13.fr > Universit? Paris 13 Sorbonne Paris Cit? > 99 avenue Jean-Baptiste Cl?ment > 93430 Villetaneuse > t?l. : (+33) 06 51.65.84.38 > email : xavier-laurent.salvador at univ-paris13.fr > site web: http://www.biblehistoriale.fr > site web: http://www.humanitesnumeriques.fr > > ------------------------------------------------------------ > Ce message peut contenir des informations r?serv?es exclusivement ? son > destinataire. Toute diffusion sans autorisation est interdite. Si vous > n'en ?tes pas le destinataire, merci de prendre contact avec l'exp?diteur > et de d?truire ce message. > > *This email may contain material for the sole use of the intended > recipient. Any forwarding without express permission is prohibited. If you > are not the intended recipient, please contact the sender and delete all > copies*. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190125/708e7e4c/attachment.htm> From christian.gruen at gmail.com Fri Jan 25 13:16:18 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Fri, 25 Jan 2019 13:16:18 +0100 Subject: [basex-talk] not() and != result In-Reply-To: <CAP+T2dzxK=fsPUhUH7K-OJKYi_qb=q0-bi6bdZFyAP60XASrLw@mail.gmail.com> References: <CAP+T2dw5rYQmbDaAAU1oUo3t7MBqanTELB0SRvt-0qq4NS3mCQ@mail.gmail.com> <20190124205253.GA18706@fenja> <CAP+T2dz07dqJq=D_PpN8iLiuU0WE+aXc674y2Us+owc9aAGWHg@mail.gmail.com> <CAP+T2dzxK=fsPUhUH7K-OJKYi_qb=q0-bi6bdZFyAP60XASrLw@mail.gmail.com> Message-ID: <CAP94bnOCaxGWc8-+xtcYKK1W8gt7joC0_A96trODGqHvMNE9aQ@mail.gmail.com> Hi Xavier, The reason for the memory usage might be an internal cache that is created for speeding up the execution time of your query. Out of interest: 1. Did the problem persist with Graydon?s alternative (and the standard of 2 GB RAM)? let $a as xs:string+ := /thesaurus/entry/synonym/term/string() let $b as xs:string+ := /thesaurus/entry/term/string() return $a[not(. = $b)] 2. If yes, can you successfully run the following query? let $b := /thesaurus/entry/term return /thesaurus/entry/synonym/term[not(.=$b)] If yes, how many thesaurus terms are stored in your database? count(/thesaurus/entry/term) Best, Christian On Fri, Jan 25, 2019 at 9:01 AM Xavier-Laurent SALVADOR <xavierlaurent.salvador at gmail.com> wrote: > > Hi, > > I checked it during night: same symptom. > And I realized I just downloaded a new 9.1.2 Basex on my good old and full hard drive while using this param: BASEX_JVM="-Xmx2g $BASEX_JVM"... > I moved it to an external drive, and ... shazam ! Hardware problem, so. > Thanks a lot ! > > Xavier-Laurent Salvador > > > Le jeu. 24 janv. 2019 ? 22:10, Xavier-Laurent SALVADOR <xavierlaurent.salvador at gmail.com> a ?crit : >> >> Awesome, thanks! >> >> Le jeu. 24 janv. 2019 ? 21:52, Graydon <graydonish at gmail.com> a ?crit : >>> >>> On Thu, Jan 24, 2019 at 09:41:18PM +0100, Xavier-Laurent SALVADOR scripsit: >>> > Hi List, >>> > >>> > I'm seing a little problem I can't understand with a small 27M thesaurus >>> > database. I created all indexes. >>> > When using the '!=' operator to compare two lists, I get a quick and wrong >>> > result: >>> > ------- >>> > *Case 1:* >>> > let $a := /thesaurus/entry/synonym/term >>> > let $b := /thesaurus/entry/term >>> > return $a[. = $b] >>> > ---> First result: "raccourcir" >>> >>> The = and != operators compare the sequences. >>> >>> = returns true if _any_ member of the left-hand sequence can be found in >>> the right-hand sequence. >>> >>> ('1','2','3') = ('1','asparagus','guillotine') >>> >>> returns true(). >>> >>> Similarly, != returns true if _any_ member of the left-hand sequence can >>> NOT be found in the right-hand sequence. >>> >>> ('1','2','3') != ('1','asparagus','guillotine') >>> >>> returns true(). >>> >>> != is an exceedingly tricksy operator; >>> >>> (1,2,3) != (1,2,3) >>> >>> is true. >>> >>> > *Case 4:* >>> > let $a := /thesaurus/entry/synonym/term >>> > let $b := /thesaurus/entry/term >>> > return $a[not(.=$b)] >>> > ------- >>> > >>> > It dies out of memory. >>> >>> That's odd. That pattern generally works pretty well for me on much >>> larger datasets than 27 MB. >>> >>> If you're trying to check that all your synonyms are terms, I'd try >>> >>> let $a as xs:string+ := /thesaurus/entry/synonym/term/string() >>> let $b as xs:string+ := /thesaurus/entry/term/string() >>> >>> return $a[not(. = $b)] >>> >>> so you do the "I want the string value of this element" only once, or if >>> for some reason you need the elements later, maybe create maps and then >>> compare the map keys? >>> >>> -- Graydon >>> >>> >> -- >> Xavier-Laurent Salvador >> Professeur Agr?g?, Ma?tre de Conf?rence HDR >> ECC TTN 2018 22868H - ?quipe "Humanit?s Num?riques" >> Coordinateur du r?seau international HiBHidEM >> http://ttn.univ-paris13.fr >> Universit? Paris 13 Sorbonne Paris Cit? >> 99 avenue Jean-Baptiste Cl?ment >> 93430 Villetaneuse >> t?l. : (+33) 06 51.65.84.38 >> email : xavier-laurent.salvador at univ-paris13.fr >> site web: http://www.biblehistoriale.fr >> site web: http://www.humanitesnumeriques.fr >> >> ------------------------------------------------------------ >> Ce message peut contenir des informations r?serv?es exclusivement ? son destinataire. Toute diffusion sans autorisation est interdite. Si vous n'en ?tes pas le destinataire, merci de prendre contact avec l'exp?diteur et de d?truire ce message. >> >> This email may contain material for the sole use of the intended recipient. Any forwarding without express permission is prohibited. If you are not the intended recipient, please contact the sender and delete all copies. From christian.gruen at gmail.com Fri Jan 25 13:40:43 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Fri, 25 Jan 2019 13:40:43 +0100 Subject: [basex-talk] BaseX 9.1.2: Maintenance Release In-Reply-To: <7933eab3-ad2f-0e15-3301-8737af9e5192@gmail.com> References: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> <55717221-de23-cc44-f46c-97b6c0d2710e@gmail.com> <CAP94bnMdVW2c0Usniquigm=+DR1CYp3g9iMZzUDQdPZkveGAjA@mail.gmail.com> <7933eab3-ad2f-0e15-3301-8737af9e5192@gmail.com> Message-ID: <CAP94bnNgQq=4W7X2Q+QyHppuwf6o7aKeAjccuwr+y_w7uQmPqA@mail.gmail.com> Hi George, I?m glad to hear you found a solution. > only doc() function will look at the correct base-uri directory, > while the import module statements, will try to import files from the > parent of the base directory. Are you sure about that? If you run the following query (after updating the base uri to your local environment)? declare base-uri 'file:///c:/users/user/desktop/'; import module namespace a = 'a' at 'a.xqm'; doc('x.xml') ?both the a.xqm module and the x.xml document will be looked up in the specified directory. If you remove the trailing slash from the URI, the files will be looked up in the parent directory (in accordance with the official specs). > Maybe a Java example showing the correct usage of the base-uri, or a > wiki page will help others in the future. Feel free to provide us with such an example, we?ll be happy to include it in our collection of Java examples. Best, Christian From gsf.greece at gmail.com Fri Jan 25 13:53:28 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Fri, 25 Jan 2019 14:53:28 +0200 Subject: [basex-talk] BaseX 9.1.2: Maintenance Release In-Reply-To: <CAP94bnNgQq=4W7X2Q+QyHppuwf6o7aKeAjccuwr+y_w7uQmPqA@mail.gmail.com> References: <CAP94bnM+0MX3LhG+6Ebxfmf-sJdhC9ehN75ZqQQLreZAFzwtQw@mail.gmail.com> <55717221-de23-cc44-f46c-97b6c0d2710e@gmail.com> <CAP94bnMdVW2c0Usniquigm=+DR1CYp3g9iMZzUDQdPZkveGAjA@mail.gmail.com> <7933eab3-ad2f-0e15-3301-8737af9e5192@gmail.com> <CAP94bnNgQq=4W7X2Q+QyHppuwf6o7aKeAjccuwr+y_w7uQmPqA@mail.gmail.com> Message-ID: <1f94132b-1466-3071-3427-1be69b3fc341@gmail.com> Hi Christian, If I'm not mistaken it only works if you declare it inside the query, which is not an option for my use case, because the scripts can run in different environments with paths that are setup in environment variables. This probably only affects the QueryProcessor constructor. But I will try to doublecheck that and send an email again if it's still doing the same. On 1/25/19 2:40 PM, Christian Gr?n wrote: > Are you sure about that? If you run the following query (after > updating the base uri to your local environment)? > > declare base-uri 'file:///c:/users/user/desktop/'; > import module namespace a = 'a' at 'a.xqm'; > doc('x.xml') > > ?both the a.xqm module and the x.xml document will be looked up in the > specified directory. If you remove the trailing slash from the URI, > the files will be looked up in the parent directory (in accordance > with the official specs). > > Feel free to provide us with such an example, we?ll be happy to > include it in our collection of Java examples. > > Best, > Christian From wd at creativosdigitales.co Mon Jan 28 18:10:42 2019 From: wd at creativosdigitales.co (wd at creativosdigitales.co) Date: Mon, 28 Jan 2019 12:10:42 -0500 Subject: [basex-talk] BaseX query performance Message-ID: <009001d4b72c$697498c0$3c5dca40$@creativosdigitales.co> Hi, We have a BaseX database of 3.2 MB size with 146,036,090 nodes on 451,124 documents Executing a simple query that returns only 4 elements is taking 6 seconds to run. Here is the execution plan: Optimized Query: *:entity Query: entity Result: - Hit(s): 4 Items - Updated: 0 Items - Printed: 549 b - Read Locking: deterioroTest - Write Locking: (none) Timing: - Parsing: 0.22 ms - Compiling: 1.13 ms - Evaluating: 4785.05 ms - Printing: 28.17 ms - Total Time: 4814.58 ms Query plan: <QueryPlan compiled="true" updating="false"> <CachedPath type="element()+" size="4"> <IterStep axis="child" test="*:entity" type="element()*"/> </CachedPath> </QueryPlan> And this is path list obtained from BaseX GUI doc(): 451124x entity: 4x name: 4x, leaf text(): 4x, 4 distinct strings, leaf active: 4x, leaf text(): 4x, string, leaf entityid: 4x, leaf text(): 4x, 4 distinct integers [1, 90086898], leaf dateadd: 3x, leaf text(): 3x, 3 distinct integers [1.536184706046E12, 1.538624904652E12], leaf description: 1x, leaf text(): 1x, string, leaf user: 3x password: 3x, leaf text(): 3x, 3 distinct strings, leaf role: 3x, leaf text(): 3x, 3 distinct integers [1, 3], leaf name: 3x, leaf text(): 3x, 3 distinct strings, leaf active: 3x, leaf text(): 3x, string, leaf entityid: 3x, leaf text(): 3x, 2 distinct integers [1, 90086898], leaf userid: 3x, leaf text(): 3x, 3 distinct integers [1, 1152438998], leaf email: 3x, leaf text(): 3x, 3 distinct strings, leaf dateadd: 2x, leaf text(): 2x, 2 distinct integers [1.537475569294E12, 1.537993241561E12], leaf project: 35x dateadd: 35x, leaf text(): 35x, 35 distinct integers [1.536184761155E12, 1.543940735205E12], leaf name: 35x, leaf text(): 35x, 28 distinct strings, leaf entityid: 35x, leaf text(): 35x, 4 distinct integers [1, 90086898], leaf projectid: 35x, leaf text(): 35x, 29 distinct integers [1, 20181030], leaf userUpId: 35x, leaf text(): 35x, 2 distinct integers [1, 123], leaf priority: 1x, leaf text(): 1x, string, leaf collection: 2x AA: 2x, leaf text(): 2x, double [0.021, 0.021], leaf BB: 2x, leaf text(): 2x, double [0.1268, 0.1268], leaf CC: 2x, leaf text(): 2x, double [0.2257, 0.2257], leaf A: 2x, leaf text(): 2x, double [0.0388, 0.0388], leaf B: 2x, leaf text(): 2x, double [0.1416, 0.1416], leaf garantianoadmisible: 2x, leaf text(): 2x, double [0.6, 0.6], leaf singarantia: 2x, leaf text(): 2x, 2 distinct doubles [0.65, 0.75], leaf entityid: 2x, leaf text(): 2x, 2 distinct integers [2, 90086898], leaf colateralfinanciero: 2x, leaf text(): 2x, double [0.12, 0.12], leaf info_asset: 181x entitytid: 181x, leaf text(): 181x, 4 distinct integers [1, 90086898], leaf total: 33x, leaf text(): 33x, 12 distinct integers [13856, 14381], leaf working: 33x, leaf text(): 33x, 2 distinct strings, leaf projectid: 181x, leaf text(): 181x, 29 distinct integers [1, 20181030], leaf status: 181x, leaf text(): 181x, 3 distinct strings, leaf asset: 450811x assetid: 450811x, leaf text(): 450811x, integers [42521, 296380], leaf lastdatepay: 450811x, leaf text(): 450811x, strings, leaf ammount: 450811x, leaf text(): 450811x, string, leaf term: 450811x, leaf text(): 450811x, 93 distinct integers [1, 180], leaf balance: 450811x, leaf text(): 450811x, doubles [1, 120901239], leaf rate: 450811x, leaf text(): 450811x, 69 distinct doubles [0.3346062573, 1.6], leaf payment: 450811x, leaf text(): 450811x, doubles [11760, 7043536], leaf pendingpayments: 450811x, leaf text(): 450811x, doubles [0, 146.41], leaf pagare: 450811x, leaf text(): 450811x, integers [42521, 296380], leaf calificacion: 450811x, leaf text(): 450811x, 5 distinct strings, leaf plazo: 450811x, leaf text(): 450811x, 93 distinct integers [1, 180], leaf linea: 450811x, leaf text(): 450811x, 12 distinct strings, leaf destino: 450811x, leaf text(): 450811x, 51 distinct strings, leaf tasacolocacion: 450811x, leaf text(): 450811x, 69 distinct doubles [0.3346062573, 1.6], leaf capitalinicial: 450811x, leaf text(): 450811x, integers [17347, 150000000], leaf saldocapital: 450811x, leaf text(): 450811x, doubles [1, 120901239], leaf anualidad: 450811x, leaf text(): 450811x, integers [11760, 7043536], leaf cedulasociado: 450811x, leaf text(): 450811x, strings, leaf codempresa: 464689x, leaf text(): 450811x, integers [0, 173767], leaf garantia: 450811x, leaf text(): 450811x, 7 distinct integers [1, 11], leaf nombre_de_la_garantia: 450811x, leaf text(): 450811x, 7 distinct strings, leaf diasmora: 450811x, leaf text(): 450811x, integers [0, 1280], leaf f_ultimop: 450811x, leaf text(): 450811x, strings, leaf nrocuotasfaltantesporcancelar: 450811x, leaf text(): 450811x, doubles [0, 146.41], leaf edad: 450811x, leaf text(): 450808x, 75 distinct doubles [18, 118], leaf Cobertura: 155142x, leaf text(): 155142x, 2 distinct integers [0, 1], leaf entityid: 450811x, leaf text(): 450811x, 4 distinct integers [1, 90086898], leaf projectid: 450811x, leaf text(): 450811x, 26 distinct integers [1, 201809], leaf diasmoracalculado: 450811x, leaf text(): 450811x, integers [-40, 2742], leaf simulation: 450811x payment: 7939701x number: 7939701x, leaf text(): 7939701x, integers [1, 655], leaf principal: 7939701x, leaf text(): 7939701x, doubles [-2334, 7019755.662381106], leaf interest: 7939701x, leaf text(): 7939701x, doubles [4.048068592674E-5, 1209057.2709579417], leaf newbalance: 7939701x, leaf text(): 7939701x, doubles [0, 1.1950670127095795E8], leaf total: 7939701x, leaf text(): 7939701x, doubles [0.0028131304069364, 7043536], leaf npv: 7939701x, leaf text(): 7939701x, doubles [0.00265466933451965, 7019849.776618443], leaf punishment: 502215x, leaf text(): 502215x, doubles [0, 3082100.44417925], leaf punishment: 450811x policyid: 450811x, leaf text(): 450811x, 20 distinct strings, leaf name: 450811x, leaf text(): 450811x, 18 distinct strings, leaf priority: 450811x, leaf text(): 450811x, 19 distinct integers [0, 65], leaf rate: 450811x, leaf text(): 450811x, 8 distinct doubles [0, 100], leaf formula: 450811x, leaf text(): 72339x, strings, leaf totalnpv: 450811x, leaf text(): 450811x, doubles [1, 1.2090123899999978E8], leaf totalpunishment: 450811x, leaf text(): 450811x, doubles [0, 9.421980520009069E7], leaf punish: 450811x, leaf text(): 450811x, doubles [-61.90886675214506, 3.097031737637267E7], leaf f_cargoprestamo: 295669x, leaf text(): 295669x, strings, leaf ESTADO: 70121x, leaf text(): 70121x, strings, leaf Rango_de_Mora: 295669x, leaf text(): 295669x, 8 distinct strings, leaf intcorriente: 295669x, leaf text(): 295669x, doubles [-93720, 4698722], leaf intcorrientenocontabilizado: 295669x, leaf text(): 295669x, integers [0, 4698106], leaf intmora: 295669x, leaf text(): 295669x, doubles [0, 84202], leaf intmoranocontabilizado: 295669x, leaf text(): 295669x, doubles [0, 2857125], leaf formapago: 295669x, leaf text(): 295669x, 2 distinct strings, leaf valorgarantia: 295669x, leaf text(): 295669x, integers [0, 419847500], leaf nrocuotascanceladas: 295669x, leaf text(): 295669x, doubles [0, 179.69], leaf salario: 295669x, leaf text(): 295669x, integers [0, 1152688304], leaf CATEGORIA: 268010x, leaf text(): 268010x, 3 distinct strings, leaf FECHA_DE_NACIMIENTO: 295669x, leaf text(): 295667x, strings, leaf Estado: 184022x, leaf text(): 184022x, 2 distinct strings, leaf _: 55836x, leaf policy: 88x policyid: 88x, leaf text(): 88x, 22 distinct strings, leaf rate: 88x, leaf text(): 88x, 8 distinct integers [0, 100], leaf name: 88x, leaf text(): 88x, 20 distinct strings, leaf formula: 88x, leaf text(): 88x, strings, leaf entityid: 88x, leaf text(): 88x, 4 distinct integers [1, 90086898], leaf priority: 88x, leaf text(): 88x, 21 distinct integers [10, 67], leaf Is there anything we could do to reduce the response time? This query is used in que UI of the application and the user is nagging about having to wait 6 seconds to open a Combo box. Thanks in advance for all the help, William David Vel?squez Creativo de Software Creativos Digitales S.A.S. Calle 30A # 83 - 53 Local 1033 Tel: 322 1730 - 311 709 8421 Medell?n, Colombia ?Necesita Integrar Sistemas? Conozca nuestra soluci?n de Integraci?n no Invasiva: http://creativosdigitales.co <http://creativosdigitales.co/> ?Preguntas sobre Facturaci?n Electr?nica? Visite: http://facturasyrespuestas.com <http://facturasyrespuestas.com/> S?ganos en Twitter <https://twitter.com/FacYRespuestas> @FacyRespuestas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190128/65df6afd/attachment.htm> From christian.gruen at gmail.com Mon Jan 28 19:48:11 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Mon, 28 Jan 2019 19:48:11 +0100 Subject: [basex-talk] BaseX query performance In-Reply-To: <009001d4b72c$697498c0$3c5dca40$@creativosdigitales.co> References: <009001d4b72c$697498c0$3c5dca40$@creativosdigitales.co> Message-ID: <CAP94bnOEyysXNohZctbH_xC4rAqmN+=yUge6Ec4s1QZSRYcGcg@mail.gmail.com> Hi William, > We have a BaseX database of 3.2 MB size with 146,036,090 nodes on 451,124 documents Is it 3.2 MB or GB? > Executing a simple query that returns only 4 elements is taking 6 seconds to run. If you have specific entity documents, you can partition your database and choose dedicated paths: /entities: entity1.xml entity2.xml ... /assets ... ?and address these documents via the db:open function: db:open('db', '/entities')/entity You can as well store these documents in a dedicated 'entity' database: db:open('entities')/entity Cheers, Christian > Here is the execution plan: > > Optimized Query: > *:entity > Query: > entity > Result: > - Hit(s): 4 Items > - Updated: 0 Items > - Printed: 549 b > - Read Locking: deterioroTest > - Write Locking: (none) > Timing: > - Parsing: 0.22 ms > - Compiling: 1.13 ms > - Evaluating: 4785.05 ms > - Printing: 28.17 ms > - Total Time: 4814.58 ms > Query plan: > <QueryPlan compiled="true" updating="false"> > <CachedPath type="element()+" size="4"> > <IterStep axis="child" test="*:entity" type="element()*"/> > </CachedPath> > </QueryPlan> > > And this is path list obtained from BaseX GUI > > doc(): 451124x > > entity: 4x > > name: 4x, leaf > > text(): 4x, 4 distinct strings, leaf > > active: 4x, leaf > > text(): 4x, string, leaf > > entityid: 4x, leaf > > text(): 4x, 4 distinct integers [1, 90086898], leaf > > dateadd: 3x, leaf > > text(): 3x, 3 distinct integers [1.536184706046E12, 1.538624904652E12], leaf > > description: 1x, leaf > > text(): 1x, string, leaf > > user: 3x > > password: 3x, leaf > > text(): 3x, 3 distinct strings, leaf > > role: 3x, leaf > > text(): 3x, 3 distinct integers [1, 3], leaf > > name: 3x, leaf > > text(): 3x, 3 distinct strings, leaf > > active: 3x, leaf > > text(): 3x, string, leaf > > entityid: 3x, leaf > > text(): 3x, 2 distinct integers [1, 90086898], leaf > > userid: 3x, leaf > > text(): 3x, 3 distinct integers [1, 1152438998], leaf > > email: 3x, leaf > > text(): 3x, 3 distinct strings, leaf > > dateadd: 2x, leaf > > text(): 2x, 2 distinct integers [1.537475569294E12, 1.537993241561E12], leaf > > project: 35x > > dateadd: 35x, leaf > > text(): 35x, 35 distinct integers [1.536184761155E12, 1.543940735205E12], leaf > > name: 35x, leaf > > text(): 35x, 28 distinct strings, leaf > > entityid: 35x, leaf > > text(): 35x, 4 distinct integers [1, 90086898], leaf > > projectid: 35x, leaf > > text(): 35x, 29 distinct integers [1, 20181030], leaf > > userUpId: 35x, leaf > > text(): 35x, 2 distinct integers [1, 123], leaf > > priority: 1x, leaf > > text(): 1x, string, leaf > > collection: 2x > > AA: 2x, leaf > > text(): 2x, double [0.021, 0.021], leaf > > BB: 2x, leaf > > text(): 2x, double [0.1268, 0.1268], leaf > > CC: 2x, leaf > > text(): 2x, double [0.2257, 0.2257], leaf > > A: 2x, leaf > > text(): 2x, double [0.0388, 0.0388], leaf > > B: 2x, leaf > > text(): 2x, double [0.1416, 0.1416], leaf > > garantianoadmisible: 2x, leaf > > text(): 2x, double [0.6, 0.6], leaf > > singarantia: 2x, leaf > > text(): 2x, 2 distinct doubles [0.65, 0.75], leaf > > entityid: 2x, leaf > > text(): 2x, 2 distinct integers [2, 90086898], leaf > > colateralfinanciero: 2x, leaf > > text(): 2x, double [0.12, 0.12], leaf > > info_asset: 181x > > entitytid: 181x, leaf > > text(): 181x, 4 distinct integers [1, 90086898], leaf > > total: 33x, leaf > > text(): 33x, 12 distinct integers [13856, 14381], leaf > > working: 33x, leaf > > text(): 33x, 2 distinct strings, leaf > > projectid: 181x, leaf > > text(): 181x, 29 distinct integers [1, 20181030], leaf > > status: 181x, leaf > > text(): 181x, 3 distinct strings, leaf > > asset: 450811x > > assetid: 450811x, leaf > > text(): 450811x, integers [42521, 296380], leaf > > lastdatepay: 450811x, leaf > > text(): 450811x, strings, leaf > > ammount: 450811x, leaf > > text(): 450811x, string, leaf > > term: 450811x, leaf > > text(): 450811x, 93 distinct integers [1, 180], leaf > > balance: 450811x, leaf > > text(): 450811x, doubles [1, 120901239], leaf > > rate: 450811x, leaf > > text(): 450811x, 69 distinct doubles [0.3346062573, 1.6], leaf > > payment: 450811x, leaf > > text(): 450811x, doubles [11760, 7043536], leaf > > pendingpayments: 450811x, leaf > > text(): 450811x, doubles [0, 146.41], leaf > > pagare: 450811x, leaf > > text(): 450811x, integers [42521, 296380], leaf > > calificacion: 450811x, leaf > > text(): 450811x, 5 distinct strings, leaf > > plazo: 450811x, leaf > > text(): 450811x, 93 distinct integers [1, 180], leaf > > linea: 450811x, leaf > > text(): 450811x, 12 distinct strings, leaf > > destino: 450811x, leaf > > text(): 450811x, 51 distinct strings, leaf > > tasacolocacion: 450811x, leaf > > text(): 450811x, 69 distinct doubles [0.3346062573, 1.6], leaf > > capitalinicial: 450811x, leaf > > text(): 450811x, integers [17347, 150000000], leaf > > saldocapital: 450811x, leaf > > text(): 450811x, doubles [1, 120901239], leaf > > anualidad: 450811x, leaf > > text(): 450811x, integers [11760, 7043536], leaf > > cedulasociado: 450811x, leaf > > text(): 450811x, strings, leaf > > codempresa: 464689x, leaf > > text(): 450811x, integers [0, 173767], leaf > > garantia: 450811x, leaf > > text(): 450811x, 7 distinct integers [1, 11], leaf > > nombre_de_la_garantia: 450811x, leaf > > text(): 450811x, 7 distinct strings, leaf > > diasmora: 450811x, leaf > > text(): 450811x, integers [0, 1280], leaf > > f_ultimop: 450811x, leaf > > text(): 450811x, strings, leaf > > nrocuotasfaltantesporcancelar: 450811x, leaf > > text(): 450811x, doubles [0, 146.41], leaf > > edad: 450811x, leaf > > text(): 450808x, 75 distinct doubles [18, 118], leaf > > Cobertura: 155142x, leaf > > text(): 155142x, 2 distinct integers [0, 1], leaf > > entityid: 450811x, leaf > > text(): 450811x, 4 distinct integers [1, 90086898], leaf > > projectid: 450811x, leaf > > text(): 450811x, 26 distinct integers [1, 201809], leaf > > diasmoracalculado: 450811x, leaf > > text(): 450811x, integers [-40, 2742], leaf > > simulation: 450811x > > payment: 7939701x > > number: 7939701x, leaf > > text(): 7939701x, integers [1, 655], leaf > > principal: 7939701x, leaf > > text(): 7939701x, doubles [-2334, 7019755.662381106], leaf > > interest: 7939701x, leaf > > text(): 7939701x, doubles [4.048068592674E-5, 1209057.2709579417], leaf > > newbalance: 7939701x, leaf > > text(): 7939701x, doubles [0, 1.1950670127095795E8], leaf > > total: 7939701x, leaf > > text(): 7939701x, doubles [0.0028131304069364, 7043536], leaf > > npv: 7939701x, leaf > > text(): 7939701x, doubles [0.00265466933451965, 7019849.776618443], leaf > > punishment: 502215x, leaf > > text(): 502215x, doubles [0, 3082100.44417925], leaf > > punishment: 450811x > > policyid: 450811x, leaf > > text(): 450811x, 20 distinct strings, leaf > > name: 450811x, leaf > > text(): 450811x, 18 distinct strings, leaf > > priority: 450811x, leaf > > text(): 450811x, 19 distinct integers [0, 65], leaf > > rate: 450811x, leaf > > text(): 450811x, 8 distinct doubles [0, 100], leaf > > formula: 450811x, leaf > > text(): 72339x, strings, leaf > > totalnpv: 450811x, leaf > > text(): 450811x, doubles [1, 1.2090123899999978E8], leaf > > totalpunishment: 450811x, leaf > > text(): 450811x, doubles [0, 9.421980520009069E7], leaf > > punish: 450811x, leaf > > text(): 450811x, doubles [-61.90886675214506, 3.097031737637267E7], leaf > > f_cargoprestamo: 295669x, leaf > > text(): 295669x, strings, leaf > > ESTADO: 70121x, leaf > > text(): 70121x, strings, leaf > > Rango_de_Mora: 295669x, leaf > > text(): 295669x, 8 distinct strings, leaf > > intcorriente: 295669x, leaf > > text(): 295669x, doubles [-93720, 4698722], leaf > > intcorrientenocontabilizado: 295669x, leaf > > text(): 295669x, integers [0, 4698106], leaf > > intmora: 295669x, leaf > > text(): 295669x, doubles [0, 84202], leaf > > intmoranocontabilizado: 295669x, leaf > > text(): 295669x, doubles [0, 2857125], leaf > > formapago: 295669x, leaf > > text(): 295669x, 2 distinct strings, leaf > > valorgarantia: 295669x, leaf > > text(): 295669x, integers [0, 419847500], leaf > > nrocuotascanceladas: 295669x, leaf > > text(): 295669x, doubles [0, 179.69], leaf > > salario: 295669x, leaf > > text(): 295669x, integers [0, 1152688304], leaf > > CATEGORIA: 268010x, leaf > > text(): 268010x, 3 distinct strings, leaf > > FECHA_DE_NACIMIENTO: 295669x, leaf > > text(): 295667x, strings, leaf > > Estado: 184022x, leaf > > text(): 184022x, 2 distinct strings, leaf > > _: 55836x, leaf > > policy: 88x > > policyid: 88x, leaf > > text(): 88x, 22 distinct strings, leaf > > rate: 88x, leaf > > text(): 88x, 8 distinct integers [0, 100], leaf > > name: 88x, leaf > > text(): 88x, 20 distinct strings, leaf > > formula: 88x, leaf > > text(): 88x, strings, leaf > > entityid: 88x, leaf > > text(): 88x, 4 distinct integers [1, 90086898], leaf > > priority: 88x, leaf > > text(): 88x, 21 distinct integers [10, 67], leaf > > > > Is there anything we could do to reduce the response time? This query is used in que UI of the application and the user is nagging about having to wait 6 seconds to open a Combo box. > > Thanks in advance for all the help, > > > > William David Vel?squez > Creativo de Software > Creativos Digitales S.A.S. > Calle 30A # 83 - 53 Local 1033 > Tel: 322 1730 - 311 709 8421 > Medell?n, Colombia > > ?Necesita Integrar Sistemas? Conozca nuestra soluci?n de Integraci?n no Invasiva: http://creativosdigitales.co > > > > ?Preguntas sobre Facturaci?n Electr?nica? Visite: http://facturasyrespuestas.com > > S?ganos en Twitter @FacyRespuestas > > > > From gsf.greece at gmail.com Tue Jan 29 22:39:20 2019 From: gsf.greece at gmail.com (George Sofianos) Date: Tue, 29 Jan 2019 23:39:20 +0200 Subject: [basex-talk] fn:serialize() behaviour Message-ID: <dcfdfaeb-b324-f449-df39-4c2febf60c89@gmail.com> Hi, This is probably a non issue, but I thought I should report it anyway. I was playing around with serialization options today and I noticed that: let $head := <head></head> let $body := <body></body> return serialize(($head, $body), map { "method": "html", "version": "5.0"}) will return <!DOCTYPE html> <head></head> <!DOCTYPE html> <body></body> I don't think fn:serialize() is defined in the xquery spec so it's implementation specific so I guess it also could be correct :) Also I have a question, I remember in the past a discussion about need for extra testing (XQuery spec wise) in BaseX? Is this still an issue? Hopefully I can find some time and help outwith that. Regards, George -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190129/67e0b45b/attachment.htm> From johannes.bauer at tanner.de Wed Jan 30 09:16:40 2019 From: johannes.bauer at tanner.de (Johannes Bauer) Date: Wed, 30 Jan 2019 09:16:40 +0100 Subject: [basex-talk] WWW-Authenticate header Message-ID: <9b7553e4-f844-91e0-8384-9dc520cecbe3@tanner.de> Hello BaseX Team, I'm trying to implement a RESTXQ service that uses JWT tokens for authorization and authentication. For this purpose I use a permission check annotated function that validates the token and returns a 401 response if the token is invalid or missing. declare %perm:check('/admin','{$perm}')function security:check-admin($perm) { let $token := $perm?authorization where empty($token) return <rest:response> <http:response status='401'> <http:header name='WWW-Authenticate' value='Bearer realm="c42"'/> </http:response> </rest:response> }; When I call the endpoint without token I get the expected response but the WWW-Authenticate header is different to the one that I have defined: WWW-Authenticate Basic realm="BaseX" Is there a way to override the BaseX provided header? Thanks for your input. Best regards Johannes -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190130/84f5b6de/attachment.htm> From christian.gruen at gmail.com Wed Jan 30 14:50:08 2019 From: christian.gruen at gmail.com (=?UTF-8?Q?Christian_Gr=C3=BCn?=) Date: Wed, 30 Jan 2019 14:50:08 +0100 Subject: [basex-talk] fn:serialize() behaviour In-Reply-To: <dcfdfaeb-b324-f449-df39-4c2febf60c89@gmail.com> References: <dcfdfaeb-b324-f449-df39-4c2febf60c89@gmail.com> Message-ID: <CAP94bnP+gDXHo4L8ROm9fUGzhstbrZOrJxH9MH1XtupdPDo_DQ@mail.gmail.com> Hi George, Your query will probably give a better result if you wrap head and body into an html element: let $head := <head></head> let $body := <body></body> return serialize( element html { $head, $body }, map { "method": "html", "version": "5.0"} ) But you are right, in general it makes no sense to output the document type declaration more than once. The nifty details are defined in a separate specification [1], The document is very comprehensive, but there are still some combinations of serialization parameters that are not discussed in full depth. I noticed that BaseX and Saxon behave differently in some cases. For example, Saxon may raise SEPM0004 error for both the "xml" and "xhtml" output method, while it?s only defined for "xml" in the spec. This error code is raised if a doctype declaration is requested, and if more than one item is to be output. BaseX followed the spec more closely, but as the Saxon solution seems more consistent to me, I just updated our code: ? The new snapshot [2] will output only one doctype declaration. ? SEPM0004 will also be raised for the xhtml method now (not for html, though, because html serialization is generally more lax than xml/xhtml serialization). ? In analogy with Saxon, I turned the implementation-defined default for the "include-content-type" parameter to "yes"; so the output of the query above will now be as follows: <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body></body> </html> Cheers, Christian [1] https://www.w3.org/TR/xslt-xquery-serialization-31/ [2] http://files.basex.org/releases/latest/ > This is probably a non issue, but I thought I should report it anyway. I was playing around with serialization options today and I noticed that: > > let $head := <head></head> > let $body := <body></body> > return serialize(($head, $body), map { "method": "html", "version": "5.0"}) > > will return > > <!DOCTYPE html> > <head></head> > <!DOCTYPE html> > <body></body> > > I don't think fn:serialize() is defined in the xquery spec so it's implementation specific so I guess it also could be correct :) > > Also I have a question, I remember in the past a discussion about need for extra testing (XQuery spec wise) in BaseX? Is this still an issue? Hopefully I can find some time and help out with that. > > Regards, > > George From johannes.bauer at tanner.de Wed Jan 30 15:42:05 2019 From: johannes.bauer at tanner.de (Johannes Bauer) Date: Wed, 30 Jan 2019 15:42:05 +0100 Subject: [basex-talk] %perm:allow annotation Message-ID: <8c1c26cc-4c5b-cfb7-a98a-18f22083a30e@tanner.de> Hi again, I've another question about the %perm:allow annotation. Is it possible to use multiple instances of this annotation? My expectation would be that any of the listed roles is allowed to call the function. For example: declare %rest:GET %rest:path("/multi") %perm:allow("role1") %perm:allow("role2") function test:multi()as item()* { <response>multi</response> }; declare %rest:GET %rest:path("/single") %perm:allow("role1") function test:single()as item()* { <response>single</response> }; declare %perm:check('/admin','{$perm}')function test:check($perm) { () }; When I call /multi the first time I get a response. But all further calls to /multi will result in a NullPointerException. Calls to /single do not have this problem. My error stacktrace is: java.lang.NullPointerException at org.basex.util.list.ObjectList.finish(ObjectList.java:235) at org.basex.query.value.seq.StrSeq.get(StrSeq.java:64) at org.basex.http.restxq.RestXqPerm.map(RestXqPerm.java:43) at org.basex.http.restxq.RestXqFunction.bind(RestXqFunction.java:263) at org.basex.http.restxq.RestXqResponse.bind(RestXqResponse.java:61) at org.basex.http.web.WebResponse.create(WebResponse.java:53) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:50) at org.basex.http.BaseXServlet.service(BaseXServlet.java:59) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) I think the error is at the location where he parses the perm:allow annotations. Best regards Johannes -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190130/a0d2b485/attachment.htm> From en-trance at yandex.ru Thu Jan 31 10:23:00 2019 From: en-trance at yandex.ru (=?utf-8?B?0JLQtdGC0L7RiNC60LjQvSDQktC70LDQtNC40LzQuNGA?=) Date: Thu, 31 Jan 2019 12:23:00 +0300 Subject: [basex-talk] ubuntu 18.04 (bionic) Message-ID: <8539011548926580@iva4-f774db2b5297.qloud-c.yandex.net> An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190131/402f0316/attachment.htm> From ms at basex.org Thu Jan 31 10:26:53 2019 From: ms at basex.org (Michael Seiferle) Date: Thu, 31 Jan 2019 10:26:53 +0100 Subject: [basex-talk] ubuntu 18.04 (bionic) In-Reply-To: <8539011548926580@iva4-f774db2b5297.qloud-c.yandex.net> References: <8539011548926580@iva4-f774db2b5297.qloud-c.yandex.net> Message-ID: <CANmxkNQq3WQohkA2CYKh=fds4VrE3kfxsnHqGvELssfWV=i2_A@mail.gmail.com> Hi, feel free to use the ZIP-file provided at http://files.basex.org/releases/9.1.2/ it should be up to date :) Best Michael On Thu, Jan 31, 2019 at 10:23 AM ???????? ???????? <en-trance at yandex.ru> wrote: > Hi! > > There is only 8.5.1 version for Ubuntu 18.04 at url > https://launchpad.net/basex/+packages > <https://launchpad.net/basex/+packages)> > Is there 9 version for that Ubuntu ? > > Thx > > -- > ? ?????????, > ???????? ???????? ???????????? > > -- -- Michael Seiferle, BaseX GmbH, http://basex.org |-- Firmensitz: Obere Laube 73, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Gesch?ftsf?hrer: | Dr. Christian Gr?n, Dr. Alexander Holupirek, Michael Seiferle `-- Fon: +49 7531 28 28 676, Fax: +49 7531 20 05 22 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190131/5183ab87/attachment.htm> From en-trance at yandex.ru Thu Jan 31 10:35:24 2019 From: en-trance at yandex.ru (=?utf-8?B?0JLQtdGC0L7RiNC60LjQvSDQktC70LDQtNC40LzQuNGA?=) Date: Thu, 31 Jan 2019 12:35:24 +0300 Subject: [basex-talk] ubuntu 18.04 (bionic) In-Reply-To: <CANmxkNQq3WQohkA2CYKh=fds4VrE3kfxsnHqGvELssfWV=i2_A@mail.gmail.com> References: <8539011548926580@iva4-f774db2b5297.qloud-c.yandex.net> <CANmxkNQq3WQohkA2CYKh=fds4VrE3kfxsnHqGvELssfWV=i2_A@mail.gmail.com> Message-ID: <8646521548927324@iva5-84bc572481fe.qloud-c.yandex.net> An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190131/88a4f5cd/attachment.htm> From joewiz at gmail.com Thu Jan 31 16:41:23 2019 From: joewiz at gmail.com (Joe Wicentowski) Date: Thu, 31 Jan 2019 10:41:23 -0500 Subject: [basex-talk] BaseX-specific serialization parameters In-Reply-To: <CAP94bnMb6id3Wr8n9xr58f8_7rNKUorzXzyPBFUgB-OaZ_Z8ug@mail.gmail.com> References: <CAHwerk2w=rWPGHEsmkaMcOCJYtjuQL=b08rRfo-JFaRcsi_XQw@mail.gmail.com> <CAP94bnMb6id3Wr8n9xr58f8_7rNKUorzXzyPBFUgB-OaZ_Z8ug@mail.gmail.com> Message-ID: <CAHwerk3ro11EwLo6J_hasURsGENAe2OfmQEwKhG+KQwc0-bWfw@mail.gmail.com> Thank you, Christian! Your edit to the documentation completely clears this up. Best, Joe On Thu, Jan 17, 2019 at 4:22 AM Christian Gr?n <christian.gruen at gmail.com> wrote: > Hi Joe, > > You are completely right; we ignored this specific rule of the > specification in the past. One of the reasons is that there are many > ways in BaseX how parameters can be supplied, and in some cases, no > namespace prefixes can be declared. I have just updated our > serialization [1]; I hope that the new wording is helpful. > > As you indicated, a change would require various changes in existing > XQuery libraries. I?ll have some more thoughts on this; maybe we could > tackle this with version 10 of BaseX. > > Thanks for your observation! > Christian > > [1] http://docs.basex.org/wiki/Serialization > > > > > As I read the serialization spec ( > https://www.w3.org/TR/xslt-xquery-serialization-31/#serparam), additional > values for the "method" parameter and additional serialization parameters > should use a non-null namespace URI (see quotes below). > > > > I believe BaseX's "csv" method or "tabulator" parameter ( > http://docs.basex.org/wiki/Serialization) would fall into this category. > Without such a non-null namespace URI, users may mistake BaseX-specific > serialization features for those supported in the spec. The solution that > the spec seems to suggest would be to use "basex:csv" or > "basex:tabulator". I bet such a change would be unwelcome to those with > code that uses the existing supported values, but aside from a disruptive > cycle of deprecating existing values, etc., an alternative would be for the > BaseX documentation to note this non-compliance with the spec. > > > > I realize this is nit-picky, but I thought it might be worth raising > given the potential for confusion mentioned above, and should this > discussion be of interest to anyone. > > > > Best, > > Joe > > > > Regarding values for the "method" parameter: > > > > > The value of the method parameter is an expanded QName. If the value > has a null namespace URI, then the local name identifies a method specified > in this document and MUST be one of xml, html, xhtml, text, json , or > adaptive; in this case, the output method specified MUST be used for > serializing. If the namespace URI is non-null, then it identifies an > implementation-defined output method; the behavior in this case is not > specified by this document. > > > > Regarding additional serialization parameters: > > > > > Implementations MAY define additional serialization parameters, and > MAY allow users to do so. For this purpose, the name of a serialization > parameter is considered to be a QName; the parameters listed above are > QNames whose expanded-QName has a null namespace URI, while any additional > serialization parameters that are either implementation-defined or defined > by the host language MUST have names that are namespace-qualified. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mailman.uni-konstanz.de/pipermail/basex-talk/attachments/20190131/c6d1ebf1/attachment.htm>