- BaseX-Talk - mailman.uni-konstanz.de

Huge CSV
by Giuseppe Celano 13 Aug '18

13 Aug '18

Hi, I am trying to work with a huge CSV file (about 380 MB), but If I built the database it seems that even simple operations cannot be evaluated. Is splitting the CSV file the only option or am I missing something here? Thanks. Giuseppe

3 8

Different interpretation of regex in eXist, Saxon and BaseX
by Andreas Mixich 10 Aug '18

10 Aug '18

Hi [rfc3986](https://tools.ietf.org/html/rfc3986#appendix-B) defines a nice regular expression, which groups any URI, including URN, by URI component. Interesting about this regex is the use of the '?' quantifier which makes every preceding group/component optional, thus matching either an URI or any other(!) string, since anything, that does not match one of the special groups, goes into a catch-all group (no.5), which keeps either the path or the full, arbitrary string. This is neglectable, since the input to this regex is guaranteed to be of the right type (a/@href/string()). Here is the relevant part from the RFC. Appendix B ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? 12 3 4 5 6 7 8 9 The numbers in the second line above are only to assist readability; they indicate the reference points for each subexpression (i.e., each paired parenthesis). We refer to the value matched for subexpression <n> as $<n>. For example, matching the above expression to http://www.ics.uci.edu/pub/ietf/uri/#Related results in the following subexpression matches: $1 = http: $2 = http $3 = //www.ics.uci.edu $4 = www.ics.uci.edu $5 = /pub/ietf/uri/ $6 = <undefined> $7 = <undefined> $8 = #Related $9 = Related where <undefined> indicates that the component is not present, as is the case for the query component in the above example. Therefore, we can determine the value of the five components as scheme = $2 authority = $4 path = $5 query = $7 fragment = $9 Going in the opposite direction, we can recreate a URI reference from its components by using the algorithm of Section 5.3. I tested this regex with Saxon, eXist and BaseX. eXist successfully parsed all the test-cases, I threw at it, into the right groups, Saxon and BaseX did not. The failure is: [FORX0003] Pattern matches empty string.. And that got me baffled, since all three processors use Java underneath and since the definition of the '?' quantifier, when used like this, seems to be: Makes the preceding item optional. Greedy, so the optional item is included in the match if possible. Which means, that *if* any of the group's contents match, they should be included, rather than producing an empty string. Why is it like that? And what can I do about it? I found no other URI parsing regex, that componentizes this way and would be compatible with XQuery. See, attached, a test-case. -- Goody Bye, Minden jót, Mit freundlichen Grüßen, Andreas Mixich

7 17

Add Command: Resource not found
by Florian Peschka 10 Aug '18

10 Aug '18

2 2

Re: [basex-talk] BaseX-Talk Digest, Vol 104, Issue 15
by Martin Lourduswamy 09 Aug '18

09 Aug '18

On Thu, Aug 9, 2018 at 2:29 AM, <basex-talk-request(a)mailman.uni-konstanz.de> wrote: > Send BaseX-Talk mailing list submissions to > basex-talk(a)mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request(a)mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner(a)mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Re: Different interpretation of regex in eXist, Saxon and > BaseX (Omar Siam) > 2. Re: BaseX insert/delete node performance (Christian Gr?n) > 3. Transaction management in BaseX 8.6.4 (Marc Coenegracht) > 4. Re: Transaction management in BaseX 8.6.4 (Christian Gr?n) > 5. Re: Transaction management in BaseX 8.6.4 (Christian Gr?n) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 8 Aug 2018 12:58:39 +0200 > From: Omar Siam <Omar.Siam(a)oeaw.ac.at> > To: basex-talk(a)mailman.uni-konstanz.de > Subject: Re: [basex-talk] Different interpretation of regex in eXist, > Saxon and BaseX > Message-ID: <91b47b0e-70a1-1336-ced6-e12eaa804cde(a)oeaw.ac.at> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi > > I think the problem is: There are numerous implemetations of regular > expressions which have a common subset but are different in the more > advanced features. > > Using the java regular expression implementation you can use greedy and > some other things. The XSL and XQuery implementation according to the > standards does not allow this and so misinterpretes the regular > expression. See here: https://www.w3.org/TR/xpath- > functions-31/#regex-syntax > > You can tell Saxon to use a different regexp engine such as the standard > Java one: > https://www.saxonica.com/html/documentation/functions/fn/matches.html > > Best regards > > Omar > > > Am 07.08.2018 um 21:38 schrieb Andreas Mixich: > > Hi > > > > [rfc3986](https://tools.ietf.org/html/rfc3986#appendix-B) defines a nice > > regular expression, which groups any URI, including URN, by URI > component. > > > > Interesting about this regex is the use of the '?' quantifier which > > makes every preceding group/component optional, thus matching either an > > URI or any other(!) string, since anything, that does not match one of > > the special groups, goes into a catch-all group (no.5), which keeps > > either the path or the full, arbitrary string. This is neglectable, > > since the input to this regex is guaranteed to be of the right type > > (a/@href/string()). > > > > Here is the relevant part from the RFC. > > > > Appendix B > > > > ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? > > 12 3 4 5 6 7 8 9 > > > > The numbers in the second line above are only to assist > > readability; they indicate the reference points for each > > subexpression (i.e., each paired parenthesis). We refer to the > > value matched for subexpression <n> as $<n>. For example, matching > > the above expression to > > > > http://www.ics.uci.edu/pub/ietf/uri/#Related > > > > results in the following subexpression matches: > > > > $1 = http: > > $2 = http > > $3 = //www.ics.uci.edu > > $4 = www.ics.uci.edu > > $5 = /pub/ietf/uri/ > > $6 = <undefined> > > $7 = <undefined> > > $8 = #Related > > $9 = Related > > > > where <undefined> indicates that the component is not present, > > as is the case for the query component in the above example. > > Therefore, we can determine the value of the five components as > > > > scheme = $2 > > authority = $4 > > path = $5 > > query = $7 > > fragment = $9 > > > > Going in the opposite direction, we can recreate a URI reference > > from its components by using the algorithm of Section 5.3. > > > > > > I tested this regex with Saxon, eXist and BaseX. eXist successfully > > parsed all the test-cases, I threw at it, into the right groups, Saxon > > and BaseX did not. The failure is: > > > > [FORX0003] Pattern matches empty string.. > > > > And that got me baffled, since all three processors use Java underneath > > and since the definition of the '?' quantifier, when used like this, > > seems to be: > > > > Makes the preceding item optional. Greedy, so the optional item > > is included in the match if possible. > > > > Which means, that *if* any of the group's contents match, they should be > > included, rather than producing an empty string. > > > > Why is it like that? And what can I do about it? I found no other URI > > parsing regex, that componentizes this way and would be compatible with > > XQuery. > > > > See, attached, a test-case. > > > > > > ------------------------------ > > Message: 2 > Date: Wed, 8 Aug 2018 19:16:51 +0200 > From: Christian Gr?n <christian.gruen(a)gmail.com> > To: BIRKNER Michael <Michael.BIRKNER(a)akwien.at> > Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> > Subject: Re: [basex-talk] BaseX insert/delete node performance > Message-ID: > <CAP94bnPj9-qHXKu6bbv_6FAiX=JQ28R8etd_oH31j=-=tPL+UQ@mail. > gmail.com> > Content-Type: text/plain; charset="utf-8" > > Michael, > > Welcome to the list. > > One thing you could try immediately is to call OPTIMIZE ? possibly followed > by the ALL flag, or db:optimize(..., true() ? and see if performance > improves. Obviously, this doesn't make sense after each single update > operation, but it could be called before a bigger number of updates is to > be performed. > > > The problem is that in my case, I have to do about 150000 inserts and > deletes, so it would take too much time. > > If you define all the insert expression (or a bigger number than just 1 or > 10) in a single XQuery expression (via a FLWOR expression), you will > benefit from various bulk optimizations. Did you try that already? > > Best, > Christian > > Hi > I did the same and the result are good, I took db:node-pre() suggestion from the website and from Christian and did a one time bulk delete first and then one bulk insert(try to use insert as last for speed) in a bulk operation in a loop. Optimize the database before and after each of the bulk operation. I use text optimization and optimize all takes longtime 1. session->"create index text" 2. for all delete pre-nodes-array session->query-->"delete statement 3. for all insert xml session->query->"insert statement" 4. session->"create index text This solved my issue of bulk update Christian suggested replace inserted of delete and insert but replace was taking a little extra time Thanks, > > BIRKNER Michael <Michael.BIRKNER(a)akwien.at> schrieb am Mi., 8. Aug. 2018, > 08:36: > > > Hello, > > > > > > I asked this question in StackOverflow concerning some performance > > problems I experienced when inserting nodes into a BaseX database: > > > > https://stackoverflow.com/questions/51595210/basex- > inserting-nodes-performance-problems > > > > I already made some progress, especially when it comes to querying all > > data I need for the updates. I work a lot with the indexes now. > > > > But I still have problems with inserting - and also deleting - nodes. It > > doesn't matter if I insert/delete nodes via a Java program or in the > editor > > of the BaseX GUI: Both is quite slow. Inserting just one node in the GUI > > with an XQuery like this one takes up to 3 seconds: > > > > insert node <related_record><title>Test title</title><author>Joe > > Lastname</author></related_record> into db:open-id('Database_Name', > 7947561) > > > > Deleting a node with the following command takes up to 7 seconds: > > delete node db:open-id('Database_Name', 88085737) > > > > The problem is that in my case, I have to do about 150000 inserts and > > deletes, so it would take too much time. > > > > Maybe my database is just too big to be performant? Or some settings are > > wrong? I'm very new to BaseX (and XML databases in general) so maybe > there > > are just some errors I don't see. I also give you some information on my > > database that I copied from the info screen of the BaseX GUI: > > > > > > Database Properties > > NAME: Database_Name > > SIZE: 2568 MB > > NODES: 135607105 > > DOCUMENTS: 1 > > BINARIES: 0 > > TIMESTAMP: 2018-08-07T07:05:56.000Z > > UPTODATE: true > > > > Resource Properties > > INPUTPATH: /path/to/file.xml > > INPUTSIZE: 1774 MB > > INPUTDATE: 2018-07-24T14:32:58.000Z > > > > Indexes > > TEXTINDEX: true > > ATTRINDEX: true > > TOKENINDEX: false > > FTINDEX: false > > TEXTINCLUDE: > > ATTRINCLUDE: > > TOKENINCLUDE: > > FTINCLUDE: > > LANGUAGE: English > > STEMMING: false > > CASESENS: false > > DIACRITICS: false > > STOPWORDS: > > UPDINDEX: true > > AUTOOPTIMIZE: false > > MAXCATS: 100 > > MAXLEN: 96 > > SPLITSIZE: 0 > > > > Best regards, > > Michael > > > > Beachten Sie, dass Sie uns ab sofort unter einer ge?nderten Rufnummer > > erreichen. Bitte speichern Sie gleich Ihren Kontakt zur AK Wien ein > unter *501 > > 65 1*, gefolgt von der gewohnten Durchwahl. > > Dieses Mail ist ausschlie?lich f?r die Verwendung durch die/den darin > > genannten AdressatInnen bestimmt und kann vertrauliche bzw rechtlich > > gesch?tzte Informationen enthalten, deren Verwendung ohne Genehmigung > durch > > den/ die AbsenderIn rechtswidrig sein kann. Falls Sie dieses Mail > > irrt?mlich erhalten haben, informieren Sie uns bitte und l?schen Sie die > > Nachricht. UID: ATU 16209706 I > > https://wien.arbeiterkammer.at/Datenschutz_(DSGVO).html > > >

1 0

Re: [basex-talk] Transaction management in BaseX 8.6.4
by Christian Grün 09 Aug '18

09 Aug '18

Hi Marc (cc to the list), If the database replacement is defined as a single REST operation, you won’t encounter any problems; other transactions will need to wait until your database has been fully created. Best, Christian Marc Coenegracht <marc(a)crosseyed.nl> schrieb am Do., 9. Aug. 2018, 00:21: > Hi Christian, > > Thanks for the quick answer. > > Defining the operations in a single query would be preferable but isn't > possible, since the read operation is simply triggered by a website > visitor, and the creation of the new DB (indeed overwriting the old DB > with db:create) is performed by an action of the CMS admin. > > So, these operations can happen at the exact same moment. The > inconvenience at the front-end will be minimal, I'm just wondering if > these concurrent operations can cause problems with BaseX or the > db:create operation. > > best, > Marc > > On Wed, 8 Aug 2018, Christian Grün wrote: > > > Hi Marc, > > > > As one XQuery expression is one transaction, the best approach is to > > define your operations in a single query. If you call db:create, an > > existing database will be overwritten, and the function allows you to > > specify some initial input. > > > > Hope this helps, > > Christian > > > > > > > > On Wed, Aug 8, 2018 at 10:31 PM Marc Coenegracht <marc(a)crosseyed.nl> > wrote: > >> > >> Hi, > >> > >> A CMS occasionally recreates some existing databases of a production > site. > >> The databases are deleted and again created with the new content within > >> a few seconds. > >> > >> What happens if a read operation is taking place during this process? > Can > >> it cause problems with the recreation of the DB or with the BaseX server > >> instance? > >> > >> Of course it is possible to update the databases instead, but this > process > >> is a lot simpler and probably faster too. > >> > >> All operations are executed running xquery scripts with REST using the > >> BaseX http server. > >> > >> > >> Marc > >

1 0

Transaction management in BaseX 8.6.4
by Marc Coenegracht 08 Aug '18

08 Aug '18

Hi, A CMS occasionally recreates some existing databases of a production site. The databases are deleted and again created with the new content within a few seconds. What happens if a read operation is taking place during this process? Can it cause problems with the recreation of the DB or with the BaseX server instance? Of course it is possible to update the databases instead, but this process is a lot simpler and probably faster too. All operations are executed running xquery scripts with REST using the BaseX http server. Marc

2 1

CSV underscore
by Giuseppe Celano 07 Aug '18

07 Aug '18

I am importing a csv file having column names with single underscores which are transformed by BaseX into double underscores. Why? Is it possible to keep the original name? Thanks. Giuseppe

2 1

BaseX and oXygen
by Andreas Mixich 02 Aug '18

02 Aug '18

Hi, I remember, that I was able to run BaseX as execution engine for the oXygenXML XQuery editor. This seems to be impossible since a few oXygen versions. All that's left is the WebDAV part. I did not manage to run my old config, creating new ones did not help either. I remember having read a post by Michael Kay, who seemed to have some problems with XQJ not being up-to-date and some licensing issues making it impossible to fork/update or something like that. The bug-tracker and mailing-list links on the XQJ site are dead as well. -- Goody Bye, Minden jót, Mit freundlichen Grüßen, Andreas Mixich

2 2

Can't get `cdata-section-elements` to work at all for XSLT output
by Hugh Guiney 02 Aug '18

02 Aug '18

Hello, First off, loving BaseX so far! Using it as the backend for an API I’m building. However, I’m running into an issue. I’m trying to transform my database XML into an RSS 2.0 feed. It’s mostly working fine, but I can’t output CDATA content at all, which I need to do for `content:encoded` elements. Specs: - BaseX 9.0.2 (started via basexserver script) - Saxon-HE 9.8.0.12J from Saxonica - java version "1.8.0_112" - basex 0.9.0 (NodeJS) - macOS Sierra 10.12.6 ### First Attempt I set `cdata-section-elements` in the XSLT. rss.xq: ``` xquery version "3.0"; declare option output:omit-xml-declaration "no"; let $in := <root> <child>hello</child> </root> let $style := doc( 'rss.xslt' ) return xslt:transform( $in, $style ) ``` rss.xslt: ``` <?xml version="1.0" encoding="UTF-8"?> <xsl:transform version="3.0" xmlns="http://backend.userland.com/rss2" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:content="http://purl.org/rss/1.0/modules/content/" > <xsl:output omit-xml-declaration="no" cdata-section-elements="content:encoded" /> <xsl:template match="/"> <rss version="2.0"> <content:encoded>hi</content:encoded> <content:encoded><xsl:text>howdy</xsl:text></content:encoded> <content:encoded><xsl:value-of select="//child" /></content:encoded> </rss> </xsl:template> </xsl:transform> ``` Result: ``` <?xml version="1.0" encoding="UTF-8"?> <rss xmlns="http://backend.userland.com/rss2" xmlns:content="http://purl.org/rss/1.0/modules/content/" version="2.0"> <content:encoded>hi</content:encoded> <content:encoded>howdy</content:encoded> <content:encoded>hello</content:encoded> </rss> ``` No CDATA sections. ### Second Attempt I set `cdata-section-elements` in the XQuery. rss.xq: ``` xquery version "3.0"; declare namespace content = "http://purl.org/rss/1.0/modules/content/"; declare option output:omit-xml-declaration "no"; declare option output:cdata-section-elements "content:encoded"; let $in := <root> <child>hello</child> </root> let $style := doc( 'rss.xslt' ) return xslt:transform( $in, $style ) ``` rss.xslt: [Unchanged] Result: [XPST0081] No namespace declared for 'content:encoded'. Clearly I declared the namespace two lines up. This looks like a bug to me, but any help appreciated if I’ve missed a step here. Thanks, Hugh

4 8

Mailing list configuration or my MUA?
by Andreas Mixich 02 Aug '18

02 Aug '18

Hi, may it be some configuration issue, I have on my side, or is the mailing list set up in a way, that each reply goes to the user's personal email-address, rather than to the list's posting address? Whenever I hit "Reply" the user's email address is put into the "To" field, whereas I would assume, the list's address to go there. I just, again, sent a reply, that should go to the list to Christian, in person (sorry about that). When I hit "Reply All" I get the list's address as a "CC". -- Goody Bye, Minden jót, Mit freundlichen Grüßen, Andreas Mixich

2 1