September 2015 - BaseX-Talk - mailman.uni-konstanz.de

duplicate declaration error-message when using dynamic module-imports ( inspect:functions)
by Rob Stapper 16 Sep '15

16 Sep '15

Hi, I get a "duplicate declaration" error-message when I using the inspect:functions-function. Consider program: testRun1-2a.xq. The program imports module: module1.xqm and module2a.xqm. Module: module1.xqm, declares a variable: $mod1:x, and a function: mod1:show( $x). Module: module2a.xqm, declare a function: mod2a:run, which calls a function by a given function-name: $name in a dynamically imported module by a given module-uri: $uri. Program: testRun1-2a.xq, calls the run-function in module2a and tells it to use variable: $mod1:x, on function: mod1:show in module: module1.xqm. This triggers an error: Duplicate declaration on static variable $mod1:x. On the other hand, program: testRun1-2b.xq, does exactly the same except that module:module2b.xqm, is statically linked to module: module1.xqm, instead of dynamically as happens in module:module2a.xqm. Shouldn't the dynamically import of modules behave in the same manner as its static counterpart? Is it possible to achieve this? Cheers, Rob PS the examples are a bit awkward but in my real-life program this construction makes sense (to my anyway). --- Dit e-mailbericht is gecontroleerd op virussen met Avast antivirussoftware. https://www.avast.com/antivirus

2 1

Stopword-related NullPointerException (in FTWords.java)
by Ron Katriel 16 Sep '15

16 Sep '15

Hi, I encountered a peculiar error with a query using a stopwords file in the context of a full text search. The query joins two XML databases: CT.gov (containing 86635503 nodes) and the 2015 MeSH dictionary (containing 12064461 nodes). I am debugging using CT.gov trial NCT00303472, hardcoded in the ‘where' clause of the following query: let $trees := db:open('MeSH')/DescriptorRecordSet/DescriptorRecord for $article in db:open('CTGov')/clinical_study where $article/id_info/nct_id = 'NCT00303472' let $mesh := $article/condition_browse/mesh_term let $tn1 := $trees[DescriptorName/String contains text { $mesh }] let $tn2 := $trees[DescriptorName/String contains text { $mesh } using stop words at "/Volumes/Extra/Documents/Standards/MeSH/stopwords.txt"]/TreeNumberList/TreeNumber return <match> { $article/id_info/nct_id, $mesh, $tn2 } </match> When the return clause contains the variable $tn2 (i.e., using stopwords - as shown above) a Java NullPointerException is generated (see the stack trace below). However, when only $tn1 is returned there is no problem (the code for $tn2 is removed by the optimizer). The issue is related to a specific stopword (“syndrome”). When the stopword is removed from the file the exception does not occur. Surprisingly, when the stopword is in uppercase (“Syndrome”) the issue does not occur - even though the target MeSH term in this CT.gov trial is in uppercase, that is <mesh_term>Syndrome</mesh_term> Am I doing something wrong, or is this a real bug in BaseX? If the former, please suggest a workaround as I would like to filter out generic MeSH terms that match the stopwords before any further processing (I removed a lot of code from the above query to make it easier to debug). Thanks, Ron Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.2 Java: Oracle Corporation, 1.8.0_20 OS: Mac OS X, x86_64 Stack Trace: java.lang.NullPointerException at org.basex.query.expr.ft.FTWords$1.next(FTWords.java:166) at org.basex.query.expr.ft.FTIndexAccess$1.next(FTIndexAccess.java:48) at org.basex.query.expr.ft.FTIndexAccess$1.next(FTIndexAccess.java:45) at org.basex.query.iter.Iter.value(Iter.java:53) at org.basex.query.expr.ParseExpr.value(ParseExpr.java:67) at org.basex.query.QueryContext.value(QueryContext.java:421) at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:41) at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:22) at org.basex.query.QueryContext.iter(QueryContext.java:410) at org.basex.query.expr.List$1.next(List.java:133) at org.basex.query.expr.constr.Constr.add(Constr.java:70) at org.basex.query.expr.constr.CElem.item(CElem.java:92) at org.basex.query.expr.constr.CElem.item(CElem.java:23) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:43) at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99) at org.basex.query.MainModule$1.next(MainModule.java:114) at org.basex.query.QueryContext.cache(QueryContext.java:660) at org.basex.query.QueryProcessor.cache(QueryProcessor.java:103) at org.basex.core.cmd.AQuery.query(AQuery.java:83) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:398) at org.basex.core.Command.execute(Command.java:100) at org.basex.gui.GUI.exec(GUI.java:472) at org.basex.gui.GUI.access$400(GUI.java:43) at org.basex.gui.GUI$7.run(GUI.java:412) Compiling: - pre-evaluating db:open("MeSH") - pre-evaluating db:open("CTGov") - inlining $trees_0 - applying full-text index for { $mesh_2 } using language 'English' - applying full-text index for { $mesh_2 } using language 'English' - inlining $tn2_4 - removing variable $tn1_3 - applying text index for "NCT00303472" - rewriting where clause(s) Query: let $trees := db:open('MeSH')/DescriptorRecordSet/DescriptorRecord for $article in db:open('CTGov')/clinical_study where $article/id_info/nct_id = 'NCT00303472' let $mesh := $article/condition_browse/mesh_term let $tn1 := $trees[DescriptorName/String contains text { $mesh }] let $tn2 := $trees[DescriptorName/String contains text { $mesh } using stop words at "/Volumes/Extra/Documents/Standards/MeSH/stopwords.txt"]/TreeNumberList/TreeNumber return <match> { $article/id_info/nct_id, $mesh, $tn2 } </match> Optimized Query: for $article_1 in db:text("CTGov", "NCT00303472")/parent::*:nct_id/parent::*:id_info/parent::*:clinical_study let $mesh_2 := $article_1/*:condition_browse/*:mesh_term return element match { (($article_1/*:id_info/*:nct_id, $mesh_2, ft:search("MeSH", { $mesh_2 } using language 'English')/parent::*:String/parent::*:DescriptorName/parent::*:DescriptorRecord/TreeNumberList/TreeNumber)) } Query plan: <QueryPlan compiled="true"> <GFLWOR> <For> <Var name="$article" id="1"/> <IterPath> <ValueAccess data="CTGov" type="TEXT" name="*:nct_id"> <Str value="NCT00303472" type="xs:string"/> </ValueAccess> <IterStep axis="parent" test="*:id_info"/> <IterStep axis="parent" test="*:clinical_study"/> </IterPath> </For> <Let> <Var name="$mesh" id="2"/> <IterPath> <VarRef> <Var name="$article" id="1"/> </VarRef> <IterStep axis="child" test="*:condition_browse"/> <IterStep axis="child" test="*:mesh_term"/> </IterPath> </Let> <CElem> <QNm value="match" type="xs:QName"/> <List> <IterPath> <VarRef> <Var name="$article" id="1"/> </VarRef> <IterStep axis="child" test="*:id_info"/> <IterStep axis="child" test="*:nct_id"/> </IterPath> <VarRef> <Var name="$mesh" id="2"/> </VarRef> <CachedPath> <FTIndexAccess data="MeSH"> <FTWords> <VarRef> <Var name="$mesh" id="2"/> </VarRef> </FTWords> </FTIndexAccess> <IterStep axis="parent" test="*:String"/> <IterStep axis="parent" test="*:DescriptorName"/> <IterStep axis="parent" test="*:DescriptorRecord"/> <IterStep axis="child" test="TreeNumberList"/> <IterStep axis="child" test="TreeNumber"/> </CachedPath> </List> </CElem> </GFLWOR> </QueryPlan> Ron Katriel, Ph.D. | Senior Data Scientist | Medidata Solutions 350 Hudson Street, 7th Floor, New York, NY 10014 rkatriel(a)mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800

2 6

Different behavior with same query in BaseX 7.9 and 8.2.3
by Antoine WOLF 15 Sep '15

15 Sep '15

*Hi,* *We are in the process of evaluating BaseX and recently upgraded from version 7.9 to 8.2.3We have a DB “TESTDB” with 50 xml files, named Fund1.xml, Fund2.xml, Fund3.xml,…. each having the same structure:Fund1.xml--------------< fund>…<properties>…<domain reference=”fund1”>…</domain>…</properties>…</fund>Fund2.xml--------------< fund>…<properties>…<domain reference=”fund2”>…</domain>…</properties>…</fund>….….The following query does not behave the same on both versions :For $d at $i in db:open(“TESTDB”)/fund/properties/domainwhere $i < 2return data($d/@reference)Result in 7.9------------------“Fund1”Result in 8.2.3-----------------“Fund1 Fund2 Fund3 Fund4 …”Is this a bug or is there something wrong with the query ?Best Regards,Antoine Wolf*

3 3

Unexpected error: Improper use? Potential bug?
by Pierre-Yves JALLUD 15 Sep '15

15 Sep '15

Hi all, I have some troubles with BaseX request and webdav access. My situation is the following: the acces to BaseX with webdav is with an oXygen (developer version 17) with Ubuntu 14.0.4 and the BaseX server is running under a Scientific Linux 64b (2.6.32-573.3.1.el6.x86_64 ... if the precision is necessary ;)). The BaseX server is behind a web cluster using a load balancer and an APACHE HTTP server. All the access passing thrue this server and the different ports access are managed with HAProxy. If the context is not clear enough, don't hesitate to ask more precisions. The symptom is that the BaseX server is freezing after a webdav request. The log's extracts you can find below track the following use case: - I try an access to http://BaseX.huma-num.fr/webdav/bvm/bvm.xml with oXygen - I recieve a HTTP error 500 - I try to access to the BaseX server with my web site and more than one minute later, I have a timeout. ... but the worst symptom is that in some cases (not always), the requested db get corrupted and emptied (totally or partly) : resources are missing or they throw End of File exceptions when we then try to reopen them. This use case can appear many times a day... but not systematically. And other colleagues have the same problems with other version of oXygen with other OS (Windows), but allways with the same BaseX server. Does anyone of you have an idea to solve my problem? Thanks Pierre-Yves Some extracts of the log file: ... 17:13:06.694 X.X.X.X:34953 admin REQUEST [GET] http://BaseX.huma-num.fr/webdav/bvm/bvm.xml 17:13:06.845 X.X.X.X:34954 admin REQUEST [LOCK] http://BaseX.huma-num.fr/webdav/bvm/bvm.xml 17:13:06.888 X.X.X.X:34954 admin 200 43.82 ms 17:13:08.437 X.X.X.X:34953 admin 500 Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.2.1 Java: Oracle Corporation, 1.8.0_31 OS: Linux, amd64 Stack Trace: java.lang.RuntimeException: org.eclipse.jetty.io.EofException at org.basex.http.webdav.BXServletResponse.close(BXServletResponse.java:80) at org.basex.http.webdav.WebDAVServlet.run(WebDAVServlet.java:46) at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextH... 1743.02 ms ... 17:15:20.041 X.X.X.X:38436 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/skepsis/Cicero_De_Finibus.xml 17:15:40.067 X.X.X.X:38770 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/skepsis/Cicero_De_Finibus.xml 17:16:00.078 X.X.X.X:39090 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/skepsis/ 17:16:00.170 X.X.X.X:39091 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/bvm/bvm.xml 17:16:20.194 X.X.X.X:39582 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/bvm/bvm.xml 17:17:00.904 X.X.X.X:39091 admin 200 60733.77 ms 17:17:00.907 X.X.X.X:39582 admin 200 40712.41 ms 17:17:00.909 X.X.X.X:38770 admin 200 80842.45 ms 17:17:00.910 X.X.X.X:38436 admin 200 100868.15 ms 17:17:01.004 X.X.X.X:39090 admin 200 60926.05 ms 17:19:37.189 Y.Y.Y.Y:39926 admin ERROR Timeout exceeded. 60000.62 ms ...

3 6

Re: [basex-talk] Xquery collections
by michele.greco2＠email.it 15 Sep '15

15 Sep '15

Hi, i tried this xquery: for $e in db:open("dbName")//w:tc[.//text()="Nome"]  return $e/./following::text()except   (for $x in db:open("dbName")//w:tc[.//text()="Indirizzo"] return $x//following::text())   but only returns the result of the first document. How do i do it for all the documents? --------- Original Message -------- Da: "Dirk Kirsten" <dk(a)basex.org> To: Cc: basex-talk(a)mailman.uni-konstanz.de Oggetto: Re: [basex-talk] Xquery collections Data: 03/09/15 18:08 Ciao Michele, welcome to the community :-) Collections in BaseX are basically databases. You can simply open a database e.g. by issueing the XQuery db:open('mydatabase'). This will use all documents in your database. You can find some more examples at http://docs.basex.org/wiki/Databases It might also be helpful for you to read some of the Tutorials given at http://docs.basex.org/wiki/Getting_Started like BaseX for Dummies, given a concise introduction to BaseX. Spero che sia d'aiuto Dirk On 09/03/2015 05:48 PM, michele.greco2(a)email.it wrote: Hello I Michele i'm new to basex, i created a collection with different documents .xml. I want to know how to query the entire collection e its documents at one xquery. I'm trying,but i can't.   Thanks. MG   ---- ZE-Light e ZE-Pro: servizi zimbra per caselle con dominio email.it, per tutti i dettagli clicca qui Sponsor: Idee regalo classiche o alternative? Trova l'offerta migliore in un click Clicca qui -- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22 -- ZE-Light e ZE-Pro: servizi zimbra per caselle con dominio email.it, per tutti i dettagli Clicca qui http://posta.email.it/caselle-di-posta-z-email-it/?utm_campaign=email_Zimbr… Sponsor: Soluzioni di email hosting per tutte le esigenze: dalle caselle gratuite a quelle professionali su piattaforma Zimbra, da quelle su proprio dominio a quelle certificate PEC. Confronta le soluzioni Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=13325&d=4-9

3 2

Validate module and catalogs
by Lizzi, Vincent 15 Sep '15

15 Sep '15

It appears that the Validate module is not using catalogs. I'm working with XML that contains a DTD DOCTYPE declaration with a public identifier and a system identifier that contains the filename of the DTD, and I have set up a catalog that provides the DTD location. Here is a rough example: declare option db:chop 'false'; declare option db:dtd 'true'; declare option db:catfile 'schemas/catalog.xml'; let $xmlFile := 'path/to/file.xml' let $parsed := doc($xmlFile) let $report := validate:dtd-report($xmlFile) return ($report, $parsed) This fails on the validate:dtd-report line. The error report indicates that BaseX is trying to find the DTD in the same folder as the XML file. I know the catalog is working because the doc() call is parsing the XML and populating default attributes, and if I remove the db:catfile option declaration the doc() call fails. If I add a path to the DTD in the second parameter in validate:dtd-report($xmlFile, 'path/to/dtd.dtd') the validation works. I'm not sure why this is needed because the XML contains a DOCTYPE and the catalog should be resolving the DTD location. Is this intentional? There are use cases to require that validation is done using a specific DTD and not simply any DTD that the XML happens to declare. In other cases however it is easier to validate using the DTD that is already identified in the DOCTYPE of the XML file. Is this a bug? Thanks, Vincent

3 3

Out of Main Memory - Inconsistent DB
by Martín Ferrari 14 Sep '15

14 Sep '15

Hi all, I think one of our DB seems to have gotten corrupt. I wonder if there's a command that will recover it. Issuing REPLACE or OPTIMIZE on it will result in "Out of Main Memory" errors. We have DBs that occupy more space and have more documents, so I assume the problem must be that this DB broke somehow. I've tried raising the -Xmx parameter to 1024m, but still get the same "Out of Main Memory" error. Thanks, Martín. INSPECT Checking main table (11292441 nodes):- 0 invalid node kinds- 0 invalid parent references- 87 wrong parent/descendant relationships (pre: 10932445,..)Warning: Database is inconsistent. INFO DB Database Properties Name: OrderInquiryV2_1 Size: 2138 MB Nodes: 11292441 Documents: 251980 Binaries: 0 Timestamp: 2015-09-10T16:40:29.000Z Resource Properties Timestamp: 2015-09-08T23:11:28.497Z Encoding: UTF-8 CHOP: false Indexes Up-to-date: false TEXTINDEX: false ATTRINDEX: false FTINDEX: false LANGUAGE: English STEMMING: false CASESENS: false DIACRITICS: false STOPWORDS: UPDINDEX: false AUTOOPTIMIZE: false MAXCATS: 100 MAXLEN: 96

2 1

Re: [basex-talk] size on collection in the time (Marc)
by cmarchand＠oxiane.com 14 Sep '15

14 Sep '15

I confirm the problem : on another database (same data, same script.xbs, different server), we have exactly the same problem : the database grows each time the script is run. I'm going to prepare a test-case with small public data tonight. Best regards, Christophe Hello Dirk, We use a BaseX script (.xbs) with a list of REPLACE. We try to investigate this point if this is reproducable or not. If I find this problem again, I take a copy of the collection before to delete her. Thank you for answer. Marc Le 03/09/2015 09:52, Dirk Kirsten a ?crit : > Hello Marc, > > sure, a replace should lead to a true replace, i.e. the old data will be > dropped. So this sounds like a bug, but it is difficult to replicate > from what you describe. > > Do you think it would be possible to make the issue reproducable, maybe > also with a smaller data set? Also, how exactly do you add documents or > do you also do any queries and maybe other updating queries? > > Cheers > Dirk > > On 09/02/2015 05:36 PM, Marc wrote: > >> Hi, >> I have a remark about the size of the files of a collection. >> We do a lot of updates (REPLACE) with an attributes index but without >> a text index. >> I see that the size of the files grows up in the time we arrive until >> 30Go until the limit of the filessytems quota. >> When I drop the collection and recreate it from the last version of >> the files the size was only 6Go. >> >> We will redo the test with the time, but I just want to know if the >> program clears the disk use when he does a REPLACE? >> >> Marc ------------------------------ Message: 4 Date: Thu, 3 Sep 2015 10:01:05 +0200 From: Marc <Marc.Liste(a)free.fr> To: Dirk Kirsten <dk(a)basex.org> Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> Subject: Re: [basex-talk] (no subject) Message-ID: <55E7FE41.5080205(a)free.fr> Content-Type: text/plain; charset=iso-8859-15; format=flowed I forgot to give this precision, we use the 7.9 version on this server. Marc Le 01/01/1970 01:00, a ?crit : > Hello Dirk, > We use a BaseX script (.xbs) with a list of REPLACE. > We try to investigate this point if this is reproducable or not. > If I find this problem again, I take a copy of the collection before to > delete her. > > Thank you for answer. > > Marc > Le 03/09/2015 09:52, Dirk Kirsten a ?crit : Hello Marc, > > sure, a replace should lead to a true replace, i.e. the old data will be > dropped. So this sounds like a bug, but it is difficult to replicate > from what you describe. > > Do you think it would be possible to make the issue reproducable, maybe > also with a smaller data set? Also, how exactly do you add documents or > do you also do any queries and maybe other updating queries? > > Cheers > Dirk > > On 09/02/2015 05:36 PM, Marc wrote: Hi, > I have a remark about the size of the files of a collection. > We do a lot of updates (REPLACE) with an attributes index but without > a text index. > I see that the size of the files grows up in the time we arrive until > 30Go until the limit of the filessytems quota. > When I drop the collection and recreate it from the last version of > the files the size was only 6Go. > > We will redo the test with the time, but I just want to know if the > program clears the disk use when he does a REPLACE? > > Marc End of BaseX-Talk Digest, Vol 69, Issue 5 *****************************************

2 1

size on collection in the time
by Marc 14 Sep '15

14 Sep '15

Hi, I have a remark about the size of the files of a collection. We do a lot of updates (REPLACE) with an attributes index but without a text index. I see that the size of the files grows up in the time we arrive until 30Go until the limit of the filessytems quota. When I drop the collection and recreate it from the last version of the files the size was only 6Go. We will redo the test with the time, but I just want to know if the program clears the disk use when he does a REPLACE? Marc

3 4

Schematron validation
by Lizzi, Vincent 14 Sep '15

14 Sep '15

I have just released a module that provides an easy way to use Schematron in BaseX. It's is available on GitHub at https://github.com/vincentml/schematron-basex. This module is a convenience wrapper around the standard ISO Schematron XSLTs. In addition, a version of this module for eXist-db is available at https://github.com/vincentml/schematron-exist Vincent "Everything should be made as simple as possible, but not simpler."

2 1