August 2015 - BaseX-Talk - mailman.uni-konstanz.de

GUI feature request "Add as import"
by Andy Bunce 03 Jan '16

03 Jan '16

Hi, I often find myself adding module imports in the GUI and I wonder if the following suggestion might speed up that process. Add a new menu item to the filelist right click context menu "Add as import" when the file selection is a XQuery module. The action would be to add a line to the current edit window source with import module namespace {$prefix} = '{$namespace}' at {$path}; Where $prefix and $namespace} are read from the source of the selected file, and $path is calculated from the relative path from the edit file source to the selected filelist source. The line would be inserted at a suitable place in the prologue or at the current location if that is too tricky. /Andy

2 4

Db creation
by meumapple 17 Sep '15

17 Sep '15

Hi, I am trying to open a collection of files where some of them contain character entity references like – which are not declared in the document and so make basex crash if I use collection() If I use file:read-text for each file and then put this functioni in file:write, the saved text will show < > instead of < and >: why? Moreover, if I create a database from the GUI and select the option "skip corrupt", the database is created. However if I follow instruction here http://docs.basex.org/wiki/Parsers In order to add single files to a db usind db:add and precede the query with declare option db:skipcorrupt 'true', the files are not added because I get an error message due to the not well-formed xml. I have tried to declare other db options in the same way but they seem not be read. Giuseppe

3 6

Stopword-related NullPointerException (in FTWords.java)
by Ron Katriel 15 Sep '15

15 Sep '15

Hi, I encountered a peculiar error with a query using a stopwords file in the context of a full text search. The query joins two XML databases: CT.gov (containing 86635503 nodes) and the 2015 MeSH dictionary (containing 12064461 nodes). I am debugging using CT.gov trial NCT00303472, hardcoded in the ‘where' clause of the following query: let $trees := db:open('MeSH')/DescriptorRecordSet/DescriptorRecord for $article in db:open('CTGov')/clinical_study where $article/id_info/nct_id = 'NCT00303472' let $mesh := $article/condition_browse/mesh_term let $tn1 := $trees[DescriptorName/String contains text { $mesh }] let $tn2 := $trees[DescriptorName/String contains text { $mesh } using stop words at "/Volumes/Extra/Documents/Standards/MeSH/stopwords.txt"]/TreeNumberList/TreeNumber return <match> { $article/id_info/nct_id, $mesh, $tn2 } </match> When the return clause contains the variable $tn2 (i.e., using stopwords - as shown above) a Java NullPointerException is generated (see the stack trace below). However, when only $tn1 is returned there is no problem (the code for $tn2 is removed by the optimizer). The issue is related to a specific stopword (“syndrome”). When the stopword is removed from the file the exception does not occur. Surprisingly, when the stopword is in uppercase (“Syndrome”) the issue does not occur - even though the target MeSH term in this CT.gov trial is in uppercase, that is <mesh_term>Syndrome</mesh_term> Am I doing something wrong, or is this a real bug in BaseX? If the former, please suggest a workaround as I would like to filter out generic MeSH terms that match the stopwords before any further processing (I removed a lot of code from the above query to make it easier to debug). Thanks, Ron Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.2 Java: Oracle Corporation, 1.8.0_20 OS: Mac OS X, x86_64 Stack Trace: java.lang.NullPointerException at org.basex.query.expr.ft.FTWords$1.next(FTWords.java:166) at org.basex.query.expr.ft.FTIndexAccess$1.next(FTIndexAccess.java:48) at org.basex.query.expr.ft.FTIndexAccess$1.next(FTIndexAccess.java:45) at org.basex.query.iter.Iter.value(Iter.java:53) at org.basex.query.expr.ParseExpr.value(ParseExpr.java:67) at org.basex.query.QueryContext.value(QueryContext.java:421) at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:41) at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:22) at org.basex.query.QueryContext.iter(QueryContext.java:410) at org.basex.query.expr.List$1.next(List.java:133) at org.basex.query.expr.constr.Constr.add(Constr.java:70) at org.basex.query.expr.constr.CElem.item(CElem.java:92) at org.basex.query.expr.constr.CElem.item(CElem.java:23) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:43) at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99) at org.basex.query.MainModule$1.next(MainModule.java:114) at org.basex.query.QueryContext.cache(QueryContext.java:660) at org.basex.query.QueryProcessor.cache(QueryProcessor.java:103) at org.basex.core.cmd.AQuery.query(AQuery.java:83) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:398) at org.basex.core.Command.execute(Command.java:100) at org.basex.gui.GUI.exec(GUI.java:472) at org.basex.gui.GUI.access$400(GUI.java:43) at org.basex.gui.GUI$7.run(GUI.java:412) Compiling: - pre-evaluating db:open("MeSH") - pre-evaluating db:open("CTGov") - inlining $trees_0 - applying full-text index for { $mesh_2 } using language 'English' - applying full-text index for { $mesh_2 } using language 'English' - inlining $tn2_4 - removing variable $tn1_3 - applying text index for "NCT00303472" - rewriting where clause(s) Query: let $trees := db:open('MeSH')/DescriptorRecordSet/DescriptorRecord for $article in db:open('CTGov')/clinical_study where $article/id_info/nct_id = 'NCT00303472' let $mesh := $article/condition_browse/mesh_term let $tn1 := $trees[DescriptorName/String contains text { $mesh }] let $tn2 := $trees[DescriptorName/String contains text { $mesh } using stop words at "/Volumes/Extra/Documents/Standards/MeSH/stopwords.txt"]/TreeNumberList/TreeNumber return <match> { $article/id_info/nct_id, $mesh, $tn2 } </match> Optimized Query: for $article_1 in db:text("CTGov", "NCT00303472")/parent::*:nct_id/parent::*:id_info/parent::*:clinical_study let $mesh_2 := $article_1/*:condition_browse/*:mesh_term return element match { (($article_1/*:id_info/*:nct_id, $mesh_2, ft:search("MeSH", { $mesh_2 } using language 'English')/parent::*:String/parent::*:DescriptorName/parent::*:DescriptorRecord/TreeNumberList/TreeNumber)) } Query plan: <QueryPlan compiled="true"> <GFLWOR> <For> <Var name="$article" id="1"/> <IterPath> <ValueAccess data="CTGov" type="TEXT" name="*:nct_id"> <Str value="NCT00303472" type="xs:string"/> </ValueAccess> <IterStep axis="parent" test="*:id_info"/> <IterStep axis="parent" test="*:clinical_study"/> </IterPath> </For> <Let> <Var name="$mesh" id="2"/> <IterPath> <VarRef> <Var name="$article" id="1"/> </VarRef> <IterStep axis="child" test="*:condition_browse"/> <IterStep axis="child" test="*:mesh_term"/> </IterPath> </Let> <CElem> <QNm value="match" type="xs:QName"/> <List> <IterPath> <VarRef> <Var name="$article" id="1"/> </VarRef> <IterStep axis="child" test="*:id_info"/> <IterStep axis="child" test="*:nct_id"/> </IterPath> <VarRef> <Var name="$mesh" id="2"/> </VarRef> <CachedPath> <FTIndexAccess data="MeSH"> <FTWords> <VarRef> <Var name="$mesh" id="2"/> </VarRef> </FTWords> </FTIndexAccess> <IterStep axis="parent" test="*:String"/> <IterStep axis="parent" test="*:DescriptorName"/> <IterStep axis="parent" test="*:DescriptorRecord"/> <IterStep axis="child" test="TreeNumberList"/> <IterStep axis="child" test="TreeNumber"/> </CachedPath> </List> </CElem> </GFLWOR> </QueryPlan> Ron Katriel, Ph.D. | Senior Data Scientist | Medidata Solutions 350 Hudson Street, 7th Floor, New York, NY 10014 rkatriel(a)mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800

2 6

Unexpected error: Improper use? Potential bug?
by Pierre-Yves JALLUD 15 Sep '15

15 Sep '15

Hi all, I have some troubles with BaseX request and webdav access. My situation is the following: the acces to BaseX with webdav is with an oXygen (developer version 17) with Ubuntu 14.0.4 and the BaseX server is running under a Scientific Linux 64b (2.6.32-573.3.1.el6.x86_64 ... if the precision is necessary ;)). The BaseX server is behind a web cluster using a load balancer and an APACHE HTTP server. All the access passing thrue this server and the different ports access are managed with HAProxy. If the context is not clear enough, don't hesitate to ask more precisions. The symptom is that the BaseX server is freezing after a webdav request. The log's extracts you can find below track the following use case: - I try an access to http://BaseX.huma-num.fr/webdav/bvm/bvm.xml with oXygen - I recieve a HTTP error 500 - I try to access to the BaseX server with my web site and more than one minute later, I have a timeout. ... but the worst symptom is that in some cases (not always), the requested db get corrupted and emptied (totally or partly) : resources are missing or they throw End of File exceptions when we then try to reopen them. This use case can appear many times a day... but not systematically. And other colleagues have the same problems with other version of oXygen with other OS (Windows), but allways with the same BaseX server. Does anyone of you have an idea to solve my problem? Thanks Pierre-Yves Some extracts of the log file: ... 17:13:06.694 X.X.X.X:34953 admin REQUEST [GET] http://BaseX.huma-num.fr/webdav/bvm/bvm.xml 17:13:06.845 X.X.X.X:34954 admin REQUEST [LOCK] http://BaseX.huma-num.fr/webdav/bvm/bvm.xml 17:13:06.888 X.X.X.X:34954 admin 200 43.82 ms 17:13:08.437 X.X.X.X:34953 admin 500 Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.2.1 Java: Oracle Corporation, 1.8.0_31 OS: Linux, amd64 Stack Trace: java.lang.RuntimeException: org.eclipse.jetty.io.EofException at org.basex.http.webdav.BXServletResponse.close(BXServletResponse.java:80) at org.basex.http.webdav.WebDAVServlet.run(WebDAVServlet.java:46) at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextH... 1743.02 ms ... 17:15:20.041 X.X.X.X:38436 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/skepsis/Cicero_De_Finibus.xml 17:15:40.067 X.X.X.X:38770 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/skepsis/Cicero_De_Finibus.xml 17:16:00.078 X.X.X.X:39090 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/skepsis/ 17:16:00.170 X.X.X.X:39091 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/bvm/bvm.xml 17:16:20.194 X.X.X.X:39582 admin REQUEST [PROPFIND] http://BaseX.huma-num.fr/webdav/bvm/bvm.xml 17:17:00.904 X.X.X.X:39091 admin 200 60733.77 ms 17:17:00.907 X.X.X.X:39582 admin 200 40712.41 ms 17:17:00.909 X.X.X.X:38770 admin 200 80842.45 ms 17:17:00.910 X.X.X.X:38436 admin 200 100868.15 ms 17:17:01.004 X.X.X.X:39090 admin 200 60926.05 ms 17:19:37.189 Y.Y.Y.Y:39926 admin ERROR Timeout exceeded. 60000.62 ms ...

3 6

Schematron validation
by Lizzi, Vincent 14 Sep '15

14 Sep '15

I have just released a module that provides an easy way to use Schematron in BaseX. It's is available on GitHub at https://github.com/vincentml/schematron-basex. This module is a convenience wrapper around the standard ISO Schematron XSLTs. In addition, a version of this module for eXist-db is available at https://github.com/vincentml/schematron-exist Vincent "Everything should be made as simple as possible, but not simpler."

2 1

Fastest way to get resources starting with path
by Martín Ferrari 14 Sep '15

14 Sep '15

Hello: I wonder if there's a fast way to get the contents or names of the resources that start with a certain path. We have something like this: string query = @"for $doc in collection() where substring-after(document-uri($doc), '/') = '" + documentName + @"' return $doc/Root"; But our db has millions of resources so this doesn't scale too well. I'd like a way to get the contents (or names) of resources in which the names start with a certain pattern without having to check all names. Is it possible? The alternative would be to add a field inside the xml resources, index on it, and perform a query. Thanks, Martín.

3 3

Erstellen eines XLSX Sheet mittels Apache POI in einer XQuery Rest Funktion
by Arthur Stocker 14 Sep '15

14 Sep '15

Hallo Zusammen Ich möchte in einer REST Funktion Daten als XLSX zurück geben. Via Java Bindings wollte ich in der Xquery Funktion die Apache POI nutzen und habe mal versucht das einfache Beispiel der JSP Seite zu übersetzen, scheitere nun aber an der Meldung "[XPST0017] Several implementations found for …. „ beim Aufruf der Funktion createSheet(). Einfacher Versuch - XQ Funktion. import module namespace xlsx = "org.apache.poi.xssf.usermodel.XSSFWorkbook"; declare %rest:path('/test') function root:test( ) { let $sheet := xlsx:createSheet() let $row := $sheet:createRow(0) let $cell := $row:createCell(0) return $cell:setCellValue("Some text“) }; JSP Source: <%@page import="org.apache.poi.xssf.usermodel.*" %><%@page import="java.io.*" %><% // create a small spreadsheet if xls then HSSF.... instead of XSSF.... for xlsx XSSFWorkbook wb = new XSSFWorkbook(); XSSFSheet sheet = wb.createSheet(); XSSFRow row = sheet.createRow(0); XSSFCell cell = row.createCell(0); cell.setCellValue("Some text"); // write it as an excel attachment ByteArrayOutputStream outByteStream = new ByteArrayOutputStream(); wb.write(outByteStream); byte [] outArray = outByteStream.toByteArray(); //response.setContentType("application/ms-excel"); response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"); response.setContentLength(outArray.length); response.setHeader("Expires:", "0"); // eliminates browser caching //response.setHeader("Content-Disposition", "attachment; filename=testxls.xls"); response.setHeader("Content-Disposition", "attachment; filename=testxls.xlsx"); OutputStream outStream = response.getOutputStream(); outStream.write(outArray); outStream.flush(); %> Die Dokumentation konnte mir das weitere Vorgehen nicht so ganz aufzeigen. Hat jemand das Apache POI schonmal in XQuery genutzt? Beim Lesen der Doku und den Beispielen bekomme ich den Eindruck, nicht um das Schreiben einer eigenen JAVA Klasse herum zu kommen. VG Arthur

2 1

DELETE Performance
by Martín Ferrari 14 Sep '15

14 Sep '15

Hi: I have a DB with around a million records. Adding resources to it takes some milliseconds. However, executing DELETE on a resource takes more than 60 seconds. Am I doing something wrong? Is this by design? Thanks! Martín.

3 3

Finding document based on filename
by Mansi Sheth 01 Sep '15

01 Sep '15

Hello, I would be having 100s of databases, with each database having 100 XML documents. I want to devise an algorithm, where given a part of XML file name, i want to know which database(s) contains it, or null if document is not currently present in any database. Based on that, add current document into the database. This is to always maintain latest version of a document in DB, and remove the older version, while adding newer version. So far, only way I could come up with is: for $db in all-databases: open $db $fileNames = list $db for eachFileName in $fileNames: if $eachFileName.contains(sub-xml filename): add to ret-list-db return ret-list-db Above algorithm, seems highly inefficient, Is there any indexing, which can be done ? Do you suggest, for each document insert, I should maintain a separate XML document, which lists each file inserted etc. Once, i get hold of above list of db, I would be eventually deleting that file and inserting a latest version of that file(which would have same sub-xml file name). So, constant updating of this external document also seems painful (Map be ?). Also, would it be faster, using XQUERY script files, thru java code, or using Java API for such operations ? How do you all deal with such operations ? - Mansi

4 5

BaseX and MySQL: Driver can't be initialized
by Ulrike Schaper - SideStep Business Solutions GmbH 31 Aug '15

31 Aug '15

Hi everybody, I know, that this issue has been discussed already, but I still have problems concerning BaseX and MySQL: Of course I read the documentation: I downloaded the Connector/J from the MySQL website and put the .jar file (mysql-connector-java-5.1.36-bin.jar) into the basex/lib folder. I closed and restarted the GUI and tried the following: import module namespace sql="http://basex.org/modules/sql"; sql:init("com.mysql.jdbc.Driver") I receive the following error message: [bxerr:BXSQ0007] Could not initialize supplied driver: 'com.mysql.jdbc.Driver' Is there something obvious I'm missing? Best regards, Ulrike.

2 1