- BaseX-Talk - mailman.uni-konstanz.de

Coding help
by Greg Kawchuk 08 Aug '19

08 Aug '19

Hi everyone, I'm wondering if someone could provide what I think is a brief script for a scientific project to do the following. The goal is to identify XML documents from a very large collection that would be too big to load into a database all at once. Here is how I see the functions provided by the code. 1. In the script, the user could enter the path of the target folder (with millions of XML documents). 2. In the script, the user would enter the number of documents to load into a database at a given time (i =. 1,000) depending on memory limitations. 3. The code would then create a temporary database from the first (i) xml files in the target folder. 4. The code would then search the 1000 xml documents in the database for a pre-defined text string. 5. If hits exist for the text string, the code would write those documents to a unique XML file. 6. Clear the database. 7. Read in the next 1000 files (or remaining files in the folder). 8. Return to #4. There would be no need to append XML files in step 5. The resulting XML files could be concatenated afterwards. Thank you in advance. If you have any questions, please feel free to email me here. Greg *************************************************** Greg Kawchuk BSC, DC, MSc, PhD. Professor, Faculty of Rehabilitation Medicine University of Alberta greg.kawchuk(a)ualberta.ca 780-492-6891

5 4

BaseX in AWS Lambda - has anyone tried it?
by James Ball 07 Aug '19

07 Aug '19

Good afternoon, I’m currently working on a project where I get lots - maybe thousands - of XML files and I have to run a series of tests on them. I get a whole batch then maybe none for a week. I have all my code in XQuery in BaseX but with some locking and update issues it can take a long time to process. So I thought I might give AWS Lambda a try as all the tests can run independently. Before I jump in I wondered if anyone else has tried it? Any thoughts, suggestions, issues? I did a search of the list but didn’t see anything. Many thanks, James

1 0

Stopping BaseX HTTP server
by Markus Wittenberg 02 Aug '19

02 Aug '19

Hi all, we have a problem trying to stop a BaseX HTTP server with the Windows task scheduler. The server ist started with a maven profile, calling mvn exec:java -Pstart pom.xml extract see below, and can also be stopped directly with the corresponding stop profile by a batch file: cd /D "%~dp0" mvn exec:java -Pstop But calling this batch profile in task scheduler works fine on our local computers, but does not work on the server (Windows Server 2012). On this machine, executing the stop profile by the task scheduler results in a message HTTP server was started (port: 8985). in the console window of the process to be stopped, but it continues running anyway. Any ideas regarding this strange behavior? Can we stop the server with a simple HTTP request? Calling localhost:8985 in the browser also results only in the message above. Thanks in advance for any hints. Best regards, Markus Extract from pom.xml: <profile> <id>start</id> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <version>1.4.0</version> <configuration> <mainClass>org.basex.BaseXHTTP</mainClass> <arguments> <argument>-h9894</argument> <argument>-s8985</argument> <argument>-p1985</argument> <argument>-Urestxq</argument> </arguments> <additionalClasspathElements> <additionalClasspathElement>src/main/resources</additionalClasspathElement> </additionalClasspathElements> </configuration> </plugin> </plugins> </build> </profile> <profile> <id>stop</id> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <version>1.4.0</version> <configuration> <mainClass>org.basex.BaseXHTTP</mainClass> <arguments> <argument>-s8985</argument> <argument>stop</argument> </arguments> </configuration> </plugin> </plugins> </build> </profile> <profile> -- Markus Wittenberg Tel +49 (0)341 248 475 36 Mail wittenberg(a)axxepta.de ---- axxepta solutions GmbH Lehmgrubenweg 17, 88131 Lindau Amtsgericht Berlin HRB 97544B Geschäftsführer: Karsten Becke, Maximilian Gärber

3 12

Forcing download of JSON doc
by Tim Thompson 01 Aug '19

01 Aug '19

Hello, I am posting a JSON doc to a RESTXQ function from XForms and trying to set a Content-Disposition header to trigger a file download. Is this possible to do without first saving the JSON doc as a file? Right now, the RESTXQ function returns the correct headers, but does not force a file download dialog. Thanks in advance! Tim -- Query: xquery version "3.1"; module namespace wrap = "http://example.org/wrap"; declare %rest:path("/wrap") %rest:POST("{$data}") %output:method("json") function wrap:process( $data as node() ) as item()* { let $name := data($data/data/upload/file/@name) let $json := json-to-xml($data/data/upload/file) let $header := ``[attachment; filename="`{$name}`";]`` return ( <rest:response> <http:response status="200" message="OK"> <http:header name="Content-Type" value="application/json"/> <http:header name="Content-Disposition" value="{$header}"/> </http:response> </rest:response>, <map xmlns="http://www.w3.org/2005/xpath-functions"> <map key="Profile"> <array key="resourceTemplates">{$json}</array> <string key="title">Wrapper_Profile</string> <string key="id">Wrapper:Profile:BF2</string> <string key="description">Profile wrapper for resource templates</string> <string key="author">Wrapper</string> <string key="schema"> https://ld4p.github.io/sinopia/schemas/0.0.9/profile.json</string> <string key="date">{current-dateTime()}</string> </map> </map> => xml-to-json() => parse-json() => serialize(map {'method': 'json', 'use-character-maps' : map {'/' : '/'}}) ) }; -- Tim A. Thompson Discovery Metadata Librarian Yale University Library

2 4

ß = ss: not equal in all cases
by Alexander Witzigmann 31 Jul '19

31 Jul '19

Hi All, the following query declare default collation 'http://basex.org/collation?lang=de;strength=secondary'; "anschließend" contains text "anschliessend" returns *false* but the following query declare default collation 'http://basex.org/collation?lang=de;strength=secondary'; "anschließend" = "anschliessend" returns *true*. and the following query declare default collation 'http://basex.org/collation?lang=de;strength=secondary'; "anschließend" eq "anschliessend" returns *true*. I wonder why 'contains text' does not return true as well? All full-text related functions doesn't support ß = ss for german collation. Alex

2 1

Full-Text Search long s
by Günter Dunz-Wolff 26 Jul '19

26 Jul '19

Hi all, I’m working since some years on a digital edition of the works of a former german author. In my transcription of those works are lots of gothic characters like the old german long s (Unicode: LATIN SMALL LETTER LONG S). For example: Büchſe (exactly Buͤchſe). In my Full-Text-Search my goal is, that the user asks for „Büchse“ and gets „Büchse“ AND „Büchſe“ (with long s). In best case, she should get „Büchse“ AND „Büchſe“ AND „Buͤchſe“. How can I achieve, that //text[. contains text { } treats s and ſ and ü and uͤ as the same character? Thanks a lot for any help. Best regards, Guenter

3 2

Managing external entities
by Majewski, Steven Dennis (sdm7g) 26 Jul '19

26 Jul '19

I have a corpus of TEI files with figures and page images encoded as external entities. It appears that even when choosing “Parse DTDs and entities” this info is lost when parsing files into database, And in any case, unparsed-entity-uri() is an XSLT only function. It would appear that I first need to transform the files first and replace @entity attributes with @url attributes while these unparsed entity values are available, before creating the database, or else generate another database to map entity names to values later. Are there any better ways to handle this case ? Is there any way to do these transforms on the fly before parsing the files into the database ? The only thing that comes to mind is to set up a local SaxonServlet to do the transforms, and load from URLs instead of file paths. ( I’ve been doing something similar for a different case, and running into memory errors that I don’t see when loading from a directory when creating a database. Increasing memory didn’t help much, but inserting ‘flush’ command between even ‘add’ commands seemed to work. ) — Steve Majewski

2 2

BaseX 9.2.1: Maintenance Release
by Christian Grün 26 Jul '19

26 Jul '19

Dear all, A new BaseX maintenance release is available: - GUI: Parsing of project files has become faster and smarter. - RESTXQ: Custom reponses are now sent as streams. - Performance tweaks (static typing: typeswitch, instance of). - Minor bugs have been fixed. Looking forward to your feedback, Christian BaseX Team

1 3

File-io:error. File lock by another process (windows-only)
by France Baril 24 Jul '19

24 Jul '19

Hi, We have this issue only on Windows system since 9.2 that we know of... could be from 902. It is still happening after upgrade to 923. I am not able to provide a sample to reproduce systematically as it never happens at the same .xqm location. It does however always seem that the 1st error message occurs on a line with a file:write or file;append. I am giving you everything I know, in case that it rings a bell. 1. Start server with basexhttp 2. Run a process that has many functions and many file:write and file:append. 3. This generally works properly. The process ends with success. 4. Run the same process again, there is a good chance that we'll get an error that says something like: [file:io-error] Resource "debug-list-imtes.xml' locked by another process. 5. Run the same process again May get file-oi:error at the same location or a different one. 6. Restarting the server brings us back to 1... but sometimes we need to manually delete the files written with function from the file module before we can go back to 1 and get 1 good run of the process. Note: Although the process runs correctly and seems to finish running (to the same point that releases all locks on Linux). The person on the Windows system also often gets: [db:open] Resource "C:\basex\data\staging-en-us\txt.basex locked by another process and then the related database also remains locked and we may get messages: Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 9.2.3 Java: Oracle Corporation, 1.8.0_211 OS: Windows 10, amd64 Stack Trace: java.nio.channels.OverlappingFileLockException at sun.nio.ch.SharedFileLockTable.checkList(Unknown Source) at sun.nio.ch.SharedFileLockTable.add(Unknown Source) at sun.nio.ch.FileChannelImpl.tryLock(Unknown Source) at org.basex.io.random.TableDiskAccess.lock(TableDiskAccess.java:147) at org.basex.io.random.TableDiskAccess.(TableDiskAccess.java:79) at org.basex.data.DiskData.init(DiskData.java:125) at org.basex.data.DiskData.(DiskData.java:84) at org.basex.core.cmd.Open.open(Open.java:99) at org.basex.query.QueryResources.database(QueryResources.java:179) at org.basex.query.func.StandardFunc.checkData(StandardFunc.java:429) at org.basex.query.func.db.DbOpen.value(DbOpen.java:17) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:45) at org.basex.query.expr.path.IterPath$1.init(IterPath.java:79) at org.basex.query.expr.path.IterPath$1.next(IterPath.java:46) at org.basex.query.expr.path.IterPath$1.next(IterPath.java:1) at org.basex.query.iter.Iter.value(Iter.java:65) at org.basex.query.expr.path.IterPath.nodes(IterPath.java:87) at org.basex.query.expr.path.AxisPath.value(AxisPath.java:39) at org.basex.query.expr.gflwor.Let$LetEval.next(Let.java:173) at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:80) at org.basex.query.func.StaticFunc.invValue(StaticFunc.java:172) at org.basex.query.func.FuncCall.invoke(FuncCall.java:117) at org.basex.query.func.FuncCall.value(FuncCall.java:56) at org.basex.query.scope.MainModule.value(MainModule.java:135) at org.basex.query.QueryContext.update(QueryContext.java:673) at org.basex.query.QueryContext.iter(QueryContext.java:332) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73) at org.basex.http.web.WebResponse.create(WebResponse.java:56) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:54) at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Unknown Source)undefined -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 10

Re: [basex-talk] Transaction Management
by Christian Grün 24 Jul '19

24 Jul '19

Hi Radim (cc to the list), > it takes so long even when it was being stored from a file on local hard drive. The xml file has about 15 to 20 MB. Wow. How large is your database? How did you add the documents? > Could you give me the XQuery or the BaseX command to move the file from the TempDB to the TargetDB the quickest possible way? Sure. There are various ways to do so, here’s just one (based on the database module [1]): 1. upload your document to database 'db-temp' 2. insert documents into final database: db:replace('db', db:list('db-temp'), db:open('db-temp')) 3. remove all your documents from the temporary database: db:delete('db-temp', '/') You can also drop the temporary database completely via db:drop. Hope this helps, Christian [1] http://docs.basex.org/wiki/Database_Module And how about database cache settings? Is something like that in BaseX? > > Thank you in advance for your response. > Best regards, > > Radim Havlicek > > > ---------- Původní e-mail ---------- > Od: Christian Grün <christian.gruen(a)gmail.com> > Komu: Radim Havlicek <radim-havlicek(a)post.cz> > Datum: 23. 7. 2019 15:12:04 > Předmět: Re: [basex-talk] Transaction Management > > Hi Radim, > > Do you have a clue how the long update time can be explained? Does it > take 20 minutes to upload your document to the BaseX server, or does > it take the same time if you update your database directly on the > server? If it’s the upload time, you could e.g. insert your document > in another temporary database and copy it to the final database in a > second step. > > Hope this helps, > Christian > > > > On Tue, Jul 23, 2019 at 10:34 AM <radim-havlicek(a)post.cz> wrote: > > > > Hello. > > > > We use client/server architecture, especially the C# client. I read that: > > > > Read transactions are executed in parallel. > > If an updating transaction comes in, it will be queued and executed after all previous read transaction have been executed. > > > > We have a web Front-End built on BaseX. We read XML documents stored in the database, and show charts with statistics calculated from those documents. Few times per day, we copy a new document to the database (via REPLACE command to make sure, the document is not stored in the database already). > > > > The problem we are facing is that while big document is being stored to the database (10 to 20 minutes), we cannot read documents from the database and display charts for users on the web interface. The web interface is waiting until the document is stored, and then read the the needed documents, and display the charts. It is very inconvenient for users to wait so long for a chart that is normally displayed in few seconds. > > > > Is there a way we could read the xml documents from the database while another document is being stored to the database? It is very, very needed. > > > > Best regards, > > > > Radim Havlicek

1 0