November 2012 - BaseX-Talk - mailman.uni-konstanz.de

slow query
by Mahlow Cerstin 20 Nov '12

20 Nov '12

Hi, I have a problem with slow queries. I'm not sure if it is due to the construction of the query or if there is something else going on. I'm querying two databases: "collect" which is opened at the beginning and contains several thousands entries like these two: <entry time="2012-05-22T11:27:09"> <node>40177618</node> <query>[text() contains text ('den' ftand 'Hals' ftand 'brachen') distance at most 10 words]</query> <person>britta</person> <phraseme>Ad0194</phraseme> <selected>yes</selected> </entry> <entry time="2012-05-22T11:27:09"> <node>40672561</node> <query>[text() contains text ('den' ftand 'Hals' ftand 'brachen') distance at most 10 words]</query> <person>britta</person> <phraseme>Ad0194</phraseme> <selected>no</selected> </entry> The node-ids refer to a second database: "TG-DTA-GerManC-stemming-ws". The queries contain the xquery the node was found with. For a particular phraseme, I display the original node using ft:mark to highlight the query terms. Additionally, I display some of the metadata for this node. The user will be able to "delete" some of the entries -- i.e., setting the value of "selected" from "yes" to "no". Therefore I render everything as a form element with a checkbox. The query is this one: for $i at $p in //entry[phraseme[text() = "Ad0194"] and selected[text() = "yes"]] let $query := $i/query let $node := $i/node let $prefix := fn:in-scope-prefixes($i) let $title := db:open-id('TG-DTA-GerManC-stemming-ws', $node) /ancestor::*:TEI[1]//*:fileDesc[1]//*:titleStmt[1]//*:title[1] let $author := db:open-id('TG-DTA-GerManC-stemming-ws', $node) /ancestor::*:TEI[1]//*:sourceDesc[1]//*:bibl[1]//*:author[1] let $note := db:open-id('TG-DTA-GerManC-stemming-ws', $node) /ancestor::*:TEI[1]//*:notesStmt//*:note let $expr := concat("ft:mark(db:open-id('TG-DTA-GerManC-stemming-ws', ", $node, ") ", $query, ")") let $time := data($i/@time) return <div> <hit count="{ $p}"> <input type="checkbox" name="NODE" value="{$node}"/> {$p} ({ if($prefix = "dta") then "DTA" else "TG"})Knoten: {$i/node} {xquery:eval($expr)} </hit> <bib> {$time} Bibliographie { data($author)}: { data($title)} Anmerkung: { data ($note) } Korpus: { if($prefix = "dta") then "Deutsches Textarchiv" else "TextGrid Digitale Bibliothek"} </bib> </div> However, for this particular phraseme ("Ad0194"), there will be 676 hits to be displayed (out of around 1000 stored) which takes around 15 seconds in the GUI. In the actual web application, the full site is shown after more than 20 seconds (there is some HTML to be displayed and another short xquery for displaying distinct queries for all entries in question), which is too slow, I think. Is there anything I can do to speed up the query? Can I somehow rewrite it to make execution faster? I think opening the second DB four times costs a lot of time. Both DBs have an optimized index already. Thanks in advance Cerstin -- Dr. phil. Cerstin Mahlow Universität Basel Deutsches Seminar Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow(a)unibas.ch Web: http://www.oldphras.net

3 6

RESTXQ: What to use for page layout / view templating?
by Daniel Kvasnička 19 Nov '12

19 Nov '12

Hi folks, another thing I'm trying to solve while working on a XQuery web app. What kind of templating system do you use to separate HTML templates from the rest of your code? - XSLT - the only problem I have with this is performance. I tested the XSLT Module with Saxon 9 and a primitive page and it was more than 3x slower compared to pure XQuery templates (XQuery fn taking $model). Other than that I think it's the best built-in option. I actually don't have any users-per-second expectations and those templates will be quite small... do I overemphasize the perf. question? - pure XQuery - not aimed at templating at all and it shows I'm afraid... just can't come up with a solution that would be clean enough (plus HTML, JS & CSS highlighting withing xq files is not good at all, at least in Vim) - I looked at Mustache.xq and quite liked it - depends on MarkLogic though :( - Any hidden gem I haven't googled yet? Leaning towards XSLT right now... Daniel -- danielkvasnicka.net

5 8

Saving xquery output to a file
by Alex Muir 19 Nov '12

19 Nov '12

Hi, I've create a standalone script which creates a database and run's an xquery although I'm not clear on how to save the result of the xquery to file. generally I would like to create a database and run a series of xqueries each going to file rather than to the console output. <?xml version="1.0" encoding="UTF-8"?> <commands> <show-databases/> <drop-db name="1997-i4enrich-10k"/> <create-db name="1997-i4enrich-10k"/> <add path="1997-i4enrich-10k">/mnt/xslt_volume/i4ContentOutput/DebugSet/10k-GHU-2009/MDAFalse/Completed/</add> <info-db/> <run file='../xquery/section-10k-metrics.xq'/> <close/> </commands> I saw this on the documentation however I couldn't get it to work Q{java:java.io.FileOutputStream}new("output.txt") btw.. It might be nice to be able to say <run file='../xquery/section-10k-metrics.xq' outputFile="filename.xml" /> Thanks -- - Alex G. Muir Software Engineering Consultant Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 Love African Kora Music? Take a moment to listen to Gambia's - Amadu Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com Your support keeps Africa's griot tradition alive... Cheers!

2 1

Re: [basex-talk] BUG# Multiple Thread try to execute some Query
by Samuel Lacmene 19 Nov '12

19 Nov '12

1 0

concurrent issues with 7.5 - OverlappingFileLockException
by Srinivasan Muthu 19 Nov '12

19 Nov '12

Hi, We started having some thread hanging issues when concurrent calls were being made to our basex 7.3 database (client-server mode) from java 6 web app on Mac (less prominent in Windows). We extracted the code into a simple java app to test this in two threads executing a simple doc('blah') command several times and were able to reproduce the thread hanging issue easily in basex 7.3. After Christian's suggestion in another forum conversation to use 7.5, we tried with 7.5, now the code does not hang but it throws java.nio.channels.OverlappingFileLockException frequently for some of the db requests. I am attaching the sample java program here as a jar file (source code inside as well). Here is a sample output of this program: Srini-MacBook-Pro:Downloads srini$ java -jar BasexTest.jar the time the xxxxxxxx 1 time the time the xxxxxxxx 2 time . . (some output lines hidden here for brevity) . the time the xxxxxxxx 30 time the time the xxxxxxxx 31 time java.io.IOException: java.nio.channels.OverlappingFileLockException at com.vmware.gts.vha.basex.BaseXClient$Query.exec(BaseXClient.java:446) at com.vmware.gts.vha.basex.BaseXClient$Query.execute(BaseXClient.java:405) at com.vmware.gts.vha.test.ParallelExecution.runXQuery(ParallelExecution.java:33) at com.vmware.gts.vha.test.ParallelExecution.access$0(ParallelExecution.java:30) at com.vmware.gts.vha.test.ParallelExecution$SampleRunnable.run(ParallelExecution.java:13) at java.lang.Thread.run(Thread.java:680) Is this a potential bug in BaseX 7.5? This issue is more prominent (happens frequently) in Mac than Windows. We tried with both DBLocking=true and false and the issue remains. Thanks, Srini

2 4

Re: [basex-talk] BUG# Multiple Thread try to execute some Query
by Samuel Lacmene 18 Nov '12

18 Nov '12

2 1

udpindex
by Mahlow Cerstin 17 Nov '12

17 Nov '12

Hi, as far as I can see, UPDINDEX cannot be set in the GUI when creating a database. Is this correct? So how would I proceed to change my existing database created without this option and updated and queried by users constantly into an database with this option to make retrieval faster? Is it correct that with UPDINDEX true there is no need anymore to "optimize" the index from time to time? Initially I thought of exporting the existing DB as XML and then using this as source for a new DB with UPDINDEX set true. How do I do this without the GUI? Oh, and could you please make it possible to choose UPDINDEX as an option in the GUI? Best regards Cerstin -- Dr. phil. Cerstin Mahlow Universität Basel Deutsches Seminar Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow(a)unibas.ch Web: http://www.oldphras.net

3 2

Updating single database locks entire server
by Srinivasan Muthu 17 Nov '12

17 Nov '12

First of all, great work you guys, thanks much. We use BaseX 7.3 in client/server mode for concurrency purposes and have several databases. We have an updating function which updates the XML database after the initial creation to make sure the attributes are setup for faster querying. Here is the function: declare updating function local:processVMs($dbName as xs:string){ let $doc := doc($dbName) let $vms := $doc/root/proj/VM return if (empty($vms/@desktopId)=true()) then ( for $vm in $vms let $desktopId := local:get-desktop-id-from-vm($vm) (: function not provided here:) return insert node (attribute {'desktopId'} {string($desktopId)}, 'text', <VM/>) into $vm) else () }; This function takes a minute to run when there are about 3000 objects. While this query runs, BaseX queues all other xqueries (including read xqueries on other databases too) and thereby rendering the server unresponsive until this update query is done. Due to this even the queries that are running on other databases become unresponsive and our application looks frozen until this update query is done.From reading old email archives here, I see that BaseX transaction monitor will queue all requests when update query is processed as multiple dbs can be updated using one xQuery. Is this really true and is there any way to hint transaction monitor to lock only the database that is being updated and not the entire server? Thanks, Srini

2 5

performance and collections
by pw＠themail.co.uk 16 Nov '12

16 Nov '12

dear list I have a web-based database (basex 7.5 running on tomcat) in which I have a collection with about 3000 files in various sizes from about 1k to 40k. Would I get better performance by splitting these files across several collections? what are the rules of thumb here? many thanks Pete

2 1

XQuery web app authentication - best practices?
by Daniel Kvasnička 15 Nov '12

15 Nov '12

Hi folks, I'd like to write an app using RESTXQ and I'd like to auth users using a regular form-based authentication and then on some XQuery functions check for an existing user session (and possibly user roles). I'd also like to add some social media login using OAuth (later). My question is - is it somehow possible to do this in a declarative way? For example custom annotations on XQuery handlers? Something like %auth:roles-allowed("admin") I definetly don't want to "if" at the beginning of every function that should be protected. No problem with implementing this in Java or XQuery. Just tell me how to approach this orthogonal concern in a reasonable way... Or should I equal app users to BaseX users and leverage Basex auth? Any tips appreciated (yes, you can even tell me BaseX RESTXQ is not a good tool for that). Daniel -- danielkvasnicka.net

4 5