BaseX-Talk December 2012

basex-talk@mailman.uni-konstanz.de

33 participants
46 discussions

DTD validation
by France Baril 18 Dec '12

18 Dec '12

Hi, We are having challenges with validation. We have built a Documentation Controller for authors that create XML content. On a regular basis, we ask them to check that all documents are valid. We are running the process on hundreds of file. the ideal situation would be to collect all errors and display them of the webpage for authors to find and correct. Ex. of dream code: *let **$db* := *app:open-db*(*$lang*) *let* *$topic-validation* := <ul>{*for **$x* *in **$db*/*[* name*()='topic' or *name*()='media'] *return if* (*contains*(* validate:dtd*(*$x*, *$app:dtd-topic*), 'BXVA0001')) *then *<li>File {*$x*/* base-uri*()} doesn't validate against the DTD.</li> *else *() } </ul> *let* *$menu-validation* := <ul>{*for **$x* *in **$db*/*[* name*()='menu'] *return if* (*contains*(* validate:dtd*(*$x*, *$app:dtd-menu*), 'BXVA0001')) *then *<li>File {*$x*/* base-uri*()} doesn't validate against the DTD.</li> *else *() }</ul> *return if* (*count*(*$topic-validation*/*li*) + *count*(* $menu-validation*/*li*[*normalize-space*(.) != '']) != 0) *then *<div>{*$topic-validation*}{*$menu-validation*} </div> *else *<ul><li *style*=*"color:green"*>All menus and topics validate against the DTDs.</li></ul> Instead, as soon as validate:dtd finds one single file with an error, it crashes the web page with an error message like this: Stopped at line 270, column 77 in /Applications/basex75/webapp/restxq/How2_Admin.xqm: [BXVA0001] Validation failed: 1355790115420.xml, 3:22: Element type "headin" must be declared. On top of this, the name of the file that is referenced in the error message doesn't exist. I've been querying the DB to find errors. Ex. Search for all files with 'headin' in this case. However, we'd really like the writer to be able to find their own faulty files. Are we misusing the function? Is this a bug? What can we do about it? Thanks in advance for you help! -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com (514) 572-0341

2 3

Binding a sequence to an external variable?
by Johannes Krampf 18 Dec '12

18 Dec '12

Hello, I would like to bind a sequence of xs:string to an external variable. I've tried various ways and after trying to read the source and the spec I fear that it might not be possible. First of all the code: Inside the xquery: declare variable $mysequence as xs:string* external; On the Java side: ClientQuery clientQuery = clientSession.query(query); clientQuery.bind(parameter, values.get(parameter)); I tried to give the type "xs:string*" as a third parameter to clientQuery.bind, but I got the error message "org.basex.core.BaseXException: [XPST0003] Unknown type 'xs:string*'", so this is not the way to do it. I furthermore tried to pass a string array as a parameter, but the variable got the value of the output of the toString method. When I tried with a list of Strings, I got again the output of the toString method. What I want is something like the following: let $mysequence := ("Hello", "world") but using sequence values transmitted from the java application using an external variable. Is there any way to achieve this? The goal is to test for membership in the sequence, i.e. allowing the parametrize the query using a variable number of possible values. Thanks for your assistance, Johannes

2 2

Full Text Index giving error for ..7.5 GB XML Data
by John Best 18 Dec '12

18 Dec '12

Dear XML Team, I have a XML DB of size 7.5GB. When creating an Full-Text Index I get an error message - * Out of Main Memory. You can try to: - increase Java's heap size with the flag -Xmx<size> - deactivate the text and attribute indexes.* I tried this with 7.3 and thru GUI... To make searching fast I need FT Index.. :( Another interesting thing I would like to highlight about this error.... In the data folder the FT Index files (ftxXX and ftyXX is the format of the files) are created... The first time when this happened, I checked the data folder and found that the above ftxXX and ftyXX files where up to ftx23 and fty23. Then I tried Full-Text index again with no success. Again I checked the same files. This time they were up to ftx36 and fty36... This might help you that is why I am mentioning..... -- Have a nice day JBest

2 1

xmas feature wishes ;-)
by Maximilian Gärber 16 Dec '12

16 Dec '12

Hi, will there be a JSONiq package under the tree (or anytime soon)? Regards, Max -- Maximilian Gärber axxepta solutions GmbH Postfach 51 02 38 13362 Berlin Tel +49 (0)30 499 147 66 Fax +49 (0)30 499 147 67 Mail gaerber(a)axxepta.de

2 2

setting multiple bindings in the GUI
by Randy reed 15 Dec '12

15 Dec '12

Hi All, I've got three external variables that I want to test in the GUI. I put Set Bindings $dname="/home/randy/..." and that works fine, but when I try to do more I get a syntax error. I've tried separating the entries with ; or , or spaces but none of that seems to work. if I do the full command separated by ; (i.e. set bindings $dname=...; set bindings $inputf=....; set bindings $inputn=....) only the last one takes. Anyone had success doing this? Thanks Randy -- Dr. Randall Reed Assistant Professor, Dept. of Philosophy and Religion Appalachian State University

3 4

Java bindings error
by Randy reed 14 Dec '12

14 Dec '12

Hi all, I switched from 6.8 to 7.5 and I'm generating an error I didn't get in 6.8 when using Java Bindings There error is :[FORG0006] Invalid call of Java function: TimeZone.inDaylightTime(java(), java()) It appears that its problem is using the output of one java function in another function. Is there some new way to do this in 7.5? Thanks Randy The code is: declare namespace tz = "java:java.util.TimeZone"; declare namespace dt = "java:java.util.Date"; declare namespace df = "java:java.text.DateFormat"; declare namespace cal = "java:java.util.Calendar"; declare namespace sdf = "java:java.text.SimpleDateFormat"; let $odate := xs:long(1345665817) let $edate := xs:long($odate)*1000 let $fdate := dt:new(xs:long($edate)) let $tmz := tz:getDefault() let $tbool := tz:inDaylightTime($tmz, $fdate) return ( <results> <jdate>{$fdate}</jdate> <Timezone>{$tmz}</Timezone> <Daylight>{$tbool}</Daylight> </results> ) -- Dr. Randall Reed Assistant Professor, Dept. of Philosophy and Religion Appalachian State University

2 3

Upcoming: XMLPrague 2013
by Christian Grün 14 Dec '12

14 Dec '12

Hi all! The XMLPrague Conference is the most important European event for XML specialists, developers and users. Next year, it will take place from Feb 8-10: http://www.xmlprague.cz/ Once again, our team will be present there, and it will be a pleasure for us to see you and talk to you live! What may be even more interesting: we are planning yet another BaseX User Meeting on the first day (Friday). We have just started to compile our agenda. As many of you are using BaseX for both exciting projects and inn productive environments, it would be great to have some of you talk about your experiences with BaseX. Would you be interested to contribute? We are looking forward to your feedback! Feel free to ask for more details, Christian

2 2

strange behavior of ft:mark() and ft:extract()
by Francis Avila 14 Dec '12

14 Dec '12

ft:mark() and ft:extract() cannot be used with any intermediate looping construct, at least in BaseX 7.3. For example: > create db ftex "<r><a>text example string</a></r>" > xquery ft:mark(db:open('ftex')/descendant::text()[. contains text 'example']) text example string > xquery db:open('ftex')/descendant::text()[. contains text 'example'] ! ft:mark(.) text example string Notice that the use of a loop means that ft:mark() no longer works. This doesn't work with ft:search() either: > create index fulltext Index 'FULLTEXT' created in 7.63 ms. > xquery for $n in ft:search('ftex','example') return ft:mark($n) text example string However this works if there's no intermediate variable binding (implicit copying) even if we do some xpath navigation. > xquery ft:mark(ft:search('ftex','example')/..) <a>text example string</a> Query executed in 0.96 ms. > xquery ft:mark(ft:search('ftex','example')/../..) <r> <a>text example string</a> </r> However (this is a separate bug), the document-node() parent/child ordering is messed up: > xquery ft:mark(ft:search('ftex','example')/../../..) <r> <a>text </a> </r> example string > xquery ft:mark(ft:search('ftex','example')/ancestor::document-node()) <r> <a>text </a> </r> example string It seems to me that probably there is some extra hidden metadata attached to the text node which is not preserved by any implicit copying, such as by loops. (Although I'm not sure what's going on with the document-node() example.) One example of where this is a big headache is when we want to know in which document was a the text match. Here is a natural implementation: QUERY: for $n in ft:search('ftex','example') return <r doc="{document-uri($n/ancestor::document-node())}">{ft:mark($n)}</r> RESULT: <r doc="ftex/ftex.xml">text example string</r> However in this case the text will not be marked. We can't even do this because the "marked" document does not have a reference to the document-node() (which is understandable, as there's probably a copy/modify/return transform under there): QUERY: xquery for $n in ft:mark(ft:search('ftex','example')/..) return <r doc="{document-uri($n/ancestor::document-node())}">{$n/(* | text())}</r> RESULT: <r doc="">text example string</r> The workaround I used is this: QUERY: let $matches := ft:search('ftex','example') for $n at $i in ft:mark(ft:search('ftex','example')/..) return <r doc="{document-uri($matches[$i]/ancestor::document-node())}">{$n/(* | text())}</r> This is extremely ugly, and shares with the last example the downside that I need to mark the *parent* node of the matched text node and then carefully unbox it in the return clause. Quite a "gotcha"! In addition to being ugly, I'm also worried that it may be incorrect because I don't know for sure that the order of search will always be exactly the same (although it *seems* to be). Even if the order is stable, what if the fulltext index is updated by another session between the first call to ft:search() and the second? I don't know what the right answer to all of this is, but the way things are seems very not-good. At the very least the problems with ft:mark() and ft:extract() should be documented with big red text! Perhaps a better method is to have a function with a data structure that contains the text matched text node (as a reference, so that node references are retained) *and* matching substrings explicitly and separately. E.g. a ft:search-with-info('ftex','string example') could return: (map { 'text' := text('test example string'), 'substrings' := (5,8, 14,6) }, ...) or maybe this: (map { 'text' := text('test example string'), 'substrings' := (<s start="5" length="8"/>, <s start="14" length="6"/> ) }, ...) or maybe just a sequence of alternating items: (text('test example string'), <matchinfo><s start="5" length="8"/><s start="14" length="6"/></matchinfo>, ...) Unfortunately I don't see how we can return a simple sequence of elements since the text node result would have to be a copy. With the results separated we could easily make a family of functions which accepts the same data structure and does mark() and extract()-like things with it. The pairs can be processed either with a tumbling window or a for loop with a "where $i mod 2" in it. -- Francis Avila

2 2

Perl BaseX API questions
by Liam R E Quin 13 Dec '12

13 Dec '12

(1) The sample perl files and the perl module implementing the API mention a readme file that explains the methods; does anyone know where one might find such a readme file? It might answer my other question... (2) if there's an error in my query and I catch the exception, the message I get is about how many milliseconds the query took, not the actual error. I can enclose my Perl script, but I'm wondering if others have seen this? Thanks, Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ (fromoldbooks.org uses BaseX e.g. for Search and for the tag cloud)

2 6

Performance Question
by Thomas Kaltofen 13 Dec '12

13 Dec '12

Hi, I have a question regarding performance, because my database shows a somehow strange behavior. I have a database with around 3000 documents, the size of each document does not really matter, because I tried it with small and big documents and there is no real difference (total size of test database is around 40 MB). I am connecting to the database through the client/server interface to execute a query over all documents (all documents have the same structure). The first time the query is executed after the server has been started takes a little longer, subsequent executions of the same query are very fast (around 250 ms). Now I leave the server running for several hours without touching the database at all (e.g. over the night). If I now execute the same query again (the database is still running but was idle for several hours), the execution takes very long (several minutes), but only for the first execution after the idle time, subsequent calls to the query are fast again (around 250 ms). This behavior always happens whenever the database was idle for several hours. What could be the cause for this delay? What I tried already: - Run the database on another computer -> same behavior - Try out different JRE versions (6, 7) -> same behavior - Try out different BaseX versions (7.3, latest 7.5 snapshot) -> same behavior - Start the VM with different options regarding performance & garbage collection -> same behavior - Use smaller or larger documents in the database -> same behavior - Execute a different query -> same behavior I am really running out of ideas and this seems to be independent of the size of the individual documents. What I noticed is that the described lag after the idle phase increases with the amount of documents in the database. Any help and suggestions are appreciated. Thank you! Best regards, Thomas P.S. When (approx.) do you plan to release the next version of BaseX?

5 11

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk December 2012