October 2016 - BaseX-Talk - mailman.uni-konstanz.de

Text resources
by Kendall Shaw 27 Oct '16

27 Oct '16

Is this how I should retrieve text that is stored using db:store, within xquery? stream:materialize(db:retrieve(‘some’,’some.txt’)) ? Kendall

1 0

Adding UIDs to all the elements in a collection
by Graydon Saunders 19 Oct '16

19 Oct '16

HI -- In the process of trying to find the differences between two collections of schemas (the XML differences, as opposed to the not-semantically-meaningful differences in the order of the definitions), I find myself wanting to stick UIDs on everything. (So I can go "oh, this element node is different, but it has descendant elements UIDs also in the list of differences, I don't want this one, the descendants are more precise" (because if I don't do something like this, I'm getting xsd:schema every time...)) I haven't done anything with updating expressions before. By itself, in the GUI, for $x in /descendant-or-self::* return insert node(attribute UID {generate-id($x)}) into $x works. Since I would like everything in one query, with a little flailing I've got as far as let $new as document-node()+ := for $each in db:open($newName) return copy $temp := $each modify ( for $x in $temp/descendant-or-self::* return insert node(attribute UID {generate-id($x)}) into $x ) return $temp return $new And there are all the elements with UID attributes associated with them. Is there a better way to do this? Thanks! Graydon

3 3

Compiler issue?
by Marco Lettere 17 Oct '16

17 Oct '16

Dear BaseX team, running the following two queries [1,2] (which should be semantically equivalent to my eyes) I get different results. In particular [2] sometimes return multiple results or empty sequence which is pretty odd, isn't it? I've tried on 8.5.3 and on a recent snapshot labeled 8.5.4 (beta 6e9bf7c). Regards, Marco. [1] let $domain := for $record in (1 to 100) where $record mod 5 = 0 return $record let $index := 1 + random:integer(count($domain)) return $domain[$index] [2] let $domain := for $record in (1 to 100) where $record mod 5 = 0 return $record return $domain[1 + random:integer(count($domain))]

2 2

Creating more than a million databases per session: Out Of Memory
by Bram Vanroy | KU Leuven 17 Oct '16

17 Oct '16

Hi all I've talked before on how we restructured our data to drastically improve search times on a 500 million token corpus. [1] Now, after some minor improvements, I am trying to import the generated XML files into BaseX. The result would be 100,00s to millions of BaseX databases - as we expect. When doing the import, though, I am running into OOM errors. We put our memory limit on 512MB. The thing is that this seems incredibly odd to me: because we are creating so many different databases, which are all really small as a consequence, I would not expect BaseX to need to store much in memory. After each database is created, the garbage collector can come along and remove everything that was needed for the previously generated database. A solution, I suppose, would be to close and open the BaseX session on each creation but I'm afraid that (on such a huge scale) the impact on speed would be too large. How it is set up now, in pseudo code: ---------------------------------------------------------------------------- ---- $session = Session->new(host, port, user, pw); # @allFiles is at least 100,000 items For $file (@allFiles) { $database_name = $file . "name"; $session->execute("CREATE DB $database_name file "); $session->execute("CLOSE"); } $session->close(); ---------------------------------------------------------------------------- ---- So all databases are created on the same session which I believe causes the issue. But why? What is still required in memory after ->execute("CLOSE")? Are the indices for the generated databases stored in memory? If so, can we force them to write to disk? ANY thoughts on this are appreciated. Enlightenment on how what is stored in a Session's memory is useful as well. Increasing the memory should be a last resort. Thank you in advance! Bram [1]: http://www.lrec-conf.org/proceedings/lrec2014/workshops/LREC2014Workshop-CML C2%20Proceedings-rev2.pdf#page=20

3 4

executing xquery on basex
by Wang, Genneva 16 Oct '16

16 Oct '16

Hi We have seen some weird issue with basex only when executing xquery via java client (the basexclient.java from https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o…) . And I was wondering if you would have any tips. The issue is happening when executing xquery to execute the function fn:current-dateTime() We have logs the queries that we are sending prior to calling the execute method in BasexClient.java, so we know that the xquerys are constructed correctly. However, we have been receiving inconsistent results from execute this xquery. Below are some examples of the basex logs. 05:42:40.907 172.16.0.43:51776 admin REQUEST XQUERY fn:current-dateTime() 0.11 ms 05:42:40.908 172.16.0.43:51776 admin OK Query executed in 0.18 ms. 0.77 ms 05:42:44.787 172.16.0.35:56245 admin REQUEST Ìº°Ì±ïERY fn:current-dateTime() 10000.3 ms 05:42:44.787 172.16.0.35:56245 admin ERROR Stopped at , 1/6: Unknown command: Ìº°Ì±ïERY. Did you mean 'ALTER'? 0.03 ms In this basex log, as you can see, that the first time that the query "XQUERY fn:current-dateTime()"went successful, and somehow the second time the the query was being altered (in red above) and the query was not able to recognize the query from basex. We were not able to reproduce this issue consistently. We have cross check the log on our applications so we know that the query has been constructed the same prior to making calls to basex. Any thoughts would be greatly appreciated. Thanks. Regards, -Genneva

2 1

Re: [basex-talk] Large CSV File (~1M rows)
by Shaun Flynn 16 Oct '16

16 Oct '16

Hello there, The parsing runs out memory. I also tried doubling the memory to 1024MB, and still the same issue. Kind regards On 16 Oct 2016 09:15, "Christian Grün" <christian.gruen(a)gmail.com> wrote: Hi Shaun, > I do this through the MS DOS prompt, but -Xmx does not appear to be a > parameter any more, If you work with the ZIP distributions of BaseX, you can adjust the memory setting in the start scripts (in the bin directory). Otherwise, you’ll need to pass it to to java, not BaseX itself. > Is there a better method for parsing large CSV files? I then want to add the > resulting file tlk.xml to a new database. Did you check if it’s the CSV parsing or the fn:put call that causes the error? Christian [1] http://docs.basex.org/wiki/Database_Module#db:add

2 3

Making connections to a database
by Thomas Daly 16 Oct '16

16 Oct '16

Hello, I'm developing an application server with node.js. When the server starts, it creates a single connection to a BaseX database, to allow subsequent user queries to the database. I notice that when I connect separately to the BaseX database with the BaseXClient command line tool, the application server connection dies with a 'broken pipe' error. Am I doing this correctly, or should the application server connect then immediately disconnect from BaseX to serve each individual user query (in the same way that PHP works with MySQL)? Thanks Thomas

2 1

(no subject)
by Shaun Flynn 16 Oct '16

16 Oct '16

Hello there, I am trying to parse a large CSV file into XML by the following xquery: let $file := fetch:text("D:\BPLAN\tlk.txt") let $convert := csv:parse($file, map { 'header' : true(), 'separator' : 'tab'}) return fn:put( <tlks> {for $row in $convert/csv/record return <tlk>{$row/*}</tlk> }</tlks>, "D:\BPLAN\tlk.xml" ) Using the GUI, it runs out of memory -- when I click on the bottom right hand corner (where the memory usage is shown), it says to increase memory, restart using -Xmx <size>. I do this through the MS DOS prompt, but -Xmx does not appear to be a parameter any more, Is there a better method for parsing large CSV files? I then want to add the resulting file tlk.xml to a new database. Kindest Regards Shaun Connelly-Flynn

2 1

retrieve a sequence of all values within an attribute index
by Alex Muir 10 Oct '16

10 Oct '16

Greetings, I need to generate a list of all the distinct values within an index. Is there a way to retrieve a sequence of all values within an attribute index? Regards Alex tech.jahtoe.com bafila.jahtoe.com

3 9

http:send-request POST xml body is inheriting the http-client namespace
by chrisis 10 Oct '16

10 Oct '16

I'm using BaseX 8.4 but have recreated the issue with 8.5.4 beta 8d90b66 I'm POSTing xml files to a RESTXQ function using the following script let $fpath := 'C:\incoming\SAQD\mo' let $targetBasex := 'http://localhost:8984/saqd/mo/' let $list := sort(file:list($fpath), function($fname) {fn:analyze-string($fname,'.*_(.*?)\.xml$')/fn:match/fn:group[@nr="1"]/string()}) for $file in $list let $abspath := $fpath||'\'||$file let $body := try {fetch:xml($abspath, map { 'chop': true() }) } catch * {()} return if ( not(empty($body)) ) then (http:send-request(<http:request href="{$targetBasex||$file}" method='POST'> <http:body media-type="text/xml">{$body}</http:body> </http:request>), file:move($abspath,$fpath||'\processed\'||$file) ) else () The XML file has no namespace declarations. The RESTXQ function does a db:update using the body received and the http-client namespace has been added to the root node. If I POST to the same function using curl the stored document is unchanged. The function is as follows. declare %rest:path("/saqd/mo/{$fname}") (: %rest:consumes("application/xml", "text/xml") :) %rest:POST("{$body}") %updating function saqd:mo($fname, $body as document-node()) { try {(db:replace('SAQD', '/MO/'||$fname, $body ), db:output(http:send-request(<http:request href="{' http://example.org:8082/home/VALIDSCHEMA/xml/current/Metoffice_Data/'||$fname }" method='PUT' username='validuser' password='validpassword' send-authorization='true'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)) ) } catch * {db:output( <rest:response> <http:response status="500" message="{$err:code||' : '||$err:description}"> </http:response> </rest:response>)} }; The file PUT inside the db:output statement also gets the namespace added i.e. it happens even when the file was POSTed with curl so the recieved document still had no namespaces. Is this expected behaviour, a bug, or am I using http:send-request incorrectly? Regards, chris sweeney

3 6