- BaseX-Talk - mailman.uni-konstanz.de

Creating more than a million databases per session: Out Of Memory
by Bram Vanroy | KU Leuven 17 Oct '16

17 Oct '16

Hi all I've talked before on how we restructured our data to drastically improve search times on a 500 million token corpus. [1] Now, after some minor improvements, I am trying to import the generated XML files into BaseX. The result would be 100,00s to millions of BaseX databases - as we expect. When doing the import, though, I am running into OOM errors. We put our memory limit on 512MB. The thing is that this seems incredibly odd to me: because we are creating so many different databases, which are all really small as a consequence, I would not expect BaseX to need to store much in memory. After each database is created, the garbage collector can come along and remove everything that was needed for the previously generated database. A solution, I suppose, would be to close and open the BaseX session on each creation but I'm afraid that (on such a huge scale) the impact on speed would be too large. How it is set up now, in pseudo code: ---------------------------------------------------------------------------- ---- $session = Session->new(host, port, user, pw); # @allFiles is at least 100,000 items For $file (@allFiles) { $database_name = $file . "name"; $session->execute("CREATE DB $database_name file "); $session->execute("CLOSE"); } $session->close(); ---------------------------------------------------------------------------- ---- So all databases are created on the same session which I believe causes the issue. But why? What is still required in memory after ->execute("CLOSE")? Are the indices for the generated databases stored in memory? If so, can we force them to write to disk? ANY thoughts on this are appreciated. Enlightenment on how what is stored in a Session's memory is useful as well. Increasing the memory should be a last resort. Thank you in advance! Bram [1]: http://www.lrec-conf.org/proceedings/lrec2014/workshops/LREC2014Workshop-CML C2%20Proceedings-rev2.pdf#page=20

3 4

executing xquery on basex
by Wang, Genneva 16 Oct '16

16 Oct '16

Hi We have seen some weird issue with basex only when executing xquery via java client (the basexclient.java from https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o…) . And I was wondering if you would have any tips. The issue is happening when executing xquery to execute the function fn:current-dateTime() We have logs the queries that we are sending prior to calling the execute method in BasexClient.java, so we know that the xquerys are constructed correctly. However, we have been receiving inconsistent results from execute this xquery. Below are some examples of the basex logs. 05:42:40.907 172.16.0.43:51776 admin REQUEST XQUERY fn:current-dateTime() 0.11 ms 05:42:40.908 172.16.0.43:51776 admin OK Query executed in 0.18 ms. 0.77 ms 05:42:44.787 172.16.0.35:56245 admin REQUEST Ìº°Ì±ïERY fn:current-dateTime() 10000.3 ms 05:42:44.787 172.16.0.35:56245 admin ERROR Stopped at , 1/6: Unknown command: Ìº°Ì±ïERY. Did you mean 'ALTER'? 0.03 ms In this basex log, as you can see, that the first time that the query "XQUERY fn:current-dateTime()"went successful, and somehow the second time the the query was being altered (in red above) and the query was not able to recognize the query from basex. We were not able to reproduce this issue consistently. We have cross check the log on our applications so we know that the query has been constructed the same prior to making calls to basex. Any thoughts would be greatly appreciated. Thanks. Regards, -Genneva

2 1

Re: [basex-talk] Large CSV File (~1M rows)
by Shaun Flynn 16 Oct '16

16 Oct '16

Hello there, The parsing runs out memory. I also tried doubling the memory to 1024MB, and still the same issue. Kind regards On 16 Oct 2016 09:15, "Christian Grün" <christian.gruen(a)gmail.com> wrote: Hi Shaun, > I do this through the MS DOS prompt, but -Xmx does not appear to be a > parameter any more, If you work with the ZIP distributions of BaseX, you can adjust the memory setting in the start scripts (in the bin directory). Otherwise, you’ll need to pass it to to java, not BaseX itself. > Is there a better method for parsing large CSV files? I then want to add the > resulting file tlk.xml to a new database. Did you check if it’s the CSV parsing or the fn:put call that causes the error? Christian [1] http://docs.basex.org/wiki/Database_Module#db:add

2 3

Making connections to a database
by Thomas Daly 16 Oct '16

16 Oct '16

Hello, I'm developing an application server with node.js. When the server starts, it creates a single connection to a BaseX database, to allow subsequent user queries to the database. I notice that when I connect separately to the BaseX database with the BaseXClient command line tool, the application server connection dies with a 'broken pipe' error. Am I doing this correctly, or should the application server connect then immediately disconnect from BaseX to serve each individual user query (in the same way that PHP works with MySQL)? Thanks Thomas

2 1

(no subject)
by Shaun Flynn 16 Oct '16

16 Oct '16

Hello there, I am trying to parse a large CSV file into XML by the following xquery: let $file := fetch:text("D:\BPLAN\tlk.txt") let $convert := csv:parse($file, map { 'header' : true(), 'separator' : 'tab'}) return fn:put( <tlks> {for $row in $convert/csv/record return <tlk>{$row/*}</tlk> }</tlks>, "D:\BPLAN\tlk.xml" ) Using the GUI, it runs out of memory -- when I click on the bottom right hand corner (where the memory usage is shown), it says to increase memory, restart using -Xmx <size>. I do this through the MS DOS prompt, but -Xmx does not appear to be a parameter any more, Is there a better method for parsing large CSV files? I then want to add the resulting file tlk.xml to a new database. Kindest Regards Shaun Connelly-Flynn

2 1

retrieve a sequence of all values within an attribute index
by Alex Muir 10 Oct '16

10 Oct '16

Greetings, I need to generate a list of all the distinct values within an index. Is there a way to retrieve a sequence of all values within an attribute index? Regards Alex tech.jahtoe.com bafila.jahtoe.com

3 9

http:send-request POST xml body is inheriting the http-client namespace
by chrisis 10 Oct '16

10 Oct '16

I'm using BaseX 8.4 but have recreated the issue with 8.5.4 beta 8d90b66 I'm POSTing xml files to a RESTXQ function using the following script let $fpath := 'C:\incoming\SAQD\mo' let $targetBasex := 'http://localhost:8984/saqd/mo/' let $list := sort(file:list($fpath), function($fname) {fn:analyze-string($fname,'.*_(.*?)\.xml$')/fn:match/fn:group[@nr="1"]/string()}) for $file in $list let $abspath := $fpath||'\'||$file let $body := try {fetch:xml($abspath, map { 'chop': true() }) } catch * {()} return if ( not(empty($body)) ) then (http:send-request(<http:request href="{$targetBasex||$file}" method='POST'> <http:body media-type="text/xml">{$body}</http:body> </http:request>), file:move($abspath,$fpath||'\processed\'||$file) ) else () The XML file has no namespace declarations. The RESTXQ function does a db:update using the body received and the http-client namespace has been added to the root node. If I POST to the same function using curl the stored document is unchanged. The function is as follows. declare %rest:path("/saqd/mo/{$fname}") (: %rest:consumes("application/xml", "text/xml") :) %rest:POST("{$body}") %updating function saqd:mo($fname, $body as document-node()) { try {(db:replace('SAQD', '/MO/'||$fname, $body ), db:output(http:send-request(<http:request href="{' http://example.org:8082/home/VALIDSCHEMA/xml/current/Metoffice_Data/'||$fname }" method='PUT' username='validuser' password='validpassword' send-authorization='true'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)) ) } catch * {db:output( <rest:response> <http:response status="500" message="{$err:code||' : '||$err:description}"> </http:response> </rest:response>)} }; The file PUT inside the db:output statement also gets the namespace added i.e. it happens even when the file was POSTed with curl so the recieved document still had no namespaces. Is this expected behaviour, a bug, or am I using http:send-request incorrectly? Regards, chris sweeney

3 6

Make BXElem return also namespace declarations
by Erdal Karaca 09 Oct '16

09 Oct '16

Hi, I am accessing the DOM tree of the collections in the embedded mode. Now, I need to process the DOM node returned by a query. For this, Item.toJava() returns a DOM Element (for my specific case/query). When calling Element#getAttributes(), I only get "direct" attributes, i.e. no namespace declarations are included in the list of attributes. Are the namespace declarations omitted intentionally? How can I make the BXElem also report them in the resulting attributes list? Thanks.

2 2

embedded mode thread-safe?
by Erdal Karaca 09 Oct '16

09 Oct '16

Hi all, Is the embedded mode thread-safe? I mean, once I have the Context object and opened a database, is it safe to access (read/write) it from multiple threads? Will the same transaction mechanism apply as if the server mode would be used? Thanks.

2 7

copy attributes of one node to another
by Erdal Karaca 09 Oct '16

09 Oct '16

Hi, I am trying to copy attributes of one node to another: copy $x := <test a="b" c="d"></test>, $y := parse-xml('<o a="x" c="y"></o>') modify ( delete nodes $x/@*, insert nodes $y/@* into $x ) return $x Unfortunately, the resulting $x is an empty <test/> node. I expected it to have the attributes of the parsed node. Am I using parse-xml() properly? Thanks!

2 2