Hi Lars
I have done some OAI-PMH fetches but never got into stack-overflow issues. I guess one workaround you can do on your part is to partition your query with date-ranges using the query parameters "from" and "until" on your initial call to the endpoint.
Regards, Johan Mörén
On Wed, May 11, 2016 at 5:07 PM Lars Johnsen yoonsen@gmail.com wrote:
The basexgui startup file now contains:
BASEX_JVM="-Xmx8g -Xss4m $BASEX_JVM"
It helped the script a long way, but eventually it had to kneel. It works fine though, on smaller datasets.
Maybe there is some other way to get the data over. I'll have a talk with the guys providing the OAI-endpoint.
Thanks for the pointer to Xss!
Lars
2016-05-11 14:38 GMT+02:00 Dirk Kirsten dk@basex.org:
Hello Lars,
if you have a deep recursion Java will at some point hit its stack size limit. Have you already tried to simply increase the Java stack size, e.g. by passing the parameter -Xss2m to the JVM?
Cheers Dirk
On 05/11/2016 01:43 PM, Lars Johnsen wrote:
The following code generates the error "Stack Overflow: try tail recursion?"
The code reads in bibliographic data using OAI-PMH and updates a database for each chunk of data. With OAI-PMH, only part of the data is available for each request, so the server returns a resumption token if there are more data available.
The xquery function making the queries is implemented recursively preceded by a database update request (see the last two lines) for each call. Is it db:add() that causes the stack overflow? The recursion cannot be placed further towards the end!
declare %updating function local:getResumption($token) { if (empty($token)) then () else let $http-request := http:send-request($http-option, $URL || $token) let $result := if ($http-request instance of node()) then $http-request else <http-err>{$http-request}</http-err>
let $resume := $result//oai:resumptionToken/text() return ( db:add($database,element chunk {$result//oai:metadata},$path) , local:getResumption($resume) ) };
Best, Lars
-- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22