Hi BaseX team!
A quick question.
Is there some known bug/common setting missing for RESTXQ and memory problems?
I have this simple module ( into /webapp, a .xqm file ) :
*module namespace exe = 'http://site.com/execute http://site.com/execute';* *declare * *%updating* * %rest:path("update")* * %rest:consumes("application/x-www-form-urlencoded")* * %rest:POST function exe:update() {*
* xquery:eval-update(request:parameter("query"))*
*};*
I use it to execute some updates against some databases from different clients.
Everything works fine by a while, but after some time I get this error [1]:
*java.lang.OutOfMemoryError: Java heap space*
I noticed that every call to *update()* the memory grows and grows until it reaches the *OutOfMemoryError*. [2]
If I stop the HTTP server, the memory is released immediately. [3]
What I'm doing wrong? Is there some command to execute the GC? Is this a problem with "*xquery:eval-update()"*? Am I using it in the wrong way?
Best regards, Sebastian. [1] https://imgur.com/DrcbwQg [2] https://imgur.com/fonmrhm [3] https://imgur.com/SYFBFK8
Hi Sebastian,
There is a prof:gc() function, but that should only be used for debugging purposes. Is your Pending Update List applied to the collections after each call (do you see your changes in the db inbetween calls) ? I had memory overflow issues during xquery update functions calls, but always when issued in the same script, because of saturated PUL.
Best, Fabrice
Profiling Module - BaseX Documentationhttps://docs.basex.org/wiki/Profiling_Module#prof:gc Signatures: prof:current-ms() as xs:integer: Summary: Returns the number of milliseconds passed since 1970/01/01 UTC. The granularity of the value depends on the underlying operating system and may be larger. docs.basex.org
________________________________ De : BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de de la part de Sebastian Guerrero chapeti@gmail.com Envoyé : jeudi 28 mai 2020 20:39 À : BaseX basex-talk@mailman.uni-konstanz.de Objet : [basex-talk] RESTXQ - java.lang.OutOfMemoryError: Java heap space
Hi BaseX team!
A quick question.
Is there some known bug/common setting missing for RESTXQ and memory problems?
I have this simple module ( into /webapp, a .xqm file ) :
module namespace exe = 'http://site.com/execute'; declare %updating %rest:path("update") %rest:consumes("application/x-www-form-urlencoded") %rest:POST function exe:update() { xquery:eval-update(request:parameter("query")) };
I use it to execute some updates against some databases from different clients.
Everything works fine by a while, but after some time I get this error [1]:
java.lang.OutOfMemoryError: Java heap space
I noticed that every call to update() the memory grows and grows until it reaches the OutOfMemoryError. [2]
If I stop the HTTP server, the memory is released immediately. [3]
What I'm doing wrong? Is there some command to execute the GC? Is this a problem with "xquery:eval-update()"? Am I using it in the wrong way?
Best regards, Sebastian. [1] https://imgur.com/DrcbwQg [2] https://imgur.com/fonmrhm [3] https://imgur.com/SYFBFK8
Hi Sebastian,
In general, artifacts of updating queries should be cleaned up after the execution.
The stack trace indicates that a very large main-memory database instance was created by one of your queries that exceeded the memory limits. Can you share the query with us?
As XQuery is more powerful than pure query languages, it may be risky to allow the execution of arbitrary client code. First of all, you should ensure that the code is run with limited user permissions; otherwise, your system can be wiped out by a single file:delete('/', true()) call. Next, you could try to limit memory usage and execution time via the 'memory' and 'timeout' parameters [1]. However, as it’s close to impossible to reliably control the memory consumption of single threads in Java, I would rather suggest providing predefined user queries.
Espero que esto ayude, Christian
[1] https://docs.basex.org/wiki/XQuery_Module#xquery:eval
On Thu, May 28, 2020 at 8:39 PM Sebastian Guerrero chapeti@gmail.com wrote:
Hi BaseX team!
A quick question.
Is there some known bug/common setting missing for RESTXQ and memory problems?
I have this simple module ( into /webapp, a .xqm file ) :
module namespace exe = 'http://site.com/execute'; declare %updating %rest:path("update") %rest:consumes("application/x-www-form-urlencoded") %rest:POST function exe:update() {
xquery:eval-update(request:parameter("query"))
};
I use it to execute some updates against some databases from different clients.
Everything works fine by a while, but after some time I get this error [1]:
java.lang.OutOfMemoryError: Java heap space
I noticed that every call to update() the memory grows and grows until it reaches the OutOfMemoryError. [2]
If I stop the HTTP server, the memory is released immediately. [3]
What I'm doing wrong? Is there some command to execute the GC? Is this a problem with "xquery:eval-update()"? Am I using it in the wrong way?
Best regards, Sebastian. [1] https://imgur.com/DrcbwQg [2] https://imgur.com/fonmrhm [3] https://imgur.com/SYFBFK8
Hi Christian,
Thank you for your reply, you always help me.
*- Can you share the query with us?*. Yes, of course.
The query is pretty much the same that I've sent to Fabrice:
*(# db:copynode false #) {* * for $case in doc('file_A.xml')/trademark-applications-daily/application-information/file-segments/action-keys/case-file[case-file-header/status-code=(410,413,616,620,624,625,630,631,638,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,663,664,665,666,667,668,672,680,681,682,686,688,689,690,692,693,694,700,701,702,703,704,705,706,707,708,717,718,719,720,721,722,724,725,730,731,732,733,734,739,740,744,745,746,748,752,753,756,757,760,762,763,764,765,766,771,772,773,774,775,777,778,779,780,790,794,800,801,802,803,804,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,969,973)] * * return insert node $case as last into doc("US")* *}*
And you're totally right about the security concerns: I don't offer to anyone the access to the RESTXQ endpoint, it's only for me but from another app ( on this case an Azure Web job instance ).
It's easy for me to do some maintenance tasks from a C# app, using the RESTXQ, instead of directly on some BaseX GUI.
At the beginning I tried to do this:
*let $path:="A:\sources\xml"*
*let $files:=* *let $parts:=("US00","US01")* *for $part in $parts* *let $dir:= $path || $part* *for $file in file:list($dir)* *return $path || $part || "" || $file*
*return* *(# db:copynode false #) {* * for $file in $files * * for $case in doc($file)/trademark-applications-daily/application-information/file-segments/action-keys/case-file[case-file-header/status-code=(410,413,616,620,624,625,630,631,638,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,663,664,665,666,667,668,672,680,681,682,686,688,689,690,692,693,694,700,701,702,703,704,705,706,707,708,717,718,719,720,721,722,724,725,730,731,732,733,734,739,740,744,745,746,748,752,753,756,757,760,762,763,764,765,766,771,772,773,774,775,777,778,779,780,790,794,800,801,802,803,804,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,969,973)] * * return insert node $case as last into doc("US")* *}*
And, of course, this ran out of memory in a matter of seconds. ( there are more than 100 files )
So, I thought:
-* "Maybe, if I execute each of them using the RESTXQ the memory will be released after each execution..." *
But seems I'm still missing something because the result is the same. But, at least, I can finish the process restarting the HTTP server when it fails.
I've uploaded two of the XML files in case you want to test something with them.[1]
Por supuesto que ayuda! Regards, Sebastian.
[1] https://easyupload.io/sd2ilu
On Fri, May 29, 2020 at 8:38 AM Christian Grün christian.gruen@gmail.com wrote:
Hi Sebastian,
In general, artifacts of updating queries should be cleaned up after the execution.
The stack trace indicates that a very large main-memory database instance was created by one of your queries that exceeded the memory limits. Can you share the query with us?
As XQuery is more powerful than pure query languages, it may be risky to allow the execution of arbitrary client code. First of all, you should ensure that the code is run with limited user permissions; otherwise, your system can be wiped out by a single file:delete('/', true()) call. Next, you could try to limit memory usage and execution time via the 'memory' and 'timeout' parameters [1]. However, as it’s close to impossible to reliably control the memory consumption of single threads in Java, I would rather suggest providing predefined user queries.
Espero que esto ayude, Christian
[1] https://docs.basex.org/wiki/XQuery_Module#xquery:eval
On Thu, May 28, 2020 at 8:39 PM Sebastian Guerrero chapeti@gmail.com wrote:
Hi BaseX team!
A quick question.
Is there some known bug/common setting missing for RESTXQ and memory
problems?
I have this simple module ( into /webapp, a .xqm file ) :
module namespace exe = 'http://site.com/execute'; declare %updating %rest:path("update") %rest:consumes("application/x-www-form-urlencoded") %rest:POST function exe:update() {
xquery:eval-update(request:parameter("query"))
};
I use it to execute some updates against some databases from different
clients.
Everything works fine by a while, but after some time I get this error
[1]:
java.lang.OutOfMemoryError: Java heap space
I noticed that every call to update() the memory grows and grows until
it reaches the OutOfMemoryError. [2]
If I stop the HTTP server, the memory is released immediately. [3]
What I'm doing wrong? Is there some command to execute the GC? Is this a problem with "xquery:eval-update()"? Am I using it in the wrong way?
Best regards, Sebastian. [1] https://imgur.com/DrcbwQg [2] https://imgur.com/fonmrhm [3] https://imgur.com/SYFBFK8
Hm… Yes, maybe the number of inserts is simply too large for single insert operations. Did you count how many new cases will be created?
into doc("US")
Are you inserting into a document or a database? In the latter case, you could replace the function call with db:open.
The db:copynode option can probably be dropped, as it only affects the construction of new nodes in XQuery (e.g. via "element abc { ... }").
It MAY be a difference if you use a single insert statement (I’m not sure if that matters for your use case):
let $nodes := ( for $file in $files return doc($file)/trademark/..... ) return insert nodes $nodes into db:open('US')
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
ETANCHAUD Fabrice -
Sebastian Guerrero