Memory footprint for a certain query is much higher usisng BaseX10 compared to 9
Hi, BaseX 10 has a different behavior in memory footprint for the following situation. To avoid passing around data you can create db using the following create (i did this with BaseX 9) declare variable $input := <root><sheet name="SAP">{for $idx in (1 to 10000) return <row>{for $e in $enames return element {$e} { attribute {"content"} {$idx}}}</row>}</sheet></root>; db:create("test-db",$input,"test.xml") Then using the following query on the created db: declare variable $input := db:open("test-db"); <root>{ for $s in $input//sheet[./@name => contains("SAP")] return <sheet>{$s/@*}{ for $r in $s//row let $no := $r/Objektkurztext/@content return <row>{$r/@*}{($r/*,<partno>{$no}</partno>)}</row> }</sheet> }</root> Using BaseX 9 the used memory for the query is round about 2 GB which is ok. Using BaseX 10 Out-of-Memory exception happens in my case used memory is > 12 GB Any thoughts what the reason is - any additional information required? Alex
Hi Alex,
declare variable $input := <root><sheet name="SAP">{for $idx in (1 to 10000) return <row>{for $e in $enames return element {$e} { attribute {"content"} {$idx}}}</row>}</sheet></root>; db:create("test-db",$input,"test.xml")
Looking at the subsequent query, I assume there must be additional "Objektkurztext" elements. Do you have a query for us that we can run out of the box? Thanks in advance, Christian
Ah I see -for the create db part I missed the list of element names. Stored in variable $enames One of them is the referenced one and the rest are any name 5-15 chars long. There are round about 30 elements / I assume the real name doesn’t make a difference… each of them are unique in the list of enames. but I can send you those next week The created db using the same characteristic and volume than real world data and reproducing the issue at least on my test client Alex
Am 12.08.2022 um 18:01 schrieb Christian Grün <christian.gruen@gmail.com>:
Diese Nachricht stammt von einem nicht vertrauenswürdigen Absender Sie haben bisher nicht mit diesem Absender korrespondiert. Hi Alex,
declare variable $input := <root><sheet name="SAP">{for $idx in (1 to 10000) return <row>{for $e in $enames return element {$e} { attribute {"content"} {$idx}}}</row>}</sheet></root>; db:create("test-db",$input,"test.xml")
Looking at the subsequent query, I assume there must be additional "Objektkurztext" elements. Do you have a query for us that we can run out of the box?
Thanks in advance, Christian
I managed to get this reproduced [1]. A first JMC analysis indicates this seems to be related to the revised data structures for main-memory updates [2]. A small overhead would be expectable and acceptable, but definitely not on this scale. Thanks for the observation. [1] https://github.com/BaseXdb/basex/issues/2134 [2] https://docs.basex.org/wiki/BaseX_10#Main-Memory_Updates
…and I’m looking forward to your feedback on the latest snapshot [1]. Best, Christian [1] https://files.basex.org/releases/latest/
works wonderful - appr. same memory footprint as using 9.x release and similar and fast response time ... thanks alex Am 12.08.2022 um 20:28 schrieb Christian Grün:
…and I’m looking forward to your feedback on the latest snapshot [1]. Best, Christian [1] https: //files. basex. org/releases/latest/
…and I’m looking forward to your feedback on the latest snapshot [1].
Best, Christian
participants (2)
-
Alexander Witzigmann -
Christian Grün