Hi Michael,
Your query looks pretty straightforward. As you have already guessed, it’s simply the big number of inserted nodes that causes the memory error.
Is there any chance to assign more memory to your BaseX Java process? If not, you may need to write an intermediate XML document with the desired structure to disk and reimport this file in a second step. You could also call your function multiple times, and insert only parts of you source data in a single run.
Hope this helps, Christian
On Fri, Sep 27, 2019 at 12:05 PM BIRKNER Michael Michael.BIRKNER@akwien.at wrote:
Hi to all,
I get an "Out of Memory" error (using the BaseX GUI on Ubuntu Linux) when I try to insert quite a lot of data into a BaseX database. The use case: I have a database (size is about 2600 MB, 13718400 nodes) with information in <record> elements that should be added to <record> elements in another database. The <record>s have a 1 to 1 connection identified by an ID that is available in both databases.
An example (simplified) of the DB with the information I want to add to the other DB:
<collection> <record> <id>1</id> <data>Some data</data> <data>More data</data> <data>More data</data> ... </record> <record> <id>2</id> <data>Some data</data> <data>More data</data> <data>More data</data> ... </record> <record> <id>3</id> <data>Some data</data> <data>More data</data> <data>More data</data> ... </record> ... many many more <record>s </collection>
Here an example (simplified) of the DB to which the above <data> elements should be added:
<collection> <record> <id>1</id> <mainData>Main data</mainData> <mainData>More main data</mainData> <mainData>More main data</mainData> ... <!-- Insert <data> elements of record with id 1 of the other database here --> </record> <record> <id>2</id> <mainData>Main data</mainData> <mainData>More main data</mainData> <mainData>More main data</mainData> ... <!-- Insert <data> elements of record with id 2 of the other database here --> </record> <record> <id>3</id> <mainData>Main data</mainData> <mainData>More main data</mainData> <mainData>More main data</mainData> ... <!-- Insert <data> elements of record with id 3 of the other database here --> </record> ... many many more <record>s </collection>
This is the XQuery I use to insert the given <data> elements from the <record> in one database to the corresponding <record> in the other database. It results in an "Out of Memory" error:
let $infoRecs := db:open('db-with-data')/collection/record let $mainRecs := db:open('db-to-insert-data')/collection/record for $infoRec in $infoRecs let $id := data($infoRec/id) let $mainRec := $mainRecs[id=$id] let $dataToInsert := $infoRec/*[not(name()='id')] return insert node ($dataToInsert) into $mainRec
I assume that the error is a result of the large amount of data that is processed. My question is if a strategy exists to work with such an amount of data without getting an "Out of Memory" error?
Thanks very much to everyone in advance for any hint and advice. If you need more information about DB setup or options just let me know.
Best regrads, Michael