Exactly, it seems to be the final MERGE step during index creation
that blows up your system. If you are restricted to the 2 GB of
main-memory, this is what you could try next:

1. Did you already try to tweak the JVM memory limit via -Xmx? What’s
the largest value that you can assign on your system?

2. If you will query only specific values of your data sets, you can
restrict your indexes to specific elements or attributes; this will
reduce memory consumption (see [1] for details). If you observe that
no indexes will be utilized in your queries anyway, you can simply
disable the text and attribute indexes, and memory usage will shrink
even more.

3. Create your database on a more powerful system [2] and move it to
your target machine (makes only sense if there’s no need for further
updates).

4. Distribute your data across multiple databases. In some way, this
is comparable to sharding; it cannot be automated, though, as the
partitioning strategy depends on the characteristics of your XML input
data (some people have huge standalone documents, others have millions
of small documents, …).

[1] http://docs.basex.org/wiki/Indexes
[2] A single CREATE call may be sufficient: CREATE DB database
sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz




On Thu, Oct 3, 2019 at 8:53 AM first name last name
<randomcoder1@gmail.com> wrote:
>
> I tried again, using SPLITSIZE = 12 in the .basex config file
> The batch(console) script I used is attached mass-import.xq
> This time I didn't do the optimize or index creation post-import, but instead, I did it as part of the import similar to what
> is described in [4].
> This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory."
> So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here?
>
> Thanks
>
> [4] http://docs.basex.org/wiki/Indexes#Performance
>
>
> On Wed, Oct 2, 2019 at 11:06 AM first name last name <randomcoder1@gmail.com> wrote:
>>
>> Hey Christian,
>>
>> Thank you for your answer :)
>> I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and
>> then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again.
>> Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step).
>>
>> To quote the statistics page from the wiki:
>>     Databases in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression.
>> This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this
>> I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them.
>>
>> Best regards,
>> Stefan
>>
>>
>> On Tue, Oct 1, 2019 at 4:22 PM Christian Grün <christian.gruen@gmail.com> wrote:
>>>
>>> Hi first name,
>>>
>>> If you optimize your database, the indexes will be rebuilt. In this
>>> step, the builder tries to guess how much free memory is still
>>> available. If memory is exhausted, parts of the index will be split
>>> (i. e., partially written to disk) and merged in a final step.
>>> However, you can circumvent the heuristics by manually assigning a
>>> static split value; see [1] for more information. If you use the DBA,
>>> you’ll need to assign this value to your .basex or the web.xml file
>>> [2]. In order to find the best value for your setup, it may be easier
>>> to play around with the BaseX GUI.
>>>
>>> As you have already seen in our statistics, an XML document has
>>> various properties that may represent a limit for a single database.
>>> Accordingly, these properties make it difficult to decide for the
>>> system when the memory will be exhausted during an import or index
>>> rebuild.
>>>
>>> In general, you’ll get best performance (and your memory consumption
>>> will be lower) if you create your database and specify the data to be
>>> imported in a single run. This is currently not possible via the DBA;
>>> use the GUI (Create Database) or console mode (CREATE DB command)
>>> instead.
>>>
>>> Hope this helps,
>>> Christian
>>>
>>> [1] http://docs.basex.org/wiki/Options#SPLITSIZE
>>> [2] http://docs.basex.org/wiki/Configuration
>>>
>>>
>>>
>>> On Mon, Sep 30, 2019 at 7:09 AM first name last name
>>> <randomcoder1@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > Let's say there's a 30GB dataset [3] containing most threads/posts from [1].
>>> > After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory).
>>> > I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any.
>>> > Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed.
>>> > In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize".
>>> > Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do?
>>> >
>>> > Thanks,
>>> > Stefan
>>> >
>>> > [1] https://www.linuxquestions.org/
>>> > [2] http://docs.basex.org/wiki/Statistics
>>> > [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2