basex OOM on 30GB database upon running /dba/db-optimize/
Hi, Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do? Thanks, Stefan [1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
Hi first name, If you optimize your database, the indexes will be rebuilt. In this step, the builder tries to guess how much free memory is still available. If memory is exhausted, parts of the index will be split (i. e., partially written to disk) and merged in a final step. However, you can circumvent the heuristics by manually assigning a static split value; see [1] for more information. If you use the DBA, you’ll need to assign this value to your .basex or the web.xml file [2]. In order to find the best value for your setup, it may be easier to play around with the BaseX GUI. As you have already seen in our statistics, an XML document has various properties that may represent a limit for a single database. Accordingly, these properties make it difficult to decide for the system when the memory will be exhausted during an import or index rebuild. In general, you’ll get best performance (and your memory consumption will be lower) if you create your database and specify the data to be imported in a single run. This is currently not possible via the DBA; use the GUI (Create Database) or console mode (CREATE DB command) instead. Hope this helps, Christian [1] http://docs.basex.org/wiki/Options#SPLITSIZE [2] http://docs.basex.org/wiki/Configuration On Mon, Sep 30, 2019 at 7:09 AM first name last name <randomcoder1@gmail.com> wrote:
Hi,
Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do?
Thanks, Stefan
[1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
Hey Christian, Thank you for your answer :) I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step). To quote the statistics page from the wiki: Databases <http://docs.basex.org/wiki/Databases> in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them. Best regards, Stefan On Tue, Oct 1, 2019 at 4:22 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi first name,
If you optimize your database, the indexes will be rebuilt. In this step, the builder tries to guess how much free memory is still available. If memory is exhausted, parts of the index will be split (i. e., partially written to disk) and merged in a final step. However, you can circumvent the heuristics by manually assigning a static split value; see [1] for more information. If you use the DBA, you’ll need to assign this value to your .basex or the web.xml file [2]. In order to find the best value for your setup, it may be easier to play around with the BaseX GUI.
As you have already seen in our statistics, an XML document has various properties that may represent a limit for a single database. Accordingly, these properties make it difficult to decide for the system when the memory will be exhausted during an import or index rebuild.
In general, you’ll get best performance (and your memory consumption will be lower) if you create your database and specify the data to be imported in a single run. This is currently not possible via the DBA; use the GUI (Create Database) or console mode (CREATE DB command) instead.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Options#SPLITSIZE [2] http://docs.basex.org/wiki/Configuration
On Mon, Sep 30, 2019 at 7:09 AM first name last name <randomcoder1@gmail.com> wrote:
Hi,
Let's say there's a 30GB dataset [3] containing most threads/posts from
[1].
After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do?
Thanks, Stefan
[1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
I tried again, using SPLITSIZE = 12 in the .basex config file The batch(console) script I used is attached mass-import.xq This time I didn't do the optimize or index creation post-import, but instead, I did it as part of the import similar to what is described in [4]. This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory.*"* So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here? Thanks [4] http://docs.basex.org/wiki/Indexes#Performance On Wed, Oct 2, 2019 at 11:06 AM first name last name <randomcoder1@gmail.com> wrote:
Hey Christian,
Thank you for your answer :) I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step).
To quote the statistics page from the wiki: Databases <http://docs.basex.org/wiki/Databases> in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them.
Best regards, Stefan
On Tue, Oct 1, 2019 at 4:22 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi first name,
If you optimize your database, the indexes will be rebuilt. In this step, the builder tries to guess how much free memory is still available. If memory is exhausted, parts of the index will be split (i. e., partially written to disk) and merged in a final step. However, you can circumvent the heuristics by manually assigning a static split value; see [1] for more information. If you use the DBA, you’ll need to assign this value to your .basex or the web.xml file [2]. In order to find the best value for your setup, it may be easier to play around with the BaseX GUI.
As you have already seen in our statistics, an XML document has various properties that may represent a limit for a single database. Accordingly, these properties make it difficult to decide for the system when the memory will be exhausted during an import or index rebuild.
In general, you’ll get best performance (and your memory consumption will be lower) if you create your database and specify the data to be imported in a single run. This is currently not possible via the DBA; use the GUI (Create Database) or console mode (CREATE DB command) instead.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Options#SPLITSIZE [2] http://docs.basex.org/wiki/Configuration
On Mon, Sep 30, 2019 at 7:09 AM first name last name <randomcoder1@gmail.com> wrote:
Hi,
Let's say there's a 30GB dataset [3] containing most threads/posts from
[1].
After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do?
Thanks, Stefan
[1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
Hi, just saying that 16 GB of DDR3 RAM cost about 40 € now. Gerrit On 03.10.2019 08:53, first name last name wrote:
I tried again, using SPLITSIZE = 12 in the .basex config file The batch(console) script I used is attached mass-import.xq This time I didn't do the optimize or index creation post-import, but instead, I did it as part of the import similar to what is described in [4]. This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory.*"* So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here?
Thanks
[4] http://docs.basex.org/wiki/Indexes#Performance
On Wed, Oct 2, 2019 at 11:06 AM first name last name <randomcoder1@gmail.com <mailto:randomcoder1@gmail.com>> wrote:
Hey Christian,
Thank you for your answer :) I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step).
To quote the statistics page from the wiki: Databases <http://docs.basex.org/wiki/Databases> in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them.
Best regards, Stefan
On Tue, Oct 1, 2019 at 4:22 PM Christian Grün <christian.gruen@gmail.com <mailto:christian.gruen@gmail.com>> wrote:
Hi first name,
If you optimize your database, the indexes will be rebuilt. In this step, the builder tries to guess how much free memory is still available. If memory is exhausted, parts of the index will be split (i. e., partially written to disk) and merged in a final step. However, you can circumvent the heuristics by manually assigning a static split value; see [1] for more information. If you use the DBA, you’ll need to assign this value to your .basex or the web.xml file [2]. In order to find the best value for your setup, it may be easier to play around with the BaseX GUI.
As you have already seen in our statistics, an XML document has various properties that may represent a limit for a single database. Accordingly, these properties make it difficult to decide for the system when the memory will be exhausted during an import or index rebuild.
In general, you’ll get best performance (and your memory consumption will be lower) if you create your database and specify the data to be imported in a single run. This is currently not possible via the DBA; use the GUI (Create Database) or console mode (CREATE DB command) instead.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Options#SPLITSIZE [2] http://docs.basex.org/wiki/Configuration
On Mon, Sep 30, 2019 at 7:09 AM first name last name <randomcoder1@gmail.com <mailto:randomcoder1@gmail.com>> wrote: > > Hi, > > Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. > After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). > I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. > Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. > In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". > Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do? > > Thanks, > Stefan > > [1] https://www.linuxquestions.org/ > [2] http://docs.basex.org/wiki/Statistics > [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke@le-tex.de, http://www.le-tex.de Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer / Managing Directors: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt
Exactly, it seems to be the final MERGE step during index creation that blows up your system. If you are restricted to the 2 GB of main-memory, this is what you could try next: 1. Did you already try to tweak the JVM memory limit via -Xmx? What’s the largest value that you can assign on your system? 2. If you will query only specific values of your data sets, you can restrict your indexes to specific elements or attributes; this will reduce memory consumption (see [1] for details). If you observe that no indexes will be utilized in your queries anyway, you can simply disable the text and attribute indexes, and memory usage will shrink even more. 3. Create your database on a more powerful system [2] and move it to your target machine (makes only sense if there’s no need for further updates). 4. Distribute your data across multiple databases. In some way, this is comparable to sharding; it cannot be automated, though, as the partitioning strategy depends on the characteristics of your XML input data (some people have huge standalone documents, others have millions of small documents, …). [1] http://docs.basex.org/wiki/Indexes [2] A single CREATE call may be sufficient: CREATE DB database sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz On Thu, Oct 3, 2019 at 8:53 AM first name last name <randomcoder1@gmail.com> wrote:
I tried again, using SPLITSIZE = 12 in the .basex config file The batch(console) script I used is attached mass-import.xq This time I didn't do the optimize or index creation post-import, but instead, I did it as part of the import similar to what is described in [4]. This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory." So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here?
Thanks
[4] http://docs.basex.org/wiki/Indexes#Performance
On Wed, Oct 2, 2019 at 11:06 AM first name last name <randomcoder1@gmail.com> wrote:
Hey Christian,
Thank you for your answer :) I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step).
To quote the statistics page from the wiki: Databases in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them.
Best regards, Stefan
On Tue, Oct 1, 2019 at 4:22 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi first name,
If you optimize your database, the indexes will be rebuilt. In this step, the builder tries to guess how much free memory is still available. If memory is exhausted, parts of the index will be split (i. e., partially written to disk) and merged in a final step. However, you can circumvent the heuristics by manually assigning a static split value; see [1] for more information. If you use the DBA, you’ll need to assign this value to your .basex or the web.xml file [2]. In order to find the best value for your setup, it may be easier to play around with the BaseX GUI.
As you have already seen in our statistics, an XML document has various properties that may represent a limit for a single database. Accordingly, these properties make it difficult to decide for the system when the memory will be exhausted during an import or index rebuild.
In general, you’ll get best performance (and your memory consumption will be lower) if you create your database and specify the data to be imported in a single run. This is currently not possible via the DBA; use the GUI (Create Database) or console mode (CREATE DB command) instead.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Options#SPLITSIZE [2] http://docs.basex.org/wiki/Configuration
On Mon, Sep 30, 2019 at 7:09 AM first name last name <randomcoder1@gmail.com> wrote:
Hi,
Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do?
Thanks, Stefan
[1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
Hi Christian, About option 4: I agree with the options you laid out. I am currently diving deeper into option 4 in the list you wrote. Regarding the partitioning strategy, I agree. I did manage however to partition the files to be imported, into separate sets, with a constraint on max partition size (on disk) and max partition file count (the number of XML documents in each partition). The tool called fpart [5] made this possible (I can imagine more sophisticated bin-packing methods, involving pre-computed node count values, and other variables, can be achieved via glpk [6] but that might be too much work). So, currently I am experimenting with a max partition size of 2.4GB and a max file count of 85k files, and fpart seems to have split the file list into 11 partitions of 33k files each and the size of a partition being ~ 2.4GB. So, I wrote a script for this, it's called sharded-import.sh and attached here. I'm also noticing that the /dba/ BaseX web interface is not blocked anymore if I run this script, as opposed to running the previous import where I run CREATE DB db_name /directory/ which allows me to see the progress or allows me to run queries before the big import finishes. Maybe the downside is that it's more verbose, and prints out a ton of lines like
ADD /share/Public/archive/tech-sites/ linuxquestions.org/threads/viewtopic_9_356613.html Resource(s) added in 47.76 ms. along the way, and maybe that's slower than before.
About option 1: Re: increase memory, I am running these experiments on a low-memory, old, network-attached storage, model QNAP TS-451+ [7] [8], which I had to take apart with a screwdriver to add 2GB of RAM (now it has 4GB of memory), and I can't seem to find around the house any additional memory sticks to take it up to 8GB (which is also the maximum memory it supports). And if I want to find like 2 x 4GB sticks of RAM, the frequency of the memory has to match what it supports, I'm having trouble finding the exact one, Corsair says it has memory sticks that would work, but I'd have to wait weeks for them to ship to Bucharest which is where I live. It seems like buying an Intel NUC that goes up to 64GB of memory would be a bit too expensive at $1639 [9] but .. people on reddit [10] were discussing some years back about this supermicro server [11] which is only $668 and would allow to add up to 64GB of memory. Basically I would buy something cheap that I can jampack with a lot of RAM, but a hands-off approach would be best here, so if it comes pre-equipped with all the memory and everything, would be nice (would spare the trouble of having to buy the memory separate, making sure it matches the motherboard specs etc). About option 2: In fact, that's a great idea. But it would require me to write something that would figure out the XPath patterns where the actual content sits. I actually wanted to look for some algorithm that's designed to do that, and try to implement it, but I had no time. It would either have to detect the repetitive bloated nodes, and build XPaths for the rest of the nodes, where the actual content sits. I think this would be equivalent to computing the "web template" of a website, given all its pages. It would definitely decrease the size of the content that would have to be indexed. By the way, here I'm writing about a more general procedure, because it's not just this dataset that I want to import.. I want to import heavy, large amounts of data :) These are my thoughts for now [5] https://github.com/martymac/fpart [6] https://www.gnu.org/software/glpk/ [7] https://www.amazon.com/dp/B015VNLGF8 [8] https://www.qnap.com/en/product/ts-451+ [9] https://www.amazon.com/Intel-NUC-NUC8I7HNK-Gaming-Mini/dp/B07WGWWSWT/ [10] https://www.reddit.com/r/sysadmin/comments/64x2sb/nuc_like_system_but_with_6... [11] https://www.amazon.com/Supermicro-SuperServer-E300-8D-Mini-1U-D-1518/dp/B01M... On Thu, Oct 3, 2019 at 1:30 PM Christian Grün <christian.gruen@gmail.com> wrote:
Exactly, it seems to be the final MERGE step during index creation that blows up your system. If you are restricted to the 2 GB of main-memory, this is what you could try next:
1. Did you already try to tweak the JVM memory limit via -Xmx? What’s the largest value that you can assign on your system?
2. If you will query only specific values of your data sets, you can restrict your indexes to specific elements or attributes; this will reduce memory consumption (see [1] for details). If you observe that no indexes will be utilized in your queries anyway, you can simply disable the text and attribute indexes, and memory usage will shrink even more.
3. Create your database on a more powerful system [2] and move it to your target machine (makes only sense if there’s no need for further updates).
4. Distribute your data across multiple databases. In some way, this is comparable to sharding; it cannot be automated, though, as the partitioning strategy depends on the characteristics of your XML input data (some people have huge standalone documents, others have millions of small documents, …).
[1] http://docs.basex.org/wiki/Indexes [2] A single CREATE call may be sufficient: CREATE DB database sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz
On Thu, Oct 3, 2019 at 8:53 AM first name last name <randomcoder1@gmail.com> wrote:
I tried again, using SPLITSIZE = 12 in the .basex config file The batch(console) script I used is attached mass-import.xq This time I didn't do the optimize or index creation post-import, but
instead, I did it as part of the import similar to what
is described in [4]. This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory." So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here?
Thanks
[4] http://docs.basex.org/wiki/Indexes#Performance
On Wed, Oct 2, 2019 at 11:06 AM first name last name < randomcoder1@gmail.com> wrote:
Hey Christian,
Thank you for your answer :) I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same
OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and
then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step).
To quote the statistics page from the wiki: Databases in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them.
Best regards, Stefan
On Tue, Oct 1, 2019 at 4:22 PM Christian Grün < christian.gruen@gmail.com> wrote:
Hi first name,
If you optimize your database, the indexes will be rebuilt. In this step, the builder tries to guess how much free memory is still available. If memory is exhausted, parts of the index will be split (i. e., partially written to disk) and merged in a final step. However, you can circumvent the heuristics by manually assigning a static split value; see [1] for more information. If you use the DBA, you’ll need to assign this value to your .basex or the web.xml file [2]. In order to find the best value for your setup, it may be easier to play around with the BaseX GUI.
As you have already seen in our statistics, an XML document has various properties that may represent a limit for a single database. Accordingly, these properties make it difficult to decide for the system when the memory will be exhausted during an import or index rebuild.
In general, you’ll get best performance (and your memory consumption will be lower) if you create your database and specify the data to be imported in a single run. This is currently not possible via the DBA; use the GUI (Create Database) or console mode (CREATE DB command) instead.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Options#SPLITSIZE [2] http://docs.basex.org/wiki/Configuration
On Mon, Sep 30, 2019 at 7:09 AM first name last name <randomcoder1@gmail.com> wrote:
Hi,
Let's say there's a 30GB dataset [3] containing most threads/posts
from [1].
After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do?
Thanks, Stefan
[1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
I had another look at the script I wrote and realized that it's not working as it's supposed to. Apparently the order of operations should be this: - turn on all the types of indexes required - create the db - the parser settings and the filter settings - add all the files to the db - run "OPTIMIZE" If I'm not doing them in this order (specifically with "OPTIMIZE" at the end) the resulting db lacks all indexes. On Fri, Oct 4, 2019 at 11:32 PM first name last name <randomcoder1@gmail.com> wrote:
Hi Christian,
About option 4: I agree with the options you laid out. I am currently diving deeper into option 4 in the list you wrote. Regarding the partitioning strategy, I agree. I did manage however to partition the files to be imported, into separate sets, with a constraint on max partition size (on disk) and max partition file count (the number of XML documents in each partition). The tool called fpart [5] made this possible (I can imagine more sophisticated bin-packing methods, involving pre-computed node count values, and other variables, can be achieved via glpk [6] but that might be too much work). So, currently I am experimenting with a max partition size of 2.4GB and a max file count of 85k files, and fpart seems to have split the file list into 11 partitions of 33k files each and the size of a partition being ~ 2.4GB. So, I wrote a script for this, it's called sharded-import.sh and attached here. I'm also noticing that the /dba/ BaseX web interface is not blocked anymore if I run this script, as opposed to running the previous import where I run CREATE DB db_name /directory/ which allows me to see the progress or allows me to run queries before the big import finishes. Maybe the downside is that it's more verbose, and prints out a ton of lines like
ADD /share/Public/archive/tech-sites/ linuxquestions.org/threads/viewtopic_9_356613.html Resource(s) added in 47.76 ms. along the way, and maybe that's slower than before.
About option 1: Re: increase memory, I am running these experiments on a low-memory, old, network-attached storage, model QNAP TS-451+ [7] [8], which I had to take apart with a screwdriver to add 2GB of RAM (now it has 4GB of memory), and I can't seem to find around the house any additional memory sticks to take it up to 8GB (which is also the maximum memory it supports). And if I want to find like 2 x 4GB sticks of RAM, the frequency of the memory has to match what it supports, I'm having trouble finding the exact one, Corsair says it has memory sticks that would work, but I'd have to wait weeks for them to ship to Bucharest which is where I live. It seems like buying an Intel NUC that goes up to 64GB of memory would be a bit too expensive at $1639 [9] but .. people on reddit [10] were discussing some years back about this supermicro server [11] which is only $668 and would allow to add up to 64GB of memory. Basically I would buy something cheap that I can jampack with a lot of RAM, but a hands-off approach would be best here, so if it comes pre-equipped with all the memory and everything, would be nice (would spare the trouble of having to buy the memory separate, making sure it matches the motherboard specs etc).
About option 2: In fact, that's a great idea. But it would require me to write something that would figure out the XPath patterns where the actual content sits. I actually wanted to look for some algorithm that's designed to do that, and try to implement it, but I had no time. It would either have to detect the repetitive bloated nodes, and build XPaths for the rest of the nodes, where the actual content sits. I think this would be equivalent to computing the "web template" of a website, given all its pages. It would definitely decrease the size of the content that would have to be indexed. By the way, here I'm writing about a more general procedure, because it's not just this dataset that I want to import.. I want to import heavy, large amounts of data :)
These are my thoughts for now
[5] https://github.com/martymac/fpart [6] https://www.gnu.org/software/glpk/ [7] https://www.amazon.com/dp/B015VNLGF8 [8] https://www.qnap.com/en/product/ts-451+ [9] https://www.amazon.com/Intel-NUC-NUC8I7HNK-Gaming-Mini/dp/B07WGWWSWT/ [10] https://www.reddit.com/r/sysadmin/comments/64x2sb/nuc_like_system_but_with_6... [11] https://www.amazon.com/Supermicro-SuperServer-E300-8D-Mini-1U-D-1518/dp/B01M...
On Thu, Oct 3, 2019 at 1:30 PM Christian Grün <christian.gruen@gmail.com> wrote:
Exactly, it seems to be the final MERGE step during index creation that blows up your system. If you are restricted to the 2 GB of main-memory, this is what you could try next:
1. Did you already try to tweak the JVM memory limit via -Xmx? What’s the largest value that you can assign on your system?
2. If you will query only specific values of your data sets, you can restrict your indexes to specific elements or attributes; this will reduce memory consumption (see [1] for details). If you observe that no indexes will be utilized in your queries anyway, you can simply disable the text and attribute indexes, and memory usage will shrink even more.
3. Create your database on a more powerful system [2] and move it to your target machine (makes only sense if there’s no need for further updates).
4. Distribute your data across multiple databases. In some way, this is comparable to sharding; it cannot be automated, though, as the partitioning strategy depends on the characteristics of your XML input data (some people have huge standalone documents, others have millions of small documents, …).
[1] http://docs.basex.org/wiki/Indexes [2] A single CREATE call may be sufficient: CREATE DB database sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz
On Thu, Oct 3, 2019 at 8:53 AM first name last name <randomcoder1@gmail.com> wrote:
I tried again, using SPLITSIZE = 12 in the .basex config file The batch(console) script I used is attached mass-import.xq This time I didn't do the optimize or index creation post-import, but
instead, I did it as part of the import similar to what
is described in [4]. This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory." So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here?
Thanks
[4] http://docs.basex.org/wiki/Indexes#Performance
On Wed, Oct 2, 2019 at 11:06 AM first name last name < randomcoder1@gmail.com> wrote:
Hey Christian,
Thank you for your answer :) I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same
OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and
then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step).
To quote the statistics page from the wiki: Databases in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them.
Best regards, Stefan
On Tue, Oct 1, 2019 at 4:22 PM Christian Grün < christian.gruen@gmail.com> wrote:
Hi first name,
If you optimize your database, the indexes will be rebuilt. In this step, the builder tries to guess how much free memory is still available. If memory is exhausted, parts of the index will be split (i. e., partially written to disk) and merged in a final step. However, you can circumvent the heuristics by manually assigning a static split value; see [1] for more information. If you use the DBA, you’ll need to assign this value to your .basex or the web.xml file [2]. In order to find the best value for your setup, it may be easier to play around with the BaseX GUI.
As you have already seen in our statistics, an XML document has various properties that may represent a limit for a single database. Accordingly, these properties make it difficult to decide for the system when the memory will be exhausted during an import or index rebuild.
In general, you’ll get best performance (and your memory consumption will be lower) if you create your database and specify the data to be imported in a single run. This is currently not possible via the DBA; use the GUI (Create Database) or console mode (CREATE DB command) instead.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Options#SPLITSIZE [2] http://docs.basex.org/wiki/Configuration
On Mon, Sep 30, 2019 at 7:09 AM first name last name <randomcoder1@gmail.com> wrote:
Hi,
Let's say there's a 30GB dataset [3] containing most threads/posts
from [1].
After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do?
Thanks, Stefan
[1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
About option 1: How much memory have you been able to assign to the Java VM? first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 01:11:
I had another look at the script I wrote and realized that it's not working as it's supposed to. Apparently the order of operations should be this: - turn on all the types of indexes required - create the db - the parser settings and the filter settings - add all the files to the db - run "OPTIMIZE"
If I'm not doing them in this order (specifically with "OPTIMIZE" at the end) the resulting db lacks all indexes.
On Fri, Oct 4, 2019 at 11:32 PM first name last name < randomcoder1@gmail.com> wrote:
Hi Christian,
About option 4: I agree with the options you laid out. I am currently diving deeper into option 4 in the list you wrote. Regarding the partitioning strategy, I agree. I did manage however to partition the files to be imported, into separate sets, with a constraint on max partition size (on disk) and max partition file count (the number of XML documents in each partition). The tool called fpart [5] made this possible (I can imagine more sophisticated bin-packing methods, involving pre-computed node count values, and other variables, can be achieved via glpk [6] but that might be too much work). So, currently I am experimenting with a max partition size of 2.4GB and a max file count of 85k files, and fpart seems to have split the file list into 11 partitions of 33k files each and the size of a partition being ~ 2.4GB. So, I wrote a script for this, it's called sharded-import.sh and attached here. I'm also noticing that the /dba/ BaseX web interface is not blocked anymore if I run this script, as opposed to running the previous import where I run CREATE DB db_name /directory/ which allows me to see the progress or allows me to run queries before the big import finishes. Maybe the downside is that it's more verbose, and prints out a ton of lines like
ADD /share/Public/archive/tech-sites/ linuxquestions.org/threads/viewtopic_9_356613.html Resource(s) added in 47.76 ms. along the way, and maybe that's slower than before.
About option 1: Re: increase memory, I am running these experiments on a low-memory, old, network-attached storage, model QNAP TS-451+ [7] [8], which I had to take apart with a screwdriver to add 2GB of RAM (now it has 4GB of memory), and I can't seem to find around the house any additional memory sticks to take it up to 8GB (which is also the maximum memory it supports). And if I want to find like 2 x 4GB sticks of RAM, the frequency of the memory has to match what it supports, I'm having trouble finding the exact one, Corsair says it has memory sticks that would work, but I'd have to wait weeks for them to ship to Bucharest which is where I live. It seems like buying an Intel NUC that goes up to 64GB of memory would be a bit too expensive at $1639 [9] but .. people on reddit [10] were discussing some years back about this supermicro server [11] which is only $668 and would allow to add up to 64GB of memory. Basically I would buy something cheap that I can jampack with a lot of RAM, but a hands-off approach would be best here, so if it comes pre-equipped with all the memory and everything, would be nice (would spare the trouble of having to buy the memory separate, making sure it matches the motherboard specs etc).
About option 2: In fact, that's a great idea. But it would require me to write something that would figure out the XPath patterns where the actual content sits. I actually wanted to look for some algorithm that's designed to do that, and try to implement it, but I had no time. It would either have to detect the repetitive bloated nodes, and build XPaths for the rest of the nodes, where the actual content sits. I think this would be equivalent to computing the "web template" of a website, given all its pages. It would definitely decrease the size of the content that would have to be indexed. By the way, here I'm writing about a more general procedure, because it's not just this dataset that I want to import.. I want to import heavy, large amounts of data :)
These are my thoughts for now
[5] https://github.com/martymac/fpart [6] https://www.gnu.org/software/glpk/ [7] https://www.amazon.com/dp/B015VNLGF8 [8] https://www.qnap.com/en/product/ts-451+ [9] https://www.amazon.com/Intel-NUC-NUC8I7HNK-Gaming-Mini/dp/B07WGWWSWT/ [10] https://www.reddit.com/r/sysadmin/comments/64x2sb/nuc_like_system_but_with_6... [11] https://www.amazon.com/Supermicro-SuperServer-E300-8D-Mini-1U-D-1518/dp/B01M...
On Thu, Oct 3, 2019 at 1:30 PM Christian Grün <christian.gruen@gmail.com> wrote:
Exactly, it seems to be the final MERGE step during index creation that blows up your system. If you are restricted to the 2 GB of main-memory, this is what you could try next:
1. Did you already try to tweak the JVM memory limit via -Xmx? What’s the largest value that you can assign on your system?
2. If you will query only specific values of your data sets, you can restrict your indexes to specific elements or attributes; this will reduce memory consumption (see [1] for details). If you observe that no indexes will be utilized in your queries anyway, you can simply disable the text and attribute indexes, and memory usage will shrink even more.
3. Create your database on a more powerful system [2] and move it to your target machine (makes only sense if there’s no need for further updates).
4. Distribute your data across multiple databases. In some way, this is comparable to sharding; it cannot be automated, though, as the partitioning strategy depends on the characteristics of your XML input data (some people have huge standalone documents, others have millions of small documents, …).
[1] http://docs.basex.org/wiki/Indexes [2] A single CREATE call may be sufficient: CREATE DB database sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz
On Thu, Oct 3, 2019 at 8:53 AM first name last name <randomcoder1@gmail.com> wrote:
I tried again, using SPLITSIZE = 12 in the .basex config file The batch(console) script I used is attached mass-import.xq This time I didn't do the optimize or index creation post-import, but
instead, I did it as part of the import similar to what
is described in [4]. This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory." So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here?
Thanks
[4] http://docs.basex.org/wiki/Indexes#Performance
On Wed, Oct 2, 2019 at 11:06 AM first name last name < randomcoder1@gmail.com> wrote:
Hey Christian,
Thank you for your answer :) I tried setting in .basex the SPLITSIZE = 24000 but I've seen the
same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and
then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step).
To quote the statistics page from the wiki: Databases in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them.
Best regards, Stefan
On Tue, Oct 1, 2019 at 4:22 PM Christian Grün < christian.gruen@gmail.com> wrote:
Hi first name,
If you optimize your database, the indexes will be rebuilt. In this step, the builder tries to guess how much free memory is still available. If memory is exhausted, parts of the index will be split (i. e., partially written to disk) and merged in a final step. However, you can circumvent the heuristics by manually assigning a static split value; see [1] for more information. If you use the DBA, you’ll need to assign this value to your .basex or the web.xml file [2]. In order to find the best value for your setup, it may be easier to play around with the BaseX GUI.
As you have already seen in our statistics, an XML document has various properties that may represent a limit for a single database. Accordingly, these properties make it difficult to decide for the system when the memory will be exhausted during an import or index rebuild.
In general, you’ll get best performance (and your memory consumption will be lower) if you create your database and specify the data to be imported in a single run. This is currently not possible via the DBA; use the GUI (Create Database) or console mode (CREATE DB command) instead.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Options#SPLITSIZE [2] http://docs.basex.org/wiki/Configuration
On Mon, Sep 30, 2019 at 7:09 AM first name last name <randomcoder1@gmail.com> wrote: > > Hi, > > Let's say there's a 30GB dataset [3] containing most threads/posts
from [1].
> After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). > I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. > Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. > In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". > Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do? > > Thanks, > Stefan > > [1] https://www.linuxquestions.org/ > [2] http://docs.basex.org/wiki/Statistics > [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
Yes, I did, with -Xmx3100m (that's the maximum amount of memory I can allocate on that system for BaseX) and I got OOM. On Sat, Oct 5, 2019 at 2:19 AM Christian Grün <christian.gruen@gmail.com> wrote:
About option 1: How much memory have you been able to assign to the Java VM?
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 01:11:
I had another look at the script I wrote and realized that it's not working as it's supposed to. Apparently the order of operations should be this: - turn on all the types of indexes required - create the db - the parser settings and the filter settings - add all the files to the db - run "OPTIMIZE"
If I'm not doing them in this order (specifically with "OPTIMIZE" at the end) the resulting db lacks all indexes.
On Fri, Oct 4, 2019 at 11:32 PM first name last name < randomcoder1@gmail.com> wrote:
Hi Christian,
About option 4: I agree with the options you laid out. I am currently diving deeper into option 4 in the list you wrote. Regarding the partitioning strategy, I agree. I did manage however to partition the files to be imported, into separate sets, with a constraint on max partition size (on disk) and max partition file count (the number of XML documents in each partition). The tool called fpart [5] made this possible (I can imagine more sophisticated bin-packing methods, involving pre-computed node count values, and other variables, can be achieved via glpk [6] but that might be too much work). So, currently I am experimenting with a max partition size of 2.4GB and a max file count of 85k files, and fpart seems to have split the file list into 11 partitions of 33k files each and the size of a partition being ~ 2.4GB. So, I wrote a script for this, it's called sharded-import.sh and attached here. I'm also noticing that the /dba/ BaseX web interface is not blocked anymore if I run this script, as opposed to running the previous import where I run CREATE DB db_name /directory/ which allows me to see the progress or allows me to run queries before the big import finishes. Maybe the downside is that it's more verbose, and prints out a ton of lines like
ADD /share/Public/archive/tech-sites/ linuxquestions.org/threads/viewtopic_9_356613.html Resource(s) added in 47.76 ms. along the way, and maybe that's slower than before.
About option 1: Re: increase memory, I am running these experiments on a low-memory, old, network-attached storage, model QNAP TS-451+ [7] [8], which I had to take apart with a screwdriver to add 2GB of RAM (now it has 4GB of memory), and I can't seem to find around the house any additional memory sticks to take it up to 8GB (which is also the maximum memory it supports). And if I want to find like 2 x 4GB sticks of RAM, the frequency of the memory has to match what it supports, I'm having trouble finding the exact one, Corsair says it has memory sticks that would work, but I'd have to wait weeks for them to ship to Bucharest which is where I live. It seems like buying an Intel NUC that goes up to 64GB of memory would be a bit too expensive at $1639 [9] but .. people on reddit [10] were discussing some years back about this supermicro server [11] which is only $668 and would allow to add up to 64GB of memory. Basically I would buy something cheap that I can jampack with a lot of RAM, but a hands-off approach would be best here, so if it comes pre-equipped with all the memory and everything, would be nice (would spare the trouble of having to buy the memory separate, making sure it matches the motherboard specs etc).
About option 2: In fact, that's a great idea. But it would require me to write something that would figure out the XPath patterns where the actual content sits. I actually wanted to look for some algorithm that's designed to do that, and try to implement it, but I had no time. It would either have to detect the repetitive bloated nodes, and build XPaths for the rest of the nodes, where the actual content sits. I think this would be equivalent to computing the "web template" of a website, given all its pages. It would definitely decrease the size of the content that would have to be indexed. By the way, here I'm writing about a more general procedure, because it's not just this dataset that I want to import.. I want to import heavy, large amounts of data :)
These are my thoughts for now
[5] https://github.com/martymac/fpart [6] https://www.gnu.org/software/glpk/ [7] https://www.amazon.com/dp/B015VNLGF8 [8] https://www.qnap.com/en/product/ts-451+ [9] https://www.amazon.com/Intel-NUC-NUC8I7HNK-Gaming-Mini/dp/B07WGWWSWT/ [10] https://www.reddit.com/r/sysadmin/comments/64x2sb/nuc_like_system_but_with_6... [11] https://www.amazon.com/Supermicro-SuperServer-E300-8D-Mini-1U-D-1518/dp/B01M...
On Thu, Oct 3, 2019 at 1:30 PM Christian Grün <christian.gruen@gmail.com> wrote:
Exactly, it seems to be the final MERGE step during index creation that blows up your system. If you are restricted to the 2 GB of main-memory, this is what you could try next:
1. Did you already try to tweak the JVM memory limit via -Xmx? What’s the largest value that you can assign on your system?
2. If you will query only specific values of your data sets, you can restrict your indexes to specific elements or attributes; this will reduce memory consumption (see [1] for details). If you observe that no indexes will be utilized in your queries anyway, you can simply disable the text and attribute indexes, and memory usage will shrink even more.
3. Create your database on a more powerful system [2] and move it to your target machine (makes only sense if there’s no need for further updates).
4. Distribute your data across multiple databases. In some way, this is comparable to sharding; it cannot be automated, though, as the partitioning strategy depends on the characteristics of your XML input data (some people have huge standalone documents, others have millions of small documents, …).
[1] http://docs.basex.org/wiki/Indexes [2] A single CREATE call may be sufficient: CREATE DB database sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz
On Thu, Oct 3, 2019 at 8:53 AM first name last name <randomcoder1@gmail.com> wrote:
I tried again, using SPLITSIZE = 12 in the .basex config file The batch(console) script I used is attached mass-import.xq This time I didn't do the optimize or index creation post-import, but
instead, I did it as part of the import similar to what
is described in [4]. This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory." So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here?
Thanks
[4] http://docs.basex.org/wiki/Indexes#Performance
On Wed, Oct 2, 2019 at 11:06 AM first name last name < randomcoder1@gmail.com> wrote:
Hey Christian,
Thank you for your answer :) I tried setting in .basex the SPLITSIZE = 24000 but I've seen the
same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and
then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step).
To quote the statistics page from the wiki: Databases in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them.
Best regards, Stefan
On Tue, Oct 1, 2019 at 4:22 PM Christian Grün < christian.gruen@gmail.com> wrote: > > Hi first name, > > If you optimize your database, the indexes will be rebuilt. In this > step, the builder tries to guess how much free memory is still > available. If memory is exhausted, parts of the index will be split > (i. e., partially written to disk) and merged in a final step. > However, you can circumvent the heuristics by manually assigning a > static split value; see [1] for more information. If you use the DBA, > you’ll need to assign this value to your .basex or the web.xml file > [2]. In order to find the best value for your setup, it may be easier > to play around with the BaseX GUI. > > As you have already seen in our statistics, an XML document has > various properties that may represent a limit for a single database. > Accordingly, these properties make it difficult to decide for the > system when the memory will be exhausted during an import or index > rebuild. > > In general, you’ll get best performance (and your memory consumption > will be lower) if you create your database and specify the data to be > imported in a single run. This is currently not possible via the DBA; > use the GUI (Create Database) or console mode (CREATE DB command) > instead. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Options#SPLITSIZE > [2] http://docs.basex.org/wiki/Configuration > > > > On Mon, Sep 30, 2019 at 7:09 AM first name last name > <randomcoder1@gmail.com> wrote: > > > > Hi, > > > > Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. > > After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). > > I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. > > Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. > > In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". > > Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do? > > > > Thanks, > > Stefan > > > > [1] https://www.linuxquestions.org/ > > [2] http://docs.basex.org/wiki/Statistics > > [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
The stack Trace indicates that you enabled the fulltext index as well. For this index, you definitely need more memory than available on your system. So I assume you didn't encounter trouble with the default index structures? first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 20:52:
Yes, I did, with -Xmx3100m (that's the maximum amount of memory I can allocate on that system for BaseX) and I got OOM.
On Sat, Oct 5, 2019 at 2:19 AM Christian Grün <christian.gruen@gmail.com> wrote:
About option 1: How much memory have you been able to assign to the Java VM?
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 01:11:
I had another look at the script I wrote and realized that it's not working as it's supposed to. Apparently the order of operations should be this: - turn on all the types of indexes required - create the db - the parser settings and the filter settings - add all the files to the db - run "OPTIMIZE"
If I'm not doing them in this order (specifically with "OPTIMIZE" at the end) the resulting db lacks all indexes.
On Fri, Oct 4, 2019 at 11:32 PM first name last name < randomcoder1@gmail.com> wrote:
Hi Christian,
About option 4: I agree with the options you laid out. I am currently diving deeper into option 4 in the list you wrote. Regarding the partitioning strategy, I agree. I did manage however to partition the files to be imported, into separate sets, with a constraint on max partition size (on disk) and max partition file count (the number of XML documents in each partition). The tool called fpart [5] made this possible (I can imagine more sophisticated bin-packing methods, involving pre-computed node count values, and other variables, can be achieved via glpk [6] but that might be too much work). So, currently I am experimenting with a max partition size of 2.4GB and a max file count of 85k files, and fpart seems to have split the file list into 11 partitions of 33k files each and the size of a partition being ~ 2.4GB. So, I wrote a script for this, it's called sharded-import.sh and attached here. I'm also noticing that the /dba/ BaseX web interface is not blocked anymore if I run this script, as opposed to running the previous import where I run CREATE DB db_name /directory/ which allows me to see the progress or allows me to run queries before the big import finishes. Maybe the downside is that it's more verbose, and prints out a ton of lines like
ADD /share/Public/archive/tech-sites/ linuxquestions.org/threads/viewtopic_9_356613.html Resource(s) added in 47.76 ms. along the way, and maybe that's slower than before.
About option 1: Re: increase memory, I am running these experiments on a low-memory, old, network-attached storage, model QNAP TS-451+ [7] [8], which I had to take apart with a screwdriver to add 2GB of RAM (now it has 4GB of memory), and I can't seem to find around the house any additional memory sticks to take it up to 8GB (which is also the maximum memory it supports). And if I want to find like 2 x 4GB sticks of RAM, the frequency of the memory has to match what it supports, I'm having trouble finding the exact one, Corsair says it has memory sticks that would work, but I'd have to wait weeks for them to ship to Bucharest which is where I live. It seems like buying an Intel NUC that goes up to 64GB of memory would be a bit too expensive at $1639 [9] but .. people on reddit [10] were discussing some years back about this supermicro server [11] which is only $668 and would allow to add up to 64GB of memory. Basically I would buy something cheap that I can jampack with a lot of RAM, but a hands-off approach would be best here, so if it comes pre-equipped with all the memory and everything, would be nice (would spare the trouble of having to buy the memory separate, making sure it matches the motherboard specs etc).
About option 2: In fact, that's a great idea. But it would require me to write something that would figure out the XPath patterns where the actual content sits. I actually wanted to look for some algorithm that's designed to do that, and try to implement it, but I had no time. It would either have to detect the repetitive bloated nodes, and build XPaths for the rest of the nodes, where the actual content sits. I think this would be equivalent to computing the "web template" of a website, given all its pages. It would definitely decrease the size of the content that would have to be indexed. By the way, here I'm writing about a more general procedure, because it's not just this dataset that I want to import.. I want to import heavy, large amounts of data :)
These are my thoughts for now
[5] https://github.com/martymac/fpart [6] https://www.gnu.org/software/glpk/ [7] https://www.amazon.com/dp/B015VNLGF8 [8] https://www.qnap.com/en/product/ts-451+ [9] https://www.amazon.com/Intel-NUC-NUC8I7HNK-Gaming-Mini/dp/B07WGWWSWT/ [10] https://www.reddit.com/r/sysadmin/comments/64x2sb/nuc_like_system_but_with_6... [11] https://www.amazon.com/Supermicro-SuperServer-E300-8D-Mini-1U-D-1518/dp/B01M...
On Thu, Oct 3, 2019 at 1:30 PM Christian Grün < christian.gruen@gmail.com> wrote:
Exactly, it seems to be the final MERGE step during index creation that blows up your system. If you are restricted to the 2 GB of main-memory, this is what you could try next:
1. Did you already try to tweak the JVM memory limit via -Xmx? What’s the largest value that you can assign on your system?
2. If you will query only specific values of your data sets, you can restrict your indexes to specific elements or attributes; this will reduce memory consumption (see [1] for details). If you observe that no indexes will be utilized in your queries anyway, you can simply disable the text and attribute indexes, and memory usage will shrink even more.
3. Create your database on a more powerful system [2] and move it to your target machine (makes only sense if there’s no need for further updates).
4. Distribute your data across multiple databases. In some way, this is comparable to sharding; it cannot be automated, though, as the partitioning strategy depends on the characteristics of your XML input data (some people have huge standalone documents, others have millions of small documents, …).
[1] http://docs.basex.org/wiki/Indexes [2] A single CREATE call may be sufficient: CREATE DB database sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz
On Thu, Oct 3, 2019 at 8:53 AM first name last name <randomcoder1@gmail.com> wrote:
I tried again, using SPLITSIZE = 12 in the .basex config file The batch(console) script I used is attached mass-import.xq This time I didn't do the optimize or index creation post-import,
but instead, I did it as part of the import similar to what
is described in [4]. This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory." So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here?
Thanks
[4] http://docs.basex.org/wiki/Indexes#Performance
On Wed, Oct 2, 2019 at 11:06 AM first name last name < randomcoder1@gmail.com> wrote: > > Hey Christian, > > Thank you for your answer :) > I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and > then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. > Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step). > > To quote the statistics page from the wiki: > Databases in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. > This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this > I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them. > > Best regards, > Stefan > > > On Tue, Oct 1, 2019 at 4:22 PM Christian Grün < christian.gruen@gmail.com> wrote: >> >> Hi first name, >> >> If you optimize your database, the indexes will be rebuilt. In this >> step, the builder tries to guess how much free memory is still >> available. If memory is exhausted, parts of the index will be split >> (i. e., partially written to disk) and merged in a final step. >> However, you can circumvent the heuristics by manually assigning a >> static split value; see [1] for more information. If you use the DBA, >> you’ll need to assign this value to your .basex or the web.xml file >> [2]. In order to find the best value for your setup, it may be easier >> to play around with the BaseX GUI. >> >> As you have already seen in our statistics, an XML document has >> various properties that may represent a limit for a single database. >> Accordingly, these properties make it difficult to decide for the >> system when the memory will be exhausted during an import or index >> rebuild. >> >> In general, you’ll get best performance (and your memory consumption >> will be lower) if you create your database and specify the data to be >> imported in a single run. This is currently not possible via the DBA; >> use the GUI (Create Database) or console mode (CREATE DB command) >> instead. >> >> Hope this helps, >> Christian >> >> [1] http://docs.basex.org/wiki/Options#SPLITSIZE >> [2] http://docs.basex.org/wiki/Configuration >> >> >> >> On Mon, Sep 30, 2019 at 7:09 AM first name last name >> <randomcoder1@gmail.com> wrote: >> > >> > Hi, >> > >> > Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. >> > After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). >> > I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. >> > Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. >> > In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". >> > Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do? >> > >> > Thanks, >> > Stefan >> > >> > [1] https://www.linuxquestions.org/ >> > [2] http://docs.basex.org/wiki/Statistics >> > [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
Attached a more complete output of ./bin/basexhttp . Judging from this output, it would seem that everything was ok, except for the full-text index. I now realize that I have another question about full-text indexes. It seems like the full-text index here is dependent on the amount of memory available (in other words, the more data to be indexed, the more RAM memory required). I was using a certain popular RDBMS, for full-text indexing, and I never bumped into problems like it running out of memory when building such indexes. I think their model uses a certain buffer in memory, and it keeps multiple files on disk where it store data, and then it assembles together the results in-memory but always keeping the constraint of using only as much memory as was declared to be allowed for it to use. Perhaps the topic would be "external memory algorithms" or "full-text search using secondary storage". I'm not an expert in this field, but.. my question here would be if this kind of thing is something that BaseX is looking to handle in the future? Thanks, Stefan On Sat, Oct 5, 2019 at 11:08 PM Christian Grün <christian.gruen@gmail.com> wrote:
The stack Trace indicates that you enabled the fulltext index as well. For this index, you definitely need more memory than available on your system.
So I assume you didn't encounter trouble with the default index structures?
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 20:52:
Yes, I did, with -Xmx3100m (that's the maximum amount of memory I can allocate on that system for BaseX) and I got OOM.
On Sat, Oct 5, 2019 at 2:19 AM Christian Grün <christian.gruen@gmail.com> wrote:
About option 1: How much memory have you been able to assign to the Java VM?
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 01:11:
I had another look at the script I wrote and realized that it's not working as it's supposed to. Apparently the order of operations should be this: - turn on all the types of indexes required - create the db - the parser settings and the filter settings - add all the files to the db - run "OPTIMIZE"
If I'm not doing them in this order (specifically with "OPTIMIZE" at the end) the resulting db lacks all indexes.
On Fri, Oct 4, 2019 at 11:32 PM first name last name < randomcoder1@gmail.com> wrote:
Hi Christian,
About option 4: I agree with the options you laid out. I am currently diving deeper into option 4 in the list you wrote. Regarding the partitioning strategy, I agree. I did manage however to partition the files to be imported, into separate sets, with a constraint on max partition size (on disk) and max partition file count (the number of XML documents in each partition). The tool called fpart [5] made this possible (I can imagine more sophisticated bin-packing methods, involving pre-computed node count values, and other variables, can be achieved via glpk [6] but that might be too much work). So, currently I am experimenting with a max partition size of 2.4GB and a max file count of 85k files, and fpart seems to have split the file list into 11 partitions of 33k files each and the size of a partition being ~ 2.4GB. So, I wrote a script for this, it's called sharded-import.sh and attached here. I'm also noticing that the /dba/ BaseX web interface is not blocked anymore if I run this script, as opposed to running the previous import where I run CREATE DB db_name /directory/ which allows me to see the progress or allows me to run queries before the big import finishes. Maybe the downside is that it's more verbose, and prints out a ton of lines like
ADD /share/Public/archive/tech-sites/ linuxquestions.org/threads/viewtopic_9_356613.html Resource(s) added in 47.76 ms. along the way, and maybe that's slower than before.
About option 1: Re: increase memory, I am running these experiments on a low-memory, old, network-attached storage, model QNAP TS-451+ [7] [8], which I had to take apart with a screwdriver to add 2GB of RAM (now it has 4GB of memory), and I can't seem to find around the house any additional memory sticks to take it up to 8GB (which is also the maximum memory it supports). And if I want to find like 2 x 4GB sticks of RAM, the frequency of the memory has to match what it supports, I'm having trouble finding the exact one, Corsair says it has memory sticks that would work, but I'd have to wait weeks for them to ship to Bucharest which is where I live. It seems like buying an Intel NUC that goes up to 64GB of memory would be a bit too expensive at $1639 [9] but .. people on reddit [10] were discussing some years back about this supermicro server [11] which is only $668 and would allow to add up to 64GB of memory. Basically I would buy something cheap that I can jampack with a lot of RAM, but a hands-off approach would be best here, so if it comes pre-equipped with all the memory and everything, would be nice (would spare the trouble of having to buy the memory separate, making sure it matches the motherboard specs etc).
About option 2: In fact, that's a great idea. But it would require me to write something that would figure out the XPath patterns where the actual content sits. I actually wanted to look for some algorithm that's designed to do that, and try to implement it, but I had no time. It would either have to detect the repetitive bloated nodes, and build XPaths for the rest of the nodes, where the actual content sits. I think this would be equivalent to computing the "web template" of a website, given all its pages. It would definitely decrease the size of the content that would have to be indexed. By the way, here I'm writing about a more general procedure, because it's not just this dataset that I want to import.. I want to import heavy, large amounts of data :)
These are my thoughts for now
[5] https://github.com/martymac/fpart [6] https://www.gnu.org/software/glpk/ [7] https://www.amazon.com/dp/B015VNLGF8 [8] https://www.qnap.com/en/product/ts-451+ [9] https://www.amazon.com/Intel-NUC-NUC8I7HNK-Gaming-Mini/dp/B07WGWWSWT/ [10] https://www.reddit.com/r/sysadmin/comments/64x2sb/nuc_like_system_but_with_6... [11] https://www.amazon.com/Supermicro-SuperServer-E300-8D-Mini-1U-D-1518/dp/B01M...
On Thu, Oct 3, 2019 at 1:30 PM Christian Grün < christian.gruen@gmail.com> wrote:
Exactly, it seems to be the final MERGE step during index creation that blows up your system. If you are restricted to the 2 GB of main-memory, this is what you could try next:
1. Did you already try to tweak the JVM memory limit via -Xmx? What’s the largest value that you can assign on your system?
2. If you will query only specific values of your data sets, you can restrict your indexes to specific elements or attributes; this will reduce memory consumption (see [1] for details). If you observe that no indexes will be utilized in your queries anyway, you can simply disable the text and attribute indexes, and memory usage will shrink even more.
3. Create your database on a more powerful system [2] and move it to your target machine (makes only sense if there’s no need for further updates).
4. Distribute your data across multiple databases. In some way, this is comparable to sharding; it cannot be automated, though, as the partitioning strategy depends on the characteristics of your XML input data (some people have huge standalone documents, others have millions of small documents, …).
[1] http://docs.basex.org/wiki/Indexes [2] A single CREATE call may be sufficient: CREATE DB database sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz
On Thu, Oct 3, 2019 at 8:53 AM first name last name <randomcoder1@gmail.com> wrote: > > I tried again, using SPLITSIZE = 12 in the .basex config file > The batch(console) script I used is attached mass-import.xq > This time I didn't do the optimize or index creation post-import, but instead, I did it as part of the import similar to what > is described in [4]. > This time I got a different error, that is, "org.basex.core.BaseXException: Out of Main Memory." > So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any difference here? > > Thanks > > [4] http://docs.basex.org/wiki/Indexes#Performance > > > On Wed, Oct 2, 2019 at 11:06 AM first name last name < randomcoder1@gmail.com> wrote: >> >> Hey Christian, >> >> Thank you for your answer :) >> I tried setting in .basex the SPLITSIZE = 24000 but I've seen the same OOM behavior. It looks like the memory consumption is moderate until when it reaches about 30GB (the size of the db before optimize) and >> then memory consumption spikes, and OOM occurs. Now I'm trying with SPLITSIZE = 1000 and will report back if I get OOM again. >> Regarding what you said, it might be that the merge step is where the OOM occurs (I wonder if there's any way to control how much memory is being used inside the merge step). >> >> To quote the statistics page from the wiki: >> Databases in BaseX are light-weight. If a database limit is reached, you can distribute your documents across multiple database instances and access all of them with a single XQuery expression. >> This to me sounds like sharding. I would probably be able to split the documents into chunks and upload them under a db with the same prefix, but varying suffix.. seems a lot like shards. By doing this >> I think I can avoid OOM, but if BaseX provides other, better, maybe native mechanisms of avoiding OOM, I would try them. >> >> Best regards, >> Stefan >> >> >> On Tue, Oct 1, 2019 at 4:22 PM Christian Grün < christian.gruen@gmail.com> wrote: >>> >>> Hi first name, >>> >>> If you optimize your database, the indexes will be rebuilt. In this >>> step, the builder tries to guess how much free memory is still >>> available. If memory is exhausted, parts of the index will be split >>> (i. e., partially written to disk) and merged in a final step. >>> However, you can circumvent the heuristics by manually assigning a >>> static split value; see [1] for more information. If you use the DBA, >>> you’ll need to assign this value to your .basex or the web.xml file >>> [2]. In order to find the best value for your setup, it may be easier >>> to play around with the BaseX GUI. >>> >>> As you have already seen in our statistics, an XML document has >>> various properties that may represent a limit for a single database. >>> Accordingly, these properties make it difficult to decide for the >>> system when the memory will be exhausted during an import or index >>> rebuild. >>> >>> In general, you’ll get best performance (and your memory consumption >>> will be lower) if you create your database and specify the data to be >>> imported in a single run. This is currently not possible via the DBA; >>> use the GUI (Create Database) or console mode (CREATE DB command) >>> instead. >>> >>> Hope this helps, >>> Christian >>> >>> [1] http://docs.basex.org/wiki/Options#SPLITSIZE >>> [2] http://docs.basex.org/wiki/Configuration >>> >>> >>> >>> On Mon, Sep 30, 2019 at 7:09 AM first name last name >>> <randomcoder1@gmail.com> wrote: >>> > >>> > Hi, >>> > >>> > Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. >>> > After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). >>> > I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. >>> > Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. >>> > In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". >>> > Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do? >>> > >>> > Thanks, >>> > Stefan >>> > >>> > [1] https://www.linuxquestions.org/ >>> > [2] http://docs.basex.org/wiki/Statistics >>> > [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2
The current full text index builder provides a similar outsourcing mechanism to that of the index builder for the default index structures; but the meta data structures are kept in main-memory, and they are more bulky. There are definitely ways to tackle this technically; it hasn't been of high priority so far, but this may change. Please note that you won't create an index over your whole data set in RDBMS. Instead, you'll usually create it for specific fields that you will query later on. It's a convenience feature in BaseX that you can build an index for all of your data. For large full-text corpora, however, it's recommendable in most cases to restrict indexing to the relevant XML elements. first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 23:28:
Attached a more complete output of ./bin/basexhttp . Judging from this output, it would seem that everything was ok, except for the full-text index. I now realize that I have another question about full-text indexes. It seems like the full-text index here is dependent on the amount of memory available (in other words, the more data to be indexed, the more RAM memory required).
I was using a certain popular RDBMS, for full-text indexing, and I never bumped into problems like it running out of memory when building such indexes. I think their model uses a certain buffer in memory, and it keeps multiple files on disk where it store data, and then it assembles together the results in-memory but always keeping the constraint of using only as much memory as was declared to be allowed for it to use. Perhaps the topic would be "external memory algorithms" or "full-text search using secondary storage". I'm not an expert in this field, but.. my question here would be if this kind of thing is something that BaseX is looking to handle in the future?
Thanks, Stefan
On Sat, Oct 5, 2019 at 11:08 PM Christian Grün <christian.gruen@gmail.com> wrote:
The stack Trace indicates that you enabled the fulltext index as well. For this index, you definitely need more memory than available on your system.
So I assume you didn't encounter trouble with the default index structures?
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 20:52:
Yes, I did, with -Xmx3100m (that's the maximum amount of memory I can allocate on that system for BaseX) and I got OOM.
On Sat, Oct 5, 2019 at 2:19 AM Christian Grün <christian.gruen@gmail.com> wrote:
About option 1: How much memory have you been able to assign to the Java VM?
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 01:11:
I had another look at the script I wrote and realized that it's not working as it's supposed to. Apparently the order of operations should be this: - turn on all the types of indexes required - create the db - the parser settings and the filter settings - add all the files to the db - run "OPTIMIZE"
If I'm not doing them in this order (specifically with "OPTIMIZE" at the end) the resulting db lacks all indexes.
On Fri, Oct 4, 2019 at 11:32 PM first name last name < randomcoder1@gmail.com> wrote:
Hi Christian,
About option 4: I agree with the options you laid out. I am currently diving deeper into option 4 in the list you wrote. Regarding the partitioning strategy, I agree. I did manage however to partition the files to be imported, into separate sets, with a constraint on max partition size (on disk) and max partition file count (the number of XML documents in each partition). The tool called fpart [5] made this possible (I can imagine more sophisticated bin-packing methods, involving pre-computed node count values, and other variables, can be achieved via glpk [6] but that might be too much work). So, currently I am experimenting with a max partition size of 2.4GB and a max file count of 85k files, and fpart seems to have split the file list into 11 partitions of 33k files each and the size of a partition being ~ 2.4GB. So, I wrote a script for this, it's called sharded-import.sh and attached here. I'm also noticing that the /dba/ BaseX web interface is not blocked anymore if I run this script, as opposed to running the previous import where I run CREATE DB db_name /directory/ which allows me to see the progress or allows me to run queries before the big import finishes. Maybe the downside is that it's more verbose, and prints out a ton of lines like > ADD /share/Public/archive/tech-sites/ linuxquestions.org/threads/viewtopic_9_356613.html Resource(s) added in 47.76 ms. along the way, and maybe that's slower than before.
About option 1: Re: increase memory, I am running these experiments on a low-memory, old, network-attached storage, model QNAP TS-451+ [7] [8], which I had to take apart with a screwdriver to add 2GB of RAM (now it has 4GB of memory), and I can't seem to find around the house any additional memory sticks to take it up to 8GB (which is also the maximum memory it supports). And if I want to find like 2 x 4GB sticks of RAM, the frequency of the memory has to match what it supports, I'm having trouble finding the exact one, Corsair says it has memory sticks that would work, but I'd have to wait weeks for them to ship to Bucharest which is where I live. It seems like buying an Intel NUC that goes up to 64GB of memory would be a bit too expensive at $1639 [9] but .. people on reddit [10] were discussing some years back about this supermicro server [11] which is only $668 and would allow to add up to 64GB of memory. Basically I would buy something cheap that I can jampack with a lot of RAM, but a hands-off approach would be best here, so if it comes pre-equipped with all the memory and everything, would be nice (would spare the trouble of having to buy the memory separate, making sure it matches the motherboard specs etc).
About option 2: In fact, that's a great idea. But it would require me to write something that would figure out the XPath patterns where the actual content sits. I actually wanted to look for some algorithm that's designed to do that, and try to implement it, but I had no time. It would either have to detect the repetitive bloated nodes, and build XPaths for the rest of the nodes, where the actual content sits. I think this would be equivalent to computing the "web template" of a website, given all its pages. It would definitely decrease the size of the content that would have to be indexed. By the way, here I'm writing about a more general procedure, because it's not just this dataset that I want to import.. I want to import heavy, large amounts of data :)
These are my thoughts for now
[5] https://github.com/martymac/fpart [6] https://www.gnu.org/software/glpk/ [7] https://www.amazon.com/dp/B015VNLGF8 [8] https://www.qnap.com/en/product/ts-451+ [9] https://www.amazon.com/Intel-NUC-NUC8I7HNK-Gaming-Mini/dp/B07WGWWSWT/ [10] https://www.reddit.com/r/sysadmin/comments/64x2sb/nuc_like_system_but_with_6... [11] https://www.amazon.com/Supermicro-SuperServer-E300-8D-Mini-1U-D-1518/dp/B01M...
On Thu, Oct 3, 2019 at 1:30 PM Christian Grün < christian.gruen@gmail.com> wrote:
> Exactly, it seems to be the final MERGE step during index creation > that blows up your system. If you are restricted to the 2 GB of > main-memory, this is what you could try next: > > 1. Did you already try to tweak the JVM memory limit via -Xmx? What’s > the largest value that you can assign on your system? > > 2. If you will query only specific values of your data sets, you can > restrict your indexes to specific elements or attributes; this will > reduce memory consumption (see [1] for details). If you observe that > no indexes will be utilized in your queries anyway, you can simply > disable the text and attribute indexes, and memory usage will shrink > even more. > > 3. Create your database on a more powerful system [2] and move it to > your target machine (makes only sense if there’s no need for further > updates). > > 4. Distribute your data across multiple databases. In some way, this > is comparable to sharding; it cannot be automated, though, as the > partitioning strategy depends on the characteristics of your XML > input > data (some people have huge standalone documents, others have > millions > of small documents, …). > > [1] http://docs.basex.org/wiki/Indexes > [2] A single CREATE call may be sufficient: CREATE DB database > sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz > > > > > On Thu, Oct 3, 2019 at 8:53 AM first name last name > <randomcoder1@gmail.com> wrote: > > > > I tried again, using SPLITSIZE = 12 in the .basex config file > > The batch(console) script I used is attached mass-import.xq > > This time I didn't do the optimize or index creation post-import, > but instead, I did it as part of the import similar to what > > is described in [4]. > > This time I got a different error, that is, > "org.basex.core.BaseXException: Out of Main Memory." > > So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make any > difference here? > > > > Thanks > > > > [4] http://docs.basex.org/wiki/Indexes#Performance > > > > > > On Wed, Oct 2, 2019 at 11:06 AM first name last name < > randomcoder1@gmail.com> wrote: > >> > >> Hey Christian, > >> > >> Thank you for your answer :) > >> I tried setting in .basex the SPLITSIZE = 24000 but I've seen the > same OOM behavior. It looks like the memory consumption is moderate until > when it reaches about 30GB (the size of the db before optimize) and > >> then memory consumption spikes, and OOM occurs. Now I'm trying > with SPLITSIZE = 1000 and will report back if I get OOM again. > >> Regarding what you said, it might be that the merge step is where > the OOM occurs (I wonder if there's any way to control how much memory is > being used inside the merge step). > >> > >> To quote the statistics page from the wiki: > >> Databases in BaseX are light-weight. If a database limit is > reached, you can distribute your documents across multiple database > instances and access all of them with a single XQuery expression. > >> This to me sounds like sharding. I would probably be able to > split the documents into chunks and upload them under a db with the same > prefix, but varying suffix.. seems a lot like shards. By doing this > >> I think I can avoid OOM, but if BaseX provides other, better, > maybe native mechanisms of avoiding OOM, I would try them. > >> > >> Best regards, > >> Stefan > >> > >> > >> On Tue, Oct 1, 2019 at 4:22 PM Christian Grün < > christian.gruen@gmail.com> wrote: > >>> > >>> Hi first name, > >>> > >>> If you optimize your database, the indexes will be rebuilt. In > this > >>> step, the builder tries to guess how much free memory is still > >>> available. If memory is exhausted, parts of the index will be > split > >>> (i. e., partially written to disk) and merged in a final step. > >>> However, you can circumvent the heuristics by manually assigning > a > >>> static split value; see [1] for more information. If you use the > DBA, > >>> you’ll need to assign this value to your .basex or the web.xml > file > >>> [2]. In order to find the best value for your setup, it may be > easier > >>> to play around with the BaseX GUI. > >>> > >>> As you have already seen in our statistics, an XML document has > >>> various properties that may represent a limit for a single > database. > >>> Accordingly, these properties make it difficult to decide for the > >>> system when the memory will be exhausted during an import or > index > >>> rebuild. > >>> > >>> In general, you’ll get best performance (and your memory > consumption > >>> will be lower) if you create your database and specify the data > to be > >>> imported in a single run. This is currently not possible via the > DBA; > >>> use the GUI (Create Database) or console mode (CREATE DB command) > >>> instead. > >>> > >>> Hope this helps, > >>> Christian > >>> > >>> [1] http://docs.basex.org/wiki/Options#SPLITSIZE > >>> [2] http://docs.basex.org/wiki/Configuration > >>> > >>> > >>> > >>> On Mon, Sep 30, 2019 at 7:09 AM first name last name > >>> <randomcoder1@gmail.com> wrote: > >>> > > >>> > Hi, > >>> > > >>> > Let's say there's a 30GB dataset [3] containing most > threads/posts from [1]. > >>> > After importing all of it, when I try to run /dba/db-optimize/ > on it (which must have some corresponding command) I get the OOM error in > the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of > memory (the machine I'm running this on doesn't have a lot of memory). > >>> > I was looking at [2] for some estimates of peak memory usage > for this "db-optimize" operation, but couldn't find any. > >>> > Actually it would be nice to know peak memory usage because.. > of course, for any database (including BaseX) a common operation is to do > server sizing, to know what kind of server would be needed. > >>> > In this case, it seems like 2GB memory is enough to import > 340k documents, weighing in at 30GB total, but it's not enough to run > "dba-optimize". > >>> > Is there any info about peak memory usage on [2] ? And are > there guidelines for large-scale collection imports like I'm trying to do? > >>> > > >>> > Thanks, > >>> > Stefan > >>> > > >>> > [1] https://www.linuxquestions.org/ > >>> > [2] http://docs.basex.org/wiki/Statistics > >>> > [3] > https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2 >
Regarding selective full-text indexing, I just tried XQUERY db:optimize("linuxquestions.org-selective", true(), map { 'ftindex': true(), 'ftinclude': 'div table td a' }) And I got OOM on that, the exact stacktrace attached in this message. I will open a separate thread regarding migrating the data from BaseX shards to PostgreSQL (for the purpose of full-text indexing). On Sun, Oct 6, 2019 at 10:19 AM Christian Grün <christian.gruen@gmail.com> wrote:
The current full text index builder provides a similar outsourcing mechanism to that of the index builder for the default index structures; but the meta data structures are kept in main-memory, and they are more bulky. There are definitely ways to tackle this technically; it hasn't been of high priority so far, but this may change.
Please note that you won't create an index over your whole data set in RDBMS. Instead, you'll usually create it for specific fields that you will query later on. It's a convenience feature in BaseX that you can build an index for all of your data. For large full-text corpora, however, it's recommendable in most cases to restrict indexing to the relevant XML elements.
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 23:28:
Attached a more complete output of ./bin/basexhttp . Judging from this output, it would seem that everything was ok, except for the full-text index. I now realize that I have another question about full-text indexes. It seems like the full-text index here is dependent on the amount of memory available (in other words, the more data to be indexed, the more RAM memory required).
I was using a certain popular RDBMS, for full-text indexing, and I never bumped into problems like it running out of memory when building such indexes. I think their model uses a certain buffer in memory, and it keeps multiple files on disk where it store data, and then it assembles together the results in-memory but always keeping the constraint of using only as much memory as was declared to be allowed for it to use. Perhaps the topic would be "external memory algorithms" or "full-text search using secondary storage". I'm not an expert in this field, but.. my question here would be if this kind of thing is something that BaseX is looking to handle in the future?
Thanks, Stefan
On Sat, Oct 5, 2019 at 11:08 PM Christian Grün <christian.gruen@gmail.com> wrote:
The stack Trace indicates that you enabled the fulltext index as well. For this index, you definitely need more memory than available on your system.
So I assume you didn't encounter trouble with the default index structures?
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 20:52:
Yes, I did, with -Xmx3100m (that's the maximum amount of memory I can allocate on that system for BaseX) and I got OOM.
On Sat, Oct 5, 2019 at 2:19 AM Christian Grün < christian.gruen@gmail.com> wrote:
About option 1: How much memory have you been able to assign to the Java VM?
first name last name <randomcoder1@gmail.com> schrieb am Sa., 5. Okt. 2019, 01:11:
I had another look at the script I wrote and realized that it's not working as it's supposed to. Apparently the order of operations should be this: - turn on all the types of indexes required - create the db - the parser settings and the filter settings - add all the files to the db - run "OPTIMIZE"
If I'm not doing them in this order (specifically with "OPTIMIZE" at the end) the resulting db lacks all indexes.
On Fri, Oct 4, 2019 at 11:32 PM first name last name < randomcoder1@gmail.com> wrote:
> Hi Christian, > > About option 4: > I agree with the options you laid out. I am currently diving deeper > into option 4 in the list you wrote. > Regarding the partitioning strategy, I agree. I did manage however > to partition the files to be imported, into separate sets, with a > constraint on max partition size (on disk) and max partition file count > (the number of XML documents in each partition). > The tool called fpart [5] made this possible (I can imagine more > sophisticated bin-packing methods, involving pre-computed node count > values, and other variables, can be achieved via glpk [6] but that might be > too much work). > So, currently I am experimenting with a max partition size of 2.4GB > and a max file count of 85k files, and fpart seems to have split the file > list into 11 partitions of 33k files each and the size of a partition being > ~ 2.4GB. > So, I wrote a script for this, it's called sharded-import.sh and > attached here. I'm also noticing that the /dba/ BaseX web interface is not > blocked anymore if I run this script, as opposed to running the previous > import where I run > CREATE DB db_name /directory/ > which allows me to see the progress or allows me to run queries > before the big import finishes. > Maybe the downside is that it's more verbose, and prints out a ton > of lines like > > ADD /share/Public/archive/tech-sites/ > linuxquestions.org/threads/viewtopic_9_356613.html > Resource(s) added in 47.76 ms. > along the way, and maybe that's slower than before. > > About option 1: > Re: increase memory, I am running these experiments on a low-memory, > old, network-attached storage, model QNAP TS-451+ [7] [8], which I had to > take apart with a screwdriver to add 2GB of RAM (now it has 4GB of memory), > and I can't seem to find around the house any additional memory sticks to > take it up to 8GB (which is also the maximum memory it supports). And if I > want to find like 2 x 4GB sticks of RAM, the frequency of the memory has to > match what it supports, I'm having trouble finding the exact one, Corsair > says it has memory sticks that would work, but I'd have to wait weeks for > them to ship to Bucharest which is where I live. > It seems like buying an Intel NUC that goes up to 64GB of memory > would be a bit too expensive at $1639 [9] but .. people on reddit [10] were > discussing some years back about this supermicro server [11] which is only > $668 and would allow to add up to 64GB of memory. > Basically I would buy something cheap that I can jampack with a lot > of RAM, but a hands-off approach would be best here, so if it comes > pre-equipped with all the memory and everything, would be nice (would spare > the trouble of having to buy the memory separate, making sure it matches > the motherboard specs etc). > > About option 2: > In fact, that's a great idea. But it would require me to write > something that would figure out the XPath patterns where the actual content > sits. I actually wanted to look for some algorithm that's designed to do > that, and try to implement it, but I had no time. > It would either have to detect the repetitive bloated nodes, and > build XPaths for the rest of the nodes, where the actual content sits. I > think this would be equivalent to computing the "web template" of a > website, given all its pages. > It would definitely decrease the size of the content that would have > to be indexed. > By the way, here I'm writing about a more general procedure, because > it's not just this dataset that I want to import.. I want to import heavy, > large amounts of data :) > > These are my thoughts for now > > [5] https://github.com/martymac/fpart > [6] https://www.gnu.org/software/glpk/ > [7] https://www.amazon.com/dp/B015VNLGF8 > [8] https://www.qnap.com/en/product/ts-451+ > [9] > https://www.amazon.com/Intel-NUC-NUC8I7HNK-Gaming-Mini/dp/B07WGWWSWT/ > [10] > https://www.reddit.com/r/sysadmin/comments/64x2sb/nuc_like_system_but_with_6... > [11] > https://www.amazon.com/Supermicro-SuperServer-E300-8D-Mini-1U-D-1518/dp/B01M... > > > On Thu, Oct 3, 2019 at 1:30 PM Christian Grün < > christian.gruen@gmail.com> wrote: > >> Exactly, it seems to be the final MERGE step during index creation >> that blows up your system. If you are restricted to the 2 GB of >> main-memory, this is what you could try next: >> >> 1. Did you already try to tweak the JVM memory limit via -Xmx? >> What’s >> the largest value that you can assign on your system? >> >> 2. If you will query only specific values of your data sets, you can >> restrict your indexes to specific elements or attributes; this will >> reduce memory consumption (see [1] for details). If you observe that >> no indexes will be utilized in your queries anyway, you can simply >> disable the text and attribute indexes, and memory usage will shrink >> even more. >> >> 3. Create your database on a more powerful system [2] and move it to >> your target machine (makes only sense if there’s no need for further >> updates). >> >> 4. Distribute your data across multiple databases. In some way, this >> is comparable to sharding; it cannot be automated, though, as the >> partitioning strategy depends on the characteristics of your XML >> input >> data (some people have huge standalone documents, others have >> millions >> of small documents, …). >> >> [1] http://docs.basex.org/wiki/Indexes >> [2] A single CREATE call may be sufficient: CREATE DB database >> sample-data-for-basex-mailing-list-linuxquestions.org.tar.gz >> >> >> >> >> On Thu, Oct 3, 2019 at 8:53 AM first name last name >> <randomcoder1@gmail.com> wrote: >> > >> > I tried again, using SPLITSIZE = 12 in the .basex config file >> > The batch(console) script I used is attached mass-import.xq >> > This time I didn't do the optimize or index creation post-import, >> but instead, I did it as part of the import similar to what >> > is described in [4]. >> > This time I got a different error, that is, >> "org.basex.core.BaseXException: Out of Main Memory." >> > So right now.. I'm a bit out of ideas. Would AUTOOPTIMIZE make >> any difference here? >> > >> > Thanks >> > >> > [4] http://docs.basex.org/wiki/Indexes#Performance >> > >> > >> > On Wed, Oct 2, 2019 at 11:06 AM first name last name < >> randomcoder1@gmail.com> wrote: >> >> >> >> Hey Christian, >> >> >> >> Thank you for your answer :) >> >> I tried setting in .basex the SPLITSIZE = 24000 but I've seen >> the same OOM behavior. It looks like the memory consumption is moderate >> until when it reaches about 30GB (the size of the db before optimize) and >> >> then memory consumption spikes, and OOM occurs. Now I'm trying >> with SPLITSIZE = 1000 and will report back if I get OOM again. >> >> Regarding what you said, it might be that the merge step is >> where the OOM occurs (I wonder if there's any way to control how much >> memory is being used inside the merge step). >> >> >> >> To quote the statistics page from the wiki: >> >> Databases in BaseX are light-weight. If a database limit is >> reached, you can distribute your documents across multiple database >> instances and access all of them with a single XQuery expression. >> >> This to me sounds like sharding. I would probably be able to >> split the documents into chunks and upload them under a db with the same >> prefix, but varying suffix.. seems a lot like shards. By doing this >> >> I think I can avoid OOM, but if BaseX provides other, better, >> maybe native mechanisms of avoiding OOM, I would try them. >> >> >> >> Best regards, >> >> Stefan >> >> >> >> >> >> On Tue, Oct 1, 2019 at 4:22 PM Christian Grün < >> christian.gruen@gmail.com> wrote: >> >>> >> >>> Hi first name, >> >>> >> >>> If you optimize your database, the indexes will be rebuilt. In >> this >> >>> step, the builder tries to guess how much free memory is still >> >>> available. If memory is exhausted, parts of the index will be >> split >> >>> (i. e., partially written to disk) and merged in a final step. >> >>> However, you can circumvent the heuristics by manually >> assigning a >> >>> static split value; see [1] for more information. If you use >> the DBA, >> >>> you’ll need to assign this value to your .basex or the web.xml >> file >> >>> [2]. In order to find the best value for your setup, it may be >> easier >> >>> to play around with the BaseX GUI. >> >>> >> >>> As you have already seen in our statistics, an XML document has >> >>> various properties that may represent a limit for a single >> database. >> >>> Accordingly, these properties make it difficult to decide for >> the >> >>> system when the memory will be exhausted during an import or >> index >> >>> rebuild. >> >>> >> >>> In general, you’ll get best performance (and your memory >> consumption >> >>> will be lower) if you create your database and specify the data >> to be >> >>> imported in a single run. This is currently not possible via >> the DBA; >> >>> use the GUI (Create Database) or console mode (CREATE DB >> command) >> >>> instead. >> >>> >> >>> Hope this helps, >> >>> Christian >> >>> >> >>> [1] http://docs.basex.org/wiki/Options#SPLITSIZE >> >>> [2] http://docs.basex.org/wiki/Configuration >> >>> >> >>> >> >>> >> >>> On Mon, Sep 30, 2019 at 7:09 AM first name last name >> >>> <randomcoder1@gmail.com> wrote: >> >>> > >> >>> > Hi, >> >>> > >> >>> > Let's say there's a 30GB dataset [3] containing most >> threads/posts from [1]. >> >>> > After importing all of it, when I try to run >> /dba/db-optimize/ on it (which must have some corresponding command) I get >> the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is >> limited to 2GB of memory (the machine I'm running this on doesn't have a >> lot of memory). >> >>> > I was looking at [2] for some estimates of peak memory usage >> for this "db-optimize" operation, but couldn't find any. >> >>> > Actually it would be nice to know peak memory usage because.. >> of course, for any database (including BaseX) a common operation is to do >> server sizing, to know what kind of server would be needed. >> >>> > In this case, it seems like 2GB memory is enough to import >> 340k documents, weighing in at 30GB total, but it's not enough to run >> "dba-optimize". >> >>> > Is there any info about peak memory usage on [2] ? And are >> there guidelines for large-scale collection imports like I'm trying to do? >> >>> > >> >>> > Thanks, >> >>> > Stefan >> >>> > >> >>> > [1] https://www.linuxquestions.org/ >> >>> > [2] http://docs.basex.org/wiki/Statistics >> >>> > [3] >> https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2 >> >
participants (3)
-
Christian Grün -
first name last name -
Imsieke, Gerrit, le-tex