Is it possible to limit a BaseX server maximum memory consumption?
Hello. Is it possible to hardly limit a maximum memory consumption of a BaseX server instance process (something like a cache_size option in MongoDB) avoiding getting Java heap space errors simultaneously? Or, in the current implementation, it's required for the process to be able to allocate memory enough for storing a whole data set of the currently accessed database? Thanks.
Hi Jacob, The memory limit of the BaseX server process can be limited via the JVM -Xmx option in the BaseX startup scripts. If the maximum amount of assigned memory is exceeded, heap space errors will be raised. I am not sure, however, if this is what you are asking for, because you were looking for an option that does not raise heap space errors, right? If you use xquery:eval, you can specify a maximum memory limit for a single client [1]. This function is e.g. used in our DBA in order to avoid that queries sent from the client will compromise the BaseX server. As all threads run in the same JVM, it’s hard to guess how much memory is really spent by a single process; but at least it works to kill real memory killers. I haven’t found much documentation on the MongoDB cache_size option. Could you possible give us some more infos what it does, and what happens if a process requires more than the assigned memory? Best, Christian [1] http://docs.basex.org/wiki/XQuery_Module#xquery:eval On Fri, Apr 7, 2017 at 4:02 AM, Jacob Borisov <jacobborisov@gmail.com> wrote:
Hello. Is it possible to hardly limit a maximum memory consumption of a BaseX server instance process (something like a cache_size option in MongoDB) avoiding getting Java heap space errors simultaneously? Or, in the current implementation, it's required for the process to be able to allocate memory enough for storing a whole data set of the currently accessed database? Thanks.
Hello, Christian. As far as I know, mainstream DBMSs (MySQL, MongoDB, etc.) have the caching mechanism: if some data were obtained from a disk during query execution, DBMS put such data and/or the query results into a memory for speeding up following queries. Configuration variables like cache_size restrict the amount of a maximum memory allocated for such purpose. So, if cache_size is exceeded and there is a demand to put more data into the cache, DBMS moves out least used data from the memory to be able to put new data into it. In other words, it's something like a disk cache in the Linux Kernel. Particularly my problem is that a BaseX process allocated memory grows gradually and finally Java heap space errors are raised, and I were guess the reason of this behavior is a cache growing. So does the BaseX have a mechanism like one described above or its architecture is completely different and a Java heap space error means exactly that my performed queries require more and more memory for execution as my database grows? Thanks. 2017-04-07 17:15 GMT+08:00 Christian Grün <christian.gruen@gmail.com>:
Hi Jacob,
The memory limit of the BaseX server process can be limited via the JVM -Xmx option in the BaseX startup scripts. If the maximum amount of assigned memory is exceeded, heap space errors will be raised.
I am not sure, however, if this is what you are asking for, because you were looking for an option that does not raise heap space errors, right?
If you use xquery:eval, you can specify a maximum memory limit for a single client [1]. This function is e.g. used in our DBA in order to avoid that queries sent from the client will compromise the BaseX server. As all threads run in the same JVM, it’s hard to guess how much memory is really spent by a single process; but at least it works to kill real memory killers.
I haven’t found much documentation on the MongoDB cache_size option. Could you possible give us some more infos what it does, and what happens if a process requires more than the assigned memory?
Best, Christian
[1] http://docs.basex.org/wiki/XQuery_Module#xquery:eval
On Fri, Apr 7, 2017 at 4:02 AM, Jacob Borisov <jacobborisov@gmail.com> wrote:
Hello. Is it possible to hardly limit a maximum memory consumption of a BaseX server instance process (something like a cache_size option in MongoDB) avoiding getting Java heap space errors simultaneously? Or, in the current implementation, it's required for the process to be able to allocate memory enough for storing a whole data set of the currently accessed database? Thanks.
Hi Jacob, Thanks for the background infos. I got the principles.
So does the BaseX have a mechanism like one described above or its architecture is completely different and a Java heap space error means exactly that my performed queries require more and more memory for execution as my database grows?
Exactly. In XQuery, it’s fairly easy to hit memory limits, as many operations are completely main-memory based. A simple yet malign example that does not access any data on disk: sort(1 to 10000000000) As XQuery is a full programming language, the complexity of caching data would be similar to a task like creating a generic caching-mechanism for applications in any other programming language. Incoming data of expressions like file:read-binary('huge.data') would probably need to be cached as well, etc. Talking about your specific use case, could you share your queries? Maybe there are simple answers how they could be rewritten to require less memory. Best, Christian
2017-04-07 17:15 GMT+08:00 Christian Grün <christian.gruen@gmail.com>:
Hi Jacob,
The memory limit of the BaseX server process can be limited via the JVM -Xmx option in the BaseX startup scripts. If the maximum amount of assigned memory is exceeded, heap space errors will be raised.
I am not sure, however, if this is what you are asking for, because you were looking for an option that does not raise heap space errors, right?
If you use xquery:eval, you can specify a maximum memory limit for a single client [1]. This function is e.g. used in our DBA in order to avoid that queries sent from the client will compromise the BaseX server. As all threads run in the same JVM, it’s hard to guess how much memory is really spent by a single process; but at least it works to kill real memory killers.
I haven’t found much documentation on the MongoDB cache_size option. Could you possible give us some more infos what it does, and what happens if a process requires more than the assigned memory?
Best, Christian
[1] http://docs.basex.org/wiki/XQuery_Module#xquery:eval
On Fri, Apr 7, 2017 at 4:02 AM, Jacob Borisov <jacobborisov@gmail.com> wrote:
Hello. Is it possible to hardly limit a maximum memory consumption of a BaseX server instance process (something like a cache_size option in MongoDB) avoiding getting Java heap space errors simultaneously? Or, in the current implementation, it's required for the process to be able to allocate memory enough for storing a whole data set of the currently accessed database? Thanks.
participants (2)
-
Christian Grün -
Jacob Borisov