Exhausting PermGen space when using xar extensions w/ Java modules?
Howdy -- I'm occasionally getting the following after a BaseX instance has been up and accepting queries for some time: [XPST0017] Class cannot be initialized: PermGen space. As I understand it, PermGen space is used during classloading. I'm using a Java extension (packaged in a xar); as BaseX has a fair bit of custom code around classloading, this seems the obvious first place to start looking. That said -- has anyone already seen this? (By the way -- without looking at the code, I would expect new classloading to only happen if the repo holding this plugin is being uninstalled and reinstalled; is this expectation accurate, or are plugins reloaded within a running instance more frequently?)
On 04/04/2012 05:40 PM, Charles Duffy wrote:
Howdy --
I'm occasionally getting the following after a BaseX instance has been up and accepting queries for some time:
[XPST0017] Class cannot be initialized: PermGen space.
As I understand it, PermGen space is used during classloading. I'm using a Java extension (packaged in a xar); as BaseX has a fair bit of custom code around classloading, this seems the obvious first place to start looking. That said -- has anyone already seen this?
(By the way -- without looking at the code, I would expect new classloading to only happen if the repo holding this plugin is being uninstalled and reinstalled; is this expectation accurate, or are plugins reloaded within a running instance more frequently?)
Using jmap and jhat, it's clear that we have a classloader leak -- org.basex.util.JarLoader has 13 instances in memory, each of which is referred to by a ModuleLoader instance, two java.lang.Package instances (one of which is referred to by my Java module), and separate copies of my 3rd-party module's classes and all their dependencies; this makes the source of the PermGen outage quite clear. After a bit of digging, it appears that each query generates a new QueryContext, which generates a new ModuleLoader, which generates a new set of Java modules -- in short, that every Java module is completely loaded anew for every query referencing an XQuery module *in the same package* (not necessarily even using the Java module at all). Is there design intent for this behavior? From where I currently stand, it seems really quite unfortunate -- not only from the perspective of making it easy to exhaust the limited PermGen space, but also for runtime performance.
Investigating the nature of the leak proper -- Any loadable module which, either directly or through its libraries, sets up thread-local data containing strong references to instances of objects whose classes were loaded via a plugin classloader will in turn have references to its classloader (presently a distinct JarLoader instance for each query) from the thread finalization routine. Bringing this into the context of my own usage for the Java interface -- the very purpose and intent of the SvnKitWrapper module is to do one-time initialization only once, stuff context away in thread-local space, and be able to efficiently leverage that context during future queries. There's still a serious bug in the BaseX Java module interface that needs to be fixed (in terms of not reusing classloaders or module instances, mooting any benefit from this kind of state caching), but let's ignore that for the moment and try to figure out what we can do differently in terms of the Java module API to make this kind of thing possible in the future. I'd like to propose that the specification for Java modules require them to put only weak references in thread-local space, and include a call which BaseX can make when initially loading a module which will return a strong reference to any state-local data, which the BaseX module interface will contract to hold until that module is uninstalled. This way, we can ensure that no strong references are held in the thread-local state (preventing module unload), while ensuring that strong references *are* held as long as a module is installed to permit caching state. Thoughts?
Wow - your activity seems to easily outperform our responsiveness on this list.
I'd like to propose that the specification for Java modules require them to put only weak references in thread-local space, and include a call which BaseX can make when initially loading a module which will return a strong reference to any state-local data, which the BaseX module interface will contract to hold until that module is uninstalled.
This would surely make sense, as I'm pretty sure the current solution will also show unexpected results as soon as concurrent queries will be performed. The main reason why classes need to be unloaded at all is the existence of the "repo delete" command, so it may be a better approach to only remove classes if this command is called. I'm pretty sure your time is limited as well, but as you seem to have some concrete ideas how to improve the current situation, you are invited to propose another patch (otherwise, I'm concerned that this issue might get lost due to the total number of open issues, some of which are listed at GitHub). Christian ___________________________ On Thu, Apr 5, 2012 at 8:21 PM, Charles Duffy <charles@dyfis.net> wrote:
Investigating the nature of the leak proper --
Any loadable module which, either directly or through its libraries, sets up thread-local data containing strong references to instances of objects whose classes were loaded via a plugin classloader will in turn have references to its classloader (presently a distinct JarLoader instance for each query) from the thread finalization routine.
Bringing this into the context of my own usage for the Java interface -- the very purpose and intent of the SvnKitWrapper module is to do one-time initialization only once, stuff context away in thread-local space, and be able to efficiently leverage that context during future queries. There's still a serious bug in the BaseX Java module interface that needs to be fixed (in terms of not reusing classloaders or module instances, mooting any benefit from this kind of state caching), but let's ignore that for the moment and try to figure out what we can do differently in terms of the Java module API to make this kind of thing possible in the future.
I'd like to propose that the specification for Java modules require them to put only weak references in thread-local space, and include a call which BaseX can make when initially loading a module which will return a strong reference to any state-local data, which the BaseX module interface will contract to hold until that module is uninstalled.
This way, we can ensure that no strong references are held in the thread-local state (preventing module unload), while ensuring that strong references *are* held as long as a module is installed to permit caching state.
Thoughts?
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
participants (2)
-
Charles Duffy -
Christian Grün