Thanks, Christian.  I haven't seen it crash recently, but I'm watching closely.

I think I will also take another approach to getting the bulk of the data into BaseX.  The slowdown that I was talking about earlier was a little annoying (haven't tried yet with the latest snapshot, sorry), and my approach of storing one document per "record" (there are millions) might not be the smartest.

So now I've built a module which builds a single big text file of those million records, and when I load that with a single BaseX command it's much much faster.  The only thing I'm missing is a nice progress bar, but I can probably live without that.

On Mon, Oct 6, 2014 at 2:34 PM, Christian Grün <christian.gruen@gmail.com> wrote:
Hi Gerald,

did you manage to write some Java code that causes the same problems?
I am afraid that this is currently all I can offer.

Christian



On Wed, Sep 24, 2014 at 5:41 PM, Gerald de Jong <gerald@delving.eu> wrote:
> It happened again.  Could I not argue that nothing I can do as a client to
> BaseX should be able to freeze it?  I mean, even if, for some reason, things
> are getting muddled up.
>
> Exception in thread "Thread-56391" java.lang.IllegalMonitorStateException
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131)
> at org.basex.core.DBLocking.release(DBLocking.java:219)
> at org.basex.core.Context.unregister(Context.java:249)
> at org.basex.core.Command.execute(Command.java:104)
>
>
> On Tue, Sep 23, 2014 at 2:17 PM, Christian Grün <christian.gruen@gmail.com>
> wrote:
>>
>> You can set the debugging flag and check out the database log files to
>> get more insight into what's going on.
>>
>>
>> On Tue, Sep 23, 2014 at 2:15 PM, Gerald de Jong <gerald@delving.eu> wrote:
>> > Can I start up the basexserver with some kind of logging so I can see
>> > what
>> > leads to the IllegalMonitorStateException?
>> >
>> > On Tue, Sep 23, 2014 at 1:53 PM, Christian Grün
>> > <christian.gruen@gmail.com>
>> > wrote:
>> >>
>> >> > Looking at these tests, it appears that none of them do large numbers
>> >> > of
>> >> > ADD
>> >> > commands.  Did I miss something?
>> >>
>> >> You may need to check the other tests. But as the (updating) add
>> >> command will never be executed in parallel, it shouldn't cause any
>> >> problems with parallelization (feel free to tell me if I'm wrong).
>> >>
>> >>
>> >> >
>> >> > On Tue, Sep 23, 2014 at 1:34 PM, Christian Grün
>> >> > <christian.gruen@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> > The fact that it's scala should be insignificant, since I'm using
>> >> >> > the
>> >> >> > Java
>> >> >> > API.
>> >> >>
>> >> >> Well, it might be theoretically insignificant, but we cannot give a
>> >> >> quick answer on what's your problem. In general, parallel client
>> >> >> requests are no problem, we have various JUnit tests doing exactly
>> >> >> that (see e.g. [1], [2] or [3]).
>> >> >>
>> >> >> [1]
>> >> >>
>> >> >>
>> >> >> https://github.com/BaseXdb/basex/blob/master/basex-core/src/test/java/org/basex/server/ConcurrencyTest.java
>> >> >> [2]
>> >> >>
>> >> >>
>> >> >> https://github.com/BaseXdb/basex/blob/master/basex-tests/src/test/java/org/basex/performance/RESTConcurrencyTest.java
>> >> >> [3]
>> >> >>
>> >> >>
>> >> >> https://github.com/BaseXdb/basex/blob/master/basex-tests/src/test/java/org/basex/performance/ServerStressTest.java
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > What's happening is that two (or more) threads are doing this at
>> >> >> > the
>> >> >> > same
>> >> >> > time:
>> >> >> >
>> >> >> > ClientSession session = new ClientSession(...);
>> >> >> > session.execute(new Open(database));
>> >> >> > session.execute(new Set("autoflush", "false"));
>> >> >> > session.add([path string], [document-bytes]);    //   a zillion
>> >> >> > times
>> >> >> > session.execute(new Flush());
>> >> >> > session.close();
>> >> >> >
>> >> >> > My first question is of course if this should work parallel with
>> >> >> > no
>> >> >> > problems
>> >> >> > "in theory".
>> >> >> >
>> >> >> > On Tue, Sep 23, 2014 at 12:46 PM, Christian Grün
>> >> >> > <christian.gruen@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> ...difficult to tell, as we don't have too much experience with
>> >> >> >> Scala.
>> >> >> >> Could you provide this example as Java code?
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On Tue, Sep 23, 2014 at 12:10 PM, Gerald de Jong
>> >> >> >> <gerald@delving.eu>
>> >> >> >> wrote:
>> >> >> >> > Dear BaseX experts,
>> >> >> >> >
>> >> >> >> > I'm wondering if anybody can see something that I'm doing
>> >> >> >> > wrong. I
>> >> >> >> > would
>> >> >> >> > expect it to be possible to add documents from multiple
>> >> >> >> > threads.
>> >> >> >> > Am
>> >> >> >> > I
>> >> >> >> > wrong
>> >> >> >> > about that??
>> >> >> >> >
>> >> >> >> > When I set two of my datasets saving documents to BaseX 7.9 at
>> >> >> >> > the
>> >> >> >> > same
>> >> >> >> > time, I get this exception:
>> >> >> >> >
>> >> >> >> > $ basexserver
>> >> >> >> > BaseX 7.9 [Server]
>> >> >> >> > Server was started (port: 1984)
>> >> >> >> > Exception in thread "Thread-2295"
>> >> >> >> > java.lang.IllegalMonitorStateException
>> >> >> >> > at
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131)
>> >> >> >> > at org.basex.core.DBLocking.release(DBLocking.java:219)
>> >> >> >> > at org.basex.core.Context.unregister(Context.java:249)
>> >> >> >> > at org.basex.core.Command.execute(Command.java:104)
>> >> >> >> > at org.basex.core.Command.execute(Command.java:117)
>> >> >> >> > at
>> >> >> >> >
>> >> >> >> > org.basex.server.ClientListener.execute(ClientListener.java:384)
>> >> >> >> > at org.basex.server.ClientListener.add(ClientListener.java:355)
>> >> >> >> > at org.basex.server.ClientListener.run(ClientListener.java:100)
>> >> >> >> >
>> >> >> >> > All I am doing is making concurrent calls in two parallel
>> >> >> >> > sessions:
>> >> >> >> >
>> >> >> >> > session = new ClientSession(...)
>> >> >> >> > session.execute(new Open(database))
>> >> >> >> > session.execute(new Set("autoflush", "false"))
>> >> >> >> > session.add([path string], [document-bytes])     //   a zillion
>> >> >> >> > times
>> >> >> >> > session.execute(new Flush())
>> >> >> >> >
>> >> >> >> > During the execution of the parallel add commands, it crashes.
>> >> >> >> >
>> >> >> >> > The relevant code:
>> >> >> >> >
>> >> >> >> > def withSession[T](block: ClientSession => T): T = {
>> >> >> >> >     val session = new ClientSession(host, port, user, pass)
>> >> >> >> >     try {
>> >> >> >> >       block(session)
>> >> >> >> >     }
>> >> >> >> >     finally {
>> >> >> >> >       session.close()
>> >> >> >> >     }
>> >> >> >> >   }
>> >> >> >> >
>> >> >> >> >   def withDbSession[T](database: String)(block: ClientSession
>> >> >> >> > =>
>> >> >> >> > T):
>> >> >> >> > T =
>> >> >> >> > {
>> >> >> >> >     withSession {
>> >> >> >> >       session =>
>> >> >> >> >         session.execute(new Open(database))
>> >> >> >> >         session.execute(new Set("autoflush", "false"))
>> >> >> >> >         val result = block(session)
>> >> >> >> >         session.execute(new Flush())
>> >> >> >> >         result
>> >> >> >> >     }
>> >> >> >> >   }
>> >> >> >> >
>> >> >> >> > then:
>> >> >> >> >
>> >> >> >> > withDbSession {
>> >> >> >> >    session =>
>> >> >> >> >       // many calls to session.add(...)
>> >> >> >> > }
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > Delving BV, Vasteland 8, Rotterdam
>> >> >> >> > http://www.delving.eu
>> >> >> >> > http://twitter.com/fluxe
>> >> >> >> > skype: beautifulcode
>> >> >> >> > +31629339805
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Delving BV, Vasteland 8, Rotterdam
>> >> >> > http://www.delving.eu
>> >> >> > http://twitter.com/fluxe
>> >> >> > skype: beautifulcode
>> >> >> > +31629339805
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Delving BV, Vasteland 8, Rotterdam
>> >> > http://www.delving.eu
>> >> > http://twitter.com/fluxe
>> >> > skype: beautifulcode
>> >> > +31629339805
>> >
>> >
>> >
>> >
>> > --
>> > Delving BV, Vasteland 8, Rotterdam
>> > http://www.delving.eu
>> > http://twitter.com/fluxe
>> > skype: beautifulcode
>> > +31629339805
>
>
>
>
> --
> Delving BV, Vasteland 8, Rotterdam
> http://www.delving.eu
> http://twitter.com/fluxe
> skype: beautifulcode
> +31629339805



--
Delving BV, Vasteland 8, Rotterdam
http://www.delving.eu
http://twitter.com/fluxe
skype: beautifulcode
+31629339805