Sorry, I realized I hadn't post this to the mailing list.

Hi Martin,
     I've just finished my test. It was a million resources in 5:31:20 hours, average of around 50 resources per second, I don't see any errors int the log. I wasn't able to get more speed as my client VM can't handle it. The only drawback I see is that when I get the list of dbs through the rest interface, the number of resources for my db is not updated until the sessions are closed. I can get the resources with no problem though.
     
     This test is perhaps similar to yours. This is what I did:


        static void DoInserts(string db)
        {
            const int numThreads = 10;
            for (int i = 0; i < threads.Length; ++i)
            {
                BaseXClient.Session baseXClientSession = new BaseXClient.Session(server, 1984, "admin", "admin");
                baseXClientSession.Execute("open " + db);
                
                threads[i] = new System.Threading.Thread(Write);
                threads[i].Start(baseXClientSession);
            }
        }

        static void Write(object param)
        {
            BaseXClient.Session baseXClientSession = (BaseXClient.Session)param;
            const int times = 100000;

            for (int i = 0; i < times; ++i)
            {
                int tempId = System.Threading.Interlocked.Increment(ref id);
                System.IO.MemoryStream ms = new System.IO.MemoryStream(System.Text.Encoding.UTF8.GetBytes(AddMetadata(fileContents, DateTime.Now)));
                baseXClientSession.Replace("prueba" + tempId.ToString("D8") + ".xml", ms);
            }
        }


Cheers,
     Martín.
From: ferrari_martin@hotmail.com
To: mar@centrum.cz
Subject: RE: [basex-talk] Performance and heavy load
Date: Thu, 20 Aug 2015 15:35:43 +0000

I haven't tried 250 thousand resources in a row. I just started a 1 million resources upload and will check.

Maybe for a quick test you can just close sessions that have been opened for a couple minutes? Have you seen a noticeable difference in performance by pooling instead of opening/closing? I have, but I'm curious as it might depend on some factors (OS, network latency, processor speed maybe, etc).

I'll let you know how my upload goes.

Cheers,
     Martín.

> Date: Thu, 20 Aug 2015 02:11:39 +0200
> To: basex-talk@mailman.uni-konstanz.de
> CC: ferrari_martin@hotmail.com
> Subject: Re: [basex-talk] Performance and heavy load
> From: mar@centrum.cz
>
> Hi, thanks for a quick answer.
>
> I have been doing something simillar -- only each thread had its own
> session (so no need to ask if it is in use) which got closed once the
> thread had been done. Multiple threads producing data (reading SQL
> database and filesystem producing XML) and multiple threads consuming data
> (ie. storing into a BaseX database).
>
> Monitoring the BaseX server JVM with JVisualVM showed plenty of live
> threads. Once it peeked with 600 or so live threads, I started to get
> SIGPIPE errors (ie. lost connections) and BaseX server has started to slow
> down. This way I was able to import about 250 thousand resources with some
> random errors, than it got much worse.
>
> Once I started to create and close the connection for each operation
> (simple Add()), everything has been working fine and I am able to import
> all my resources, but with slight performance penalty.
>
> I have about 650 thousand resources with various sizes 2k-700k each.
>
> I may try to use your approach, at least just to verify that the BaseX
> server behaves the same way.
>
> Thanks again,
> Martin.
>
> On Wed, Aug 19, 2015 at 03:04:22PM +0000, Martín Ferrari wrote:
> > Hi Martin, I'm not familiar with the Java client, I believe there's one that connects to BaseX directly without using the network?. I'm using the C# client found at https://github.com/BaseXdb/basex/blob/master/basex-api/src/main/c%23/BaseXClient.cs. This C# client connects to the server using tcp connections.
> > What I do is implement a pool of sessions. So, if a thread asks for a session and there's one already in the pool and not being currently in use, the thread gets that session, which will be marked as in use. If there's no available session, a new one is created and returned. Periodically, sessions that have been inactive for a certain amount of time are closed. This way, sending 10000 resources required only around 13 actual sessions (and corresponding tcp connections) in my tests.
> > I've inserted 100000 10k resources at around 60 resources per second (this was all one client was able to handle, BaseX server was able to handle more than that) with no issues.
> > I only need this as we have a huge live flow, otherwise I wouldn't have bothered :).
> >
> > I'm not sure if it helps, but this is my code for getting a session from the pool (I've added timeout to the BaseXClient.cs code). The whole session pool file is 380 lines, I can send it to you if you want.
> >
> >
> > public SessionEntry GetSession(string password, int timeout) { SessionEntry sessionEntry = null; lock(sessionList) { foreach(SessionEntry se in sessionList) { if (se.InUse == false) { sessionEntry = se; sessionEntry.InUse = true; break; } } }
> > if (sessionEntry == null) { sessionEntry = new SessionEntry(); sessionEntry.BaseXSession = new BaseXClient.Session(server, port, userName, password, timeout); if (dbName != null) { try { sessionEntry.BaseXSession.Execute("open " + dbName); } catch (Exception) { try { sessionEntry.BaseXSession.Close(); } catch (Exception) { } throw; } } sessionEntry.InUse = true; lock (sessionList) { sessionList.Add(sessionEntry); } } else { sessionEntry.BaseXSession.Timeout = timeout; }
> > return sessionEntry; }
> > Cheers, Martín.
> >
> > > Date: Wed, 19 Aug 2015 13:41:34 +0200
> > > To: basex-talk@mailman.uni-konstanz.de
> > > From: mar@centrum.cz
> > > Subject: Re: [basex-talk] Performance and heavy load
> > >
> > > Hi, I would like to know more about "keep the session opened" as you state
> > > it -- I am using Java/Groovy client populating a large database (over half
> > > a million resources) and if I keep the session opened, so it could be
> > > reused within the thread, after a while it starts to cause problems. The
> > > only solution I was able to come up with was to close each connection after
> > > I add/replace a resource and open a new one. Than it behaves correctly.
> > >
> > > JVM running the BaseX server is keeping threads alive somehow not releasing
> > > the resources properly (I have been monitoring the JVM through JVisualVM)
> > > -- I stil plan to debug it a little, but I had no chance.
> > >
> > > Performance is quite important, so I would like to know more about your
> > > solution, could you tell me more about your code?
> > >
> > > Regards,
> > > Martin
> >