This C# client connects to the server using tcp connections.
What I do is implement a pool of sessions. So, if a thread asks for a session and there's one already in the pool and not being currently in use, the thread gets that session, which will be marked as in use. If there's no available session, a new one is created and returned. Periodically, sessions that have been inactive for a certain amount of time are closed. This way, sending 10000 resources required only around 13 actual sessions (and corresponding tcp connections) in my tests.
I've inserted 100000 10k resources at around 60 resources per second (this was all one client was able to handle, BaseX server was able to handle more than that) with no issues.
I only need this as we have a huge live flow, otherwise I wouldn't have bothered :).
I'm not sure if it helps, but this is my code for getting a session from the pool (I've added timeout to the BaseXClient.cs code). The whole session pool file is 380 lines, I can send it to you if you want.
public SessionEntry GetSession(string password, int timeout)
{
SessionEntry sessionEntry = null;
lock(sessionList)
{
foreach(SessionEntry se in sessionList)
{
if (se.InUse == false)
{
sessionEntry = se;
sessionEntry.InUse = true;
break;
}
}
}
if (sessionEntry == null)
{
sessionEntry = new SessionEntry();
sessionEntry.BaseXSession = new BaseXClient.Session(server, port, userName, password, timeout);
if (dbName != null)
{
try
{
sessionEntry.BaseXSession.Execute("open " + dbName);
}
catch (Exception)
{
try
{
sessionEntry.BaseXSession.Close();
}
catch (Exception)
{
}
throw;
}
}
sessionEntry.InUse = true;
lock (sessionList)
{
sessionList.Add(sessionEntry);
}
}
else
{
sessionEntry.BaseXSession.Timeout = timeout;
}
return sessionEntry;
}
Cheers,
Martín.
> Date: Wed, 19 Aug 2015 13:41:34 +0200
> To: basex-talk@mailman.uni-konstanz.de
> From: mar@centrum.cz
> Subject: Re: [basex-talk] Performance and heavy load
>
> Hi, I would like to know more about "keep the session opened" as you state
> it -- I am using Java/Groovy client populating a large database (over half
> a million resources) and if I keep the session opened, so it could be
> reused within the thread, after a while it starts to cause problems. The
> only solution I was able to come up with was to close each connection after
> I add/replace a resource and open a new one. Than it behaves correctly.
>
> JVM running the BaseX server is keeping threads alive somehow not releasing
> the resources properly (I have been monitoring the JVM through JVisualVM)
> -- I stil plan to debug it a little, but I had no chance.
>
> Performance is quite important, so I would like to know more about your
> solution, could you tell me more about your code?
>
> Regards,
> Martin