Reusing connections to BaseX
Hi, I have a question about the BaseX server. We're using it from a J2EE application (which is the client), and so we've implemented a connection pool in the style done by many other database drivers. We are getting Broken Pipe errors, indicating that a connection has been disconnected. My question: when does BaseX disconnect, and is there a way for a client to learn this? For instance, if there's a syntax error in a XQUERY, should BaseX disconnect? If a BaseXException is thrown by the Java driver? Currently, we mark the connection as dead when an IOException or EOF occurs during the handling of one request, but we keep it alive when a BaseXException occurs, such as when we submit an ill-formed XQUERY. We could avoid pooling, connecting and disconnecting for every query, but obviously that would be expensive. Or we could "ping" the server before issueing the actual query, but that's awkward, too. How do others handle this problem? - Godmar
Hello Godmar, your email mentions topics that are of high relevance for us as well. Last year we started to use BaseX as the database of a master data server, it plays a central role in maintaining the product data of more than 100 catalogs in a sector of German industry. The concrete processing is done in individual local instance of BaxeX for each catalog. And it works great. A big "Thank you" to the BaseX team. You do an outstanding job. Now we want to make portions of the data available via webservices by JBoss, and access BaseX servers on remote boxes (with an LVM load balancer in between). * Which of the BaseX interfaces do you use: XQJ or just the plain ClientSession's? * Have you found an example to use XQJ for a remote BaseX - Database? * Have you found some documentation about the Java-client and transactional support (for example to use BaseX even for a 'logging table')? * Are there substantial benefits of the Java-client versus the rest client for such an application? * Have you already some experiences with the Java-client and BaseX under high concurrent load? Thank you for your attention and any help/feedback is highly appreciated! With best regards Andreas Am 11.02.2012 14:30, schrieb Godmar Back:
Hi,
I have a question about the BaseX server. We're using it from a J2EE application (which is the client), and so we've implemented a connection pool in the style done by many other database drivers.
We are getting Broken Pipe errors, indicating that a connection has been disconnected. My question: when does BaseX disconnect, and is there a way for a client to learn this? For instance, if there's a syntax error in a XQUERY, should BaseX disconnect? If a BaseXException is thrown by the Java driver? Currently, we mark the connection as dead when an IOException or EOF occurs during the handling of one request, but we keep it alive when a BaseXException occurs, such as when we submit an ill-formed XQUERY.
We could avoid pooling, connecting and disconnecting for every query, but obviously that would be expensive. Or we could "ping" the server before issueing the actual query, but that's awkward, too.
How do others handle this problem?
- Godmar
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Dear Andreas, thanks for your mail. Some answers (even if they may have been addressed to Godmar):
* Which of the BaseX interfaces do you use: XQJ or just the plain ClientSession's?
In general, the plain client sessions will give you better performance, as they are very light-weight and comply better with the internal BaseX representation.
* Have you found an example to use XQJ for a remote BaseX - Database?
Currently, XQJ does only work with local database instances. This may change soon, as Charles Foster will soon offer a 100% compliant and client-/server-based driver for his XQJ implementation for BaseX (see xqj.net for more)!
* Have you found some documentation about the Java-client and transactional support (for example to use BaseX even for a 'logging table')?
The BaseX documentation includes some basic information on transactions in BaseX [1]. Feel free to ask for more details.
* Are there substantial benefits of the Java-client versus the rest client for such an application?
Again it's mainly about performance: the REST API is a little bit slower due to the HTTP communication overhead and its statelessness (each time, a new session has to be created, the addressed database has to be opened, etc). Depending on the way your queries look like, this may be more or less relevant: if you need to run hundreds or thousands of queries each minute, the overhead could impose a limit at some point; if you have a few number of queries that take considerable time, the overhead should hardly be noticeable.
* Have you already some experiences with the Java-client and BaseX under high concurrent load?
To give an example: In one use cases, BaseX stores converts new JSON Twitter data to XML and stores it in a database appr. 300 times per second. Each hour, a new database is created; a single instance takes 4 GB in average (depending on the time of the day). Feedback from everyone else is welcome; feel free to browse the mailing list archive [2] to find other usage scenarios. Christian [1] http://docs.basex.org/wiki/Transaction_Management [2] http://basex.org/service/mailinglist-search/
Hello Christian, thank you for your very detailed answer! The twitter use case that you have mentioned as an example for concurrent updates is quite impressive. From the perspecitive of J2EE one might ask for features like rollback or even 2 phase commit protocol. But even the JBoss team had issues with the two phase commit when they developed their HornetQ messaging server [2]. I was astonished to see them in JBoss 6 release. So it seems to be a wise decision to focus on the specific strength of an XML Database. And for our special use case the features mentioned in [1] are sufficient. With best regards, Andreas [1] http://docs.basex.org/wiki/Transaction_Management#Update_Transactions [2] http://www.jboss.org/hornetq Am 14.02.2012 23:49, schrieb Christian Grün:
Dear Andreas,
thanks for your mail. Some answers (even if they may have been addressed to Godmar):
* Which of the BaseX interfaces do you use: XQJ or just the plain ClientSession's? In general, the plain client sessions will give you better performance, as they are very light-weight and comply better with the internal BaseX representation.
* Have you found an example to use XQJ for a remote BaseX - Database? Currently, XQJ does only work with local database instances. This may change soon, as Charles Foster will soon offer a 100% compliant and client-/server-based driver for his XQJ implementation for BaseX (see xqj.net for more)!
* Have you found some documentation about the Java-client and transactional support (for example to use BaseX even for a 'logging table')? The BaseX documentation includes some basic information on transactions in BaseX [1]. Feel free to ask for more details.
* Are there substantial benefits of the Java-client versus the rest client for such an application? Again it's mainly about performance: the REST API is a little bit slower due to the HTTP communication overhead and its statelessness (each time, a new session has to be created, the addressed database has to be opened, etc). Depending on the way your queries look like, this may be more or less relevant: if you need to run hundreds or thousands of queries each minute, the overhead could impose a limit at some point; if you have a few number of queries that take considerable time, the overhead should hardly be noticeable.
* Have you already some experiences with the Java-client and BaseX under high concurrent load? To give an example: In one use cases, BaseX stores converts new JSON Twitter data to XML and stores it in a database appr. 300 times per second. Each hour, a new database is created; a single instance takes 4 GB in average (depending on the time of the day).
Feedback from everyone else is welcome; feel free to browse the mailing list archive [2] to find other usage scenarios.
Christian
[1]http://docs.basex.org/wiki/Transaction_Management [2]http://basex.org/service/mailinglist-search/
-- Nexoma GmbH Theodorus Weg 7 59755 Arnsberg Tel. + 49 (0) 52 51 1613-0 Fax + 49 (0) 52 51 1613-99 mailto:andreas.rulle@nexoma.de Geschäftsführer: Guido Sauerland Sitz der Gesellschaft: Arnsberg Registergericht: Arnsberg, HRB 9365
To follow up on my own question, which wasn't answered. Closer examination of the Java driver (*) reveals that it doesn't throw BaseXExceptions (anymore?). Our code catches those (BaseXException is a subtype of IOException) and IOExceptions separately. I assume that an earlier version of the driver did (?). Looking through the code, IOException are thrown by most all methods. Is it correct to assume that *any* IOException thrown leaves the BaseXClient object in an unusable state, thus requiring its replacement in the pool? May I repeat my suggestion that you move this *crucial* piece of code out of the "examples" section and make it part of the properly supported/documented code base? As far as I'm concerned, with the concern and limitations about XQJ, *this* should be part of the officially sanctioned Java API. - Godmar (*) https://github.com/BaseXdb/basex-examples/blob/master/src/main/java/org/base... On Sat, Feb 11, 2012 at 8:30 AM, Godmar Back <godmar@gmail.com> wrote:
Hi,
I have a question about the BaseX server. We're using it from a J2EE application (which is the client), and so we've implemented a connection pool in the style done by many other database drivers.
We are getting Broken Pipe errors, indicating that a connection has been disconnected. My question: when does BaseX disconnect, and is there a way for a client to learn this? For instance, if there's a syntax error in a XQUERY, should BaseX disconnect? If a BaseXException is thrown by the Java driver? Currently, we mark the connection as dead when an IOException or EOF occurs during the handling of one request, but we keep it alive when a BaseXException occurs, such as when we submit an ill-formed XQUERY.
We could avoid pooling, connecting and disconnecting for every query, but obviously that would be expensive. Or we could "ping" the server before issueing the actual query, but that's awkward, too.
How do others handle this problem?
- Godmar
participants (4)
-
Andreas Rulle -
andreas.rulle＠intermoves.de
-
Christian Grün -
Godmar Back