Hi This topic was mentioned few times before, but it is key aspect of my application, so I will try anyway.
*Question*: What is recommended approach for application, where exists multiple document producers which shall store/modify documents in one, share collection.
*Background*
- Having *feeds*, which might deliver some new messages. - *Multiple preprocessing processes* - preprocessing the messages before storing them in collection (Python, celery and RabbitMQ to be ready for processing bigger amount of incoming messages in parallel) - Having *single process for write access to my BaseX collection*(arranging this is not very easy for me so far) - There are some read only queries for stored messages, there are no problems with this as it shall be easy to run multiple read-only queries concurrently.
*Assumptions* If I am wrong here, please, correct me.
- one collection can be written/modified only by one process at given moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?) - read only queries are easy as they can be run concurrently without blocking/rejecting
*Current approach*
1. Multiple preprocessing processes are concurrently sending documents into single basex-storage queue in RabbitMQ 2. There is single one-process worker, consuming documents from queue basex-storage which does all the work with adding documents for given collection.
*Wishes* Managing single write process adds quite complexity to whole solution. It would be nice, if I can write to single collection by multiple processes at once and could expect success.
Any comments are welcome
Best regards
Jan
Hi Jan,
if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time.
Kind regards, Andreas
Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD):
one collection can be written/modified only by one process at given moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?)
Hi Andreas It seems, like I was too smart (and learned a lot about concurrency in AMQP this very long night) :-)
What are starting/ending points of one transaction
1. creating connection - closing it. probably not 2. open database - close ?? 3. performing one command (like xquery, add, delete etc.) This would be the best
Jan
2010/12/31 Andreas Weiler andreas.weiler@uni-konstanz.de
Hi Jan,
if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time.
Kind regards, Andreas
Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD):
one collection can be written/modified only by one process at given
moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?)
performing one command (like xquery, add, delete etc.) This would be the best
Thats the way it is. Note that all commands are atomic transactions, like a alter database -> closes the database, renames it and reopens it...but you havent care about that.
Kind regards and happy new year (with BaseX), Andreas
Am 31.12.10 11:06, schrieb Jan Vlčinský (CAD):
Hi Andreas It seems, like I was too smart (and learned a lot about concurrency in AMQP this very long night) :-)
What are starting/ending points of one transaction
- creating connection - closing it. probably not
- open database - close ??
- performing one command (like xquery, add, delete etc.) This would be the best
Jan
2010/12/31 Andreas Weiler <andreas.weiler@uni-konstanz.de mailto:andreas.weiler@uni-konstanz.de>
Hi Jan, if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time. Kind regards, Andreas Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD): one collection can be written/modified only by one process at given moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?)-- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky@gmail.com mailto:jan.vlcinsky@gmail.com http://cz.linkedin.com/in/vlcinsky
Thanks "for the best". I tested it and storing 100 documents by 3 client processes in parallel works very well.
BaseX makes me happy.
Happy New Year to all of you.
Jan
2010/12/31 Andreas Weiler andreas.weiler@uni-konstanz.de
performing one command (like xquery, add, delete etc.) This would be the best
Thats the way it is. Note that all commands are atomic transactions, like a alter database -> closes the database, renames it and reopens it...but you havent care about that.
Kind regards and happy new year (with BaseX), Andreas
Am 31.12.10 11:06, schrieb Jan Vlčinský (CAD):
Hi Andreas It seems, like I was too smart (and learned a lot about concurrency in AMQP this very long night) :-)
What are starting/ending points of one transaction
- creating connection - closing it.
probably not 2. open database - close ?? 3. performing one command (like xquery, add, delete etc.) This would be the best
Jan
2010/12/31 Andreas Weiler andreas.weiler@uni-konstanz.de
Hi Jan,
if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time.
Kind regards, Andreas
Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD):
one collection can be written/modified only by one process at given
moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?)
-- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky@gmail.com http://cz.linkedin.com/in/vlcinsky
basex-talk@mailman.uni-konstanz.de
-
Andreas Weiler -
Jan Vlčinský (CAD)