Multiple producers/writers for one collection and concurrency
Hi This topic was mentioned few times before, but it is key aspect of my application, so I will try anyway. *Question*: What is recommended approach for application, where exists multiple document producers which shall store/modify documents in one, share collection. *Background* - Having *feeds*, which might deliver some new messages. - *Multiple preprocessing processes* - preprocessing the messages before storing them in collection (Python, celery and RabbitMQ to be ready for processing bigger amount of incoming messages in parallel) - Having *single process for write access to my BaseX collection*(arranging this is not very easy for me so far) - There are some read only queries for stored messages, there are no problems with this as it shall be easy to run multiple read-only queries concurrently. *Assumptions* If I am wrong here, please, correct me. - one collection can be written/modified only by one process at given moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?) - read only queries are easy as they can be run concurrently without blocking/rejecting *Current approach* 1. Multiple preprocessing processes are concurrently sending documents into single basex-storage queue in RabbitMQ 2. There is single one-process worker, consuming documents from queue basex-storage which does all the work with adding documents for given collection. *Wishes* Managing single write process adds quite complexity to whole solution. It would be nice, if I can write to single collection by multiple processes at once and could expect success. Any comments are welcome Best regards Jan
Hi Jan, if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time. Kind regards, Andreas Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD):
one collection can be written/modified only by one process at given moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?)
Hi Andreas It seems, like I was too smart (and learned a lot about concurrency in AMQP this very long night) :-) What are starting/ending points of one transaction 1. creating connection - closing it. probably not 2. open database - close ?? 3. performing one command (like xquery, add, delete etc.) This would be the best Jan 2010/12/31 Andreas Weiler <andreas.weiler@uni-konstanz.de>
Hi Jan,
if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time.
Kind regards, Andreas
Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD):
one collection can be written/modified only by one process at given
moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?)
-- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky@gmail.com http://cz.linkedin.com/in/vlcinsky
performing one command (like xquery, add, delete etc.) This would be the best Thats the way it is. Note that all commands are atomic transactions, like a alter database -> closes the database, renames it and reopens it...but you havent care about that.
Kind regards and happy new year (with BaseX), Andreas Am 31.12.10 11:06, schrieb Jan Vlčinský (CAD):
Hi Andreas It seems, like I was too smart (and learned a lot about concurrency in AMQP this very long night) :-)
What are starting/ending points of one transaction
1. creating connection - closing it. probably not 2. open database - close ?? 3. performing one command (like xquery, add, delete etc.) This would be the best
Jan
2010/12/31 Andreas Weiler <andreas.weiler@uni-konstanz.de <mailto:andreas.weiler@uni-konstanz.de>>
Hi Jan,
if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time.
Kind regards, Andreas
Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD):
one collection can be written/modified only by one process at given moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?)
-- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky@gmail.com <mailto:jan.vlcinsky@gmail.com> http://cz.linkedin.com/in/vlcinsky
Thanks "for the best". I tested it and storing 100 documents by 3 client processes in parallel works very well. BaseX makes me happy. Happy New Year to all of you. Jan 2010/12/31 Andreas Weiler <andreas.weiler@uni-konstanz.de>
performing one command (like xquery, add, delete etc.) This would be the best
Thats the way it is. Note that all commands are atomic transactions, like a alter database -> closes the database, renames it and reopens it...but you havent care about that.
Kind regards and happy new year (with BaseX), Andreas
Am 31.12.10 11:06, schrieb Jan Vlčinský (CAD):
Hi Andreas It seems, like I was too smart (and learned a lot about concurrency in AMQP this very long night) :-)
What are starting/ending points of one transaction
1. creating connection - closing it. probably not 2. open database - close ?? 3. performing one command (like xquery, add, delete etc.) This would be the best
Jan
2010/12/31 Andreas Weiler <andreas.weiler@uni-konstanz.de>
Hi Jan,
if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time.
Kind regards, Andreas
Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD):
one collection can be written/modified only by one process at given
moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?)
-- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky@gmail.com http://cz.linkedin.com/in/vlcinsky
-- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky@gmail.com http://cz.linkedin.com/in/vlcinsky
participants (2)
-
Andreas Weiler -
Jan Vlčinský (CAD)