Maybe. But then again I want to use a very large amount of threads. I read fairlock hurts there.
Not necessarily; but if you have critical non-locking operations that need to be executed in any case and without delays, it may be too restrictive. In both cases, you should probably increase the number of parallel jobs (via the PARALLEL option).
And I actually like the idea of always having a way to
know the status of the job system. I will look into that.
Best regards
Omar Siam
Am 29.11.2017 um 16:30 schrieb Christian Grün:
Maybe it helps you to enable the FAIRLOCK option?
On Wed, Nov 29, 2017 at 4:24 PM, Omar Siam Omar.Siam@oeaw.ac.at wrote:
No, I don't think this is possible without the RESTXQ job waiting for a result of another job it started itself.
I'm quite sure this is to be expected but then again it puzzeled me that it is possible to send jobs to the RESTXQ endpoint so fast it is unable to start it's slave script but just the RESTXQ jobs pile up.
I think it is a side effect of allowing any number of non locking jobs so there is always a possibility to send a jobs:stop request or have a look at the situation and get things on track again.
If you could make sure that jobs started by RESTXQ jobs get a chance to be unqueued before accepting new RESTXQ jobs when the parallel limit is reached would be a way to improve the situation. I'm thinking about a very little perhaps increasing delay for accepting new RESTXQ jobs when the parallel limit is reached. Just so queued jobs get a chance to start. Or a yield function to explicitly give away some execution time so another (non RESTXQ) job can execute.
Best regards Omar Siam
Am 29.11.2017 um 16:07 schrieb Christian Grün:
Hi Omar,
jobs that are started via jobs:eval are independent from the original job, but most probably the deadlocks occur because you are retrieving the result in the same query (see the last comment in [1]). If you are calling jobs:eval to get better performance, xquery:fork-join may be a better choice.
Still, if you believe that the deadlock can also be triggered without jobs:eval and jobs:result in the same query, we should definitely have a look at this.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Jobs_Module#jobs:result
On Wed, Nov 29, 2017 at 3:56 PM, Omar Siam Omar.Siam@oeaw.ac.at wrote:
Slave scripts and jobs are scripts to me I generate on the fly to hard code the DB name so locking and optimzing know what they deal with.
Some of the code I'm working with right now (now including an error so this kind of lock up does not happen:
declare %private function wde:eval($query as xs:string, $bindings as map(*)?, $jobName as xs:string, $dontCheckQuery as xs:boolean) as item()* { let $too-many-jobs := if (count(jobs:list()) >= xs:integer(db:system()//parallel)) then error(xs:QName('wde:too-many-parallel-requests'), 'Too many parallel requests!') else (), $query-is-sane := $dontCheckQuery or wde:query-is-sane($query), $j := jobs:eval($query, $bindings, map { 'cache': true(), 'id': 'wde:'||$jobName||'-'||jobs:current(), 'base-uri': $wde:basePath||'/wde_'||$jobName||'.xq'}), $_ := jobs:wait($j) return jobs:result($j) };
declare %rest:path("wde/{$dict}/{$index}") %rest:query-param("q", "{$q}") %rest:query-param("type", "{$type}") %rest:GET function wde:query-index($dict as xs:string, $index as xs:string, $q as xs:string*, $type as xs:string*) { let $query-index-script := ``[ import module namespace wde = "http://acdh.oeaw.ac.at/wde" at 'rest.xqm'; declare variable $q external; declare variable $index external; declare variable $type external; wde:do-query-index(collection("`{$dict}`"), $index, $q, $type) ]`` return wde:eval($query-index-script, map { 'q': $q, 'index': $index, 'type': $type }, 'query-index', true()) };
And the last part you might need is something that is faster than a human when it comes to start requests like node with a bunch of superagent/supertest request promises:
Promise.all(requests) .then(docs => {})
14:24:56.545 127.0.0.1:52592 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.546 127.0.0.1:52593 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.546 127.0.0.1:52595 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.546 127.0.0.1:52594 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.546 127.0.0.1:52596 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.547 127.0.0.1:52597 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.547 127.0.0.1:52598 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.548 127.0.0.1:52599 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.548 127.0.0.1:52600 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.549 127.0.0.1:52601 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.549 127.0.0.1:52602 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.550 127.0.0.1:52603 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.551 127.0.0.1:52604 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.551 127.0.0.1:52605 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.552 127.0.0.1:52606 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.552 127.0.0.1:52607 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.552 127.0.0.1:52608 admin REQUEST [GET] http://localhost:8984/wde/... 14:24:56.554 127.0.0.1:52609 admin REQUEST [GET] http://localhost:8984/wde/...
Best regards Omar Siam
Am 29.11.2017 um 15:42 schrieb Christian Grün:
Hi Omar,
Thanks for reporting your observation.
What do you exactly mean by ”slave jobs”? Do you have some guidance for us to make this reproducible?
Best, Christian
On Wed, Nov 29, 2017 at 3:20 PM, Omar Siam Omar.Siam@oeaw.ac.at wrote:
Hi!
I just observed the following: If I start more RESTXQ requests than the PARALLEL settings allows jobs, which start slave jobs and wait for them, the RESTXQ jobs never finish, hang.
The RESTXQ jobs don't lock anything but start a slave job each which does lock some DB for reading.
It seems to me that the RESTXQ requests are started regardless of the number of parallel jobs already running but the slave jobs never start because they are over the limit of parallel running jobs.
Is this a known situation? Is there some way to stop accepting RESTXQ requests or not starting slave jobs until the parallel limit permits it?
Best regards
Omar Siam