Hi Bridger,
I'm pulling data back from an OAI-PMH endpoint that is slow; i.e. response times are ~1/minute.
I’ve tried the example you have attached (thanks). It’s seems to be much faster. Do you think that’s just my geographic proximity to the Konstanz-based server, or did you use a different setting for your slow tests?
- Is there a better way, using the BaseX GUI (or the command line), to get feedback on a querying process like this?
If you use the BaseX GUI and if you restart a query or run a second one, the first one will be interrupted, so I guess you’ll have similar experiences with IntelliJ. But…
Something... asynchronous, or something clever with builtin functions in the `jobs` or `xquery` modules?
You could create multiple query jobs, which run in parallel, with the jobs:eval function. They will only be interrupted if the IDE is stopped, but your IDE won’t notify you when the queries terminate normally or expectedly.
A promising alternative for you could be xquery:fork-join [1]. In fact we mostly use it for running multiple slow HTTP requests in parallel:
xquery:fork-join( for $segment in 1 to 4 let $url := 'http://url.com/path/' || $segment return function() { http:send-request((), $url) } )
The function will terminate once all parallel requests have returned a response (and the results will be returned in the expected order).
Next, you could run a script multiple times on command line and e.g. assign different arguments:
basex -bvar=1 query.xq basex -bvar=2 query.xq ...
query.xq: declare variable $var external; file:write($var || '.xml', ...)
- If this can be addressed relatively directly with RESTXQ
RESTXQ can be helpful if you write web applications, or if you want to define custom REST endpoints. It’s true that such endpoints can then be called multiple times as well, and will run in parallel as long as the queries don’t write to the same databases [2]. Maybe it’s overkill if you only want to run scripts in parallel, though. The more basic client/server architecture could be an alternative [3]; it can be used similar to the command line solution.
I've attached a simple SSCCE, where the basic idea is: query an API for some data, and get a response like so:
You indicated that you are sending two requests. Is it the first one that’s slow? Does the first response create all input elements for the second requests, or do you have twice the number of requests in total?
Hope this helps, Christian
[1] https://docs.basex.org/wiki/XQuery_Module#xquery:fork-join [2] https://docs.basex.org/wiki/Transaction_Management [3] https://docs.basex.org/wiki/Database_Server