Also, you can have the best of both worlds, saving results in a pipeline if it's helpful for debugging, just running the updates in place once you know it all works well. This is particularly helpful if your data is complex and your datasets are largish.

I am currently merging various sources that were not originally meant to work together, creating a common reference system for fairly complex data. I find it helpful to write a bunch of simple updates that can be executed sequentially, running them with a .bxs file. If I want to see the output after any of these updates, I can do so with an OUTPUT statement. Most of these stages are only useful for debugging purposes, so once a pipeline is working well, I get rid of the OUTPUT statement.

If your data is large, it can also be helpful to have two different statements for importing the data, one that imports everything, another that imports only a small test subset that runs quickly. Comment one of them out and use the other. For rapid development, use the small, quick subset, then run it on the whole dataset once it works.

Jonathan

On Sun, Feb 13, 2022 at 6:19 PM Christian Grün christian.gruen@gmail.com wrote:

Hi Eric,

That’s perfectly feasible. You can bind the dynamic query string to an external variable and invoke it on extracted data via xquery:eval. Here’s one way to do it:

(: db.xml :) <path><to><source><data/></source></to></path>

(: query.xq :) declare variable $query external;

for $data in db:open('db')/path/to/source/data let $updated := xquery:eval($query, map { '': $data }) return replace node $data with $updated

The query could e.g. be invoked via

basex -b query="<new>{ . }</new>" query.xq

Hope this helps, Christian

On Sun, Feb 13, 2022 at 11:26 PM Eric Levy contact@ericlevy.name wrote:

...

I am completely new to BaseX, and trying to conceive of a way to use it to process a compound query that combines an external query with a static one.

The core idea is a very simple application, expressed as an XQuery file, that results in the following sequence of actions:

1. Extraction of data from a database, by some query.
2. Modification of the results of (1), by some query.
3. Persistence of the results from (2), back into the database from which the data was originally extracted.

The modifications given in (2) would be expressed external to the application, supplied as input per invocation, in a separate file or in a command-line parameter.

Thus, the overall operation would entail static components, the fixed parts of the application, represented by (1) and (3), and dynamic components, passed by the invocation, represented by (2).

I realize such a processing pipeline may be achieved through queries supplied in a full application accessing the database by one of the language-specific APIs, but I am trying to consider an approach that requires use only of XQuery statements, without involving a separate programming language or formal build process.

Does the design of BaseX accommodate an application having the structure as described?

I've thought of using a build system or XProc for this. So far, I just create a different .bxs for each of various flows. I am adding one or two people to help me soon, I may get more organized before that happens ...

Jonathan

On Mon, Feb 14, 2022 at 9:49 AM Eliot Kimber eliot.kimber@servicenow.com wrote:

I’m doing more less the same thing. I create various reports and indexes over content stored in a primary database and use small XQuery scripts to perform actions that must be individual transactions (db:create, db:drop, db:alter, etc.), do the heavy data processing and large document loading in temporary databases and then swap the temporary databases with the production databases when appropriate.

I’m doing the orchestration with bash scripts that call the basex client—maybe I’ve made it too complicated? But I didn’t see a way to pass parameters to BaseX scripts, thus shell scripts that just call the basex client to run small .xq files, specifying any needed parameters.

Cheers,

E.

---

*Eliot Kimber*

Sr Staff Content Engineer

O: 512 554 9368

M: 512 554 9368

servicenow.com https://www.servicenow.com

LinkedIn https://www.linkedin.com/company/servicenow | Twitter https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Facebook https://www.facebook.com/servicenow

*From: *BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de on behalf of Jonathan Robie jonathan.robie@gmail.com *Date: *Monday, February 14, 2022 at 8:17 AM *To: *Christian Grün christian.gruen@gmail.com *Cc: *BaseX basex-talk@mailman.uni-konstanz.de *Subject: *Re: [basex-talk] combining queries from mutiple sources

*[External Email]*

Also, you can have the best of both worlds, saving results in a pipeline if it's helpful for debugging, just running the updates in place once you know it all works well. This is particularly helpful if your data is complex and your datasets are largish.

I am currently merging various sources that were not originally meant to work together, creating a common reference system for fairly complex data. I find it helpful to write a bunch of simple updates that can be executed sequentially, running them with a .bxs file. If I want to see the output after any of these updates, I can do so with an OUTPUT statement. Most of these stages are only useful for debugging purposes, so once a pipeline is working well, I get rid of the OUTPUT statement.

If your data is large, it can also be helpful to have two different statements for importing the data, one that imports everything, another that imports only a small test subset that runs quickly. Comment one of them out and use the other. For rapid development, use the small, quick subset, then run it on the whole dataset once it works.

Jonathan

On Sun, Feb 13, 2022 at 6:19 PM Christian Grün christian.gruen@gmail.com wrote:

Hi Eric,

That’s perfectly feasible. You can bind the dynamic query string to an external variable and invoke it on extracted data via xquery:eval. Here’s one way to do it:

(: db.xml :) <path><to><source><data/></source></to></path>

(: query.xq :) declare variable $query external;

for $data in db:open('db')/path/to/source/data let $updated := xquery:eval($query, map { '': $data }) return replace node $data with $updated

The query could e.g. be invoked via

basex -b query="<new>{ . }</new>" query.xq

Hope this helps, Christian

On Sun, Feb 13, 2022 at 11:26 PM Eric Levy contact@ericlevy.name wrote:

...

I am completely new to BaseX, and trying to conceive of a way to use it to process a compound query that combines an external query with a static one.

The core idea is a very simple application, expressed as an XQuery file, that results in the following sequence of actions:

1. Extraction of data from a database, by some query.
2. Modification of the results of (1), by some query.
3. Persistence of the results from (2), back into the database from which the data was originally extracted.

The modifications given in (2) would be expressed external to the application, supplied as input per invocation, in a separate file or in a command-line parameter.

Thus, the overall operation would entail static components, the fixed parts of the application, represented by (1) and (3), and dynamic components, passed by the invocation, represented by (2).

I realize such a processing pipeline may be achieved through queries supplied in a full application accessing the database by one of the language-specific APIs, but I am trying to consider an approach that requires use only of XQuery statements, without involving a separate programming language or formal build process.

Does the design of BaseX accommodate an application having the structure as described?