Importing TSV files with a command file
I have a command file that configures a database, throwing a bunch of sources into various paths. I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path. How do I do that? Jonathan
Hi Jonathan, for $T in file:list($path-to-your-files) let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) return db:add($your-db, $TSV) maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? Best, Bridger On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
I have a command file that configures a database, throwing a bunch of sources into various paths.
I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path.
How do I do that?
Jonathan
On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi Jonathan,
for $T in file:list($path-to-your-files) let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) return db:add($your-db, $TSV)
maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? Best, Bridger
On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
I have a command file that configures a database, throwing a bunch of sources into various paths.
I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path.
How do I do that?
Jonathan
Thanks! I was getting hung up trying to do this in a command. Jonathan On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith < bdysonsmith@gmail.com> wrote:
Hi Jonathan,
for $T in file:list($path-to-your-files) let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) return db:add($your-db, $TSV)
maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? Best, Bridger
On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
I have a command file that configures a database, throwing a bunch of sources into various paths.
I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path.
How do I do that?
Jonathan
Jonathan - apologies for misreading your email! I'm not well-versed in the .bxs COMMAND syntax, but maybe this is closer? I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :) Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things! Best, Bridger test-tsv.bxs ``` CREATE DB test-tsv RUN /home/bridger/test-tsv.xq LIST test-tsv ``` test-tsv.xq ``` for $T in file:children("/home/bridger/tsvs/") let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) let $TSV-NAME := file:name($T) return db:add("test-tsv", $TSV, "/" || $TSV-NAME) ``` note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME) I've attached my two sample TSV files if that helps. On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
Thanks! I was getting hung up trying to do this in a command.
Jonathan
On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith < bdysonsmith@gmail.com> wrote:
Hi Jonathan,
for $T in file:list($path-to-your-files) let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) return db:add($your-db, $TSV)
maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? Best, Bridger
On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
I have a command file that configures a database, throwing a bunch of sources into various paths.
I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path.
How do I do that?
Jonathan
Hi Bridger, hi Jonathan, When using the graphical user interface, you can learn more about commands by a) enabling the Info View panel b) performing a GUI operation, such as adding a directory with specific options, and c) checking the contents of the Info View. Here’s a .bxs solution for adding TSV files: SET CREATEFILTER *.tsv SET PARSER tsv SET CSVPARSER header=true,separator=tab CREATE DB test ADD csv-dir As usual, CREATE and ADD can be combined whenever appropriate: CREATE DB test tsv-dir Hope this helps, Christian On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Jonathan - apologies for misreading your email! I'm not well-versed in the .bxs COMMAND syntax, but maybe this is closer? I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :)
Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things!
Best, Bridger
test-tsv.bxs ``` CREATE DB test-tsv RUN /home/bridger/test-tsv.xq LIST test-tsv ``` test-tsv.xq ``` for $T in file:children("/home/bridger/tsvs/") let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) let $TSV-NAME := file:name($T) return db:add("test-tsv", $TSV, "/" || $TSV-NAME) ``` note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME)
I've attached my two sample TSV files if that helps.
On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
Thanks! I was getting hung up trying to do this in a command.
Jonathan
On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi Jonathan,
for $T in file:list($path-to-your-files) let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) return db:add($your-db, $TSV)
maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? Best, Bridger
On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
I have a command file that configures a database, throwing a bunch of sources into various paths.
I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path.
How do I do that?
Jonathan
Excellent, thanks! Is there a similar incantation for JSON? Jonathan On Tue, Oct 12, 2021 at 6:22 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi Bridger, hi Jonathan,
When using the graphical user interface, you can learn more about commands by
a) enabling the Info View panel b) performing a GUI operation, such as adding a directory with specific options, and c) checking the contents of the Info View.
Here’s a .bxs solution for adding TSV files:
SET CREATEFILTER *.tsv SET PARSER tsv SET CSVPARSER header=true,separator=tab CREATE DB test ADD csv-dir
As usual, CREATE and ADD can be combined whenever appropriate:
CREATE DB test tsv-dir
Hope this helps, Christian
On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Jonathan - apologies for misreading your email! I'm not well-versed in the .bxs
COMMAND syntax, but maybe this is closer?
I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :)
Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things!
Best, Bridger
test-tsv.bxs ``` CREATE DB test-tsv RUN /home/bridger/test-tsv.xq LIST test-tsv ``` test-tsv.xq ``` for $T in file:children("/home/bridger/tsvs/") let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) let $TSV-NAME := file:name($T) return db:add("test-tsv", $TSV, "/" || $TSV-NAME) ``` note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME)
I've attached my two sample TSV files if that helps.
On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie < jonathan.robie@gmail.com> wrote:
Thanks! I was getting hung up trying to do this in a command.
Jonathan
On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <
jonathan.robie@gmail.com> wrote:
On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <
bdysonsmith@gmail.com> wrote:
Hi Jonathan,
for $T in file:list($path-to-your-files) let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(),
'format': 'direct' })
return db:add($your-db, $TSV)
maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? Best, Bridger
On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie < jonathan.robie@gmail.com> wrote:
I have a command file that configures a database, throwing a bunch
of sources into various paths.
I have a set of TSV files. I want to do the equivalent of ADD TO,
pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path.
How do I do that?
Jonathan
Is there a similar incantation for JSON?
Absolutely; it’s *.json and JSONPARSER ;)
On Tue, Oct 12, 2021 at 6:22 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi Bridger, hi Jonathan,
When using the graphical user interface, you can learn more about commands by
a) enabling the Info View panel b) performing a GUI operation, such as adding a directory with specific options, and c) checking the contents of the Info View.
Here’s a .bxs solution for adding TSV files:
SET CREATEFILTER *.tsv SET PARSER tsv SET CSVPARSER header=true,separator=tab CREATE DB test ADD csv-dir
As usual, CREATE and ADD can be combined whenever appropriate:
CREATE DB test tsv-dir
Hope this helps, Christian
On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Jonathan - apologies for misreading your email! I'm not well-versed in the .bxs COMMAND syntax, but maybe this is closer? I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :)
Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things!
Best, Bridger
test-tsv.bxs ``` CREATE DB test-tsv RUN /home/bridger/test-tsv.xq LIST test-tsv ``` test-tsv.xq ``` for $T in file:children("/home/bridger/tsvs/") let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) let $TSV-NAME := file:name($T) return db:add("test-tsv", $TSV, "/" || $TSV-NAME) ``` note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME)
I've attached my two sample TSV files if that helps.
On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
Thanks! I was getting hung up trying to do this in a command.
Jonathan
On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi Jonathan,
for $T in file:list($path-to-your-files) let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) return db:add($your-db, $TSV)
maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? Best, Bridger
On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie@gmail.com> wrote: > > I have a command file that configures a database, throwing a bunch of sources into various paths. > > I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path. > > How do I do that? > > Jonathan
Thanks. And I guess this is the documentation I was looking for: https://docs.basex.org/wiki/Parsers Jonathan On Wed, Oct 13, 2021 at 9:35 AM Christian Grün <christian.gruen@gmail.com> wrote:
Is there a similar incantation for JSON?
Absolutely; it’s *.json and JSONPARSER ;)
On Tue, Oct 12, 2021 at 6:22 PM Christian Grün < christian.gruen@gmail.com> wrote:
Hi Bridger, hi Jonathan,
When using the graphical user interface, you can learn more about
commands by
a) enabling the Info View panel b) performing a GUI operation, such as adding a directory with specific options, and c) checking the contents of the Info View.
Here’s a .bxs solution for adding TSV files:
SET CREATEFILTER *.tsv SET PARSER tsv SET CSVPARSER header=true,separator=tab CREATE DB test ADD csv-dir
As usual, CREATE and ADD can be combined whenever appropriate:
CREATE DB test tsv-dir
Hope this helps, Christian
On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Jonathan - apologies for misreading your email! I'm not well-versed in the .bxs
COMMAND syntax, but maybe this is closer?
I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :)
Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things!
Best, Bridger
test-tsv.bxs ``` CREATE DB test-tsv RUN /home/bridger/test-tsv.xq LIST test-tsv ``` test-tsv.xq ``` for $T in file:children("/home/bridger/tsvs/") let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) let $TSV-NAME := file:name($T) return db:add("test-tsv", $TSV, "/" || $TSV-NAME) ``` note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME)
I've attached my two sample TSV files if that helps.
On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie < jonathan.robie@gmail.com> wrote:
Thanks! I was getting hung up trying to do this in a command.
Jonathan
On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <
jonathan.robie@gmail.com> wrote:
On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <
bdysonsmith@gmail.com> wrote:
> > Hi Jonathan, > > for $T in file:list($path-to-your-files) > let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) > return db:add($your-db, $TSV) > > maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? > Best, > Bridger > > On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie < jonathan.robie@gmail.com> wrote: >> >> I have a command file that configures a database, throwing a bunch of sources into various paths. >> >> I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path. >> >> How do I do that? >> >> Jonathan
[ Unasked for comment from the gallery - this series of "parsers" - which hopefully will be further extended - is the reason because XQuery has a revolutionary character, because it turns the wealth of structured information surrounding us into a uniform and homogeneous substrate of information defined with mathematical precision, in which XPath/XQuery acts as a superconductor. ] Am Mittwoch, 13. Oktober 2021, 20:35:35 MESZ hat Jonathan Robie <jonathan.robie@gmail.com> Folgendes geschrieben: Thanks. And I guess this is the documentation I was looking for: https://docs.basex.org/wiki/Parsers Jonathan On Wed, Oct 13, 2021 at 9:35 AM Christian Grün <christian.gruen@gmail.com> wrote:
Is there a similar incantation for JSON?
Absolutely; it’s *.json and JSONPARSER ;)
On Tue, Oct 12, 2021 at 6:22 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi Bridger, hi Jonathan,
When using the graphical user interface, you can learn more about commands by
a) enabling the Info View panel b) performing a GUI operation, such as adding a directory with specific options, and c) checking the contents of the Info View.
Here’s a .bxs solution for adding TSV files:
SET CREATEFILTER *.tsv SET PARSER tsv SET CSVPARSER header=true,separator=tab CREATE DB test ADD csv-dir
As usual, CREATE and ADD can be combined whenever appropriate:
CREATE DB test tsv-dir
Hope this helps, Christian
On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Jonathan - apologies for misreading your email! I'm not well-versed in the .bxs COMMAND syntax, but maybe this is closer? I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :)
Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things!
Best, Bridger
test-tsv.bxs ``` CREATE DB test-tsv RUN /home/bridger/test-tsv.xq LIST test-tsv ``` test-tsv.xq ``` for $T in file:children("/home/bridger/tsvs/") let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) let $TSV-NAME := file:name($T) return db:add("test-tsv", $TSV, "/" || $TSV-NAME) ``` note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME)
I've attached my two sample TSV files if that helps.
On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
Thanks! I was getting hung up trying to do this in a command.
Jonathan
On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie@gmail.com> wrote:
On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi Jonathan,
for $T in file:list($path-to-your-files) let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) return db:add($your-db, $TSV)
maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? Best, Bridger
On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie@gmail.com> wrote: > > I have a command file that configures a database, throwing a bunch of sources into various paths. > > I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path. > > How do I do that? > > Jonathan
participants (4)
-
Bridger Dyson-Smith -
Christian Grün -
Hans-Juergen Rennau -
Jonathan Robie