Hello-
Just want to start out with KUDOS on the project, contributors, and technology. Amazing.
We are importing 300K+ xml & binary files from a file system with 30k in nested hierarchical folder structures into basex. The auto-magical capabilities of basex to suck in this data (and the speed at which is does it) is kick-a%$.
Question: All of the 300k+ xml files do NOT have a file extension. Is there a way to have BaseX default to xml if there is no file type? Is there perhaps some other way to handle this other than recursively renaming all the files?
Fyi -Here is the simple batch file I am using to import the data:
java -Xmx1024m -cp %devbase%\basex65\BaseX.jar org.basex.BaseX -Vc "drop db metadata"
java -Xmx1024m -cp %devbase%\basex65\BaseX.jar org.basex.BaseX -Vc "create db metadata; alter user admin xxxxxxx"
java -Xmx1024m -cp %devbase%\basex65\BaseX.jar org.basex.BaseX -Vc "open metadata; SET INTPARSE ON; add to \set1 %devbase%\code\basex\import\webcat"
java -Xmx1024m -cp %devbase%\basex65\BaseX.jar org.basex.BaseX -Vc "open metadata; SET INTPARSE ON; add to \set2 %devbase%\code\basex\import\rpd"
java -Xmx1024m -cp %devbase%\basex65\BaseX.jar org.basex.BaseX -Vc "open metadata; create index text; create index attribute; create index fulltext; create index path"
Regards,
Chad
chad@helmetcameracentral.com
Hi Chad,
Just want to start out with KUDOS on the project, contributors, and technology. Amazing.
thanks for the kudos, always welcome!
Question: All of the 300k+ xml files do NOT have a file extension. Is there a way to have BaseX default to xml if there is no file type? Is there perhaps some other way to handle this other than recursively renaming all the files?
You're probably looking for the "createfilter" option, which allows you to define a glob syntax for fitering the input files; please try this:
basex -Vc "create db metadata; alter user admin xxxxxxx" basex -Vc "open metadata; set createfilter *; add to \set1 ...
Some more information may be found here:
http://docs.basex.org/wiki/Options http://docs.basex.org/wiki/CreateCollection
Hope this helps, Christian ___________________________
Christian Grün Uni KN, Box 188 78457 Konstanz, Germany http://www.inf.uni-konstanz.de/~gruen
Thanks Christian. Just tried that. The xml files do not have a file extension such as "file1" "file2", so * would work, but there are also *.ATR files as well - which are binary format. When using *, BaseX chokes on the binary (Error: "root.atr" (Line 1): No text allowed before root tag.). Also, tried variations of FORCECREATE, PARSER.
So, tried using a standard wildcard like "*." which returns all files without an extension. This resulted in BaseX not loading any documents.
I do have some work-arounds for now...
Thx/chad
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Tuesday, February 01, 2011 12:40 PM To: Chad Lancour Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Force default file type to XML during "add to" command?
Hi Chad,
Just want to start out with KUDOS on the project, contributors, and technology. Amazing.
thanks for the kudos, always welcome!
Question: All of the 300k+ xml files do NOT have a file extension. Is
there
a way to have BaseX default to xml if there is no file type? Is there perhaps some other way to handle this other than recursively renaming all the files?
You're probably looking for the "createfilter" option, which allows you to define a glob syntax for fitering the input files; please try this:
basex -Vc "create db metadata; alter user admin xxxxxxx" basex -Vc "open metadata; set createfilter *; add to \set1 ...
Some more information may be found here:
http://docs.basex.org/wiki/Options http://docs.basex.org/wiki/CreateCollection
Hope this helps, Christian ___________________________
Christian Grün Uni KN, Box 188 78457 Konstanz, Germany http://www.inf.uni-konstanz.de/~gruen
Thanks Chad, I wasn't aware of the "*." syntax. Unfortunately, Java (version <= 6) has no class for parsing the glob syntax, but I've now extended our parser syntax to also support patterns with trailing dots. Feel free to check out the sources, or give me a message to send you a jar build.
Best, Christian
On Wed, Feb 2, 2011 at 2:38 AM, Chad Lancour chad@helmetcameracentral.com wrote:
Thanks Christian. Just tried that. The xml files do not have a file extension such as "file1" "file2", so * would work, but there are also *.ATR files as well - which are binary format. When using *, BaseX chokes on the binary (Error: "root.atr" (Line 1): No text allowed before root tag.). Also, tried variations of FORCECREATE, PARSER.
So, tried using a standard wildcard like "*." which returns all files without an extension. This resulted in BaseX not loading any documents.
I do have some work-arounds for now...
Thx/chad
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Tuesday, February 01, 2011 12:40 PM To: Chad Lancour Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Force default file type to XML during "add to" command?
Hi Chad,
Just want to start out with KUDOS on the project, contributors, and technology. Amazing.
thanks for the kudos, always welcome!
Question: All of the 300k+ xml files do NOT have a file extension. Is
there
a way to have BaseX default to xml if there is no file type? Is there perhaps some other way to handle this other than recursively renaming all the files?
You're probably looking for the "createfilter" option, which allows you to define a glob syntax for fitering the input files; please try this:
basex -Vc "create db metadata; alter user admin xxxxxxx" basex -Vc "open metadata; set createfilter *; add to \set1 ...
Some more information may be found here:
http://docs.basex.org/wiki/Options http://docs.basex.org/wiki/CreateCollection
Hope this helps, Christian ___________________________
Christian Grün Uni KN, Box 188 78457 Konstanz, Germany http://www.inf.uni-konstanz.de/~gruen
…btw, we have just extended our download page with the option to download a latest build; see:
http://basex.org/products/download/all-downloads/
Christian ___________________________
On Wed, Feb 2, 2011 at 10:57 AM, Christian Grün christian.gruen@gmail.com wrote:
Thanks Chad, I wasn't aware of the "*." syntax. Unfortunately, Java (version <= 6) has no class for parsing the glob syntax, but I've now extended our parser syntax to also support patterns with trailing dots. Feel free to check out the sources, or give me a message to send you a jar build.
Best, Christian
On Wed, Feb 2, 2011 at 2:38 AM, Chad Lancour chad@helmetcameracentral.com wrote:
Thanks Christian. Just tried that. The xml files do not have a file extension such as "file1" "file2", so * would work, but there are also *.ATR files as well - which are binary format. When using *, BaseX chokes on the binary (Error: "root.atr" (Line 1): No text allowed before root tag.). Also, tried variations of FORCECREATE, PARSER.
So, tried using a standard wildcard like "*." which returns all files without an extension. This resulted in BaseX not loading any documents.
I do have some work-arounds for now...
Thx/chad
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Tuesday, February 01, 2011 12:40 PM To: Chad Lancour Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Force default file type to XML during "add to" command?
Hi Chad,
Just want to start out with KUDOS on the project, contributors, and technology. Amazing.
thanks for the kudos, always welcome!
Question: All of the 300k+ xml files do NOT have a file extension. Is
there
a way to have BaseX default to xml if there is no file type? Is there perhaps some other way to handle this other than recursively renaming all the files?
You're probably looking for the "createfilter" option, which allows you to define a glob syntax for fitering the input files; please try this:
basex -Vc "create db metadata; alter user admin xxxxxxx" basex -Vc "open metadata; set createfilter *; add to \set1 ...
Some more information may be found here:
http://docs.basex.org/wiki/Options http://docs.basex.org/wiki/CreateCollection
Hope this helps, Christian ___________________________
Christian Grün Uni KN, Box 188 78457 Konstanz, Germany http://www.inf.uni-konstanz.de/~gruen
Christian What filter/parser shall I use to import other people minds? Yesterday I was about to propose something like that (download page with latest builds) and today you deliver it. Awesome.
Jan
2011/2/2 Christian Grün christian.gruen@gmail.com
…btw, we have just extended our download page with the option to download a latest build; see:
http://basex.org/products/download/all-downloads/
Christian ___________________________
On Wed, Feb 2, 2011 at 10:57 AM, Christian Grün christian.gruen@gmail.com wrote:
Thanks Chad, I wasn't aware of the "*." syntax. Unfortunately, Java (version <= 6) has no class for parsing the glob syntax, but I've now extended our parser syntax to also support patterns with trailing dots. Feel free to check out the sources, or give me a message to send you a jar build.
Best, Christian
On Wed, Feb 2, 2011 at 2:38 AM, Chad Lancour chad@helmetcameracentral.com wrote:
Thanks Christian. Just tried that. The xml files do not have a file extension such as "file1" "file2", so * would work, but there are also
*.ATR
files as well - which are binary format. When using *, BaseX chokes on
the
binary (Error: "root.atr" (Line 1): No text allowed before root tag.).
Also,
tried variations of FORCECREATE, PARSER.
So, tried using a standard wildcard like "*." which returns all files without an extension. This resulted in BaseX not loading any documents.
I do have some work-arounds for now...
Thx/chad
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Tuesday, February 01, 2011 12:40 PM To: Chad Lancour Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Force default file type to XML during "add to" command?
Hi Chad,
Just want to start out with KUDOS on the project, contributors, and technology. Amazing.
thanks for the kudos, always welcome!
Question: All of the 300k+ xml files do NOT have a file extension. Is
there
a way to have BaseX default to xml if there is no file type? Is there perhaps some other way to handle this other than recursively renaming
all
the files?
You're probably looking for the "createfilter" option, which allows you to define a glob syntax for fitering the input files; please try this:
basex -Vc "create db metadata; alter user admin xxxxxxx" basex -Vc "open metadata; set createfilter *; add to \set1 ...
Some more information may be found here:
http://docs.basex.org/wiki/Options http://docs.basex.org/wiki/CreateCollection
Hope this helps, Christian ___________________________
Christian Grün Uni KN, Box 188 78457 Konstanz, Germany http://www.inf.uni-konstanz.de/~gruen
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
basex-talk@mailman.uni-konstanz.de
-
Chad Lancour -
Christian Grün -
Jan Vlčinský (CAD)