SET ADDARCHIVES ON/OFF (I'm not shouting at you, it's just that uppercase convention thing…) Default should be OFF. Gerrit On 2011-04-07 22:31, Christian Grün wrote:

Dear Gerrit,

thanks for your observation. Yes, indeed you've come across a rather hidden feature, but I can understand pretty well that the behavior is not always desired. I've added a GitHub entry to document this:

https://github.com/BaseXdb/basex/issues/60

If you have some preferences how to resolve this issue, feel free to give more feedback.

Christian

On Thu, Apr 7, 2011 at 10:01 PM, Imsieke, Gerrit, le-tex <gerrit.imsieke@le-tex.de> wrote:

...

When I add a directory to a database in 6.6.1 using the plain *.xml createfilter, and when there's a zip file below this directory, the zip file's contents will be indexed, too, as if they reside in the server's working directory. If this is a feature, how can I switch it off?

Gerrit _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke@le-tex.de, http://www.le-tex.de Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt, Dr. Reinhard Vöckler

SET ADDARCHIVES ON/OFF (I'm not shouting at you, it's just that uppercase convention thing…) Default should be OFF.

...got it ;) It's resolved as proposed (well, pretty close, as we decided to stick with ON as default): http://files.basex.org/releases/latest/ http://docs.basex.org/wiki/Options Christian

On 2011-04-07 22:31, Christian Grün wrote:

...

Dear Gerrit,

thanks for your observation. Yes, indeed you've come across a rather hidden feature, but I can understand pretty well that the behavior is not always desired. I've added a GitHub entry to document this:

  https://github.com/BaseXdb/basex/issues/60

If you have some preferences how to resolve this issue, feel free to give more feedback.

Christian

On Thu, Apr 7, 2011 at 10:01 PM, Imsieke, Gerrit, le-tex <gerrit.imsieke@le-tex.de>  wrote:

...

When I add a directory to a database in 6.6.1 using the plain *.xml createfilter, and when there's a zip file below this directory, the zip file's contents will be indexed, too, as if they reside in the server's working directory. If this is a feature, how can I switch it off?

Gerrit _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke@le-tex.de, http://www.le-tex.de

Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930

Geschäftsführer: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt, Dr. Reinhard Vöckler _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Hi Jan, Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven:

I like the feature, this is something, I was dreaming about, as we archive old xml files in zip files and importing them this way would easy the work quite well. It does not work perfecktly as my test is failing on some archives with Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest Error: "..." (Line 1): The processing instruction target matching "[xX][mM][lL]" is not allowed.

Looks like there's an error in one of your XML files, there's probably some whitespace before the XML declaration, that would trigger this error.

Another wish for importing: sometime I have hundreds of xml files and some might be broken xml documents. Currently import always fails. It would be great to have an option "ignore invalid documents" which would allow quick import

Oooo yes! I want that too. Usecase: Sometimes during development some crud ends up in the folder I'm trying to import, f.i. from a bad export from another system. Then I have to clean everything first before importing into BaseX. But really, I don't care if a couple of documents fail to import, it's development and I don't need all of the documents in the DB.

and which would print out file names of those invalid documents.

So for cleaning I use a shell script with a for loop and 'xmllint -noout "$f"'. That automates it, but it would be a "nice to have" anyway if I can tell BaseX to just ignore files with errors, that would also work with .zip files, a shell script doesn't. Hartelijke groet, Huib. -- Drs. Huib Verweij Senior software developer - The Language Archive Max Planck Institute for Psycholinguistics P.O. Box 310 6500 AH Nijmegen The Netherlands t +31-24-3521911 e huib.verwey@mpi.nl w http://www.mpi.nl/

I voted, but isn't it the same issue as #12? Hartelijke groet, Huib Verweij. -- Drs. Huib Verweij Senior software developer - The Language Archive Max Planck Institute for Psycholinguistics P.O. Box 310 6500 AH Nijmegen The Netherlands t +31-24-3521911 e huib.verwey@mpi.nl w http://www.mpi.nl/ Op 8 apr 2011, om 09:10 heeft Jan Vlčinský (CAD) het volgende geschreven:

Hi Huib I added the request "Import xml files - add option to allow ignoring invalid files" to Issues: https://github.com/BaseXdb/basex/issues/61

Vote for it, if you like.

Jan

I voted, but isn't it the same issue as #12?

…dito! As so many have already voted for #61, I've now closed issue #12, and added some comments.

Hartelijke groet, Huib Verweij.

-- Drs. Huib Verweij Senior software developer - The Language Archive Max Planck Institute for Psycholinguistics P.O. Box 310 6500 AH Nijmegen The Netherlands t +31-24-3521911 e huib.verwey@mpi.nl w http://www.mpi.nl/

Op 8 apr 2011, om 09:10 heeft Jan Vlčinský (CAD) het volgende geschreven:

Hi Huib I added the request "Import xml files - add option to allow ignoring invalid files" to Issues: https://github.com/BaseXdb/basex/issues/61 Vote for it, if you like. Jan

_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Fine, so it quite looks as we don't get bored.. ;) I've added two issues in the bug tracker to memorize your observations.

To add some more items to the wish list:

Currently basexclient doesn’t exit with an error code (≠0) when it encounters a file that throws an exception on the server side during import. This is unfortunate in our use case where we issue several svn checkout and BaseX ADD statements in a Makefile. Since there is no non-null exit status, make will always continue, and the errors will not be dealt with unless someone watches the process closely.

In addition, the server process will lock the file that it was unsuccessfully trying to import. I'll have to restart the server in order to make it (=server) unlock the file so that it (=file) can be deleted. So there should be some exception handler that at least closes the file handles.

And, although you (=Christian) expressed your feeling that there are already too many options around, it should be controllable by an option whether an error should be issued or whether corrupt files should be ignored (with a warning) during import, as Jan was asking for.

I don’t think it’s a bad thing to have many options, as long as they a) make sense b) are documented.

Gerrit

On 2011-04-08 08:50, Huib Verweij wrote:

...

Hi Jan,

Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven:

...

I like the feature, this is something, I was dreaming about, as we archive old xml files in zip files and importing them this way would easy the work quite well. It does not work perfecktly as my test is failing on some archives with Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest Error: "..." (Line 1): The processing instruction target matching "[xX][mM][lL]" is not allowed.

Looks like there's an error in one of your XML files, there's probably some whitespace before the XML declaration, that would trigger this error.

...

Another wish for importing: sometime I have hundreds of xml files and some might be broken xml documents. Currently import always fails. It would be great to have an option "ignore invalid documents" which would allow quick import

Oooo yes! I want that too. Usecase: Sometimes during development some crud ends up in the folder I'm trying to import, f.i. from a bad export from another system. Then I have to clean everything first before importing into BaseX. But really, I don't care if a couple of documents fail to import, it's development and I don't need all of the documents in the DB.

...

and which would print out file names of those invalid documents.

So for cleaning I use a shell script with a for loop and 'xmllint -noout "$f"'. That automates it, but it would be a "nice to have" anyway if I can tell BaseX to just ignore files with errors, that would also work with .zip files, a shell script doesn't.

Hartelijke groet,

Huib.

-- Drs. Huib Verweij Senior software developer - The Language Archive Max Planck Institute for Psycholinguistics P.O. Box 310 6500 AH Nijmegen The Netherlands t +31-24-3521911 e huib.verwey@mpi.nl <mailto:huib.verwey@mpi.nl> w http://www.mpi.nl/

_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke@le-tex.de, http://www.le-tex.de

Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930

Geschäftsführer: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt, Dr. Reinhard Vöckler _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Currently basexclient doesn’t exit with an error code (≠0) when it encounters a file that throws an exception on the server side during import. This is unfortunate in our use case where we issue several svn checkout and BaseX ADD statements in a Makefile. Since there is no non-null exit status, make will always continue, and the errors will not be dealt with unless someone watches the process closely.

This has now been resolved for all command-line APIs; note that the Windows batch scripts (basex.bat, etc) were updated as well. Christian