xslt:transform function not working with XML Catalog
I've encountered a problem using xslt:transform in to transform some old XML that contains a DTD DOCTYPE system literal pointing to a non-working URI and also uses ENTITYREF attributes to refer to image files. I have the XML Catalog configured correctly using CATFILE. The XSLT is using unparsed-entity-uri() to convert the ENTITYREF into a file name for an href attribute. I've tested the XSLT and the XML Catalog in oXygen and in XSpec, so I know they work. I've tried a few alternatives to get this to work in BaseX and have not arrived at a working solution. The BaseX setup uses BaseX 9.3.3, XML Resolver 1.2, and Saxon HE 9.9.1-7. Passing the XML directly to xslt:transform does not use the XML Catalog for parsing the XM. I've tried making the first parameter of xslt:transform a path to a XML file, an xs:anyURI pointing to the XML file, or string containing the XML. Each time the XML fails to parse due to the non-working URI in the system literal. When I change the system literal to a URI that works the transformation works and the unparsed-entity-uri() function produces the expected file name for the href attribute. I also tried parsing the XML using doc() or fetch:xml() first and then using the parsed XML as the first parameter of xslt:transform(). The doc() or fetch:xml does use the XML Catalog to parse the XML. However, unparsed-entity-uri() function produces an empty string because the DTD information is no longer available. Hoping to be able to spot the problem I looked through the BaseX code. XsltTransform is using CatalogWrapper to set a URIResolver, but is not setting an EntityResolver. SAXWrapper is using CatalogWrapper to set an EntityResolver before parsing XML. It looks like this should work, but it's not. The next alternative is to pre-process the XML using a regular expression to replace the non-working URI in the DOCTYPE system literal with a working URI prior to xslt:transform(). This works, but it seems like this is just working around a problem. Has anyone else encountered this problem and found a better solution? Thanks, Vincent ______________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group 530 Walnut St., Suite 850, Philadelphia, PA 19106 E-Mail: vincent.lizzi@taylorandfrancis.com Phone: 215-606-4221 Web: www.tandfonline.com Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler." Information Classification: General
On Wed, 2020-07-08 at 22:46 +0000, Lizzi, Vincent wrote:
I've encountered a problem using xslt:transform in to transform some old XML that contains a DTD DOCTYPE system literal pointing to a non- working URI and also uses ENTITYREF attributes to refer to image files. I have the XML Catalog configured correctly using CATFILE.
If this is on Linux, using strace can help check which catalog file is being used; you can also turn on debugging in a CatalogManager.properties file containing the line verbosity = 999 (thee file needs to be in your Java classpath). There's also a BaseX pragma, (# db:catfile path/to/catalog.xml #) { transform(...) } You need to turn off the BaseX internal parser. Make sure that the resolver library and of course saxon are in your class path. You may need to add, declare option db:catfile "path/relative/to/cwd/catalog.xml"; to your query. Liam -- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
Hi Liam, Thanks for the reply and suggestions. Based on your suggestion I tried pragmas and strace, and had another go at CatalogManager.properties, but they've not had any effect. (I'm using Windows 10 but was able to run strace in Ubuntu via WSL). This query: try { (# db:catfile 'schemas/catalog.xml' #) (# db:intparse false #) (# db:dtd true #) (# db:chop false #) { xslt:transform('file.xml', 'stylesheet.xsl')//inlinegraphic } } catch * { $err:description } Produces the same error again due to the DTD not being available at the system literal URI. I did try setting verbosity 99 in a CatalogManager.properties file on the classpath, but this did not produce any additional messages. I also tried setting the same properties directly when launching BaseX this did not work either. Specifically, I set the following system properties when launching BaseX, and then used proc:property() in a query to confirm that these system properties were in fact set. 'xml.catalog.verbosity': '99' 'xml.catalog.ignoreMissing': 'no' 'xml.catalog.catalog-class-name': 'org.apache.xml.resolver.Resolver' 'xml.catalog.files': 'schemas/catalog.xml' xml-resolver-1.2.jar and Saxon are definitely on the classpath. Thanks, VIncent _____________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Information Classification: General From: Liam R. E. Quin <liam@fromoldbooks.org> Sent: Wednesday, July 8, 2020 10:28 PM To: Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com>; BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] xslt:transform function not working with XML Catalog On Wed, 2020-07-08 at 22:46 +0000, Lizzi, Vincent wrote:
I've encountered a problem using xslt:transform in to transform some old XML that contains a DTD DOCTYPE system literal pointing to a non- working URI and also uses ENTITYREF attributes to refer to image files. I have the XML Catalog configured correctly using CATFILE.
If this is on Linux, using strace can help check which catalog file is being used; you can also turn on debugging in a CatalogManager.properties file containing the line verbosity = 999 (thee file needs to be in your Java classpath). There's also a BaseX pragma, (# db:catfile path/to/catalog.xml #) { transform(...) } You need to turn off the BaseX internal parser. Make sure that the resolver library and of course saxon are in your class path. You may need to add, declare option db:catfile "path/relative/to/cwd/catalog.xml"; to your query. Liam -- Liam Quin, https://www.delightfulcomputing.com/<https://www.delightfulcomputing.com> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org<http://www.fromoldbooks.org>
Hi Vincent, I feel your pain. Maybe this comment helps: https://github.com/BaseXdb/basex/issues/1793#issuecomment-579134499 (omit the quotes in the pragma). I documented it here, too: https://docs.basex.org/wiki/Catalog_Resolver#Additional_Notes “The catalog location in the pragma can be given relative to the current working directory (the directory that is returned by file:current-dir()) or as an absolute operating system path. The catalog location in the pragma is not an XQuery expression; no concatenation or other operations may occur in the pragma, and the location string must not be surrounded by quotes.” Gerrit On 09.07.2020 06:32, Lizzi, Vincent wrote:
Hi Liam,
Thanks for the reply and suggestions. Based on your suggestion I tried pragmas and strace, and had another go at CatalogManager.properties, but they’ve not had any effect. (I’m using Windows 10 but was able to run strace in Ubuntu via WSL). This query:
try {
(# db:catfile 'schemas/catalog.xml' #)
(# db:intparse false #)
(# db:dtd true #)
(# db:chop false #)
{ xslt:transform('file.xml’, 'stylesheet.xsl')//inlinegraphic }
} catch * { $err:description }
Produces the same error again due to the DTD not being available at the system literal URI.
I did try setting verbosity 99 in a CatalogManager.properties file on the classpath, but this did not produce any additional messages. I also tried setting the same properties directly when launching BaseX this did not work either. Specifically, I set the following system properties when launching BaseX, and then used proc:property() in a query to confirm that these system properties were in fact set.
'xml.catalog.verbosity': '99'
'xml.catalog.ignoreMissing': 'no'
'xml.catalog.catalog-class-name': 'org.apache.xml.resolver.Resolver'
'xml.catalog.files': 'schemas/catalog.xml'
xml-resolver-1.2.jar and Saxon are definitely on the classpath.
Thanks,
VIncent
_____________________________________________
*Vincent M. Lizzi*
Head of Information Standards | Taylor & Francis Group
vincent.lizzi@taylorandfrancis.com <mailto:vincent.lizzi@taylorandfrancis.com>
Information Classification: General
*From:* Liam R. E. Quin <liam@fromoldbooks.org> *Sent:* Wednesday, July 8, 2020 10:28 PM *To:* Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com>; BaseX <basex-talk@mailman.uni-konstanz.de> *Subject:* Re: [basex-talk] xslt:transform function not working with XML Catalog
On Wed, 2020-07-08 at 22:46 +0000, Lizzi, Vincent wrote:
I've encountered a problem using xslt:transform in to transform some old XML that contains a DTD DOCTYPE system literal pointing to a non- working URI and also uses ENTITYREF attributes to refer to image files. I have the XML Catalog configured correctly using CATFILE.
If this is on Linux, using strace can help check which catalog file is being used; you can also turn on debugging in a CatalogManager.properties file containing the line verbosity = 999 (thee file needs to be in your Java classpath).
There's also a BaseX pragma, (# db:catfile path/to/catalog.xml #) { transform(...) }
You need to turn off the BaseX internal parser.
Make sure that the resolver library and of course saxon are in your class path.
You may need to add, declare option db:catfile "path/relative/to/cwd/catalog.xml"; to your query.
Liam
-- Liam Quin, https://www.delightfulcomputing.com/ <https://www.delightfulcomputing.com> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke@le-tex.de, http://www.le-tex.de Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer / Managing Directors: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt
Hi Gerrit, Thank you for the hint! Removing quotes from the pragma did not work in this case. (# db:catfile schemas/catalog.xml #) The catalog file is also configured at the beginning of the query: declare option db:catfile 'schemas/catalog.xml'; This detail about not needing quotes in a pragma is worth remembering though!. Vincent _____________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Information Classification: General From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> On Behalf Of Imsieke, Gerrit, le-tex Sent: Thursday, July 9, 2020 1:18 AM To: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] xslt:transform function not working with XML Catalog Hi Vincent, I feel your pain. Maybe this comment helps: https://github.com/BaseXdb/basex/issues/1793#issuecomment-579134499<https://github.com/BaseXdb/basex/issues/1793#issuecomment-579134499> (omit the quotes in the pragma). I documented it here, too: https://docs.basex.org/wiki/Catalog_Resolver#Additional_Notes<https://docs.basex.org/wiki/Catalog_Resolver#Additional_Notes> "The catalog location in the pragma can be given relative to the current working directory (the directory that is returned by file:current-dir()) or as an absolute operating system path. The catalog location in the pragma is not an XQuery expression; no concatenation or other operations may occur in the pragma, and the location string must not be surrounded by quotes." Gerrit On 09.07.2020 06:32, Lizzi, Vincent wrote:
Hi Liam,
Thanks for the reply and suggestions. Based on your suggestion I tried pragmas and strace, and had another go at CatalogManager.properties, but they've not had any effect. (I'm using Windows 10 but was able to run strace in Ubuntu via WSL). This query:
try {
(# db:catfile 'schemas/catalog.xml' #)
(# db:intparse false #)
(# db:dtd true #)
(# db:chop false #)
{ xslt:transform('file.xml', 'stylesheet.xsl')//inlinegraphic }
} catch * { $err:description }
Produces the same error again due to the DTD not being available at the system literal URI.
I did try setting verbosity 99 in a CatalogManager.properties file on the classpath, but this did not produce any additional messages. I also tried setting the same properties directly when launching BaseX this did not work either. Specifically, I set the following system properties when launching BaseX, and then used proc:property() in a query to confirm that these system properties were in fact set.
'xml.catalog.verbosity': '99'
'xml.catalog.ignoreMissing': 'no'
'xml.catalog.catalog-class-name': 'org.apache.xml.resolver.Resolver'
'xml.catalog.files': 'schemas/catalog.xml'
xml-resolver-1.2.jar and Saxon are definitely on the classpath.
Thanks,
VIncent
_____________________________________________
*Vincent M. Lizzi*
Head of Information Standards | Taylor & Francis Group
vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> <mailto:vincent.lizzi@taylorandfrancis.com>
Information Classification: General
*From:* Liam R. E. Quin <liam@fromoldbooks.org<mailto:liam@fromoldbooks.org>> *Sent:* Wednesday, July 8, 2020 10:28 PM *To:* Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com<mailto:Vincent.Lizzi@taylorandfrancis.com>>; BaseX <basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>> *Subject:* Re: [basex-talk] xslt:transform function not working with XML Catalog
On Wed, 2020-07-08 at 22:46 +0000, Lizzi, Vincent wrote:
I've encountered a problem using xslt:transform in to transform some old XML that contains a DTD DOCTYPE system literal pointing to a non- working URI and also uses ENTITYREF attributes to refer to image files. I have the XML Catalog configured correctly using CATFILE.
If this is on Linux, using strace can help check which catalog file is being used; you can also turn on debugging in a CatalogManager.properties file containing the line verbosity = 999 (thee file needs to be in your Java classpath).
There's also a BaseX pragma, (# db:catfile path/to/catalog.xml #) { transform(...) }
You need to turn off the BaseX internal parser.
Make sure that the resolver library and of course saxon are in your class path.
You may need to add, declare option db:catfile "path/relative/to/cwd/catalog.xml"; to your query.
Liam
-- Liam Quin, https://www.delightfulcomputing.com/<https://www.delightfulcomputing.com> <https://www.delightfulcomputing.com<https://www.delightfulcomputing.com>> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org<http://www.fromoldbooks.org>
-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke@le-tex.de<mailto:gerrit.imsieke@le-tex.de>, http://www.le-tex.de<http://www.le-tex.de> Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer / Managing Directors: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt
Is the catalog schemas/catalog.xml residing in file:current-dir()? On 09.07.2020 16:02, Lizzi, Vincent wrote:
Hi Gerrit,
Thank you for the hint! Removing quotes from the pragma did not work in this case.
(# db:catfile schemas/catalog.xml #)
The catalog file is also configured at the beginning of the query:
declare option db:catfile 'schemas/catalog.xml';
This detail about not needing quotes in a pragma is worth remembering though!.
Vincent
Yes, which is confirmed by file:exists(file:current-dir() || 'schemas/catalog.xml') returns true. Thanks, Vincent _____________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group vincent.lizzi@taylorandfrancis.com Information Classification: General -----Original Message----- From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> On Behalf Of Imsieke, Gerrit, le-tex Sent: Thursday, July 9, 2020 10:13 AM To: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] xslt:transform function not working with XML Catalog Is the catalog schemas/catalog.xml residing in file:current-dir()? On 09.07.2020 16:02, Lizzi, Vincent wrote:
Hi Gerrit,
Thank you for the hint! Removing quotes from the pragma did not work in this case.
(# db:catfile schemas/catalog.xml #)
The catalog file is also configured at the beginning of the query:
declare option db:catfile 'schemas/catalog.xml';
This detail about not needing quotes in a pragma is worth remembering though!.
Vincent
On Thu, 2020-07-09 at 04:32 +0000, Lizzi, Vincent wrote:
Hi Liam,
Thanks for the reply and suggestions. Based on your suggestion I tried pragmas and strace, and had another go at CatalogManager.properties, but they've not had any effect.
use, strace -f java.... >& hugelogfile.txt and after, grep -i catalogmanager.properties hugelogfile.txt and you should see where it's looking. If it doesn't look for that file, check to see if it opened the jar file containing the resolver. If you're running BaseX from Oxygen, Oxygen needs to have it in its classpath too i think. Also, of course, see if the catalog file is actually being opened! I actually wrote some of the code in BaseX that makes XML catalogs work with transform(), or provided a rough draft that Christian improved :), and debugging it was... interesting. I'd also try an absolute path for the catalog file - if you are using the BaseX server, relative paths will be relative to the directory (folder) where the server itself is running. (and of course the server needs the resolver in its classpath). Messages from the catalog manager seem to go (oddly) to standard output interleaved with any XML output. The command-line i used for testing this (well, one of the tests) was, R=$HOME/lib/xmlcatalog/xml-commons-resolver-1.2/resolver.jar MAIN=$HOME/packages/basex/basex java -Dxml.catalog.files=saxlog.xml -D' http://saxon.sf.net/feature/uriResolverClass=org.apache.xml.resolver.tools.C...' -cp $R/resolver.jar:/home/lee/packages/basex/basex/BaseX.jar:$MAIN/lib/cust om/*:$MAIN/lib/*: org.basex.BaseX try.xq (Saxon was in $MAIN)
-- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
Hi Liam, Thanks for the helpful suggestions. After trying everything you suggested and then also trying a few of Saxon's configuration options, unfortunately I'm still having the same problem. Trying a shell script that contains the following: MAIN="$( cd -P "$(dirname "$FILE")/../basex" && pwd )" CP=$MAIN/BaseX.jar:$MAIN/lib/custom/*:$MAIN/lib/*:$CLASSPATH echo 1 Saxon java -cp "$CP" net.sf.saxon.Transform -s:input1.xml -xsl:transform.xsl -catalog:schemas/catalog.xml echo 2 BaseX transform java -cp "$CP" org.basex.BaseX -q"(# db:catfile schemas/catalog.xml #) (# db:intparse false #) (# db:dtd true #) (# db:chop false #) { xslt:transform('input1.xml', 'transform.xsl') }" echo 3 BaseX transform with Saxon features configured java -Dhttp://saxon.sf.net/feature/entityResolverClass=org.apache.xml.resolver.tool... -Dhttp://saxon.sf.net/feature/uriResolverClass=org.apache.xml.resolver.tools.C... -cp "$CP" org.basex.BaseX -q"(# db:catfile schemas/catalog.xml #) (# db:intparse false #) (# db:dtd true #) (# db:chop false #) { xslt:transform('input1.xml', 'transform.xsl') }" echo 4 BaseX doc to show XML Catalog is configured correctly to parse XML java -cp "$CP" org.basex.BaseX -q"(# db:catfile schemas/catalog.xml #) (# db:intparse false #) (# db:dtd true #) (# db:chop false #) { doc('input1.xml') }" The classpath includes BaseX 9.3.3, Saxon HE 9.9, xml-resolver-1.2.jar, and CatalogManager.properties 1. The transformation works in Saxon and uses the catalog file to locate the DTD when parsing the XML input1.xml. 2. The BaseX xslt:transform should work the same as #1, but fails because the DTD cannot be read 3. Adding Saxon configuration for Entity Resolver Class and URI Resolve Class did not help 4. Simply parsing the XML using doc() in BaseX with the same configuration shows that the XML catalog is configured correctly within BaseX Using strace -f, the log shows that BaseX xslt:transform is reading the catalog.xml file from disk, and then is trying (and failing) to read the DTD from the non-working URIL. This might be a bug in xslt:transform, so the workaround of using a regular expression replace on the DOCTYPE system URI is probably the practical solution. Many thanks, Vincent _____________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Information Classification: General From: Liam R. E. Quin <liam@fromoldbooks.org> Sent: Thursday, July 9, 2020 12:55 PM To: Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com>; BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] xslt:transform function not working with XML Catalog On Thu, 2020-07-09 at 04:32 +0000, Lizzi, Vincent wrote:
Hi Liam,
Thanks for the reply and suggestions. Based on your suggestion I tried pragmas and strace, and had another go at CatalogManager.properties, but they've not had any effect.
use, strace -f java.... >& hugelogfile.txt and after, grep -i catalogmanager.properties hugelogfile.txt and you should see where it's looking. If it doesn't look for that file, check to see if it opened the jar file containing the resolver. If you're running BaseX from Oxygen, Oxygen needs to have it in its classpath too i think. Also, of course, see if the catalog file is actually being opened! I actually wrote some of the code in BaseX that makes XML catalogs work with transform(), or provided a rough draft that Christian improved :), and debugging it was... interesting. I'd also try an absolute path for the catalog file - if you are using the BaseX server, relative paths will be relative to the directory (folder) where the server itself is running. (and of course the server needs the resolver in its classpath). Messages from the catalog manager seem to go (oddly) to standard output interleaved with any XML output. The command-line i used for testing this (well, one of the tests) was, R=$HOME/lib/xmlcatalog/xml-commons-resolver-1.2/resolver.jar MAIN=$HOME/packages/basex/basex java -Dxml.catalog.files=saxlog.xml -D' http://saxon.sf.net/feature/uriResolverClass=org.apache.xml.resolver.tools.CatalogResolver<http://saxon.sf.net/feature/uriResolverClass=org.apache.xml.resolver.tools.CatalogResolver>' -cp $R/resolver.jar:/home/lee/packages/basex/basex/BaseX.jar:$MAIN/lib/cust om/*:$MAIN/lib/*: org.basex.BaseX try.xq (Saxon was in $MAIN)
-- Liam Quin, https://www.delightfulcomputing.com/<https://www.delightfulcomputing.com> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org<http://www.fromoldbooks.org>
Hi Vincent, can you show us your catalog? Since you mention that it chokes on finding the DTD, it might be that you need rewriteSystem instead of rewriteURI for the DTD locations. Also if you don't resolve by public ID and refer to the DTD by relative path, this path will be made absolute before being catalog-resolved, so instead of <system> your can use <systemSuffix> in order to only match the tail of the path. On the other hand, since you say it’s running with standalone Saxon, the same DTD resolution issues should be expected there. It may well be the case that our efforts to make DTD resolution available to xslt:transform() only focused on supporting xsl:import and xsl:include while not passing the resolver to the doc() function. Maybe Liam can investigate this in more depth. I then suggest to ask for a budget at Taylor & Francis. We paid Liam to explore and enable the use of catalogs for xsl:import and xsl:include, and he dug through the mess of different interfaces etc. successfully for this limited but important use case. So he is *the* expert in this field and I’d like to warmly recommend paying him so that he can explore and fix the issue. Gerrit On 10.07.2020 07:55, Lizzi, Vincent wrote:
Hi Liam,
Thanks for the helpful suggestions. After trying everything you suggested and then also trying a few of Saxon’s configuration options, unfortunately I’m still having the same problem. Trying a shell script that contains the following:
MAIN="$( cd -P "$(dirname "$FILE")/../basex" && pwd )"
CP=$MAIN/BaseX.jar:$MAIN/lib/custom/*:$MAIN/lib/*:$CLASSPATH
echo 1 Saxon
java -cp "$CP" net.sf.saxon.Transform -s:input1.xml -xsl:transform.xsl -catalog:schemas/catalog.xml
echo 2 BaseX transform
java -cp "$CP" org.basex.BaseX -q"(# db:catfile schemas/catalog.xml #) (# db:intparse false #) (# db:dtd true #) (# db:chop false #) { xslt:transform('input1.xml', 'transform.xsl') }"
echo 3 BaseX transform with Saxon features configured
java -Dhttp://saxon.sf.net/feature/entityResolverClass=org.apache.xml.resolver.tool... -Dhttp://saxon.sf.net/feature/uriResolverClass=org.apache.xml.resolver.tools.C... -cp "$CP" org.basex.BaseX -q"(# db:catfile schemas/catalog.xml #) (# db:intparse false #) (# db:dtd true #) (# db:chop false #) { xslt:transform('input1.xml', 'transform.xsl') }"
echo 4 BaseX doc to show XML Catalog is configured correctly to parse XML
java -cp "$CP" org.basex.BaseX -q"(# db:catfile schemas/catalog.xml #) (# db:intparse false #) (# db:dtd true #) (# db:chop false #) { doc('input1.xml') }"
The classpath includes BaseX 9.3.3, Saxon HE 9.9, xml-resolver-1.2.jar, and CatalogManager.properties
1. The transformation works in Saxon and uses the catalog file to locate the DTD when parsing the XML input1.xml. 2. The BaseX xslt:transform should work the same as #1, but fails because the DTD cannot be read 3. Adding Saxon configuration for Entity Resolver Class and URI Resolve Class did not help 4. Simply parsing the XML using doc() in BaseX with the same configuration shows that the XML catalog is configured correctly within BaseX
Using strace -f, the log shows that BaseX xslt:transform is reading the catalog.xml file from disk, and then is trying (and failing) to read the DTD from the non-working URIL.
This might be a bug in xslt:transform, so the workaround of using a regular expression replace on the DOCTYPE system URI is probably the practical solution.
Many thanks,
Vincent
_____________________________________________
*Vincent M. Lizzi*
Head of Information Standards | Taylor & Francis Group
vincent.lizzi@taylorandfrancis.com <mailto:vincent.lizzi@taylorandfrancis.com>
Information Classification: General
*From:* Liam R. E. Quin <liam@fromoldbooks.org> *Sent:* Thursday, July 9, 2020 12:55 PM *To:* Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com>; BaseX <basex-talk@mailman.uni-konstanz.de> *Subject:* Re: [basex-talk] xslt:transform function not working with XML Catalog
On Thu, 2020-07-09 at 04:32 +0000, Lizzi, Vincent wrote:
Hi Liam,
Thanks for the reply and suggestions. Based on your suggestion I tried pragmas and strace, and had another go at CatalogManager.properties, but they've not had any effect.
use, strace -f java.... >& hugelogfile.txt and after, grep -i catalogmanager.properties hugelogfile.txt and you should see where it's looking. If it doesn't look for that file, check to see if it opened the jar file containing the resolver.
If you're running BaseX from Oxygen, Oxygen needs to have it in its classpath too i think.
Also, of course, see if the catalog file is actually being opened!
I actually wrote some of the code in BaseX that makes XML catalogs work with transform(), or provided a rough draft that Christian improved :), and debugging it was... interesting.
I'd also try an absolute path for the catalog file - if you are using the BaseX server, relative paths will be relative to the directory (folder) where the server itself is running. (and of course the server needs the resolver in its classpath).
Messages from the catalog manager seem to go (oddly) to standard output interleaved with any XML output.
The command-line i used for testing this (well, one of the tests) was,
R=$HOME/lib/xmlcatalog/xml-commons-resolver-1.2/resolver.jar MAIN=$HOME/packages/basex/basex
java -Dxml.catalog.files=saxlog.xml -D' http://saxon.sf.net/feature/uriResolverClass=org.apache.xml.resolver.tools.C...' -cp $R/resolver.jar:/home/lee/packages/basex/basex/BaseX.jar:$MAIN/lib/cust om/*:$MAIN/lib/*: org.basex.BaseX try.xq
(Saxon was in $MAIN)
-- Liam Quin, https://www.delightfulcomputing.com/ <https://www.delightfulcomputing.com> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke@le-tex.de, http://www.le-tex.de Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer / Managing Directors: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt
On 10.07.2020 08:23, Imsieke, Gerrit, le-tex wrote:
It may well be the case that our efforts to make DTD resolution available to xslt:transform() only focused on supporting xsl:import and xsl:include while not passing the resolver to the doc() function.
s/DTD resolution/catalog resolution/
On Fri, 2020-07-10 at 08:23 +0200, Imsieke, Gerrit, le-tex wrote:
I’d like to warmly recommend paying him so that he can explore and fix the issue.
:-) Thank you for the recommendation! The trick is to find the resolver output from setting verbose, as then you will see the strings that are being sent to the resolver. If you have your strace log, you can see which CatalogManager.properties files were read, set verbose=999 in one, and then look for output, but it's possible it's getting eaten by saxon as the buffered output of XSLT. I should look into being able to capture those messages separately. Liam -- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
Gerrit and Liam, Thanks for that background information and the recommendation for sponsoring development. I tried adding rewriteSystem to the XML Catalog but it did not help. The XML Catalog already uses systemSuffix. It is good to see that the option for sponsoring development is available and has already resulted in improvements to XML Catalog support in BaseX. As you pointed out that the initial work on catalog support in xslt:transform focused on supporting xsl:import and xsl:include so the scenario for using the catalog resolver when parsing XML was probably not in scope. I've logged a ticket with a small self-contained example: https://github.com/BaseXdb/basex/issues/1903 For now, the workaround to replace the URI in the DOCTYPE using a regular expression on the XML before xslt:transform() looks like it will work. Many thanks, Vincent _____________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Information Classification: General From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> On Behalf Of Liam R. E. Quin Sent: Friday, July 10, 2020 12:56 PM To: Imsieke, Gerrit, le-tex <gerrit.imsieke@le-tex.de>; basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] xslt:transform function not working with XML Catalog On Fri, 2020-07-10 at 08:23 +0200, Imsieke, Gerrit, le-tex wrote:
I'd like to warmly recommend paying him so that he can explore and fix the issue.
:-) Thank you for the recommendation! The trick is to find the resolver output from setting verbose, as then you will see the strings that are being sent to the resolver. If you have your strace log, you can see which CatalogManager.properties files were read, set verbose=999 in one, and then look for output, but it's possible it's getting eaten by saxon as the buffered output of XSLT. I should look into being able to capture those messages separately. Liam -- Liam Quin, https://www.delightfulcomputing.com/<https://www.delightfulcomputing.com> Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org<http://www.fromoldbooks.org>
participants (3)
-
Imsieke, Gerrit, le-tex -
Liam R. E. Quin -
Lizzi, Vincent