Hello Michael,

You are certainly right that with mixed content and the example you have given here chopping does make a semantic difference. However, you can disable this behaviour so BaseX does what you want. So the only reason I see why one should change the default behaviour would be because the default is not confirmant to some XML standard. However, I can not find any specifics in the spec about which is the expected behaviour, so in my opinion BaseX is doing nothing wrong here. I see that this behaviour might be surprising for some users, but this might as well be the case if it were the other way round. Additionally, if we would change this now it would break application code and unless there is a good reason (i.e. BaseX is actually doing something wrong or non-compliant) I don't see why one should change the default. So if you could point out some details as why this is not conforming behaviour, this would be interesting.

Cheers, Dirk

On Fri, Apr 5, 2013 at 11:15 AM, Michael Piotrowski mxp@cl.uzh.ch wrote:

On 2013-04-05, Michael Seiferle ms@basex.org wrote:

...

As chopping does not change any semantics (at least with regards to what XML thinks of semantically important) but only aesthetics this is enabled by default.

I'm sorry to disagree, but chopping certainly *does* change the semantics--that's precisely why I've argued before that it shouldn't be on by default.

The problem becomes obvious with mixed content, e.g., with chopping enabled

<doc> <p>Lorem ipsum <em>dolor</em> <x>sit</x> amet ...</p> </doc>

becomes

<doc> <p>Lorem ipsum<em>dolor</em><x>sit</x>amet ...</p> </doc>

which is *not* the same, and AFAIKT this is not conforming behavior (and BaseX doesn't honor xml:space either).

I do understand that whitespace chopping as currently implemented is useful for some data-oriented applications, even if it is not conforming, but by default, the behavior should conform to the XML standard.

Best regards

-- Dr.-Ing. Michael Piotrowski, M.A. mxp@cl.uzh.ch Institute of Computational Linguistics, University of Zurich Phone +41 44 63-54313 | OpenPGP public key ID 0x1614A044

• OUT NOW: Natural Language Processing for Historical Texts
• http://morganclaypool.com/doi/abs/10.2200/S00436ED1V01Y201207HLT017

---

BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Michael (other than me :-)) you are obviously right.

— Mit freundlichen Grüßen Michael Seiferle

On Fri, Apr 5, 2013 at 12:29 PM, Michael Piotrowski mxp@cl.uzh.ch wrote:

Dirk, On 2013-04-05, Dirk Kirsten dk@basex.org wrote:

...

You are certainly right that with mixed content and the example you have given here chopping does make a semantic difference. However, you can disable this behaviour so BaseX does what you want. So the only reason I see why one should change the default behaviour would be because the default is not confirmant to some XML standard. However, I can not find any specifics in the spec about which is the expected behaviour, so in my opinion BaseX is doing nothing wrong here.

Well, if you agree that chopping may alter the semantics of a document, wouldn't you agree that applying such a transformation *by default* is a bad idea? With respect to the XML specification, section 2.10 "White Space Handling" says: An XML processor MUST always pass all characters in a document that are not markup through to the application. Yes, the spec is vague wrt. to whitespace handling, and the existence of the xml:space attribute shows that different behaviors--including potentially corrupting ones--are possible. I would therefore interpret the spec to mean that by default all characters should be preserved, but that other behaviors are possible.

...

I see that this behaviour might be surprising for some users, but this might as well be the case if it were the other way round.

No, because their documents wouldn't be corrupted. You can easily remove all whitespace afterwards if you decide you don't need it, but once it's gone, it's gone and cannot be restored. That's the problem.

...

Additionally, if we would change this now it would break application code and unless there is a good reason (i.e. BaseX is actually doing something wrong or non-compliant) I don't see why one should change the default.

Well, I'm not on a crusade or anything, so if you believe that it's a good idea to corrupt, by default, all documents containing mixed content on import, or if this behavior must be kept for compatiblity, so be it. I just wanted to point out that whitespace chopping may, in fact, alter the semantics of documents--it's not as harmless as it may seem. Best regards -- Dr.-Ing. Michael Piotrowski, M.A. mxp@cl.uzh.ch Institute of Computational Linguistics, University of Zurich Phone +41 44 63-54313 | OpenPGP public key ID 0x1614A044

• OUT NOW: Natural Language Processing for Historical Texts
• http://morganclaypool.com/doi/abs/10.2200/S00436ED1V01Y201207HLT017

---

BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

On Fri, 2013-04-05 at 11:31 +0200, Dirk Kirsten wrote:

So if you could point out some details as why this is not conforming behaviour, this would be interesting.

It's a requirement in the XML Spec that the XML parser pass all whitespace back to the application. Some whitespace may be marked as not significant - that is only possible if there's a DTD and the space is in a context where only elements would be valid, not #PCDATA. There's no formal specification, although constructing an XDM instance from an infoset, and constructing an infoset from XML, does not entail discarding these spaces: Chopping internal whitespace nodes in mixed content contexts is not sanctioned by any version of any XML specification, with any setting of xml:space. I think the onus would be on you to justify the non-standard behaviour.

On the other hand I can see its uses too. But I don't want it, and always turn it off with BaseX :-)

Best,

Liam

On Fri, 2013-04-05 at 11:15 +0200, Michael Piotrowski wrote:

On 2013-04-05, Michael Seiferle ms@basex.org wrote: chopping certainly *does* change the semantics--that's precisely why I've argued before that it shouldn't be on by default.

Agreed, but Christian has already said it will be off by default in the next release.

I have seen a commercial SGML formatter that had a similar behaviour used for aircraft manuals, where there was actually a possibility of lives lost and unlimited civil damage liability as a result of numbers run together, but I failed to get the people in charge to understand why it made a difference.

(and BaseX doesn't honor xml:space either).

The latest snapshot does.

Liam

http://www.w3.org/TR/REC-xml/#sec-white-space ...On the other hand, "significant" white space that should be preserved...

So since your parser by default creates significant whitespace where there was none, and removes it where there was, perhaps it could be fixed please, without the user needing to take special steps. Also that would make doc() agree with let:= as I mentioned above.

[Why did this not get posted...] OK but do you admit that this * wrecks HTML jammingwordstogether * wrecks KML jammingcoordinatestogether https://developers.google.com/kml/documentation/kmlreference#gxlatlonquad in fact I bet it wrecks all the other *ML languages. You can compress the whitespace down to one, but any furtherisjustplainnuts.

http://www.w3.org/TR/REC-xml/#sec-white-space ...On the other hand, "significant" white space that should be preserved...

So since your parser by default creates significant whitespace where there was none, and removes it where there was, perhaps it could be fixed please, without the user needing to take special steps. Also that would make doc() agree with let:= as I mentioned above.

http://help.adobe.com/en_US/ColdFusion/9.0/Developing/WSc3ff6d0ea77859461172... "if an XML comment is in the middle of a block of text, the DOM node view represents its position in the text while the basic view does not."

http://www.w3.org/TR/html401/struct/text.html#idx-white_space-2 "Thus, authors, and in particular authoring tools, should write:

<P>We offer free <A>technical support</A> for subscribers.</P>

and not:

<P>We offer free<A> technical support </A>for subscribers.</P>"

So we see that they are different. So the parser should not munch them down the same way.

Hi Jidanni, hi Michael,

our CHOP flag is subject to frequent discussions, which is why we will eventually change the default to FALSE. For now, we are still a little bit resistant, as such a change will change the behavior of existing BaseX applications out there, so we’ll probably combine the switch with the next major release.

For now, you can preserve whitespaces by e.g..

-- adding the line CHOP=false in your .basex configuration file -- using the basex command-line flag -w -- using "set chop false" as first command, or setting the options in any other way described in our Wiki [1].

Hope this helps, Christian

[1] http://docs.basex.org/wiki/Options

The problem is, that you will be aware of this only AFTER you created a DB and worked with it. Unfortunately, users are not informed when creating a DB that they should think about whitespace. And there is no reason a user should assume that creating a DB would semantically change their data. [...]

Yes, I absolutely agree. After all, it’s always tricky to handle issues that have some historical roots.

To improve things a little, I have added support for the xml:space attributes in the latest snapshot [1]. If you add this attribute to an element, all whitespaces in the descendant text nodes will be preserved:

<a xml:space="preserve"> <b> abc </b> </a>

Note that the XML snippet above now contains three text nodes instead of one, which means that the generated database will obviously take more space. If you want to reduce memory consumption, the xml:space attributes should either be added to the relevant elements..

<a> <b xml:space="preserve"> abc </b> </a>

..or the XML indentations should be removed from the document:

<a xml:space="preserve"><b> abc </b></a>

Hope this helps, Christian

[1] http://files.basex.org/releases/latest/

declare option output:omit-xml-declaration "no"; import module namespace j='http://jidanni.org/' at "./j.xq";

causes a never ending nightmare of

Stopped at line 2, column 8 in /home/jidanni/millerliu/air/rcr7.xq: [XPST0003] Unexpected end of query: 'module namespac...'.

The error message should simply tell the user to put the imports first.