parse-json() function reorder keys in the resulting map{}
Hi team, Found one "issue" with key ordering in the resulting map. Need to clarify, is it normal behavior or a bug. json:parse('{"b":1, "a":3}', { 'format': 'xquery' }) returns {"a":3.0e0,"b":1.0e0} but json:parse('{"b":1, "a":3}') <json type="object"><b type="number">1</b><a type="number">3</a></json> As you see, the standard function returns keys not in the order as it was set in the original JSON string. I didn't find any requirement in xQuery spec that the resulting map entries should be sorted by the key name. Is it a bug or a feature? If a feature I would be really happy to see one additional param to preserve key ordering. Thanks! -- s0rr0w
Am 16.12.2024 um 13:42 schrieb Oleksandr Shpak:
Hi team,
Found one "issue" with key ordering in the resulting map. Need to clarify, is it normal behavior or a bug.
json:parse('{"b":1, "a":3}', { 'format': 'xquery' }) returns {"a":3.0e0,"b":1.0e0}
but json:parse('{"b":1, "a":3}') <json type="object"><b type="number">1</b><a type="number">3</a></json>
As you see, the standard function returns keys not in the order as it was set in the original JSON string. I didn't find any requirement in xQuery spec that the resulting map entries should be sorted by the key name.
Is it a bug or a feature? If a feature I would be really happy to see one additional param to preserve key ordering.
There is a big debate in the working group developing XQuery, XPath and XSLT 4 on whether to introduce a new map type that preserves key insertion/creation order or whether even change the existing map type to do that as an option or by default. Interestingly enough only recently Christian Grün announced that the BaseX fiddle has experimental support for maps preserving key creation order so there <https://fiddle.basex.org/?share=%28%27query%21%27let.1*b6157a638%2C0.2*a6357b618Areturn0+%7B91%2C.2%7D+%3D%27%3A%3E+serialize%7B%28%22method%22%3A%22C%225%22indent%22%3Atrue%7B%7D%294%21%27%27%7Emode%21%27XQuery+%7BBaseX4Type%21%27xml%27%29*+%3A%3D+parse-C%7B%22%287.+90A+++4%7D%27%7Econtext5%2C+6%5C%217%5C%278%29%22%7D9%24mapA%5CnCjson%01CA9876540.*_>, with standard XPath functions, I currently get the result { "b": 1, "a": 3 } { "a": 3, "b": 1 } for let $map1 := parse-json('{"b":1, "a":3}'), $map2 := parse-json('{"a":3, "b":1}') return ($map1, $map2) =!> serialize({'method':'json', 'indent':true()}) In terms of the current version 3.1 of XPath and XQuery the XDM 3.1 map type basically doesn't prescribe any order of map/map entries so for that version it is a feature that you can't control or preserve the order. But as I said, in the future it might change and the BaseX fiddle with some BaseX trunk or 12 implementation already has experimental support. I don't know about the BaseX specific functions you seem to be using, wait for Christian or someone else of BaseX to chime in.
Am 16.12.2024 um 14:15 schrieb Martin Honnen:
There is a big debate in the working group developing XQuery, XPath and XSLT 4 on whether to introduce a new map type that preserves key insertion/creation order
Hi Martin, I found even more weird situation parse-json('{"1:aa": 1, "2:bb": 2}') returns {"2:bb":2.0e0,"1:aa":1.0e0} Seems like order is inversed But, parse-json('{"aa": 1, "b": 2, "ab": 3}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0} and, finally parse-json('{"aa": 1, "b": 2, "ab": 3, "1:aa": 1, "2:bb": 2}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0,"2:bb":2.0e0,"1:aa":1.0e0} 🤯 It seems like a bug to me ;) If we are talking about the JSON standard, not xQuery, devs will expect the function which behaves in the same way as it does in many other languages. On Mon, Dec 16, 2024 at 3:31 PM Martin Honnen <martin.honnen@gmx.de> wrote:
Am 16.12.2024 um 14:15 schrieb Martin Honnen:
There is a big debate in the working group developing XQuery, XPath and XSLT 4 on whether to introduce a new map type that preserves key insertion/creation order
-- s0rr0w
Am 16.12.2024 um 15:19 schrieb Oleksandr Shpak:
Hi Martin,
I found even more weird situation parse-json('{"1:aa": 1, "2:bb": 2}') returns {"2:bb":2.0e0,"1:aa":1.0e0}
Seems like order is inversed But, parse-json('{"aa": 1, "b": 2, "ab": 3}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0}
and, finally parse-json('{"aa": 1, "b": 2, "ab": 3, "1:aa": 1, "2:bb": 2}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0,"2:bb":2.0e0,"1:aa":1.0e0}
🤯
It seems like a bug to me ;)
As I said, in XPath 3.1 and XQuery 3.1 maps are unordered collections, you might get any order when parsing a particular JSON into an XDM 3.1 map. If it is any help, in the BaseX fiddle you probably get/see the order preservation you want/expect <https://fiddle.basex.org/?share=%28%27query%21%275.**8.A6%21%27%27%7Emode%21%27XQuery+%7BBaseX6Type%21%27xml%27%29*A%7D%2C%5Cn579b024ab03.71%3A92%3Abb020%5C%21+4875parse-json%7B%22%286%7D%27%7Econtext7%5C%278%2C+9aa014A%29%22%01A9876540.*_>.
If we are talking about the JSON standard, not xQuery, devs will expect the function which behaves in the same way as it does in many other languages.
Well, I have been around long enough to remember JavaScript/ECMAScript objects as unordered so some features change in the course of time, I also think, Python, for instance only recently switched to some order preservation in its dictionary type. I don't know what to suggest for the current BaseX release, unfortunately http://www.woerteler.de/xquery/modules/ordered-map.html is not accessible.
Since ECMAScript v5 if I remember correctly object keys are ordered, because under the hood List object should be used to store keys. On Mon, Dec 16, 2024 at 4:37 PM Martin Honnen <martin.honnen@gmx.de> wrote:
Am 16.12.2024 um 15:19 schrieb Oleksandr Shpak:
Hi Martin,
I found even more weird situation parse-json('{"1:aa": 1, "2:bb": 2}') returns {"2:bb":2.0e0,"1:aa":1.0e0}
Seems like order is inversed But, parse-json('{"aa": 1, "b": 2, "ab": 3}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0}
and, finally parse-json('{"aa": 1, "b": 2, "ab": 3, "1:aa": 1, "2:bb": 2}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0,"2:bb":2.0e0,"1:aa":1.0e0}
🤯
It seems like a bug to me ;)
As I said, in XPath 3.1 and XQuery 3.1 maps are unordered collections, you might get any order when parsing a particular JSON into an XDM 3.1 map.
If it is any help, in the BaseX fiddle you probably get/see the order preservation you want/expect <https://fiddle.basex.org/?share=%28%27query%21%275.**8.A6%21%27%27%7Emode%21%27XQuery+%7BBaseX6Type%21%27xml%27%29*A%7D%2C%5Cn579b024ab03.71%3A92%3Abb020%5C%21+4875parse-json%7B%22%286%7D%27%7Econtext7%5C%278%2C+9aa014A%29%22%01A9876540.*_> .
If we are talking about the JSON standard, not xQuery, devs will expect the function which behaves in the same way as it does in many other languages.
Well, I have been around long enough to remember JavaScript/ECMAScript objects as unordered so some features change in the course of time, I also think, Python, for instance only recently switched to some order preservation in its dictionary type.
I don't know what to suggest for the current BaseX release, unfortunately http://www.woerteler.de/xquery/modules/ordered-map.html is not accessible.
-- s0rr0w
Hi Oleksandr, As Martin has summarized, the the current behavior is not a bug as the XQuery 3.x specification does not define order on maps. It is up to the implementation to choose an internal order for entries (usually realized via an immutable/persistent hash structure) that maximizes access time. With XQuery 4, as indicated, this is about to change. As you may have seen if you have clicked the Fiddle link, a proof-of-concept implementation is available now that demonstrates the new behavior. It is also included in the latest snapshot of BaseX [1]. We will further pursue it in BaseX, no matter how the spec evolves. As for JavaScript, the JSON specification (RFC 8259) states that the order of key/value pairs is not guaranteed. In ECMAScript, however, orderedness was introduced with ES6. Hope this helps, Christian [1] https://files.basex.org/releases/latest/ On Mon, Dec 16, 2024 at 4:01 PM Oleksandr Shpak <shadowkin@gmail.com> wrote:
Since ECMAScript v5 if I remember correctly object keys are ordered, because under the hood List object should be used to store keys.
On Mon, Dec 16, 2024 at 4:37 PM Martin Honnen <martin.honnen@gmx.de> wrote:
Am 16.12.2024 um 15:19 schrieb Oleksandr Shpak:
Hi Martin,
I found even more weird situation parse-json('{"1:aa": 1, "2:bb": 2}') returns {"2:bb":2.0e0,"1:aa":1.0e0}
Seems like order is inversed But, parse-json('{"aa": 1, "b": 2, "ab": 3}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0}
and, finally parse-json('{"aa": 1, "b": 2, "ab": 3, "1:aa": 1, "2:bb": 2}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0,"2:bb":2.0e0,"1:aa":1.0e0}
🤯
It seems like a bug to me ;)
As I said, in XPath 3.1 and XQuery 3.1 maps are unordered collections, you might get any order when parsing a particular JSON into an XDM 3.1 map.
If it is any help, in the BaseX fiddle you probably get/see the order preservation you want/expect <https://fiddle.basex.org/?share=%28%27query%21%275.**8.A6%21%27%27%7Emode%21%27XQuery+%7BBaseX6Type%21%27xml%27%29*A%7D%2C%5Cn579b024ab03.71%3A92%3Abb020%5C%21+4875parse-json%7B%22%286%7D%27%7Econtext7%5C%278%2C+9aa014A%29%22%01A9876540.*_> .
If we are talking about the JSON standard, not xQuery, devs will expect the function which behaves in the same way as it does in many other languages.
Well, I have been around long enough to remember JavaScript/ECMAScript objects as unordered so some features change in the course of time, I also think, Python, for instance only recently switched to some order preservation in its dictionary type.
I don't know what to suggest for the current BaseX release, unfortunately http://www.woerteler.de/xquery/modules/ordered-map.html is not accessible.
-- s0rr0w
Thanks Christian, Martin for the explanation and good news! Yes, I've seed BXFiddle, it looks promising! Will switch to the 12th version as soon as you release it. On Mon, Dec 16, 2024 at 5:18 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi Oleksandr,
As Martin has summarized, the the current behavior is not a bug as the XQuery 3.x specification does not define order on maps. It is up to the implementation to choose an internal order for entries (usually realized via an immutable/persistent hash structure) that maximizes access time. With XQuery 4, as indicated, this is about to change. As you may have seen if you have clicked the Fiddle link, a proof-of-concept implementation is available now that demonstrates the new behavior. It is also included in the latest snapshot of BaseX [1]. We will further pursue it in BaseX, no matter how the spec evolves.
As for JavaScript, the JSON specification (RFC 8259) states that the order of key/value pairs is not guaranteed. In ECMAScript, however, orderedness was introduced with ES6.
Hope this helps, Christian
[1] https://files.basex.org/releases/latest/
On Mon, Dec 16, 2024 at 4:01 PM Oleksandr Shpak <shadowkin@gmail.com> wrote:
Since ECMAScript v5 if I remember correctly object keys are ordered, because under the hood List object should be used to store keys.
On Mon, Dec 16, 2024 at 4:37 PM Martin Honnen <martin.honnen@gmx.de> wrote:
Am 16.12.2024 um 15:19 schrieb Oleksandr Shpak:
Hi Martin,
I found even more weird situation parse-json('{"1:aa": 1, "2:bb": 2}') returns {"2:bb":2.0e0,"1:aa":1.0e0}
Seems like order is inversed But, parse-json('{"aa": 1, "b": 2, "ab": 3}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0}
and, finally parse-json('{"aa": 1, "b": 2, "ab": 3, "1:aa": 1, "2:bb": 2}') returns {"aa":1.0e0,"ab":3.0e0,"b":2.0e0,"2:bb":2.0e0,"1:aa":1.0e0}
🤯
It seems like a bug to me ;)
As I said, in XPath 3.1 and XQuery 3.1 maps are unordered collections, you might get any order when parsing a particular JSON into an XDM 3.1 map.
If it is any help, in the BaseX fiddle you probably get/see the order preservation you want/expect <https://fiddle.basex.org/?share=%28%27query%21%275.**8.A6%21%27%27%7Emode%21%27XQuery+%7BBaseX6Type%21%27xml%27%29*A%7D%2C%5Cn579b024ab03.71%3A92%3Abb020%5C%21+4875parse-json%7B%22%286%7D%27%7Econtext7%5C%278%2C+9aa014A%29%22%01A9876540.*_> .
If we are talking about the JSON standard, not xQuery, devs will expect the function which behaves in the same way as it does in many other languages.
Well, I have been around long enough to remember JavaScript/ECMAScript objects as unordered so some features change in the course of time, I also think, Python, for instance only recently switched to some order preservation in its dictionary type.
I don't know what to suggest for the current BaseX release, unfortunately http://www.woerteler.de/xquery/modules/ordered-map.html is not accessible.
-- s0rr0w
-- s0rr0w
participants (3)
-
Christian Grün -
Martin Honnen -
Oleksandr Shpak