I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587... I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
Dear Wray, I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9 Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want So in a nutshell, in XQuery 3.1, something like the following:
fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want. Hope this helps ;-) Best from Konstanz Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.com>:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587... <https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json>
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008
On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.org> wrote:
Dear Wray,
I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9
Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want
So in a nutshell, in XQuery 3.1, something like the following:
fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.com>:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
Hi, Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively. Kendall From: <basex-talk-bounces@mailman.uni-konstanz.de> on behalf of "E. Wray Johnson" <wray.johnson@gmail.com> Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle <ms@basex.org> Cc: BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008 On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.org<mailto:ms@basex.org>> wrote: Dear Wray, I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=> Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :) I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=> As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want. Hope this helps ;-) Best from Konstanz Michael Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.com<mailto:wray.johnson@gmail.com>>: I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
I may need to load hundreds of thousands of items each with dozens of sub-nodes. I am looking for a solution that will not be at risk of running out of heap memory. On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw@workday.com> wrote:
Hi,
Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively.
Kendall
*From: *<basex-talk-bounces@mailman.uni-konstanz.de> on behalf of "E. Wray Johnson" <wray.johnson@gmail.com> *Date: *Sunday, November 19, 2017 at 5:15 AM *To: *Michael Seiferle <ms@basex.org> *Cc: *BaseX <basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Loading data
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works.
Wray Johnson
(m) 704-293-9008 <(704)%20293-9008>
On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.org> wrote:
Dear Wray,
I tried to come up with a sketch of what–I think–might help you:
https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9 <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=>
Basically it boils down to:
1. Fetch the JSON
2. Convert that JSON to an XQuery item
3. Iterate over each array entry and explicitly construct the XML representation you want
So in a nutshell, in XQuery 3.1, something like the following:
fetch:text('https://gist.githubusercontent.com/ Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 1d25e2c6f6/cities.json <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> ')
=> parse-json() (: Convert to XQuery item representation :)
=> array:for-each(function($map){ (: For each entry in that array, do :)
element item { (: Construct an XML element named item :)
$map => map:keys()
=> for-each(function($key){ (: For each key in the map, do: :)
element { $key } { (: Return an element named $key :)
$map($key) (: …and the value of $map($key) :)
}
})
}
})
=> array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
fetch:text('https://gist.githubusercontent.com/ Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 1d25e2c6f6/cities.json <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> ')
=> json:parse()
For more info on our BaseX JSON-Module see: http://docs.basex.org/ wiki/JSON_Module <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=>
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.com>:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/ 2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
I hope that the tuple stream of a FLWOR expression does not accumulate heap memory. Is that the case, Christian, or anyone? Kendall From: "E. Wray Johnson" <wray.johnson@gmail.com> Date: Monday, November 20, 2017 at 10:37 AM To: Kendall Shaw <kendall.shaw@workday.com> Cc: BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data I may need to load hundreds of thousands of items each with dozens of sub-nodes. I am looking for a solution that will not be at risk of running out of heap memory. On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw@workday.com<mailto:kendall.shaw@workday.com>> wrote: Hi, Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e=> it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively. Kendall From: <basex-talk-bounces@mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de>> on behalf of "E. Wray Johnson" <wray.johnson@gmail.com<mailto:wray.johnson@gmail.com>> Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle <ms@basex.org<mailto:ms@basex.org>> Cc: BaseX <basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Loading data Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008<tel:(704)%20293-9008> On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.org<mailto:ms@basex.org>> wrote: Dear Wray, I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=> Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :) I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=> As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want. Hope this helps ;-) Best from Konstanz Michael Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.com<mailto:wray.johnson@gmail.com>>: I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
Also, I would think that map:merge would be the typical FP idea where conceptually each step is a copy of the entire map, but it’s implemented as only what has changed being new. If that isn’t the case that would be important to know. Kendall From: <basex-talk-bounces@mailman.uni-konstanz.de> on behalf of Kendall Shaw <kendall.shaw@workday.com> Date: Monday, November 20, 2017 at 12:13 PM To: BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data I hope that the tuple stream of a FLWOR expression does not accumulate heap memory. Is that the case, Christian, or anyone? Kendall From: "E. Wray Johnson" <wray.johnson@gmail.com> Date: Monday, November 20, 2017 at 10:37 AM To: Kendall Shaw <kendall.shaw@workday.com> Cc: BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data I may need to load hundreds of thousands of items each with dozens of sub-nodes. I am looking for a solution that will not be at risk of running out of heap memory. On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw@workday.com<mailto:kendall.shaw@workday.com>> wrote: Hi, Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e=> it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively. Kendall From: <basex-talk-bounces@mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de>> on behalf of "E. Wray Johnson" <wray.johnson@gmail.com<mailto:wray.johnson@gmail.com>> Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle <ms@basex.org<mailto:ms@basex.org>> Cc: BaseX <basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Loading data Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008<tel:(704)%20293-9008> On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.org<mailto:ms@basex.org>> wrote: Dear Wray, I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=> Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :) I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=> As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want. Hope this helps ;-) Best from Konstanz Michael Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.com<mailto:wray.johnson@gmail.com>>: I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
The source/input data structure (i.e. element/property names) will not typically match the output/database structure. Also some fields (e.g. longitude and latitude will be transformed into a gml:Point/gml:coordinate node. On Mon, Nov 20, 2017 at 3:43 PM, Kendall Shaw <kendall.shaw@workday.com> wrote:
Also, I would think that map:merge would be the typical FP idea where conceptually each step is a copy of the entire map, but it’s implemented as only what has changed being new. If that isn’t the case that would be important to know.
Kendall
*From: *<basex-talk-bounces@mailman.uni-konstanz.de> on behalf of Kendall Shaw <kendall.shaw@workday.com> *Date: *Monday, November 20, 2017 at 12:13 PM *To: *BaseX <basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Loading data
I hope that the tuple stream of a FLWOR expression does not accumulate heap memory. Is that the case, Christian, or anyone?
Kendall
*From: *"E. Wray Johnson" <wray.johnson@gmail.com> *Date: *Monday, November 20, 2017 at 10:37 AM *To: *Kendall Shaw <kendall.shaw@workday.com> *Cc: *BaseX <basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Loading data
I may need to load hundreds of thousands of items each with dozens of sub-nodes.
I am looking for a solution that will not be at risk of running out of heap memory.
On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw@workday.com> wrote:
Hi,
Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e=> it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively.
Kendall
*From: *<basex-talk-bounces@mailman.uni-konstanz.de> on behalf of "E. Wray Johnson" <wray.johnson@gmail.com> *Date: *Sunday, November 19, 2017 at 5:15 AM *To: *Michael Seiferle <ms@basex.org> *Cc: *BaseX <basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Loading data
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works.
Wray Johnson
(m) 704-293-9008 <(704)%20293-9008>
On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.org> wrote:
Dear Wray,
I tried to come up with a sketch of what–I think–might help you:
https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9 <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=>
Basically it boils down to:
1. Fetch the JSON
2. Convert that JSON to an XQuery item
3. Iterate over each array entry and explicitly construct the XML representation you want
So in a nutshell, in XQuery 3.1, something like the following:
fetch:text('https://gist.githubusercontent.com/ Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 1d25e2c6f6/cities.json <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> ')
=> parse-json() (: Convert to XQuery item representation :)
=> array:for-each(function($map){ (: For each entry in that array, do :)
element item { (: Construct an XML element named item :)
$map => map:keys()
=> for-each(function($key){ (: For each key in the map, do: :)
element { $key } { (: Return an element named $key :)
$map($key) (: …and the value of $map($key) :)
}
})
}
})
=> array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
fetch:text('https://gist.githubusercontent.com/ Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 1d25e2c6f6/cities.json <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> ')
=> json:parse()
For more info on our BaseX JSON-Module see: http://docs.basex.org/ wiki/JSON_Module <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=>
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.com>:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/ 2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
participants (3)
-
E. Wray Johnson -
Kendall Shaw -
Michael Seiferle