I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
Dear Wray,
I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9
Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want
So in a nutshell, in XQuery 3.1, something like the following:
fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...') => json:parse()
For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson wray.johnson@gmail.com:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587... https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works.
Wray Johnson (m) 704-293-9008
On Nov 19, 2017, at 6:30 AM, Michael Seiferle ms@basex.org wrote:
Dear Wray,
I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9
Basically it boils down to:
- Fetch the JSON
- Convert that JSON to an XQuery item
- Iterate over each array entry and explicitly construct the XML representation you want
So in a nutshell, in XQuery 3.1, something like the following:
fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...') => json:parse()
For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson wray.johnson@gmail.com:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
Hi,
Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively.
Kendall
From: basex-talk-bounces@mailman.uni-konstanz.de on behalf of "E. Wray Johnson" wray.johnson@gmail.com Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle ms@basex.org Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Loading data
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008
On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.orgmailto:ms@basex.org> wrote: Dear Wray,
I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=
Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want
So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Modulehttps://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.commailto:wray.johnson@gmail.com>:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e= I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
I may need to load hundreds of thousands of items each with dozens of sub-nodes.
I am looking for a solution that will not be at risk of running out of heap memory.
On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw kendall.shaw@workday.com wrote:
Hi,
Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively.
Kendall
*From: *basex-talk-bounces@mailman.uni-konstanz.de on behalf of "E. Wray Johnson" wray.johnson@gmail.com *Date: *Sunday, November 19, 2017 at 5:15 AM *To: *Michael Seiferle ms@basex.org *Cc: *BaseX basex-talk@mailman.uni-konstanz.de *Subject: *Re: [basex-talk] Loading data
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works.
Wray Johnson
(m) 704-293-9008 <(704)%20293-9008>
On Nov 19, 2017, at 6:30 AM, Michael Seiferle ms@basex.org wrote:
Dear Wray,
I tried to come up with a sketch of what–I think–might help you:
https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9 https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=
Basically it boils down to:
Fetch the JSON
Convert that JSON to an XQuery item
Iterate over each array entry and explicitly construct the XML
representation you want
So in a nutshell, in XQuery 3.1, something like the following:
fetch:text('https://gist.githubusercontent.com/ Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 1d25e2c6f6/cities.json https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e= ')
=> parse-json() (: Convert to XQuery item representation :)
=> array:for-each(function($map){ (: For each entry in that array, do :)
element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) }})
=> array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
fetch:text('https://gist.githubusercontent.com/ Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 1d25e2c6f6/cities.json https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e= ')
=> json:parse()
For more info on our BaseX JSON-Module see: http://docs.basex.org/ wiki/JSON_Module https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson wray.johnson@gmail.com:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/ 2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
I hope that the tuple stream of a FLWOR expression does not accumulate heap memory. Is that the case, Christian, or anyone?
Kendall
From: "E. Wray Johnson" wray.johnson@gmail.com Date: Monday, November 20, 2017 at 10:37 AM To: Kendall Shaw kendall.shaw@workday.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Loading data
I may need to load hundreds of thousands of items each with dozens of sub-nodes.
I am looking for a solution that will not be at risk of running out of heap memory.
On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw@workday.commailto:kendall.shaw@workday.com> wrote: Hi,
Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Modulehttps://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e= it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively.
Kendall
From: <basex-talk-bounces@mailman.uni-konstanz.demailto:basex-talk-bounces@mailman.uni-konstanz.de> on behalf of "E. Wray Johnson" <wray.johnson@gmail.commailto:wray.johnson@gmail.com> Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle <ms@basex.orgmailto:ms@basex.org> Cc: BaseX <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008tel:(704)%20293-9008
On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.orgmailto:ms@basex.org> wrote: Dear Wray,
I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=
Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want
So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Modulehttps://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.commailto:wray.johnson@gmail.com>:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e= I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
Also, I would think that map:merge would be the typical FP idea where conceptually each step is a copy of the entire map, but it’s implemented as only what has changed being new. If that isn’t the case that would be important to know.
Kendall
From: basex-talk-bounces@mailman.uni-konstanz.de on behalf of Kendall Shaw kendall.shaw@workday.com Date: Monday, November 20, 2017 at 12:13 PM To: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Loading data
I hope that the tuple stream of a FLWOR expression does not accumulate heap memory. Is that the case, Christian, or anyone?
Kendall
From: "E. Wray Johnson" wray.johnson@gmail.com Date: Monday, November 20, 2017 at 10:37 AM To: Kendall Shaw kendall.shaw@workday.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Loading data
I may need to load hundreds of thousands of items each with dozens of sub-nodes.
I am looking for a solution that will not be at risk of running out of heap memory.
On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw@workday.commailto:kendall.shaw@workday.com> wrote: Hi,
Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Modulehttps://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e= it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively.
Kendall
From: <basex-talk-bounces@mailman.uni-konstanz.demailto:basex-talk-bounces@mailman.uni-konstanz.de> on behalf of "E. Wray Johnson" <wray.johnson@gmail.commailto:wray.johnson@gmail.com> Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle <ms@basex.orgmailto:ms@basex.org> Cc: BaseX <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008tel:(704)%20293-9008
On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.orgmailto:ms@basex.org> wrote: Dear Wray,
I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=
Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want
So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Modulehttps://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.commailto:wray.johnson@gmail.com>:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf2587...https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e= I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
The source/input data structure (i.e. element/property names) will not typically match the output/database structure. Also some fields (e.g. longitude and latitude will be transformed into a gml:Point/gml:coordinate node.
On Mon, Nov 20, 2017 at 3:43 PM, Kendall Shaw kendall.shaw@workday.com wrote:
Also, I would think that map:merge would be the typical FP idea where conceptually each step is a copy of the entire map, but it’s implemented as only what has changed being new. If that isn’t the case that would be important to know.
Kendall
*From: *basex-talk-bounces@mailman.uni-konstanz.de on behalf of Kendall Shaw kendall.shaw@workday.com *Date: *Monday, November 20, 2017 at 12:13 PM *To: *BaseX basex-talk@mailman.uni-konstanz.de *Subject: *Re: [basex-talk] Loading data
I hope that the tuple stream of a FLWOR expression does not accumulate heap memory. Is that the case, Christian, or anyone?
Kendall
*From: *"E. Wray Johnson" wray.johnson@gmail.com *Date: *Monday, November 20, 2017 at 10:37 AM *To: *Kendall Shaw kendall.shaw@workday.com *Cc: *BaseX basex-talk@mailman.uni-konstanz.de *Subject: *Re: [basex-talk] Loading data
I may need to load hundreds of thousands of items each with dozens of sub-nodes.
I am looking for a solution that will not be at risk of running out of heap memory.
On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw kendall.shaw@workday.com wrote:
Hi,
Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e= it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively.
Kendall
*From: *basex-talk-bounces@mailman.uni-konstanz.de on behalf of "E. Wray Johnson" wray.johnson@gmail.com *Date: *Sunday, November 19, 2017 at 5:15 AM *To: *Michael Seiferle ms@basex.org *Cc: *BaseX basex-talk@mailman.uni-konstanz.de *Subject: *Re: [basex-talk] Loading data
Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works.
Wray Johnson
(m) 704-293-9008 <(704)%20293-9008>
On Nov 19, 2017, at 6:30 AM, Michael Seiferle ms@basex.org wrote:
Dear Wray,
I tried to come up with a sketch of what–I think–might help you:
https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9 https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=
Basically it boils down to:
Fetch the JSON
Convert that JSON to an XQuery item
Iterate over each array entry and explicitly construct the XML
representation you want
So in a nutshell, in XQuery 3.1, something like the following:
fetch:text('https://gist.githubusercontent.com/ Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 1d25e2c6f6/cities.json https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e= ')
=> parse-json() (: Convert to XQuery item representation :)
=> array:for-each(function($map){ (: For each entry in that array, do :)
element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) }})
=> array:flatten() (: Converts the array to a sequence :)
I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
fetch:text('https://gist.githubusercontent.com/ Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 1d25e2c6f6/cities.json https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e= ')
=> json:parse()
For more info on our BaseX JSON-Module see: http://docs.basex.org/ wiki/JSON_Module https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=
As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.
Hope this helps ;-)
Best from Konstanz
Michael
Am 19.11.2017 um 06:25 schrieb E. Wray Johnson wray.johnson@gmail.com:
I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/ 2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.
basex-talk@mailman.uni-konstanz.de
-
E. Wray Johnson -
Kendall Shaw -
Michael Seiferle