Re: [basex-talk] Loading data

20 Nov 2017

      I hope that the tuple stream of a FLWOR expression does not accumulate heap memory. Is that the case, Christian, or anyone?

Kendall

From: "E. Wray Johnson" <wray.johnson@gmail.com>
Date: Monday, November 20, 2017 at 10:37 AM
To: Kendall Shaw <kendall.shaw@workday.com>
Cc: BaseX <basex-talk@mailman.uni-konstanz.de>
Subject: Re: [basex-talk] Loading data

I may need to load hundreds of thousands of items each with dozens of sub-nodes.

I am looking for a solution that will not be at risk of running out of heap memory.

On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw@workday.com<mailto:kendall.shaw@workday.com>> wrote:
Hi,

Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e=> it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be  a map, rather than updating a map, iteratively.

Kendall

From: <basex-talk-bounces@mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de>> on behalf of "E. Wray Johnson" <wray.johnson@gmail.com<mailto:wray.johnson@gmail.com>>
Date: Sunday, November 19, 2017 at 5:15 AM
To: Michael Seiferle <ms@basex.org<mailto:ms@basex.org>>
Cc: BaseX <basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>>
Subject: Re: [basex-talk] Loading data

Thanks. I tried doing #2 inside the iteration and that did not work.  I will try it your way and let you know if it works.
Wray Johnson
(m) 704-293-9008<tel:(704)%20293-9008>

On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms@basex.org<mailto:ms@basex.org>> wrote:
Dear Wray,

I tried to come up with a sketch of what–I think–might help you:
https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=>

Basically it boils down to:
1. Fetch the JSON
2. Convert that JSON to an XQuery item
3. Iterate over each array entry and explicitly construct the XML representation you want

So in a nutshell, in XQuery 3.1, something like the following:
  fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>')
  => parse-json()  (: Convert to XQuery item representation :)
  => array:for-each(function($map){ (: For each entry in that array, do :)
    element item { (: Construct an XML element named item :)
      $map => map:keys()
      => for-each(function($key){  (: For each key in the map, do: :)
        element { $key } {         (: Return an element named $key :)
          $map($key)               (: …and the value of $map($key) :)
        }
      })
    }
  })
  => array:flatten() (: Converts the array to a sequence :)

I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away:
  fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>')
  => json:parse()
For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=>

As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want.

Hope this helps ;-)

Best from Konstanz

Michael

Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson@gmail.com<mailto:wray.johnson@gmail.com>>:

 I have a JSON file
https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>
I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements.  However my for loop does not emit individual XML elements for each json object in the loaded array.