Hi,
We have requirement to group elements on specific criteria. We are grouping by creating a new XML and grouping similar elements by creating a new element under which we are putting the similar elements. I just wanted to know whether all fetched element resides in Main Memory or some kind of reference is stored in basex which later retrieve the elements form XML database.
Suppose we have in some element in database. We are firing queries in database and grouping by append some new element.
let $a := /data/product[@category = "abc"] let $b := /data/product[@category = "bcd"]
let $c := <product><abc>{$b}</abc><bcd>$b</bcd></product>
My question is, Will $c hold all the xml data in main memory or will it contains reference to $a , $b which points to the data in database ?.
Thanks $ Regards Ankit Kumar
Hi Ankit,
let $c := <product><abc>{$b}</abc><bcd>$b</bcd></product>
My question is, Will $c hold all the xml data in main memory or will it contains reference to $a , $b which points to the data in database ?.
The XQuery spec. doesn't actually allow us to point to the original database node, because the embedded node has a different node id:
<abc>{$b}</abc>/* is $b → false
However, the representation of the embedded node resembles the data structure of the original node, so it will take much less memory than an XML fragment that has completely been created in main memory.
Best, Christian
Hi,
I have a large xml database around 2 GB. I have to group elements on the basis of some attribute. As i mention in my previous mail, I tried to created a new element and embedding the element as a child element. I am getting Out of Memory because of this. If i never create a new element just using that element it is not showing any Out of Memory Error. As per my understanding is it happenning because it is creating a totally new element in the main memory itself. Please help me out. Please go through the scenario.
Scenario :1
=== Without Any Out of Memory Error=====================
long t1=System.currentTimeMillis(); QueryProcessor pr = new QueryProcessor("let $a := //elements return $c",context); pr.context(itr); Iter first=pr.iter(); System.out.println(System.currentTimeMillis() - t1);
long t2=System.currentTimeMillis(); QueryProcessor pr1 = new QueryProcessor("declare variable $elements external; let $a := $elements/* return $a",context); pr1.bind("elements",first); Iter second=pr1.iter(); System.out.println(System.currentTimeMillis() - t2);
Scenario : 2
=== With Out of Memory Error=====================
long t1=System.currentTimeMillis(); QueryProcessor pr = new QueryProcessor("let $a := //elements return <Nelement>$c</Nelement>",context); // Here i am creating new element pr.context(itr); Iter first=pr.iter(); System.out.println(System.currentTimeMillis() - t1);
long t2=System.currentTimeMillis(); QueryProcessor pr1 = new QueryProcessor("declare variable $elements external; let $a := $elements/* return $a",context); pr1.context(first); pr1.bind("elements",first); Iter second=pr1.iter(); // Here Out of Memory Error Comes. System.out.println(System.currentTimeMillis() - t2);
On 6 February 2015 at 00:01, Christian Grün christian.gruen@gmail.com wrote:
Hi Ankit,
let $c := <product><abc>{$b}</abc><bcd>$b</bcd></product>
My question is, Will $c hold all the xml data in main memory or will it contains reference to $a , $b which points to the data in database ?.
The XQuery spec. doesn't actually allow us to point to the original database node, because the embedded node has a different node id:
<abc>{$b}</abc>/* is $b → false
However, the representation of the embedded node resembles the data structure of the original node, so it will take much less memory than an XML fragment that has completely been created in main memory.
Best, Christian
Hello Ankit,
as Christian and yourself said, the element is created in main memory and this is required by the specification.
So if you hold 2 GB data in main memory, you will need sufficient memory to hold this data. Is there any problem or limit you encounter when increasing the memory for BaseX (using -Xmx)?
Cheers, Dirk
On 02/06/2015 10:29 AM, ankit kumar wrote:
Hi,
I have a large xml database around 2 GB. I have to group elements on the basis of some attribute. As i mention in my previous mail, I tried to created a new element and embedding the element as a child element. I am getting Out of Memory because of this. If i never create a new element just using that element it is not showing any Out of Memory Error. As per my understanding is it happenning because it is creating a totally new element in the main memory itself. Please help me out. Please go through the scenario.
Scenario :1
=== Without Any Out of Memory Error=====================
long t1=System.currentTimeMillis(); QueryProcessor pr = new QueryProcessor("let $a := //elements return $c",context); pr.context(itr); Iter first=pr.iter(); System.out.println(System.currentTimeMillis() - t1);
long t2=System.currentTimeMillis(); QueryProcessor pr1 = new QueryProcessor("declare variable $elements external; let $a := $elements/* return $a",context); pr1.bind("elements",first); Iter second=pr1.iter(); System.out.println(System.currentTimeMillis() - t2);
Scenario : 2
=== With Out of Memory Error=====================
long t1=System.currentTimeMillis(); QueryProcessor pr = new QueryProcessor("let $a := //elements return <Nelement>$c</Nelement>",context); // Here i am creating new element pr.context(itr); Iter first=pr.iter(); System.out.println(System.currentTimeMillis() - t1);
long t2=System.currentTimeMillis(); QueryProcessor pr1 = new QueryProcessor("declare variable $elements external; let $a := $elements/* return $a",context); pr1.context(first); pr1.bind("elements",first); Iter second=pr1.iter(); // Here Out of Memory Error Comes. System.out.println(System.currentTimeMillis() - t2);
On 6 February 2015 at 00:01, Christian Grün christian.gruen@gmail.com wrote:
Hi Ankit,
let $c := <product><abc>{$b}</abc><bcd>$b</bcd></product>
My question is, Will $c hold all the xml data in main memory or will it contains reference to $a , $b which points to the data in database ?.
The XQuery spec. doesn't actually allow us to point to the original database node, because the embedded node has a different node id:
<abc>{$b}</abc>/* is $b → false
However, the representation of the embedded node resembles the data structure of the original node, so it will take much less memory than an XML fragment that has completely been created in main memory.
Best, Christian
basex-talk@mailman.uni-konstanz.de
-
ankit kumar -
Christian Grün -
Dirk Kirsten