Hello,
I am writing my code in Java and the result after querying xml files in my BaseX database is very large, about 500MB. Thus, I would like to stream the output to another xml file in the BaseX database. Could you tell me how to do this?
Many thanks,
Huy Vu
Dear Huy,
have you looked at the Session classes (LocalSession/ClientSession)? These classes supports input and output streams to do pipelining, such as querying results and sending them to another target without first materializing all data.
Hope this helps, Christian
On Sun, Dec 18, 2011 at 10:27 AM, Huy Vu qhuyvu@gmail.com wrote:
Hello,
I am writing my code in Java and the result after querying xml files in my BaseX database is very large, about 500MB. Thus, I would like to stream the output to another xml file in the BaseX database. Could you tell me how to do this?
Many thanks,
Huy Vu
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
Dear Christian,
Thank you very much for your answer. I have tried to use input and output streams but I could not pipeline the input stream to an output stream.
I have to explicitly create an output stream
ByteArrayOutputStream out = new ByteArrayOutputStream(); new XQuery(inputQuery).execute(context, out);
InputStream in = new ByteArrayInputStream(out.toByteArray());
then I store it to an xml file in my database:
session.execute("open " + Constants.DB);
session.add(fileName, in);
Could you tell me and/or give me the links to the instructions of pipelining the output stream to a database file?
Thank you very much.
Huy
Dear Huy,
your questions are welcome. I forgot to mention (and think of the fact) that the current version of BaseX does not allow simultaneous read and write operations, which is why it won't be possible to read data and write it back into a database (find more at [1]). We are currently implementing a more granular locking concept (database-wise), which is a little bit tricky, as a single XQuery can dynamically address arbitrary databses; for now, you'll probably have to cache the results to avoid a deadlock, or use a single XQuery for evaluating and storing results. An example (I'm using the DB module here [2]):
let $xml := db:open('db1')/your/query return db:add('db1', $xml)
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Transaction_Management [2] http://docs.basex.org/wiki/Database_Module#db:add
On Sun, Dec 18, 2011 at 8:56 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your answer. I have tried to use input and output streams but I could not pipeline the input stream to an output stream.
I have to explicitly create an output stream
ByteArrayOutputStream out = new ByteArrayOutputStream();
new XQuery(inputQuery).execute(context, out);
InputStream in = new ByteArrayInputStream(out.toByteArray());
then I store it to an xml file in my database:
session.execute("open " + Constants.DB);
session.add(fileName, in);
Could you tell me and/or give me the links to the instructions of pipelining the output stream to a database file?
Thank you very much.
Huy
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
On 18 Dec 2011, at 10:52, BaseX Support wrote:
Dear Huy,
have you looked at the Session classes (LocalSession/ClientSession)? These classes supports input and output streams to do pipelining, such as querying results and sending them to another target without first materializing all data.
Hope this helps, Christian
On Sun, Dec 18, 2011 at 10:27 AM, Huy Vu qhuyvu@gmail.com wrote:
Hello,
I am writing my code in Java and the result after querying xml files in my BaseX database is very large, about 500MB. Thus, I would like to stream the output to another xml file in the BaseX database. Could you tell me how to do this?
Many thanks,
Huy Vu
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
Dear Christian,
Thank you very much for your explanation and the example. I have tried with your example and it works very well in BaseX application itself. However, when I try to run the query from Java code (I'm using Eclipse) by the following code:
BaseXClient session = new BaseXClient("localhost", 1984, "admin", "admin");
try { Context context = new Context();
String q = "let $xml := db:open('Collection','test/xmark.xml')//name let $y := for $x in $xml return $x return db:add('Collection', document {<b>{ $y }</b>} ,'doc1.xml') ";
new XQuery(input).execute(context);
} finally { session.close(); }
I have got the errors:
org.basex.core.BaseXException: Out of Main Memory.
at org.basex.core.Command.execute(Command.java:77)
at org.basex.core.Command.execute(Command.java:89)
I have changed the memory in Eclipse by adding: -vmargs -Xms512m -Xmx1024m
to the eclipse.ini file but it does not help.
Do you think these errors are due to the fact that I have called the query in a wrong way, i.e. new XQuery(input).execute(context);
Thank you very much.
Huy
On 19 Dec 2011, at 00:15, BaseX Support wrote:
Dear Huy,
your questions are welcome. I forgot to mention (and think of the fact) that the current version of BaseX does not allow simultaneous read and write operations, which is why it won't be possible to read data and write it back into a database (find more at [1]). We are currently implementing a more granular locking concept (database-wise), which is a little bit tricky, as a single XQuery can dynamically address arbitrary databses; for now, you'll probably have to cache the results to avoid a deadlock, or use a single XQuery for evaluating and storing results. An example (I'm using the DB module here [2]):
let $xml := db:open('db1')/your/query return db:add('db1', $xml)
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Transaction_Management [2] http://docs.basex.org/wiki/Database_Module#db:add
On Sun, Dec 18, 2011 at 8:56 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your answer. I have tried to use input and output streams but I could not pipeline the input stream to an output stream.
I have to explicitly create an output stream
ByteArrayOutputStream out = new ByteArrayOutputStream();
new XQuery(inputQuery).execute(context, out);
InputStream in = new ByteArrayInputStream(out.toByteArray());
then I store it to an xml file in my database:
session.execute("open " + Constants.DB);
session.add(fileName, in);
Could you tell me and/or give me the links to the instructions of pipelining the output stream to a database file?
Thank you very much.
Huy
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
On 18 Dec 2011, at 10:52, BaseX Support wrote:
Dear Huy,
have you looked at the Session classes (LocalSession/ClientSession)? These classes supports input and output streams to do pipelining, such as querying results and sending them to another target without first materializing all data.
Hope this helps, Christian
On Sun, Dec 18, 2011 at 10:27 AM, Huy Vu qhuyvu@gmail.com wrote:
Hello,
I am writing my code in Java and the result after querying xml files in my BaseX database is very large, about 500MB. Thus, I would like to stream the output to another xml file in the BaseX database. Could you tell me how to do this?
Many thanks,
Huy Vu
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
Hi Huy,
have you tried the latest snapshot of BaseX as well [1]? If the problem persists, it may be that the instance is too large and needs indeed to be cached.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Releases _________________________
On Mon, Dec 19, 2011 at 2:40 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your explanation and the example. I have tried with your example and it works very well in BaseX application itself. However, when I try to run the query from Java code (I'm using Eclipse) by the following code:
BaseXClient session = new BaseXClient("localhost", 1984, "admin", "admin");
try { Context context = new Context();
String q = "let $xml := db:open('Collection','test/xmark.xml')//name let $y := for $x in $xml return $x return db:add('Collection', document {<b>{ $y }</b>} ,'doc1.xml') ";
new XQuery(input).execute(context);
} finally { session.close(); }
I have got the errors:
org.basex.core.BaseXException: Out of Main Memory.
at org.basex.core.Command.execute(Command.java:77)
at org.basex.core.Command.execute(Command.java:89)
I have changed the memory in Eclipse by adding: -vmargs -Xms512m -Xmx1024m
to the eclipse.ini file but it does not help.
Do you think these errors are due to the fact that I have called the query in a wrong way, i.e. new XQuery(input).execute(context);
Thank you very much.
Huy
On 19 Dec 2011, at 00:15, BaseX Support wrote:
Dear Huy,
your questions are welcome. I forgot to mention (and think of the fact) that the current version of BaseX does not allow simultaneous read and write operations, which is why it won't be possible to read data and write it back into a database (find more at [1]). We are currently implementing a more granular locking concept (database-wise), which is a little bit tricky, as a single XQuery can dynamically address arbitrary databses; for now, you'll probably have to cache the results to avoid a deadlock, or use a single XQuery for evaluating and storing results. An example (I'm using the DB module here [2]):
let $xml := db:open('db1')/your/query return db:add('db1', $xml)
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Transaction_Management [2] http://docs.basex.org/wiki/Database_Module#db:add
On Sun, Dec 18, 2011 at 8:56 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your answer. I have tried to use input and output streams but I could not pipeline the input stream to an output stream.
I have to explicitly create an output stream
ByteArrayOutputStream out = new ByteArrayOutputStream();
new XQuery(inputQuery).execute(context, out);
InputStream in = new ByteArrayInputStream(out.toByteArray());
then I store it to an xml file in my database:
session.execute("open " + Constants.DB);
session.add(fileName, in);
Could you tell me and/or give me the links to the instructions of pipelining the output stream to a database file?
Thank you very much.
Huy
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
On 18 Dec 2011, at 10:52, BaseX Support wrote:
Dear Huy,
have you looked at the Session classes (LocalSession/ClientSession)? These classes supports input and output streams to do pipelining, such as querying results and sending them to another target without first materializing all data.
Hope this helps, Christian
On Sun, Dec 18, 2011 at 10:27 AM, Huy Vu qhuyvu@gmail.com wrote:
Hello,
I am writing my code in Java and the result after querying xml files in my BaseX database is very large, about 500MB. Thus, I would like to stream the output to another xml file in the BaseX database. Could you tell me how to do this?
Many thanks,
Huy Vu
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
Hi Christian,
After playing around with the memory heap allocated in Eclipse I have fixed the problems. Everything works perfectly now. Thank you very much for all your help.
Merry Christmas.
Huy
On 19 Dec 2011, at 14:29, BaseX Support wrote:
Hi Huy,
have you tried the latest snapshot of BaseX as well [1]? If the problem persists, it may be that the instance is too large and needs indeed to be cached.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Releases _________________________
On Mon, Dec 19, 2011 at 2:40 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your explanation and the example. I have tried with your example and it works very well in BaseX application itself. However, when I try to run the query from Java code (I'm using Eclipse) by the following code:
BaseXClient session = new BaseXClient("localhost", 1984, "admin", "admin");
try { Context context = new Context();
String q = "let $xml := db:open('Collection','test/xmark.xml')//name let $y := for $x in $xml return $x return db:add('Collection', document {<b>{ $y }</b>} ,'doc1.xml') ";
new XQuery(input).execute(context);
} finally { session.close(); }
I have got the errors:
org.basex.core.BaseXException: Out of Main Memory.
at org.basex.core.Command.execute(Command.java:77) at org.basex.core.Command.execute(Command.java:89)I have changed the memory in Eclipse by adding: -vmargs -Xms512m -Xmx1024m
to the eclipse.ini file but it does not help.
Do you think these errors are due to the fact that I have called the query in a wrong way, i.e. new XQuery(input).execute(context);
Thank you very much.
Huy
On 19 Dec 2011, at 00:15, BaseX Support wrote:
Dear Huy,
your questions are welcome. I forgot to mention (and think of the fact) that the current version of BaseX does not allow simultaneous read and write operations, which is why it won't be possible to read data and write it back into a database (find more at [1]). We are currently implementing a more granular locking concept (database-wise), which is a little bit tricky, as a single XQuery can dynamically address arbitrary databses; for now, you'll probably have to cache the results to avoid a deadlock, or use a single XQuery for evaluating and storing results. An example (I'm using the DB module here [2]):
let $xml := db:open('db1')/your/query return db:add('db1', $xml)
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Transaction_Management [2] http://docs.basex.org/wiki/Database_Module#db:add
On Sun, Dec 18, 2011 at 8:56 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your answer. I have tried to use input and output streams but I could not pipeline the input stream to an output stream.
I have to explicitly create an output stream
ByteArrayOutputStream out = new ByteArrayOutputStream();
new XQuery(inputQuery).execute(context, out);
InputStream in = new ByteArrayInputStream(out.toByteArray());
then I store it to an xml file in my database:
session.execute("open " + Constants.DB);
session.add(fileName, in);
Could you tell me and/or give me the links to the instructions of pipelining the output stream to a database file?
Thank you very much.
Huy
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
On 18 Dec 2011, at 10:52, BaseX Support wrote:
Dear Huy,
have you looked at the Session classes (LocalSession/ClientSession)? These classes supports input and output streams to do pipelining, such as querying results and sending them to another target without first materializing all data.
Hope this helps, Christian
On Sun, Dec 18, 2011 at 10:27 AM, Huy Vu qhuyvu@gmail.com wrote:
Hello,
I am writing my code in Java and the result after querying xml files in my BaseX database is very large, about 500MB. Thus, I would like to stream the output to another xml file in the BaseX database. Could you tell me how to do this?
Many thanks,
Huy Vu
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
Hello again,
Sorry to disturb you again. I forgot to ask you if BaseX has a function to estimate the output size or running time of an XQuery before running it. This function is similar to the function EXPLAIN for SQL queries in Posgres [1].
Many thanks,
Huy
[1] http://www.postgresql.org/docs/8.1/static/sql-explain.html
On 19 Dec 2011, at 14:29, BaseX Support wrote:
Hi Huy,
have you tried the latest snapshot of BaseX as well [1]? If the problem persists, it may be that the instance is too large and needs indeed to be cached.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Releases _________________________
On Mon, Dec 19, 2011 at 2:40 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your explanation and the example. I have tried with your example and it works very well in BaseX application itself. However, when I try to run the query from Java code (I'm using Eclipse) by the following code:
BaseXClient session = new BaseXClient("localhost", 1984, "admin", "admin");
try { Context context = new Context();
String q = "let $xml := db:open('Collection','test/xmark.xml')//name let $y := for $x in $xml return $x return db:add('Collection', document {<b>{ $y }</b>} ,'doc1.xml') ";
new XQuery(input).execute(context);
} finally { session.close(); }
I have got the errors:
org.basex.core.BaseXException: Out of Main Memory.
at org.basex.core.Command.execute(Command.java:77) at org.basex.core.Command.execute(Command.java:89)I have changed the memory in Eclipse by adding: -vmargs -Xms512m -Xmx1024m
to the eclipse.ini file but it does not help.
Do you think these errors are due to the fact that I have called the query in a wrong way, i.e. new XQuery(input).execute(context);
Thank you very much.
Huy
On 19 Dec 2011, at 00:15, BaseX Support wrote:
Dear Huy,
your questions are welcome. I forgot to mention (and think of the fact) that the current version of BaseX does not allow simultaneous read and write operations, which is why it won't be possible to read data and write it back into a database (find more at [1]). We are currently implementing a more granular locking concept (database-wise), which is a little bit tricky, as a single XQuery can dynamically address arbitrary databses; for now, you'll probably have to cache the results to avoid a deadlock, or use a single XQuery for evaluating and storing results. An example (I'm using the DB module here [2]):
let $xml := db:open('db1')/your/query return db:add('db1', $xml)
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Transaction_Management [2] http://docs.basex.org/wiki/Database_Module#db:add
On Sun, Dec 18, 2011 at 8:56 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your answer. I have tried to use input and output streams but I could not pipeline the input stream to an output stream.
I have to explicitly create an output stream
ByteArrayOutputStream out = new ByteArrayOutputStream();
new XQuery(inputQuery).execute(context, out);
InputStream in = new ByteArrayInputStream(out.toByteArray());
then I store it to an xml file in my database:
session.execute("open " + Constants.DB);
session.add(fileName, in);
Could you tell me and/or give me the links to the instructions of pipelining the output stream to a database file?
Thank you very much.
Huy
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
On 18 Dec 2011, at 10:52, BaseX Support wrote:
Dear Huy,
have you looked at the Session classes (LocalSession/ClientSession)? These classes supports input and output streams to do pipelining, such as querying results and sending them to another target without first materializing all data.
Hope this helps, Christian
On Sun, Dec 18, 2011 at 10:27 AM, Huy Vu qhuyvu@gmail.com wrote:
Hello,
I am writing my code in Java and the result after querying xml files in my BaseX database is very large, about 500MB. Thus, I would like to stream the output to another xml file in the BaseX database. Could you tell me how to do this?
Many thanks,
Huy Vu
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
Dear Huy,
as a matter of fact, estimating the costs for an XQuery expression is a very sophisticated issue (at least compared to the evaluation and estimation of relational database queries). This is why we don't a feature similar to the explain command. What we offer are profiling functions such as util:mb() and util:ms() [1] and, if there's more interest, we could introduce new commands to explicitly request the query plan of a query instead of evaluating it.
[1] http://docs.basex.org/wiki/Utility_Module ______________________________
On Mon, Dec 19, 2011 at 4:42 PM, Huy Vu qhuyvu@gmail.com wrote:
Hello again,
Sorry to disturb you again. I forgot to ask you if BaseX has a function to estimate the output size or running time of an XQuery before running it. This function is similar to the function EXPLAIN for SQL queries in Posgres [1].
Many thanks,
Huy
[1] http://www.postgresql.org/docs/8.1/static/sql-explain.html
On 19 Dec 2011, at 14:29, BaseX Support wrote:
Hi Huy,
have you tried the latest snapshot of BaseX as well [1]? If the problem persists, it may be that the instance is too large and needs indeed to be cached.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Releases _________________________
On Mon, Dec 19, 2011 at 2:40 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your explanation and the example. I have tried with your example and it works very well in BaseX application itself. However, when I try to run the query from Java code (I'm using Eclipse) by the following code:
BaseXClient session = new BaseXClient("localhost", 1984, "admin", "admin");
try { Context context = new Context();
String q = "let $xml := db:open('Collection','test/xmark.xml')//name let $y := for $x in $xml return $x return db:add('Collection', document {<b>{ $y }</b>} ,'doc1.xml') ";
new XQuery(input).execute(context);
} finally { session.close(); }
I have got the errors:
org.basex.core.BaseXException: Out of Main Memory.
at org.basex.core.Command.execute(Command.java:77)
at org.basex.core.Command.execute(Command.java:89)
I have changed the memory in Eclipse by adding: -vmargs -Xms512m -Xmx1024m
to the eclipse.ini file but it does not help.
Do you think these errors are due to the fact that I have called the query in a wrong way, i.e. new XQuery(input).execute(context);
Thank you very much.
Huy
On 19 Dec 2011, at 00:15, BaseX Support wrote:
Dear Huy,
your questions are welcome. I forgot to mention (and think of the fact) that the current version of BaseX does not allow simultaneous read and write operations, which is why it won't be possible to read data and write it back into a database (find more at [1]). We are currently implementing a more granular locking concept (database-wise), which is a little bit tricky, as a single XQuery can dynamically address arbitrary databses; for now, you'll probably have to cache the results to avoid a deadlock, or use a single XQuery for evaluating and storing results. An example (I'm using the DB module here [2]):
let $xml := db:open('db1')/your/query return db:add('db1', $xml)
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Transaction_Management [2] http://docs.basex.org/wiki/Database_Module#db:add
On Sun, Dec 18, 2011 at 8:56 PM, Huy Vu qhuyvu@gmail.com wrote:
Dear Christian,
Thank you very much for your answer. I have tried to use input and output streams but I could not pipeline the input stream to an output stream.
I have to explicitly create an output stream
ByteArrayOutputStream out = new ByteArrayOutputStream();
new XQuery(inputQuery).execute(context, out);
InputStream in = new ByteArrayInputStream(out.toByteArray());
then I store it to an xml file in my database:
session.execute("open " + Constants.DB);
session.add(fileName, in);
Could you tell me and/or give me the links to the instructions of pipelining the output stream to a database file?
Thank you very much.
Huy
-- Huy Vu Oxford University, United Kingdom http://www.cs.ox.ac.uk/people/huy.vu/
On 18 Dec 2011, at 10:52, BaseX Support wrote:
Dear Huy,
have you looked at the Session classes (LocalSession/ClientSession)? These classes supports input and output streams to do pipelining, such as querying results and sending them to another target without first materializing all data.
Hope this helps, Christian
On Sun, Dec 18, 2011 at 10:27 AM, Huy Vu qhuyvu@gmail.com wrote: > Hello, > > I am writing my code in Java and the result after querying xml files in my BaseX database is very large, about 500MB. Thus, I would like to stream the output to another xml file in the BaseX database. Could you tell me how to do this? > > Many thanks, > > Huy Vu > > > -- > Huy Vu > Oxford University, United Kingdom > http://www.cs.ox.ac.uk/people/huy.vu/ >
basex-talk@mailman.uni-konstanz.de
-
BaseX Support -
Huy Vu