Hello all,
my basex-script should fetch 10.000something XML-files automatically from a website and insert their content into a external PostgreSQL-database. After about 8.000 files my script stops and I get "Out of Main Memory". I found your discussion with "kgfhjjgrn" [1] regarding this issue, but I'm not sure whether these options apply to my problem - I do not build a basex-database but an external one. Will autoflush=false and flushing by myself help with this?
Second, I want to insert some xs:base64Binary into my PostgreSQL database, but I cannot find the correct sql:parameter type for the bytea-column.
Could you please help me with my issues?
Thank you very much, Alexander
[1] http://comments.gmane.org/gmane.text.xml.basex.talk/2540
Hi Alexander,
how does your XQuery/BaseX script look like? If you use the XQuery doc() function, you could try to replace it with parse-xml(fetch:text(...)), because the latter approach will close your documents and free memory if the processed document is not required anymore.
Best, Christian ___________________________
2013/6/24 Alexander von Bernuth alexander.von-bernuth@student.uni-tuebingen.de:
Hello all,
my basex-script should fetch 10.000something XML-files automatically from a website and insert their content into a external PostgreSQL-database. After about 8.000 files my script stops and I get "Out of Main Memory". I found your discussion with "kgfhjjgrn" [1] regarding this issue, but I'm not sure whether these options apply to my problem - I do not build a basex-database but an external one. Will autoflush=false and flushing by myself help with this?
Second, I want to insert some xs:base64Binary into my PostgreSQL database, but I cannot find the correct sql:parameter type for the bytea-column.
Could you please help me with my issues?
Thank you very much, Alexander
[1] http://comments.gmane.org/gmane.text.xml.basex.talk/2540
-- | Alexander von Bernuth | alexander.von-bernuth@student.uni-tuebingen.de
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Christian,
thank you very much, I am going to test this as soon as I am at home. I think this is going to fix my issue. However, do you happen to know which sql:parameter type I have to use when I try to insert xs:base64binary into my bytea-column in postgres?
Thank you again, Alex
Hi Alex,
the mapping of types is defined in the BaseX FNSql class [1]. "bytea" seems to be a PostgreSQL-specific data type, so I’m not sure which mapping would be appropriate here. Could you do some research for us and try to find out which SQL types may give satifying results (see [2] for the existing setters)?
Thanks, Christian
[1] https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/query/f... [2] http://docs.oracle.com/javase/6/docs/api/java/sql/PreparedStatement.html ___________________________
thank you very much, I am going to test this as soon as I am at home. I think this is going to fix my issue. However, do you happen to know which sql:parameter type I have to use when I try to insert xs:base64binary into my bytea-column in postgres?
Thank you again, Alex
-- | Alexander von Bernuth | alexander.von-bernuth@student.uni-tuebingen.de
Am 24.06.2013 um 11:48 schrieb Christian Grün:
Hi Alexander,
how does your XQuery/BaseX script look like? If you use the XQuery doc() function, you could try to replace it with parse-xml(fetch:text(...)), because the latter approach will close your documents and free memory if the processed document is not required anymore.
Best, Christian ___________________________
2013/6/24 Alexander von Bernuth alexander.von-bernuth@student.uni-tuebingen.de:
Hello all,
my basex-script should fetch 10.000something XML-files automatically from a
website and insert their content into a external PostgreSQL-database. After
about 8.000 files my script stops and I get "Out of Main Memory".
I found your discussion with "kgfhjjgrn" [1] regarding this issue, but I'm
not sure whether these options apply to my problem - I do not build a
basex-database but an external one. Will autoflush=false and flushing by
myself help with this?
Second, I want to insert some xs:base64Binary into my PostgreSQL database,
but I cannot find the correct sql:parameter type for the bytea-column.
Could you please help me with my issues?
Thank you very much,
Alexander
[1] http://comments.gmane.org/gmane.text.xml.basex.talk/2540
--
| Alexander von Bernuth
| alexander.von-bernuth@student.uni-tuebingen.de
BaseX-Talk mailing list
BaseX-Talk@mailman.uni-konstanz.de
Hi Christian,
after further reasearch I found one way it works:
let $prepared := sql:prepare($conn, "INSERT INTO imagetest VALUES (?, decode(?, 'base64'))") let $params := sql:parameters <sql:parameter type='string'>{$id}</sql:parameter> <sql:parameter type='string'>{$image}</sql:parameter> </sql:parameters> return sql:execute-prepared($prepared, $params)
where $image contains a xs:base64Binary. This way PostgreSQL itself handles its type bytea and there is no need for a special parameter type. To receive your data from the table again, you may use
SELECT i.id, encode(i.image, 'base64') FROM imagetest i;
Nonetheless, thank you very much for your support.
Best, Alex
basex-talk@mailman.uni-konstanz.de