Hi All,
I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks!
db:create("myDB", "sourceDirectory", "destinationDirectory", map{"ftindex": true(), "language": false()} )
Best, Giuseppe
Hi Giuseppe,
I think it is because you are using xquery and your backfile is huge with respect to your java memory settings. Db:create function is an updating function, so all your inserted data has to go to the pending update list (see [1] for explanation) before database creation, causing out of memory. If you want to automate database creation/update you could have a look at the xml command files, a way to create a batch of BaseX commands ([2]). In my use cases, I found it easier to write XQuery queries, which generate XML Command files, to get rid of memory concerns.
Hoping it helps,
Best regards,
[1] : http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List
[2] : http://docs.basex.org/wiki/Commands
-----Message d'origine----- De : basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] De la part de Giuseppe Celano Envoyé : lundi 23 avril 2018 15:03 À : BaseX Objet : [basex-talk] database creation baseX.8.6.7
Hi All,
I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks!
db:create("myDB", "sourceDirectory", "destinationDirectory", map{"ftindex": true(), "language": false()} )
Best, Giuseppe
Hi Giuseppe,
Apart rom Fabrice’s helpful hints, you could try to enable the ADDCACHE option [1].
Cheers, Christian
[1] http://docs.basex.org/wiki/Database_Module#db:create
On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano celano@informatik.uni-leipzig.de wrote:
Hi All,
I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks!
db:create("myDB", "sourceDirectory", "destinationDirectory", map{"ftindex": true(), "language": false()} )
Best, Giuseppe
Yes, with "addcache" it works! Thanks to both of you.
On Apr 23, 2018, at 3:56 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Giuseppe,
Apart rom Fabrice’s helpful hints, you could try to enable the ADDCACHE option [1].
Cheers, Christian
[1] http://docs.basex.org/wiki/Database_Module#db:create
On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano celano@informatik.uni-leipzig.de wrote:
Hi All,
I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks!
db:create("myDB", "sourceDirectory", "destinationDirectory", map{"ftindex": true(), "language": false()} )
Best, Giuseppe
Dear Christian, I am a curious man, Is the Pending Update List bypassed in some way when using addcache ?
Best regards, Fabrice
-----Message d'origine----- De : basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] De la part de Giuseppe Celano Envoyé : lundi 23 avril 2018 16:53 À : Christian Grün Cc : BaseX Objet : Re: [basex-talk] database creation baseX.8.6.7
Yes, with "addcache" it works! Thanks to both of you.
On Apr 23, 2018, at 3:56 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Giuseppe,
Apart rom Fabrice’s helpful hints, you could try to enable the ADDCACHE option [1].
Cheers, Christian
[1] http://docs.basex.org/wiki/Database_Module#db:create
On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano celano@informatik.uni-leipzig.de wrote:
Hi All,
I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks!
db:create("myDB", "sourceDirectory", "destinationDirectory", map{"ftindex": true(), "language": false()} )
Best, Giuseppe
Hi Fabrice,
I am a curious man,
…a good thing ;)
Is the Pending Update List bypassed in some way when using addcache ?
If addcache is set, the nodes to be added will be written to a temporary database on disk. This way you can circumvent the main memory limitations, but it takes a bit longer (this is is why it’s an optional feature).
Cheers, Christian
-----Message d'origine----- De : basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] De la part de Giuseppe Celano Envoyé : lundi 23 avril 2018 16:53 À : Christian Grün Cc : BaseX Objet : Re: [basex-talk] database creation baseX.8.6.7
Yes, with "addcache" it works! Thanks to both of you.
On Apr 23, 2018, at 3:56 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Giuseppe,
Apart rom Fabrice’s helpful hints, you could try to enable the ADDCACHE option [1].
Cheers, Christian
[1] http://docs.basex.org/wiki/Database_Module#db:create
On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano celano@informatik.uni-leipzig.de wrote:
Hi All,
I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks!
db:create("myDB", "sourceDirectory", "destinationDirectory", map{"ftindex": true(), "language": false()} )
Best, Giuseppe
Yes, I confirm it is a bit longer than using the command via the GUI.
On Apr 23, 2018, at 5:03 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Fabrice,
I am a curious man,
…a good thing ;)
Is the Pending Update List bypassed in some way when using addcache ?
If addcache is set, the nodes to be added will be written to a temporary database on disk. This way you can circumvent the main memory limitations, but it takes a bit longer (this is is why it’s an optional feature).
Cheers, Christian
-----Message d'origine----- De : basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] De la part de Giuseppe Celano Envoyé : lundi 23 avril 2018 16:53 À : Christian Grün Cc : BaseX Objet : Re: [basex-talk] database creation baseX.8.6.7
Yes, with "addcache" it works! Thanks to both of you.
On Apr 23, 2018, at 3:56 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Giuseppe,
Apart rom Fabrice’s helpful hints, you could try to enable the ADDCACHE option [1].
Cheers, Christian
[1] http://docs.basex.org/wiki/Database_Module#db:create
On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano celano@informatik.uni-leipzig.de wrote:
Hi All,
I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks!
db:create("myDB", "sourceDirectory", "destinationDirectory", map{"ftindex": true(), "language": false()} )
Best, Giuseppe
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Fabrice ETANCHAUD -
Giuseppe Celano