BaseX-Talk April 2013

basex-talk@mailman.uni-konstanz.de

38 participants
54 discussions

Adding millions of XML files
by freesoft 18 Apr '13

18 Apr '13

Hi, I'm new to BaseX and to XQuery. I already knew XPath. I'm evaluating BaseX to store our XML files and make queries on them. We have to store about 1 million of XML files per month. The XML files are little (~1 KB to 5 KB). So our case is: High number of files, little size. I've read that BaseX is scalable and has high performance, so it is probably a good tool for us. But, in the tests I'm doing, I'm getting an "Out of Main Memory" error when loading high number of XML files. For exaple, if I create a new database ("testdb"), and add 3 XML files, no problem occurs. Files are stored correctly, and I can make queries on them. Then, if I try to add 18000 XML files to the same database ("testdb") (by using GUI > Database > Properties > Add Resources), then I see how the coloured memory bar grows and grows... until an error appears: Out of Main Memory. You can try to: - increase Java's heap size with the flag -Xmx<size> - deactivate the text and attribute indexes. The text and attribute indexes are disabled, so it is not the cause. And I increased the Java size with the flag -Xmx<size> (by editing the basexgui.bat script), and same error happens. Probaly BaseX loads all files to main memory first, and then dumps them to the database files. That shouldn't be done in that way. For each XML file, it should be loaded into main memory, then procesed and then dumped to the db files. For each file, independently from the rest. So I have two questions: 1. Do I have to use an special way to add high number of XML files? 2. Is BaseX sufficiently stable to store and manage our data (about 1 million of files will be added per month)? Thank you for our help and for your great software, and excuse me if I am asking for recurrent questions.

3 7

timezone formatting
by Marco Lettere 17 Apr '13

17 Apr '13

Hi again, when, in Basex 7.6, formatting a datetime with format-dateTime(current-dateTime(), "[Y0001][M01][D01][h01][m01][s01][Z0000]") I was expecting something like 20130314035118+0100 for my area (Central Europe) and instead I get 201303140351180060. Looks like minutes are not translated to hours and the sign is missing. For documentation I'm referring to http://www.w3.org/TR/xpath-functions-30/#rules-for-datetime-formatting. Thanks, M.

2 3

Re: [basex-talk] Adding millions of XML files
by pw＠themail.co.uk 17 Apr '13

17 Apr '13

Responding to the last question about querying over collections: I had the same issue and Lukas Kircher provided the answer - http://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg0210 0.html > > > >---- Original Message ---- >From: kqfhjjgrn(a)yahoo.es >To: christian.gruen(a)gmail.com, fetanchaud(a)questel.com >Subject: Re: [basex-talk] Adding millions of XML files >Date: Mon, 15 Apr 2013 13:12:55 +0100 (BST) > >>Worked! :-) >> >>I have uninstalled 7.6 and installed 7.7 beta. Then, created the >empty db, added the 3 files, run the "set addcache true" command, >added the 17828 files... and no "out of memory" error, just the >processing info: >> >> Path "everything" added in 462943.7 ms. >> >>that is ~8 minutes (in my development machine, not in our server). >> >>Now I'm going to do some more tests (both for adding and for >quering), and I'm going to try the "basex" command, in order to add >XML files automatically to the db. >> >>Anyway I would ask some more questions: >> >>1. Is 7.7 beta sufficiently stable to be used in our production >server? Shoud I wait for the final 7.7 release? >> >>2. Is the "addcache" property value permanently saved to the db? >Should I run the "set addcache true" command everytime I add files? >> >>3. Should I keep disabled the Text & Attribute indexes? Is the >"addcache=on" option sufficient to allow the adition of XML files, so >I can enable those indexes? Will my queries be slow with those >indexes disabled? >> >>4. Should I run Optimize after every massive insertion (even with >"addcache=on")? >> >>Thank you for the information on limits, it is very useful. In >particular, the following limits: >> >>FileSize: 512 GiB >>#Files: 536,870,912 >> >>mean a medium value of exactly 1 KB/file. Since my files are bigger >than 1 KB (in medium), then the size limit will be reached first (512 >GiB). So my Perl scripts will have to detect the size of the db, and >if it is bigger than ~500 GB, then they will create a new db and add >new XML files to it. >> >>Please show me an easy example of how to use several databases in >the same query. Perhaps something like: >> >> for $doc in (collection("db1"), collection("db2")) >> for $node in $doc/$a_node_path >> etc... >> >>Well, thank you very much for your help. And excuse me for the huge >amount of questions from a newbie like me :-) >> >> >>freesoft >> >> >> >>________________________________ >> De: Christian Grün <christian.gruen(a)gmail.com> >>Para: Fabrice Etanchaud <fetanchaud(a)questel.com> >>CC: freesoft <kqfhjjgrn(a)yahoo.es>; >"basex-talk(a)mailman.uni-konstanz.de" ><basex-talk(a)mailman.uni-konstanz.de> >>Enviado: Lunes 15 de abril de 2013 12:12 >>Asunto: Re: [basex-talk] Adding millions of XML files >> >> >>Hi kgfhjjgrn, >> >>I believe that Fabrice already mentioned all details that should >help >>you to build larger databases. The ADDCACHE option [1] (included in >>the latest stable snapshot [2]) may already be sufficient to add >your >>documents via the GUI: simply run the "set addcache true" command >via >>the input bar of the main window before opening the Properties >dialog. >> >>Note that you can access multiple databases with a single XQuery >call, >>so if you know that youll exceed the limits of a single database at >>some time (see [3]), simply create new databases in certain >intervals. >> >>Hope this helps, >>Christian >> >>[1] http://docs.basex.org/wiki/Options#ADDCACHE >>[2] http://files.basex.org/releases/latest/ >>[3] http://docs.basex.org/wiki/Statistics >>_________________________________________ >> >>> The size of your test should not cause any problem to BaseX (18 >000 files >>> from 1 up to 5 KB) >>> >>> >>> >>> 1. Did you try to set the ADDCACHE option ? >>> >>> 2. You should OPTIMIZE your collection after each batch of >ADD >>> commands, even if no index is set. >>> >>> 3. Did you try to unset the AUTOFLUSH option, and explicitly >FLUSH the >>> updates at batchs end ? >>> >>> 4. The GUI may not be the best place to run updates, did you >try the >>> basex command line tools ? >>> >>> >>> >>> Opening a collection containing a huge number of documents may >take a long >>> time from my experience. >>> >>> It seems to be related to the kind of memory data structure used >to store >>> the document names. >>> >>> A workaround could be to insert your documents under a common root >xml >>> element with XQuery Update. >>> >>> >>> >>> >>> >>> >>> Best, >>> >>> Fabrice Etanchaud >>> >>> Questel-Orbit >>> >>> >>> >>> >>> >>> De : basex-talk-bounces(a)mailman.uni-konstanz.de >>> [mailto:basex-talk-bounces@mailman.uni-konstanz.de] De la part de >freesoft >>> Envoyé : lundi 15 avril 2013 10:19 >>> À : basex-talk(a)mailman.uni-konstanz.de >>> Objet : [basex-talk] Adding millions of XML files >>> >>> >>> >>> Hi, I'm new to BaseX and to XQuery. I already knew XPath. I'm >evaluating >>> BaseX to store our XML files and make queries on them. We have to >store >>> about 1 million of XML files per month. The XML files are little >(~1 KB to 5 >>> KB). So our case is: High number of files, little size. >>> >>> I've read that BaseX is scalable and has high performance, so it >is probably >>> a good tool for us. But, in the tests I'm doing, I'm getting an >"Out of Main >>> Memory" error when loading high number of XML files. >>> >>> For exaple, if I create a new database ("testdb"), and add 3 XML >files, no >>> problem occurs. Files are stored correctly, and I can make queries >on them. >>> Then, if I try to add 18000 XML files to the same database >("testdb") (by >>> using GUI > Database > Properties > Add Resources), then I see how >the >>> coloured memory bar grows and grows... until an error appears: >>> >>> Out of Main Memory. >>> You can try to: >>> - increase Java's heap size with the flag -Xmx<size> >>> - deactivate the text and attribute indexes. >>> >>> The text and attribute indexes are disabled, so it is not the >cause. And I >>> increased the Java size with the flag -Xmx<size> (by editing the >>> basexgui.bat script), and same error happens. >>> >>> Probaly BaseX loads all files to main memory first, and then dumps >them to >>> the database files. That shouldn't be done in that way. For each >XML file, >>> it should be loaded into main memory, then procesed and then >dumped to the >>> db files. For each file, independently from the rest. >>> >>> So I have two questions: >>> 1. Do I have to use an special way to add high number of XML >files? >>> 2. Is BaseX sufficiently stable to store and manage our data >(about 1 >>> million of files will be added per month)? >>> >>> Thank you for our help and for your great software, and excuse me >if I am >>> asking for recurrent questions. >>> >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk(a)mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>

2 1

Proposition. Execution feedback
by Alexander Shpack 15 Apr '13

15 Apr '13

Hi! It would be perfect to take execution feedback from GUI or server. Execution plan: N nodes Processed: K nodes It may help in script optimization and understanding complexity of xQuery task -- s0rr0w

2 1

Use xquery:eval in insert statement
by Lauener Jonas 15 Apr '13

15 Apr '13

I want to use xquery:eval in an insert statement: declare updating function local:insert($document as xs:string, $input as element()) { insert node $input after xquery:eval(fn:concat($document, '//', node-name($input), '[last()]')) }; let $input := <Customer CustomerID="IZZTT"><CompanyName>App</CompanyName><ContactName>Steve Jobs</ContactName><ContactTitle>CEO in Heaven</ContactTitle><Phone>(777) 7777777777</Phone><FullAddress><Address>Apple Road 3</Address><City>Cuppertino</City><Region>OR</Region><PostalCode>00000</PostalCode><Country>Heaven</Country></FullAddress></Customer> return local:insert("doc('CustomersOrders.xml')", $input) On the BaseX GUI it works fine, but if I want to use it with the Java Client, it fails. I use this code in Java: public final class InsertExample { /** Hidden default constructor. */ private InsertExample() {} /** Session reference. */ static ClientSession session; /** * Main method. * @param args command-line arguments */ public static void main(final String[] args) { try { // Create a client session with host name, port, user name and password session = new ClientSession("localhost", 1984, "admin", "admin"); String command = "declare updating function local:insert($document as xs:string, $input as element()) {\n insert node $input after xquery:eval(fn:concat($document, '//', node-name($input), '[last()]'))\n};\n\nlet $input := <Customer CustomerID=\"IZZTT\"><CompanyName>App</CompanyName><ContactName>Steve Jobs</ContactName><ContactTitle>CEO in Heaven</ContactTitle><Phone>(777) 7777777777</Phone><FullAddress><Address>Apple Road 3</Address><City>Cuppertino</City><Region>OR</Region><PostalCode>00000</PostalCode><Country>Heaven</Country></FullAddress></Customer>\n\nreturn local:insert(\"doc('CustomersOrders')\", $input)"; System.out.println(command); XQuery xquery = new XQuery(command); System.out.println(session.execute(xquery)); // close session session.close(); } catch(IOException e) { e.printStackTrace(); } } } When I run it, I get this exception: Version: BaseX 7.6 Java: Oracle Corporation, 1.7.0_11 OS: Windows 8, amd64 Stack Trace: java.lang.RuntimeException: Exclusive lock could not be acquired. org.basex.util.Util.notexpected(Util.java:53) org.basex.io.random.TableDiskAccess.lock(TableDiskAccess.java:151) org.basex.data.DiskData.startUpdate(DiskData.java:202) org.basex.query.up.DatabaseUpdates.startUpdate(DatabaseUpdates.java:158) org.basex.query.up.ContextModifier.apply(ContextModifier.java:86) org.basex.query.up.Updates.apply(Updates.java:120) org.basex.query.QueryContext.update(QueryContext.java:270) org.basex.query.QueryContext.value(QueryContext.java:255) org.basex.query.QueryContext.iter(QueryContext.java:240) org.basex.query.QueryProcessor.iter(QueryProcessor.java:76) org.basex.core.cmd.AQuery.query(AQuery.java:84) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.server.ClientListener.run(ClientListener.java:145) at org.basex.server.ClientSession.receive(ClientSession.java:259) at org.basex.server.ClientSession.execute(ClientSession.java:162) at org.basex.server.ClientSession.execute(ClientSession.java:167) at org.basex.server.Session.execute(Session.java:37) at org.basex.examples.api.InsertExample.main(InsertExample.java:42) Is this a bug in BaseX? Or how can I solve this issue? Thanks, Jonas

2 2

NegativeArraySizeException
by Francis Avila 14 Apr '13

14 Apr '13

Received the following stack trace after running a .bxs script using basexclient from the command line: Version: BaseX 7.6 Java: Sun Microsystems Inc., 1.6.0_27 OS: Linux, amd64 Stack Trace: java.lang.NegativeArraySizeException org.basex.io.random.TableDiskAccess.insert(TableDiskAccess.java:390) org.basex.data.Data.insert(Data.java:931) org.basex.data.Data.insert(Data.java:820) org.basex.data.atomic.Insert.apply(Insert.java:31) org.basex.data.atomic.AtomicUpdateList.applyStructuralUpdates(AtomicUpdateList.java:297) org.basex.data.atomic.AtomicUpdateList.execute(AtomicUpdateList.java:285) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:183) org.basex.query.up.ContextModifier.apply(ContextModifier.java:90) org.basex.query.up.Updates.apply(Updates.java:120) org.basex.query.QueryContext.update(QueryContext.java:270) org.basex.query.QueryContext.value(QueryContext.java:255) org.basex.query.QueryContext.iter(QueryContext.java:240) org.basex.query.QueryProcessor.iter(QueryProcessor.java:76) org.basex.core.cmd.AQuery.query(AQuery.java:84) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.server.ClientListener.run(ClientListener.java:145) What the script does is very complex (large xquery functions, db:node-pre() indexing and lookups, etc) so if you need it to isolate this issue we can discuss directly. However I get a similar stack trace when I attempt to OPTIMIZE ALL: > open deepbills Database 'deepbills' was opened in 114.99 ms. > optimize all Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.6 Java: Sun Microsystems Inc., 1.6.0_27 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 65 org.basex.util.Compress.pull(Compress.java:156) org.basex.util.Compress.unpack(Compress.java:112) org.basex.data.DiskData.txt(DiskData.java:268) org.basex.data.DiskData.text(DiskData.java:235) org.basex.io.serial.Serializer.node(Serializer.java:343) org.basex.io.serial.Serializer.serialize(Serializer.java:99) org.basex.core.cmd.OptimizeAll$DBParser.parse(OptimizeAll.java:199) org.basex.build.Builder.parse(Builder.java:73) org.basex.build.DiskBuilder.build(DiskBuilder.java:90) org.basex.core.cmd.OptimizeAll.optimizeAll(OptimizeAll.java:124) org.basex.core.cmd.OptimizeAll.run(OptimizeAll.java:44) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.server.ClientListener.run(ClientListener.java:145) > info database Database Properties Name: deepbills Size: 901 MB Nodes: 25598080 Documents: 15528 Binaries: 0 Timestamp: 2013-04-08-15-02-07 Resource Properties Timestamp: 2013-04-08-15-02-07 Encoding: UTF-8 Whitespace Chopping: ON Indexes Up-to-date: false Text Index: OFF Attribute Index: OFF Full-Text Index: OFF UPDINDEX: OFF MAXCATS: 100 MAXLEN: 96 If I restore from a backup (taken after this issue appeared), I still can't "optimize all", but the stack trace appears truncated: > restore deepbills 'deepbills-2013-04-08-15-54-32.zip' was restored in 40918.34 ms. > optimize all No database opened. > open deepbills Database 'deepbills' was opened in 136.22 ms. > optimize all Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.6 Java: Sun Microsystems Inc., 1.6.0_27 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException > If I restore a backup that is just a little bit older (a little over an hour), it seems fine: > restore deepbills-2013-04-08-14-22-22 'deepbills-2013-04-08-14-22-22.zip' was restored in 38982.91 ms. > optimize all Database 'deepbills' was optimized in 129658.46 ms. So I have two similar databases, one 363MB zip, one 372MB zip, one of which is fine and the other which seems to encounter this problem. Could I be looking at database corruption here? If so, how to fix? -- Francis Avila Senior Developer Dancing Mammoth, Inc. (Formerly PJ Doland Web Design, Inc.) P: 703.621.0990 E: favila(a)dancingmammoth.com http://dancingmammoth.com

3 5

Re: [basex-talk] Tokenize..
by Christian Grün 14 Apr '13

14 Apr '13

> With "Info Node", I mean that I want to have tokens for a specific node. !! Got it. You’ll have to retrieve the text of this node and call ft:tokenize(). An example: for $x in //text() return ft:tokenize($x) Please note that the tokenize function won’t aggregate the tokens. You could try the following XQuery code to get a similar output (note that this particular example only works with 7.7, which includes support for general FLWOR expressions): let $text := 'a b a' for $words in ft:tokenize($text) group by $w := $words let $count := count($words) order by $count descending return <entry count="{ $count }">{ $w }</entry> _____________________________________________ > On Sun, Apr 14, 2013 at 2:56 AM, Christian Grün <christian.gruen(a)gmail.com> > wrote: >> >> Hi John, >> >> to give you a somewhat satisfying answer, I’m afraid I need more details. >> >> > I want to create tokens, so I will be using ft:tokens('Docs'). >> > This will create tokens for all the available nodes. >> > I want to create tokens for the Info Node only... How to ,,? >> >> The result of ft:tokens() will contain all tokens that are stored in >> the full-text index of the addressed database. At this point I’m >> stuck, because I can’t guess what you mean with "Info Node"? >> >> Best, >> Christian > > > > > -- > Have a nice day > JBest > >

1 0

Tokenize..
by John Best 13 Apr '13

13 Apr '13

Hi there, The structure of my XML DB (Docs) is - <Docs> <Doc> <Info></Info> <Desc></Desc> </Doc> </Docs> I want to create tokens, so I will be using ft:tokens('Docs'). This will create tokens for all the available nodes. I want to create tokens for the Info Node only... How to ,,? -- Have a nice day JBest

2 1

Multiple queries in client API
by Michael CLEMENT 13 Apr '13

13 Apr '13

Hello, I am building a Django application using BaseX as its database, using a custom model layer with the Python API. Currently, I am using a single Session instance to connect to the BaseX server, but I am facing problems when multiple queries are sent at the same time. I didn't check how the client API works in details, but is it made in such a way that only one query at a time can be sent? Should I use a new Session instance for every single query sent to the server? Thanks in advance. Regards, Michael Clement

3 4

Proposition for the GUI features
by Alexander Shpack 13 Apr '13

13 Apr '13

Hi everyone! I have a nice idea, how to stop to crack the database, when user use GUI and server simultaneously. When GUI will found launched server, every updating functions should be executed through port, not directly. User may setup different connections (require list of connections), select the default connection, and choose method of connection: direct access or via saved one Sorry for my bad English. Alexander

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk April 2013