March 2011 - BaseX-Talk - mailman.uni-konstanz.de

Vastly different database creation times
by David leigh 15 Mar '11

15 Mar '11

My application starts the basex server (and only stops the server when the application closes) and attempts to open the database that it expects to find. If it doesn't find it, it will create it with the source XML files. When a user wants to update the database, it simply drops it and recreates it from the XML files.actually executing the exact same code. I see that the database creation time for those two tasks is quite different. The first one takes quite a bit longer. If, when the application starts, it DOES find the database, it opens it. If the user then clicks "update" it does the drop/recreate process. If it does it again, it does the drop/recreate process. It seems that in these cases, the first one also takes noticeably longer. Subsequent updates are faster. My conclusion is that the first CREATE DB after starting the server takes longer than all the subsequent ones before the server is stopped. Is this a "memory acquisition" issue? I.e. the server starts with a small memory footprint by default and then simply asks (frequently) for more when it needs it? If so, can it be started with a larger amount initially or tell it to grab it in larger quantities in order to reduce the time for the initial database creation? If that has nothing to do with it, any other ideas? Here is my basex log illustrating these two examples: 18:19:15.419 Server was started. 18:19:15.905 [127.0.0.1:60244] LOGIN admin OK 18:19:16.032 [127.0.0.1:60244] OPEN OpenSongSearch Error: Database 'OpenSongSearch' was not found. 32.63 ms 18:19:16.049 [127.0.0.1:60244] LOGOUT admin OK 18:19:16.469 [127.0.0.1:60245] LOGIN admin OK 18:19:16.493 [127.0.0.1:60245] DROP DB OpenSongSearch OK 0.77 ms 18:19:16.493 [127.0.0.1:60245] SET CREATEFILTER * OK 0.66 ms 18:19:16.494 [127.0.0.1:60245] SET ENTITY true OK 0.42 ms 18:19:16.494 [127.0.0.1:60245] SET FTINDEX true OK 0.36 ms 18:19:16.495 [127.0.0.1:60245] SET INTPARSE true OK 0.36 ms 18:19:16.495 [127.0.0.1:60245] SET PARSER xml OK 0.32 ms 18:19:16.496 [127.0.0.1:60245] SET WILDCARDS true OK 0.34 ms 18:20:31.458 [127.0.0.1:60245] CREATE DB OpenSongSearch C:\Users\David\DOCUME~1\OpenSong\Songs OK 74961.71 ms 18:20:31.460 [127.0.0.1:60245] LOGOUT admin OK 18:20:33.146 [127.0.0.1:60262] LOGIN admin OK 18:20:33.175 [127.0.0.1:60262] OPEN OpenSongSearch OK 9.2 ms 18:20:33.177 [127.0.0.1:60262] INFO DB OK 1.66 ms 18:20:33.177 [127.0.0.1:60262] CLOSE OK 0.38 ms 18:20:33.178 [127.0.0.1:60262] LOGOUT admin OK 18:21:10.314 [127.0.0.1:60270] LOGIN admin OK 18:21:10.340 [127.0.0.1:60270] DROP DB OpenSongSearch OK 7.85 ms 18:21:10.340 [127.0.0.1:60270] SET CREATEFILTER * OK 0.25 ms 18:21:10.341 [127.0.0.1:60270] SET ENTITY true OK 0.29 ms 18:21:10.341 [127.0.0.1:60270] SET FTINDEX true OK 0.23 ms 18:21:10.341 [127.0.0.1:60270] SET INTPARSE true OK 0.23 ms 18:21:10.342 [127.0.0.1:60270] SET PARSER xml OK 0.25 ms 18:21:10.342 [127.0.0.1:60270] SET WILDCARDS true OK 0.25 ms 18:21:17.190 [127.0.0.1:60270] CREATE DB OpenSongSearch C:\Users\David\DOCUME~1\OpenSong\Songs OK 6848.68 ms 18:21:17.195 [127.0.0.1:60270] LOGOUT admin OK 18:21:18.140 [127.0.0.1:60272] LOGIN admin OK 18:21:18.167 [127.0.0.1:60272] OPEN OpenSongSearch OK 9.26 ms 18:21:18.168 [127.0.0.1:60272] INFO DB OK 1.05 ms 18:21:18.169 [127.0.0.1:60272] CLOSE OK 0.36 ms 18:21:18.169 [127.0.0.1:60272] LOGOUT admin OK 18:21:20.636 Server was stopped. 19:11:23.757 Server was started. 19:11:24.319 [127.0.0.1:61004] LOGIN admin OK 19:11:24.899 [127.0.0.1:61004] OPEN OpenSongSearch OK 517.02 ms 19:11:24.903 [127.0.0.1:61004] INFO DB OK 3.75 ms 19:11:24.905 [127.0.0.1:61004] CLOSE OK 2.23 ms 19:11:24.906 [127.0.0.1:61004] LOGOUT admin OK 19:11:25.358 [127.0.0.1:61005] LOGIN admin OK 19:11:25.724 [127.0.0.1:61005] OPEN OpenSongSearch OK 348.05 ms 19:11:25.727 [127.0.0.1:61005] INFO DB OK 2.81 ms 19:11:25.728 [127.0.0.1:61005] CLOSE OK 0.88 ms 19:11:25.729 [127.0.0.1:61005] LOGOUT admin OK 19:11:49.961 [127.0.0.1:61013] LOGIN admin OK 19:11:49.985 [127.0.0.1:61013] DROP DB OpenSongSearch OK 6.82 ms 19:11:49.986 [127.0.0.1:61013] SET CREATEFILTER * OK 0.67 ms 19:11:49.986 [127.0.0.1:61013] SET ENTITY true OK 0.32 ms 19:11:49.987 [127.0.0.1:61013] SET FTINDEX true OK 0.37 ms 19:11:49.987 [127.0.0.1:61013] SET INTPARSE true OK 0.34 ms 19:11:49.988 [127.0.0.1:61013] SET PARSER xml OK 0.4 ms 19:11:49.988 [127.0.0.1:61013] SET WILDCARDS true OK 0.34 ms 19:12:09.591 [127.0.0.1:61013] CREATE DB OpenSongSearch C:\Users\David\DOCUME~1\OpenSong\Songs OK 19602.28 ms 19:12:09.592 [127.0.0.1:61013] LOGOUT admin OK 19:12:10.547 [127.0.0.1:61018] LOGIN admin OK 19:12:10.598 [127.0.0.1:61018] OPEN OpenSongSearch OK 8.42 ms 19:12:10.599 [127.0.0.1:61018] INFO DB OK 0.96 ms 19:12:10.600 [127.0.0.1:61018] CLOSE OK 0.39 ms 19:12:10.601 [127.0.0.1:61018] LOGOUT admin OK 19:12:20.716 [127.0.0.1:61021] LOGIN admin OK 19:12:20.742 [127.0.0.1:61021] DROP DB OpenSongSearch OK 8.13 ms 19:12:20.742 [127.0.0.1:61021] SET CREATEFILTER * OK 0.27 ms 19:12:20.743 [127.0.0.1:61021] SET ENTITY true OK 0.26 ms 19:12:20.743 [127.0.0.1:61021] SET FTINDEX true OK 0.25 ms 19:12:20.744 [127.0.0.1:61021] SET INTPARSE true OK 0.27 ms 19:12:20.744 [127.0.0.1:61021] SET PARSER xml OK 0.25 ms 19:12:20.745 [127.0.0.1:61021] SET WILDCARDS true OK 0.26 ms 19:12:26.555 [127.0.0.1:61021] CREATE DB OpenSongSearch C:\Users\David\DOCUME~1\OpenSong\Songs OK 5811.0 ms 19:12:26.557 [127.0.0.1:61021] LOGOUT admin OK 19:12:27.677 [127.0.0.1:61024] LOGIN admin OK 19:12:27.705 [127.0.0.1:61024] OPEN OpenSongSearch OK 7.94 ms 19:12:27.706 [127.0.0.1:61024] INFO DB OK 0.99 ms 19:12:27.707 [127.0.0.1:61024] CLOSE OK 0.37 ms 19:12:27.707 [127.0.0.1:61024] LOGOUT admin OK 19:12:29.820 Server was stopped. Thanks!

2 2

Re: [basex-talk] Updates and indexing
by Gary Larsen 15 Mar '11

15 Mar '11

Hi Dimitar, Thanks for the info. I see that adding a document changes the database indexing status to off. A query to locate a document by node value in a database with 54k documents went from 3 seconds to 4 minutes. I can image its difficult to balance performance in different aspects of the application and maintain database stability at the same time. For the past couple of weeks Ive tested query performance on a half dozen databases and BaseX queries were almost always quickest and never crashed even when running with lower memory. Depending on how things go with my research I may be able to help test the index updating feature when you are ready for that. Thanks, Gary >Hi Gary, >I'm the one who's working on the index updating. As Michael said it's a work >in progress, but I expect to finish my work soon. Most probably it will be >released at some point after 6.6 (which should be available on March 23). >Best regards, >Dimitar Am Dienstag 15 März 2011, 10:01:20 schrieb Michael Seiferle: > Hi Gary, > > yes this is true either way, still in 6.5.1. > We are currently working on incremental index updates, but I unfortunately > can't give you any hint on a release date. > > For most use-cases recreating the indexes (OPTIMIZE [1]) works fine in day > to day use, but we are aware that this might get a little annoying under > some circumstances. > > Thanks a lot for your feedback, feel free to discuss or ask for more > information. -- > > Kind regards > Michael From: Gary Larsen [mailto:gary.larsen@envisn.com] Sent: Monday, March 14, 2011 8:08 PM To: 'basex-talk(a)mailman.uni-konstanz.de' Subject: Updates and indexing Hi, I was reading a thread in the mail list from Oct 2010 which states updates are not indexed: https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-October/000707.htm l Is this still the case with version 6.5.1? When talking of updates does that mean whole document replacement (or deletion) as well as XQuery updates? Thanks for any clarification, gary

1 0

Best practice approach to data security between orgs while maintaining ability to combine data from multiple orgs
by Charles F. Munat 15 Mar '11

15 Mar '11

Hello, I'm building a Web-based application that tracks students in various schools. Much of the data is tree-structured, and, after numerous attempts, I've given up on using an RDBMS. It's a hammer and I need a wrench. The schema for each school's data will be the same. Sitting above the schools is the "project" for which the application is being built. The project needs to be able to query across multiple or all schools in order to run various analyses. So I want all the data in a single database, if possible. But each school, I think, should be it's own "segment" of the database. That way, when a project manager logs in, he or she can look across all the segments, but when a school coordinator logs in, he or she accesses only the data for that school. I hope this makes sense. I am not really very familiar with how the BaseX DB is organized. I don't want individual XML files (do I?). I just want a database filled with data. So how do "collections" work? How would I segment the database so that schools are separate whiles still providing easy cross-school access for project personnel? I could just do something like this: <project> <project-data/> <school id="1"/> <school id="2"/> </project> Where each school is simply an element under the root "project" element (with project data stashed away in a "project-data" element, or equivalent). But is there a better way? What is the best practice here? I am running the DB unembedded. Thanks! Chas. Munat Somewhere in South America

5 7

Re: [basex-talk] Updates and indexing
by Michael Seiferle 15 Mar '11

15 Mar '11

Hi Gary, yes this is true either way, still in 6.5.1. We are currently working on incremental index updates, but I unfortunately can't give you any hint on a release date. For most use-cases recreating the indexes (OPTIMIZE [1]) works fine in day to day use, but we are aware that this might get a little annoying under some circumstances. Thanks a lot for your feedback, feel free to discuss or ask for more information. -- Kind regards Michael [1] http://docs.basex.org/wiki/Update#Indexes Am 15.03.2011 um 01:08 schrieb Gary Larsen: > Hi, > > > > I was reading a thread in the mail list from Oct 2010 which states updates are not indexed: > > > > https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-October/000707.ht… > > > > Is this still the case with version 6.5.1? When talking of updates does that mean whole document replacement (or deletion) as well as XQuery updates? > > > > Thanks for any clarification, > > gary > >

2 1

Updates and indexing
by Gary Larsen 15 Mar '11

15 Mar '11

Hi, I was reading a thread in the mail list from Oct 2010 which states updates are not indexed: https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-October/000707.htm l Is this still the case with version 6.5.1? When talking of updates does that mean whole document replacement (or deletion) as well as XQuery updates? Thanks for any clarification, gary

1 0

Re: [basex-talk] Query exception
by Leonard Wörteler 11 Mar '11

11 Mar '11

Hi Gary, Am 11.03.2011 14:54, schrieb Gary Larsen: > I thought that this might have something > to do with distinct-values, but using group by syntax produced the same > error. I’ve included a few queries with these errors. could you provide us with an SSCCE [1] or the actual XML files of your database? I wasn't able to reproduce the reported bug(s) yet. > I ran into exception loading data last night but that was on a different > database so I don’t think it’s related. No, that shouldn't affect other databases at all. Cheers, Leo __________ [1] http://sscce.org/

1 0

Query exception
by Gary Larsen 11 Mar '11

11 Mar '11

I'm testing out BaseX and getting remarkable query performance, first on a collection of ~3k documents, and then on ~27k docs (61m nodes). But there were some queries which ran on the small set but had exceptions on the large set. I thought that this might have something to do with distinct-values, but using group by syntax produced the same error. I've included a few queries with these errors. I ran into exception loading data last night but that was on a different database so I don't think it's related. Thanks, Gary Query: declare default element namespace 'http://developer.envisn.com/xmlns/envisn/netvisn/'; for $allInfo in distinct-values(collection('content2')//DbRefImpl/database) order by $allInfo return $allInfo Error: Possible bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 6.5.1 Java: Sun Microsystems Inc., 1.6.0_20 OS: Windows XP, x86 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 28194 org.basex.util.TokenSet.key(TokenSet.java:126) org.basex.data.Data.name(Data.java:386) org.basex.query.item.DBNode.nname(DBNode.java:117) org.basex.query.item.DBNode.qname(DBNode.java:130) org.basex.query.path.NameTest.eval(NameTest.java:107) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.AxisPath.iter(AxisPath.java:437) org.basex.query.path.AxisPath.iter(AxisPath.java:406) org.basex.query.func.FNSeq$3.<init>(FNSeq.java:146) org.basex.query.func.FNSeq.distinctValues(FNSeq.java:144) org.basex.query.func.FNSeq.iter(FNSeq.java:47) org.basex.query.QueryContext.iter(QueryContext.java:301) org.basex.query.expr.For$1.init(For.java:127) org.basex.query.expr.For$1.next(For.java:92) org.basex.query.expr.GFLWOR.iter(GFLWOR.java:289) org.basex.query.expr.GFLWOR.iter(GFLWOR.java:266) org.basex.query.QueryContext.iter(QueryContext.java:301) org.basex.query.QueryContext.iter(QueryContext.java:265) org.basex.query.QueryContext.eval(QueryContext.java:223) org.basex.query.QueryProcessor.execute(QueryProcessor.java:111) org.basex.core.cmd.AQuery.query(AQuery.java:83) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:292) org.basex.core.Command.exec(Command.java:274) org.basex.core.Command.execute(Command.java:67) org.basex.gui.GUI.exec(GUI.java:390) org.basex.gui.GUI$5.run(GUI.java:351) Query: declare default element namespace 'http://developer.envisn.com/xmlns/envisn/netvisn/'; for $allInfo in collection('content2')//DbRefImpl let $db := $allInfo/database group by $db order by $db return $db/text() Error: Possible bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 6.5.1 Java: Sun Microsystems Inc., 1.6.0_20 OS: Windows XP, x86 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 28194 org.basex.util.TokenSet.key(TokenSet.java:126) org.basex.data.Data.name(Data.java:386) org.basex.query.item.DBNode.nname(DBNode.java:117) org.basex.query.item.DBNode.qname(DBNode.java:130) org.basex.query.path.NameTest.eval(NameTest.java:107) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.AxisPath.iter(AxisPath.java:437) org.basex.query.path.AxisPath.iter(AxisPath.java:406) org.basex.query.QueryContext.iter(QueryContext.java:301) org.basex.query.expr.For$1.init(For.java:127) org.basex.query.expr.For$1.next(For.java:92) org.basex.query.expr.GFLWOR.iter(GFLWOR.java:289) org.basex.query.expr.GFLWOR.iter(GFLWOR.java:266) org.basex.query.QueryContext.iter(QueryContext.java:301) org.basex.query.QueryContext.iter(QueryContext.java:265) org.basex.query.QueryContext.eval(QueryContext.java:223) org.basex.query.QueryProcessor.execute(QueryProcessor.java:111) org.basex.core.cmd.AQuery.query(AQuery.java:83) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:292) org.basex.core.Command.exec(Command.java:274) org.basex.core.Command.execute(Command.java:67) org.basex.gui.GUI.exec(GUI.java:390) org.basex.gui.GUI$5.run(GUI.java:351) Query: declare default element namespace 'http://developer.envisn.com/xmlns/envisn/netvisn/'; for $allInfo in distinct-values(collection('content')//DbRefImpl[database='BOEING'][table='D _PROJ']/column) order by $allInfo return $allInfo Error: Possible bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 6.5.1 Java: Sun Microsystems Inc., 1.6.0_20 OS: Windows XP, x86 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 7176 org.basex.io.TableDiskAccess.read4(TableDiskAccess.java:123) org.basex.data.Data.size(Data.java:349) org.basex.data.DataPaths.doc(DataPaths.java:40) org.basex.data.DataPaths.doc(DataPaths.java:57) org.basex.data.Data.doc(Data.java:213) org.basex.query.QueryResources.addCollection(QueryResources.java:243) org.basex.query.QueryResources.collection(QueryResources.java:175) org.basex.query.func.FNGen.collection(FNGen.java:111) org.basex.query.func.FNGen.iter(FNGen.java:55) org.basex.query.QueryContext.iter(QueryContext.java:301) org.basex.query.expr.ParseExpr.value(ParseExpr.java:73) org.basex.query.func.Fun.comp(Fun.java:47) org.basex.query.path.Path.comp(Path.java:40) org.basex.query.expr.Arr.comp(Arr.java:35) org.basex.query.func.Fun.comp(Fun.java:43) org.basex.query.expr.For.comp(For.java:58) org.basex.query.expr.For.comp(For.java:1) org.basex.query.expr.GFLWOR.comp(GFLWOR.java:86) org.basex.query.QueryContext.compile(QueryContext.java:203) org.basex.query.QueryProcessor.compile(QueryProcessor.java:89) org.basex.core.cmd.AQuery.query(AQuery.java:75) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:292) org.basex.core.Command.exec(Command.java:274) org.basex.core.Command.execute(Command.java:67) org.basex.gui.GUI.exec(GUI.java:390) org.basex.gui.GUI$5.run(GUI.java:351)

1 0

limit on number of attributes
by C. M. Sperberg-McQueen 10 Mar '11

10 Mar '11

Trying to use BaseX to index the XML version of the Unicode Database, I got a message saying an element had been encountered with too many attributes (more than 32). Is there a way to lift this limit for a particular database? If one wants to work with the Unicode database in XML (and let's face it, it will be a lot nicer in XML than in the semicolon-delimited form of the early Unicode releases), then one is going to have to work with elements which have a lot of attributes (haven't counted, probably fewer than a hundred). -- C. M. Sperberg-McQueen **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

2 2

renaming a database
by Godmar Back 09 Mar '11

09 Mar '11

Last year [1], the functionality to rename a database was requested. We need this functionality as well. The option to rename the folder would require shutting down, and restarting, the BaseX server, which is not a good idea. The other alternative would be to retrieve the entire XML content of the source db, then create the target db with that content; this is wasteful and inefficient for large db. Please add a db rename functionality to the Java client binding. - Godmar [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-May/000298.html

4 5

Re: [basex-talk] How to delete a node with PHP?
by Christian Grün 08 Mar '11

08 Mar '11

Dear Yoann Maingon, your code snippet looks fine to me. I would assume that a third command won't return any results, as they should have been deleted from your database: print $session->execute(“XQUERY //definition/item[@type=’test’]”); If that's not the case, or if I didn't answer your question, feel free to give us more details on your problem. Christian On Tue, Mar 8, 2011 at 12:46 AM, Yoann Maingon <yoann.maingon(a)prodeos.com> wrote: > > Hi, > > I’ve started using BaseX as a database for a PHP project. I can open the database and add nodes but I can’t delete nodes. > I’ve tested the XQUERY on the GUI and it works just fine. Any clue? > > Here is the xquery: > > Here is the php code: > […] > $session->execute(“OPEN data”); > $session->execute(“XQUERY delete nodes (//definition/item[@type=’test’])”); > > I’ve tried without “XQUERY” but gives the same result. > > Would be nice to have a working example of such use. > > Best Regards, > Yoann Maingon > > --------------------------------------- > Yoann Maingon > Prodeos > +33 6.64.32.49.66 > yoann.maingon(a)prodeos.com > www.prodeos.com

2 1