- BaseX-Talk - mailman.uni-konstanz.de

DB inconsistency
by Marco Lettere 26 Jan '15

26 Jan '15

Hi all, we have an installation that has been running in production for nearly two months now. It's very insertion intensive and from time to time we need to run a db:optimize to keep it on track because the time it takes to serve a request turns from about 2 seconds to 100k seconds. But after the optimize it again runs with the expected performance. Today we noticed again a strange difficulty in serving requests so we tried the usual approach but we got this exception during the optimize: [SENR0001] Attributes cannot be serialized:attribute determinerCode {"INSTA.... Upon running an inspect command, we got a long list of EOF exceptions followed by the resume stating about 85K wrong parent/descendants relationships. Is there any idea on what could have gone wrongly and if possible any way to recover from this? Thank you very much. Regards, Marco. java.io.EOFException at java.io.RandomAccessFile.readFully(RandomAccessFile.java:446) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) at org.basex.io.random.TableDiskAccess.readBlock(TableDiskAccess.java:525) at org.basex.io.random.TableDiskAccess.readPage(TableDiskAccess.java:489) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:148) at org.basex.data.Data.kind(Data.java:290) at org.basex.core.cmd.Inspect.inspect(Inspect.java:50) at org.basex.core.cmd.Inspect.run(Inspect.java:27) at org.basex.core.Command.run(Command.java:360) at org.basex.core.Command.execute(Command.java:94) at org.basex.api.client.LocalSession.execute(LocalSession.java:122) at org.basex.api.client.Session.execute(Session.java:36) at org.basex.core.CLI.execute(CLI.java:106) at org.basex.core.CLI.execute(CLI.java:90) at org.basex.BaseX.console(BaseX.java:197) at org.basex.BaseX.<init>(BaseX.java:171) at org.basex.BaseX.main(BaseX.java:42) Checking main table (43626334 nodes): - 0 invalid node kinds - 0 invalid parent references - 84663 wrong parent/descendant relationships (pre: 6299342,..) Warning: Database is inconsistent. 'db' inspected in 21789.24 ms.

2 3

Automatic replication and fail-over scenario
by Andreas Jung 25 Jan '15

25 Jan '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, we have a customer requirement that an application server will talk to an XML database. The production system (app server + db) will reside in a data center near the customer. However the customer requires having a fail-over installation in a different data center (in a different geographical location) in the case of disaster or whatever. As far as I see BaseX does not support multi-master replication. What are the best approaches for dealing with a fail-over where the fail-over database should have a copy of the production database that is as recent as possible? Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJUxKtJAAoJEADcfz7u4AZjCjULwN2BnpIqhiBnaJc7dPVtbY+v aaeCYmbuCpACMZyY7qLpYBVpBFghAvEA9VIf7HSmNspGBIf+818vOIj2/Ure9keU jcgWIM53qa30Kv28TSt4R6sMPKFStrmWU1EqeDfklQo7bncU9XEL+iH4Z+CdsSZj FrIejK0abVgcHhgRluM80pSeJGkjelmgzMld67UQOv9tOPJueYezb72TtUyajchN EsObhAge6dk4oHwdIi+N2Ek4V9f/GqIvVrgWbXKpSapQxM07mX/HN+yEDWbX0oUZ SxcWzwJ+JS1vzKJtlI9+FZJpnP/Ta8pkqxFHKldWAt/lofdt1p7CxNxFdl9RZPQr yhchWZOs1lYHuYhN8iUCe8kO04TjDE7eoOKmZuc9AmgUZcd/pzLaVuiy4nyjLECZ gVkQpuDe1A2NtICBlh7Kjq4Wr+/OiTz+dnjdaZhbjJ9M2EN0I9SZIrTjq/VOjHD2 rdh0sD+vRwq6OeDM/WIYSx2cuV1riIk= =h79D -----END PGP SIGNATURE-----

2 1

infelicity in the default web.xml settings? and failure logging?
by C. M. Sperberg-McQueen 24 Jan '15

24 Jan '15

Experimenting a bit with the RESTXQ interface yesterday, using basexhttp I found that the default web.xml file shipped with BaseX 7.9 has an unfortunate property. The first servlet mapping listed is <servlet-mapping> <servlet-name>RESTXQ</servlet-name> <url-pattern>/*</url-pattern> </servlet-mapping> and the last servlet mapping is the default servlet, used for static data: <servlet-mapping> <servlet-name>default</servlet-name> <url-pattern>/static/*</url-pattern> </servlet-mapping> If I understand the Servlet specification correctly, this pattern of mappings ensures that a request for (say) a static resource will invariably be passed to the RESTXQ interface and not to RESTXQ. Note that there is some doubt as to whether I do understand the spec correctly: on my reading, a request for /rest/ should also be passed to RESTXQ, and it's clearly not. But I was unable to get Jetty to serve any static content with the default web.xml. I began to succeed when I moved the servlet mappings and changed their patterns, along the lines of the web.xml file I found with Andy Bunce's GraphXQ example, so the first and last are: <servlet-mapping> <servlet-name>default</servlet-name> <url-pattern>/</url-pattern>  </servlet-mapping> <servlet-mapping> <servlet-name>RESTXQ</servlet-name> <url-pattern>/restxq/*</url-pattern>  </servlet-mapping> There may be other ways to set the web.xml file up so that static resources are served (in particular, I don't think it's required to change "/static/*" to "/" -- but I have not experimented systematically with all options, because I'm interested in making my web application work, not in figuring out what relationship holds between the Servlet spec and Jetty's behavior. Is this a problem others have run into? Is there a better solution? And a related question -- before I changed the web.xml file (and after several unsuccessful changes), requests for any of /test.xhtml /mydirectory/test.xhtml /static/test.xhtml /static/mydirectory/test.xhtml elicited the response "Not found" (or sometimes "no function found"), but I found no entries for these requests in the BaseX logs. (And I was unsuccessful in attempting to turn Jetty's stderr or request logging on.) Are failures of this kind not written to the BaseX logs? If they aren't written to the BaseX logs, but would be written to Jetty logs, then I wonder if it would be helpful to have a few hints on turning Jetty logging on (and finding out where the log files get written) either in the BaseX documentation wiki or in comments in the default jetty.xml and web.xml. Since Jetty's documentation mostly assumes that its readers are Java programmers actively developing Java web apps, I for one found it unhelpful when trying to solve my problem. -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

3 5

Fw: Extracting a collection from BaseX
by Alex R. Loayza 23 Jan '15

23 Jan '15

Hello, I'm finding no easy way to extract the contents of a collection to a filesystem directory. For example, In database "myDB", I have a number of XML files inside a collection named "myCollection" (a subfolder of "myDB"). I'm looking for a command or function to extract those files to "myHardDriveDirectory" in my filesystem. I tried file:write(), but it works only with a single file. I also tried db:export(), but it outputs the entire database, which is overkill. So, I ended up doing this: for $doc in collection('myDB/myCollection')return file:write('D:\myHardDriveDirectory\'||$doc/pathToADateAttribute||' Report.xml', $doc) It works (I get a bunch of "2015-01-20 Report.xml" files), but I had to use a FLWOR and dynamically construct each of the names of the files to be written to disk. This could be made much simpler with something like: db:export('myDB/myCollection', 'D:\myHardDriveDirectory') I may be missing some other way to do this. But, in case there is no alternative, would you please consider enhancing db:export() to support outputting collections (or any equivalent solution you may deem better)? It would be great to have that shipped in BaseX 8. I had also set up Webdav to try the quick drag-and-drop way, but unfortunately it has several bugs and had to stop using it. Bonus question: Is there a way to do the reverse operation with some function/command? That is, get the XML files in a filesystem directory to replace any namesake XML files in a specific collection? db:replace() works with a single file only, according to the documentation. db:add() works with filesystem directories, but would create duplicate resources in the database, instead of replacing the existing ones. The only ways I've found so far are: First by using the GUI; I have to delete the entire collection and rebuild it with the "Add resources" dialog box. The second way is to use db:delete() on the collection and then db:add(). But both are lengthy/verbose ways of doing this. Is there already a better way? If not, do you think it may be possible to also enhance db:replace() for it to work with filesystem directories and database collections? Thank you very much for any help. Alex

2 1

BaseX 8 and oXygen via XQJ
by Lizzi, Vincent 23 Jan '15

23 Jan '15

After upgrading to BaseX 8.0 I'm no longer able to run queries on BaseX from oXygen. Connecting to BaseX from oXygen using WebDav still works, but trying to run a transformation in BaseX using the XQJ driver fails with a message "Access denied". It appears that the XQJ driver needs to be updated for the new authentication mechanism. Will an updated version of the BaseX XQJ driver be available soon? Thank you, Vincent

2 2

Pre-compilled Query in Base-X
by ankit kumar 23 Jan '15

23 Jan '15

Hi, I was exploring BaseX for my XML based product. I was wondering whether there is Prepared Statement Like facility incorporated in BaseX, With the same QueryProcessor instance, is it possible to execute it multiple times with different binding values for given variable. I am executing following code. QueryProcessor pr = new QueryProcessor("declare variable $a external ; $a",context); pr.context(itr); pr.bind("a","ABC"); Resultfinalr=pr.execute(); System.out.println("Final Execution Result 1 ::" + finalr); // rebinding $a with different value pr.bind("a", "XYZ"); Result out1 = pr.execute(); System.out.println("Final Execution Result 2 ::" + out1); ******************************************************************************* I am getting "ABC" when I am executing second time. I need this because I would be executing the same query multiple times. Thanks Anky

2 7

text() in XQuery
by Ana Victória Ladeira 22 Jan '15

22 Jan '15

Hi! I'm relatively new to BaseX/XQuery and currently using a XML with the following format: <?xml version="1.0" encoding="UTF-8" standalone="no"?> <modsCollection xmlns="http://www.loc.gov/mods/v3" xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.loc.gov/mods/v3 http://www.loc.gov/standards/mods/v3/mods-3-0.xsd "> <mods ID="ISI:000330282600027" version="3.0"> <titleInfo> <title>Minimum Relative Entropy for Quantum Estimation: Feasibility and General Solution</title> </titleInfo> The file is called "quantuminformation.xml", inside the DB "ExemploBibtex" I'm trying to extract the titles of all articles in the document, using the following query: for $doc in collection('ExemploBibtex') where matches(document-uri($doc), 'quantuminformation.xml') return $doc/modsCollection/mods/titleInfo/title/text() It doesn't matter if I use the text() or fn:string, the result is always empty. What am I doing wrong? Sorry for bothering and thank you for the attention.

2 1

Unable to get started
by Master Boyer 22 Jan '15

22 Jan '15

Hi Folks, I'm running Windows 7, IIS, and PHP5, with BaseX 7.9. The GUI works great, and I'm running test using localhost:8984 Even after I've tried [Start BaseX Server], or basexhttp.bat .. still no luck. I've tried the examples and all I get is a Blank Screen, not even a header or error. All settings are stock, nothing has been changed. This isn't working just yet: $session = new Session("localhost", 1984, "admin", "admin"); I have all the files, BaseXClient.php and so on. If anyone knows a way for me to determine whether or not the server is operational, please help. I've also tried NETSTAT, but not sure correct for this test, but was hoping to see something listening on port 1984. Thanks - Master Boyer

2 1

XQuery Optimization suggestions
by Mansi Sheth 20 Jan '15

20 Jan '15

Hello, I am doing some performance analysis on size of XML files in DB, no of records in a result set and how much time it takes to get me results. Currently, I have 150GB worth of XML documents imported into BaseXDB. It took roughly 21 minutes to return back result set worth 5.3 million records. Queries are of below form: http://localhost:8984/rest?run=get_query.xq&n=/Archives/*/descendant::c/des…, "xyz")]/@name/data() XQUERY File: for $db in db:list() (: Assign dynamic variables to generate query, to be used in eval :) let $query := "declare variable $db external; " || "db:open($db)" || $n return xquery:eval($query,map { 'db': $db, 'query': $n }) I have been few questions around this. 1. I have been routinely advices on this email chain, to avoid serialization from XPATH and let xquery handle it. I tried a few things like replacing strings() with data(), adding serialization option on REST call, in XQUERY file etc. But, I don't see any performance gain. Is there something else I can try or something, I am doing wrong ? 2. Does anyone have any resource to compare this performance to other NoSQL databases. I am just very curious, how above performance numbers compares to other DBs ? - Mansi

3 6

Re: [basex-talk] Linux lessons
by Dirk Kirsten 19 Jan '15

19 Jan '15

Hi Wendell, On 01/10/2015 10:31 PM, Wendell Piez wrote: > Hi Dirk, > > I am still looking at this - I hope you'll be patient with another > couple of questions. If you don't have the time to instruct me please > let me know and accept my apologies. :-) Sure, no problem. We are happy to help, but please always cc the mailing list, so others can benefit from this questions as well. I am quite sure you are not the only one wondering about stuff like this. > > All the projects are local, and they typically rely on other > applications in addition to BaseX (they tend to be XML/XSLT > applications). So for convenience I want to run a single copy of BaseX > with a single data directory, with different databases for the > different projects. > > It's fine if multiple servers run the different projects on multiple > ports, but it's also fine if they all live together on one (8984 is > fine). So there could be a single startup script for all the projects > (as long as the code supporting each project can be distributed), or > each project could have its own server. > > I guess that means running something like the basexhttp shell script > to run the server from a different directory, setting it up with its > own webapp and thus all the advantages (and encapsulation) you cite? > > How do I go about doing this? I have tried copying and modifying the > basexhttp script, but I can't get the classpaths correct. (It seems to > be designed to depend on its location in the file system relative to > BaseX.) I am afraid I am missing a simple way to start up BaseX to run > restXQ, except keep the latter code in a project directory. -- whether > with a completely different webapp, or sharing the "space" among > several projects - as long as their restXQ files can be found. I would also advise against copying the basexhttp script. It is not really a clean solution, involves lots of manually copying if we change the script in the next version (and for the 8.0 snapshots this did happen). Also, copying the binary itself by principle sounds like not the best solution. > > So -- I can give the project its own webapp directory, no problem. I > can put a jetty.xml file in there. But how do I start BaseX (installed > in /home/wendell/basex) to run from this webapp? (If my script calls > ~/bin/basexhttp, BaseX starts up with its own webapp and .basex > configuration etc.) So if I understood your setup correctly, all BaseX settings in your different projects are the same except the RESTXQ path? So one option could be to set the RESTXQ path manually before starting basexhttp. Using the latest 8.0 snapshot, this is done using the global variable BASEX_JVM. So executing export BASEX_JVM=-Dorg.basex.RESTXQPATH=/my/restxq/directory && basexhttp should use the path provided on the command line. If you want to startup all your projects this way you could write a little script which does this for you. If you call baexhttp from a subdirectory (e.g. /home/wendell/superawesomeproject), it shouldn't startup the standard BaseX webapp, but instead look for a web.xml in /home/wendell/superawesomeproject/src/main/webapp/WEB-INF/. I am not sure right now how to change this path or if this is even possible using basexhttp. Another option - which I personally prefer, also because it avoids creating your own scripts and provides easy configuration options - is using Maven. This way, a simple mvn jetty:run inside your project directory should be enough. The pom.xml configures the Maven execution. For example, the jetty maven plugin configuration looks something like this: <plugin> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty-maven-plugin</artifactId> <version>${jettyVersion}</version> <configuration> <webAppConfig> <contextPath>/</contextPath> <descriptor>WEB-INF/${webXml}</descriptor> <tempDirectory>.tmp</tempDirectory> </webAppConfig> <webAppSourceDirectory>${basedir}/${webapp}/public</webAppSourceDirectory> <jettyXml>WEB-INF/jetty.xml</jettyXml> <stopKey>keyToStopJetty</stopKey> <stopPort>${stopPort}</stopPort> </configuration> </plugin> Notice, that in this case the web.xml is actually in a variable. This way, you could startup different web applications using the command mvn jetty:run -Dwebapp=project1-web.xml and so on. So you can use different web.xml (where you can set the RESTXQ path, see http://docs.basex.org/wiki/Web_Application#Configuration for details) for your different projects. Of yourse, you could also use complete separate pom.xml using mvn -f myotherpom.xml jetty:run I hope this helps. Cheers, Dirk > > Thanks again for the hints -- > > Cheers, Wendell > > > > > > On Sat, Jan 10, 2015 at 3:31 PM, Wendell Piez <wapiez(a)wendellpiez.com> wrote: > > Dirk, thanks! > > > > Cheers, Wendell > > > > > > On Thu, Jan 8, 2015 at 4:05 AM, Dirk Kirsten <dk(a)basex.org> wrote: > >> Hello Wendell, > >> > >> your last guess is correct: BaseX can only handle a single RESTXQ directory. So solutions using a colon a separator will not work, simply because we do not support it. It does not depend on the OS (although I personally think you did a wise thing in trying out GNU/Linux ;). > >> > >> The usual approach would be to have a separate .basex in your webapp directory. So if you start basex from this directory it will use this values instead your default ones. More likely you might want to store the BaseX specific values in your jetty.xml (if using Jetty), so they are used for the application container. This has several advantages: > >> - Your web application can not access some default functionality, which shouldn't really be available. > >> - You can use different BaseX version for different projects by including BaseX within the webapp itself > >> - You can also define different configurations for your different webapps (e.g. all your projects run on a different port, so you can run them simultanously) > >> > >> Cheers, > >> Dirk > >> > >> On 01/07/2015 11:32 PM, Wendell Piez wrote: > >>> Hi again, > >>> > >>> Correction: of course, under the "composite" path, neither path works. > >>> > >>> Perhaps BaseX can only have a single RestXQ directory at a time? > >>> > >>> Thanks again for your patience with me. :-) > >>> > >>> Cheers, Wendell > >>> > >>> On Wed, Jan 7, 2015 at 5:29 PM, Wendell Piez <wapiez(a)wendellpiez.com> wrote: > >>>> Dear BaseX people, > >>>> > >>>> I am a hopeless newbie when it comes to Linux. Seriously, my ignorance > >>>> is astonishing. I am now running Linux Mint. It's a huge amount of fun > >>>> but I'm left guessing about many simply things. > >>>> > >>>> I am running the current BaseX 7.9. I am using it to provide a > >>>> pipeline architecture calling lots of XSLT2, producing HTML and SVG. > >>>> It works well for this. > >>>> > >>>> To my RESTXQPATH I wish to append to - but not override (if possible) - a path. > >>>> > >>>> By editing my .basex configuration I have managed to reset it, but not > >>>> simply to append to it. > >>>> > >>>> That is, > >>>> > >>>> RESTXQPATH = /home/wendell/basex/webapp:/home/wendell/XQuery-project/restxq > >>>> > >>>> With this, the server skips the basex/webapp directory. If I remove > >>>> the second segment and restart, it goes live again. > >>>> > >>>> What is the simplest way for me to set up my BaseX so that I can > >>>> maintain my project xqm files (including a project's RestXQ), in the > >>>> project, but keep BaseX otherwise (including its data directories) > >>>> separate? Over the longer term I can see several such directories > >>>> where I'd like BaseX to look. > >>>> > >>>> On another operating system I used to start up using different shell > >>>> scripts; is that the best way here (and if so does my shell script > >>>> have to call Java or can I use the scripts in the basex/bin and pass > >>>> in a value somehow)? > >>>> > >>>> In general, what recommendations can you offer for no-fuss > >>>> configuration of RestXQ for separate projects? > >>>> > >>>> Thanks! Wendell > >>>> > >>>> > >>>> -- > >>>> Wendell Piez | http://www.wendellpiez.com > >>>> XML | XSLT | electronic publishing > >>>> Eat Your Vegetables > >>>> _____oo_________o_o___ooooo____ooooooo_^ > >>> > >>> > >>> > >> > >> -- > >> Dirk Kirsten, BaseX GmbH, http://basexgmbh.de > >> |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > >> |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: > >> | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle > >> `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22 > >> > > > > > > > > -- > > Wendell Piez | http://www.wendellpiez.com > > XML | XSLT | electronic publishing > > Eat Your Vegetables > > _____oo_________o_o___ooooo____ooooooo_^ > > > -- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22

2 4