BaseX-Talk April 2013

basex-talk@mailman.uni-konstanz.de

38 participants
54 discussions

xquery:eval yields NullPointerException
by Marco Lettere 29 Apr '13

29 Apr '13

Hi all, on the way to writing a small DSL I stumbled upon the following BaseX 7.6 issue. I'm not able to explain to myself why the first two expressions of the following xquery module return correctly whereas the last yields NullPointerException (as shown in the appended stacktrace). What I'm mtrying to do is to wrap up some modification code into userfriendly function calls that can be used in mini xquery scripts to formalize xml workflow. Thank you for any help, Marco. declare variable $globals := map { 'change' := local:change#1, 'generate' := local:gen#0 }; declare function local:gen(){ }; declare function local:change($doc){ copy $newdoc := $doc modify( insert node <a/> as last into $newdoc ) return $newdoc }; ( xquery:eval("copy $newdoc := modify( insert node <a/> as last into $newdoc ) return $newdoc",$globals), xquery:eval("$generate()", $globals), xquery:eval("$change($doc)", map:new( ($globals, map { "doc" := }))) ) The stacktrace and query plan: Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.6 Java: Oracle Corporation, 1.7.0_15 OS: Linux, i386 Stack Trace: java.lang.NullPointerException org.basex.query.up.expr.Transform.value(Transform.java:68) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.func.UserFunc.value(UserFunc.java:175) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:68) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:104) org.basex.query.func.DynamicFunc.value(DynamicFunc.java:56) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.QueryContext.value(QueryContext.java:254) org.basex.query.func.FNXQuery.eval(FNXQuery.java:95) org.basex.query.func.FNXQuery.eval(FNXQuery.java:71) org.basex.query.func.FNXQuery.iter(FNXQuery.java:37) org.basex.query.QueryContext.iter(QueryContext.java:285) org.basex.query.expr.List$1.next(List.java:100) org.basex.query.QueryContext.execute(QueryContext.java:524) org.basex.query.QueryProcessor.execute(QueryProcessor.java:96) org.basex.core.cmd.AQuery.query(AQuery.java:77) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.gui.GUI.exec(GUI.java:397) org.basex.gui.GUI$7.run(GUI.java:349) Compiling: - pre-evaluating map { "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } } Optimized Query: declare function local:change($doc) { copy $newdoc as node() := $doc modify insert node element a { () } into $newdoc return $newdoc }; declare function local:gen() { element b { () } }; (xquery:eval("copy $newdoc := $generate() modify( insert node <a/> as last into $newdoc ) return $newdoc", map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }), xquery:eval("$generate()", map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }), xquery:eval("$change($doc)", map:new((map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }, map { "doc":=element b { () } })))) Query plan: <QueryPlan> <UserFuncs> <UserFunc name="local:change" arg0="doc"> <Transform> <Let var="$newdoc as node()"> <VarRef> <Var name="$doc" id="2"/> </VarRef> </Let> <Insert> <VarRef> <Var name="$newdoc as node()" id="3"/> </VarRef> <CElem> <QNm value="a" type="xs:QName"/> </CElem> </Insert> <VarRef> <Var name="$newdoc as node()" id="3"/> </VarRef> </Transform> </UserFunc> <UserFunc name="local:gen"> <CElem> <QNm value="b" type="xs:QName"/> </CElem> </UserFunc> </UserFuncs> <VarStack> <Var name="$globals" id="0"/> </VarStack> <List> <FNXQuery name="eval(string[,bindings])"> <Str value="copy $newdoc := $generate()
 modify(
 insert node <a/> as last into $newdoc 
 )
 return $newdoc" type="xs:string"/> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> </FNXQuery> <FNXQuery name="eval(string[,bindings])"> <Str value="$generate()" type="xs:string"/> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> </FNXQuery> <FNXQuery name="eval(string[,bindings])"> <Str value="$change($doc)" type="xs:string"/> <FNMap name="new([maps[,coll]])"> <List> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> <LitMap> <Str value="doc" type="xs:string"/> <CElem> <QNm value="b" type="xs:QName"/> </CElem> </LitMap> </List> </FNMap> </FNXQuery> </List> </QueryPlan>

2 2

validation that returns node identifiers?
by Graydon Saunders 26 Apr '13

26 Apr '13

Hi! Using BaseX77-20130417.002917 though I don't think that's relevant to the question. This may well be an inherent property of the library being used to do validation, but right now, when I use validate:xsd-info, I get error messages that look like: Error: 25:59: cvc-complex-type.2.4.a: Invalid content was found starting with element 'link'. One of '{bold, footnote, italic, superscript}' is expected. These identify the location of the error by line and column number, which is great for code and less than helpful for XML content, since any format-and-indent getting applied to the XML file destroys its usefulness. Is is possible to have the error return a node identifier of some kind, instead? /root/bucket[2]/block[4]/para[3]/emphasis[4] sort of thing? Thanks! Graydon

2 1

Bypassing file locking in BaseX
by Scott Martens 25 Apr '13

25 Apr '13

Hi all, I have an application that's embedding BaseX in Java and generates all xqueries internally. These queries are *always* read-only. The problem I'm running into is that file locking prevents more than one user from reading the same file at the same time - something that BaseX 7.2 didn't do, but BaseX 7.6 does. I've tried using the client-server approach to solve the problem, and it works, sort-of. But it has a number of other problems for the conditions that it's being run under, and the embedded solution was faster and more robust. It looks to me like the BaseX standalone command line app and basexgui just ignore file-locking, and as long as I'm just using read-only queries, I have no trouble issuing two queries at once on the same database. Is this true? And if so, how is it done? Because this app is doing only read-queries, it's a lot better a solution for me than client-server, but I can't figure out where in the source code the database is being opened and how it's being done in a way that bypasses locks. Can anyone give me pointers? -- Scott Martens

3 3

Indexes creation fails on big database
by Erdal Karaca 23 Apr '13

23 Apr '13

Hi all, I have downloaded/started the latest dev snapshot of 7.7. My database consists of an extract of the OpenStreetMap data (only Germany) which is 29GB in size (xml/OSM format). I have managed to load the file into the database but the indexes creation procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI is right, I would need the 10x of heap space to create the attribute indexes. Any ideas/workarounds to create the attribute indexes for such a big database? Thanks!

2 3

Geospatial indexing
by Erdal Karaca 23 Apr '13

23 Apr '13

Hi all, Just noticed that basex supports geospatial queries since 7.6. But, how do I index geospatial data in my database? Or is that a missing feature? Thanks! Best regards, Erdal

2 1

inserting ampersand
by marco fregonese 22 Apr '13

22 Apr '13

Hi there, I have the following xquery function: *declare updating function local:insert ($a) { insert node <title>{$a}</title> into /web }; local:insert ("&")* but if I try to insert an apersand, basex says "invalid entity '&' ". So, my question: is there any way to insert an ampersand? Thank you.

2 1

Full-Text search, how to mark whole phrases
by Bartosz Marciniak 19 Apr '13

19 Apr '13

Hi, I have placed small document in database with enabled full-text index: <text> Text with hello world phrase. </text> The following expression: ft:mark(db:open('simple')//*[text() contains text 'hello world']) produces: <text>Text with hello world phrase.</text> Is it possible to enclose result in one element hello world ? Regards Bartosz Marciniak

3 2

Fwd: whitespace around comments
by Wendell Piez 19 Apr '13

19 Apr '13

Oops -- just sent this to Christian only, when it was meant for the list -- ---------- Forwarded message ---------- From: Wendell Piez <wapiez(a)wendellpiez.com> Date: Tue, Apr 16, 2013 at 11:15 AM Subject: Re: [basex-talk] whitespace around comments To: Christian Grün <christian.gruen(a)gmail.com> Hi, Thanks to Cerstin for once again urging that CHOP not be set by default, and to Liam for reminding readers that this is *not* a new issue in markup processing -- indeed it was in view of problems with whitespace in earlier technologies (can anyone say "pernicious mixed content"?) that the XML Rec says what it does -- including section http://www.w3.org/TR/REC-xml/#sec-white-space. And thanks also to Christian for reminding us why much of the time, whitespace chopping is innocuous and helpful. Personally, I'm in an uneasy place with respect to these issues. I wholeheartedly agree with Cerstin and Liam that chopping should not be on by default. Leaving it on by default is wrong and non-conformant. However, I'm also working with an application in which it's very convenient to have it -- not only is the data highly structured and controlled -- no mixed content here -- but also part of its job is to offer clean representations of external data that is much dirtier and cruftier. It is supposed to clear bad data away and clean it up, so managing the whitespace on ingest is just part of what we are doing. For both scaling and usability issues (as Christian has suggested), whitespace chopping is nice to have. Yet at the same time, we are already calling XSLT for uses in which the chopping is destructive, and the day is not very far off when we will have mixed content in our primary data set too. Liam points out something very important: it is possible in principle to distinguish between whitespace that can be safely discarded (by design) and whitespace that can't -- if you have a schema or other specification that represents this design. As he notes, the XML Rec distinguishes between "significant" and "insignificant" whitespace by reference to content models that do and don't include #PCDATA (that is, whitespace that appears in "element content" or "mixed content"; cf http://www.w3.org/TR/REC-xml/#dt-elemcontent). If your content model for div says (p+), then whitespace between the 'p' element children of a 'div' (but not inside them) may often be judged safe to discard. (At least in a system in which a schema is used as a warrant of fitness for processing.) When technologies such as XQuery or XSLT are designed to work with and without schemas, however -- or where schemas cannot be considered as reliable indicators of markup semantics -- even relying on this mechanism can't solve the problem (to say nothing of deciding which schema languages you support). However, it can help to mitigate it. Then too, even XSLT 1.0 has strip-space and preserve-space configuration to indicate to a processor where it can "chop" whitespace. While it's a bit crude (it treats all elements with the same name the same), it can be useful. Over the longer term, therefore, I think that (1) CHOP needs to be "false" by default, (2) it should be possible to turn it on (just as I am learning how to turn it off), and also (3) that we also need more flexible and configurable means for discriminating how it should work, with and without schemas to reference. Cheers, Wendell On Sat, Apr 13, 2013 at 7:05 AM, Christian Grün <christian.gruen(a)gmail.com> wrote: > I’d like to add some more info on why we initially decided to chop > whitespaces, and why a sudden change of the default value may break > existing applications (if you know the details, simply skip this > section..): > > Many XML documents contain whitespace-only text nodes for properly > indenting elements. In highly structured data (i.e., when not working > with mixed content), these nodes are in fact completely irrelevant. > For example, if the following document… > > <xml> > <a>X</a> > </xml> > > …is parsed with CHOP set to true, we will get a document with a single > text node. The following query… > > for $t in //text() > return replace node $t with 'x' > > …will generate the following result: > > <xml> > <a>x</a> > </xml> > > If we set CHOP to false, the document will have three text nodes, two > of them whitespace-only, and the same query will create the following > result document: > > <xml>x<a>x</a>x</xml> > > This is just one example to demonstrate that a sudden change of the > default for chop would most probably lead to unwanted side effects in > existing applications. Another side effect: databases are expected to > increase in size, as all whitespace nodes will get their own node ids, > will be fully stored and indexed, etc. > > However, I completely agree that the removal of whitespaces may lead > to serious changes in mixed contents, and I easily admit that we > haven’t been aware of all the implications some years ago when we > started off designing the database. While I still believe that our > storage copes pretty well with nowaday’s requirements, I would love to > have some weeks off to completely rebuild it, and include > optimizations for all kinds of features that are relevant today > (including larger ranges for node ids and namespaces, or support for > other tree formats such as json). > > Thanks for reading, > Christian > ___________________________ > > On Sat, Apr 13, 2013 at 8:28 AM, Liam R E Quin <liam(a)w3.org> wrote: >> On Fri, 2013-04-05 at 11:31 +0200, Dirk Kirsten wrote: >> >>> So if you could point out some details as why this is not conforming >>> behaviour, this would be interesting. >> >> It's a requirement in the XML Spec that the XML parser pass all >> whitespace back to the application. Some whitespace may be marked as not >> significant - that is only possible if there's a DTD and the space is in >> a context where only elements would be valid, not #PCDATA. There's no >> formal specification, although constructing an XDM instance from an >> infoset, and constructing an infoset from XML, does not entail >> discarding these spaces: >> Chopping internal whitespace nodes in mixed content contexts is not >> sanctioned by any version of any XML specification, with any setting of >> xml:space. I think the onus would be on you to justify the non-standard >> behaviour. >> >> On the other hand I can see its uses too. But I don't want it, and >> always turn it off with BaseX :-) >> >> Best, >> >> Liam >> >> -- >> Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ >> Pictures from old books: http://fromoldbooks.org/ >> Ankh: irc.sorcery.net irc.gnome.org freenode/#xml >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk(a)mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk -- Wendell Piez | http://www.wendellpiez.com XML | XSLT | electronic publishing Eat Your Vegetables _____oo_________o_o___ooooo____ooooooo_^ -- Wendell Piez | http://www.wendellpiez.com XML | XSLT | electronic publishing Eat Your Vegetables _____oo_________o_o___ooooo____ooooooo_^

2 3

java.lang.NoClassDefFoundError while deploying on Tomcat server
by Carlos MARCOS LAGUNAR 19 Apr '13

19 Apr '13

Hi, I've downloaded last BaseX version from the download page and what I've done it is just put the BaseX76.war into webapps deployment directory of Tomcat. As far as I know, it has deployed correctly, since I can make REST requests successfully like http://MiServer/BaseX76/rest?command=show+users. The problem comes when I try to access it from my Java Application, where I've tried with two methods: JavaClient and XQJ. With JavaClient it works, but with XQJ the response is always the same (log from [TOMCAT_DEPLOYMENT_DIRECTORY]/BaseX76/data/.logs): 13:32:40.053 172.24.41.93:59521 admin ERROR Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.6 Java: Oracle Corporation, 1.7.0 OS: Linux, i386 Stack Trace: java.lang.NoClassDefFoundError: org/basex/index/value/DiskValues org.basex.data.DiskData.<init>(DiskData.java:88) org.basex.core.cmd.Open.open(Open.java:70) org.basex.core.cmd.Open.run(Open.java:36) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.server.ClientListener.run(ClientListener.java:145) 453.12 ms The block of Java code where the Exception is thrown is the following (for the case of XQJ): log.debug("[BaseX] executing query:"+query); XQDataSource xqs = new BaseXXQDataSource(); xqs.setProperty("databaseName", dbName); xqs.setProperty("serverName", serverNameBaseX); xqs.setProperty("port", portBaseX); xqs.setProperty("user", usuario); xqs.setProperty("password", clave); xqs.setLogWriter(new PrintWriter(System.out, true)); XQConnection xqc = xqs.getConnection(); ß Here is where the Exception is thrown XQExpression xqe = xqc.createExpression(); XQSequence xq = xqe.executeQuery(query); StringBuilder res = new StringBuilder(); while (xq.next()) { res.append(xq.getItemAsString(null)); } log.debug("[BaseX] result:"+res); xqc.close(); It seems that it is not finding the jars inside the WAR where DiskData class is located, but I've entered into the Tomcat directory where the WAR has been deployed (/BaseX76/WEB-INF/lib) and all of them are there: -rw-rw---- 1 2558399 2013-02-05 18:22 basex-7.6.jar -rw-rw---- 1 295821 2013-01-30 11:42 basex-xqj-1.2.3.jar -rw-rw---- 1 58160 2013-01-30 11:42 commons-codec-1.4.jar -rw-rw---- 1 59590 2013-01-30 11:42 commons-fileupload-1.2.2.jar -rw-rw---- 1 109043 2013-01-30 11:42 commons-io-1.4.jar -rw-rw---- 1 153115 2013-01-30 11:42 jdom-1.1.jar -rw-rw---- 1 794991 2013-01-31 15:34 jts-1.13.jar -rw-rw---- 1 410933 2013-01-30 11:42 milton-api-1.7.2.jar -rw-rw---- 1 119180 2013-01-30 11:42 mime-util-2.1.3.jar -rw-rw---- 1 23445 2013-01-30 11:42 slf4j-api-1.5.8.jar -rw-rw---- 1 5284 2013-01-30 11:42 slf4j-nop-1.5.8.jar -rw-rw---- 1 9260 2013-01-30 11:42 xmldb-api-1.0.jar -rw-rw---- 1 7188 2013-01-30 11:45 xmlpull-1.1.3.1.jar -rw-rw---- 1 24956 2013-01-30 11:45 xpp3_min-1.1.4c.jar -rw-rw---- 1 4578 2013-01-30 11:42 xqj2-0.1.0.jar -rw-rw---- 1 16221 2013-01-30 11:42 xqj-api-1.0.jar -rw-rw---- 1 481672 2013-01-30 11:42 xstream-1.4.2.jar Any ideas?. Other projects deployed into that server worked without any problem. I know that I could use JavaClient method for accessing but then the problem is that I cannot recover the result of the query execution, while with XQJ I can: JavaClient: log.debug("[BaseX] executing query:"+query); ClientSession cSes = new ClientSession(serverNameBaseX, Integer.valueOf(portBaseX), user, pass); cSes.setOutputStream(System.out); ClientQuery cQuery = cSes.query(query); cQuery.execute(); String res = cQuery.info(); ß This res only stores the execution time and not the result of the query, so it is useless (for me :-) ) cQuery.close(); XQJ: XQSequence xq = xqe.executeQuery(query); ß Instead, this do stores the result of the query (exactly what I want) Thank you and best regards, C. Carlos Marcos Lagunar Health Technical Consultant Research & Innovation carlos.marcos(a)atosresearch.eu <mailto:carlos.marcos@atosresearch.eu> Valladolid - Spain www.atosresearch.eu <http://www.atosresearch.eu/> es.atos.net ------------------------------------------------------------------ This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Este mensaje y los ficheros adjuntos pueden contener informacion confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente pueden estar protegidos por secreto profesional. Si usted recibe este correo electronico por error, gracias por informar inmediatamente al remitente y destruir el mensaje. Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningun compromiso para el grupo Atos, salvo ratificacion escrita por ambas partes. Aunque se esfuerza al maximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no sera responsable de cualesquiera danos que puedan resultar de una transmision de virus. ------------------------------------------------------------------

2 2

Running and configuring update operations outside the GUI
by Wendell Piez 18 Apr '13

18 Apr '13

Hi, I too have a question about database updates, in particular Fabrice's remark that "The GUI may not be the best place to run updates, did you try the basex command line tools ?" I have an update that is working okay from the GUI, but I wouldn't mind it if it were faster. :-) Is simply invoking the same XQuery using RUN from the command line all I have to do to get the benefits of using the CL? In addition: my update query is configured with a bit of XML, something like declare variable $build-db := <build db="new"> <source db="old1"/> <source db="old2"/> <source db="old3"/> </build>; (Of course it's more complicated than this but you get the essence.) But I can't pass this XML into a function to perform the updates, since I can't put update operations into a function at all. (So the XML is piled into a global variable, which the update operations reference.) Do you have any suggestions for how I can encapsulate the updating logic and also make it configurable externally like this? And run it from the command line or a script for speed and efficiency? Thanks, Wendell -- Wendell Piez | http://www.wendellpiez.com XML | XSLT | electronic publishing Eat Your Vegetables _____oo_________o_o___ooooo____ooooooo_^

3 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk April 2013