- BaseX-Talk - mailman.uni-konstanz.de

Who is using... ?
by Fabrice Etanchaud 27 May '12

27 May '12

Dear Christian, 1. Will BaseX clients still able to connect to the BaseX server serving HHTP requests ? This XQuery Client API Module is a very good idea ! 2. IMHO, BaseX should concentrate on processing xml, I stopped using ZIP module when I realized db:add() was doing the job, in streaming mode. Great job ! Best regards, Fabrice ________________________________ Hi all, we are interested in your opinion! As one of our main focus is to keep BaseX small and cute (..sad but true: the BaseX core library already exceeds 2mb.. ;), we're currently contemplating the removal of two existing BaseX features, namely.. 1) the BaseXHTTP client mode (flag -c). Currently, we offer three operation modes: default, local, and client [1]. In client mode, all requests are supposed to be passed on to yet another BaseX server instance. Currently, however, WebDAV is the only service which is 100% compatible with this mode, while RESTXQ isn't tailored at all to handle remote database requests. Instead, we are contemplating on adding a light-weight XQuery Client API Module, which can then be used to explicitly address remote BaseX instances from within XQuery whenever databases are not stored locally anyway. 2) the current ZIP Module. The functionality won't disappear, though: we will replace it with a new, more flexible proposal specified by Matthias Branter (28msec) and me [2]. In a nutshell: who of you is using either 1) or 2) and would run into big troubles if those features were ousted and replaced with the approaches we believe are more promising for future development? Thanks as usual for your feedback, either to the list (preferred) or in private, Christian [1] http://docs.basex.org/wiki/Startup_Options#BaseX_HTTP_Server [2] http://spex.basex.org/index.php?title=ZIP_Module Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

Re: [basex-talk] Too Many Open Files
by Andreas Rulle 26 May '12

26 May '12

Hello Andreas, thank you very much for your valuable hints. sudo lsof -p basex-ps-no delivers many lines of java 11489 root 962u sock 0,6 0t0 10449645 can't identify protocol and (1) sudo lsof -p 11489 | wc -l gets reports an increasing number of used "files". The increase of the number in (1) in a given time interval correlates to the number of requests that the LVM load balancer sends to the basex port 1984 in that interval. There are about 4 requests from the LVM load balancer per minute. At the time of this writing (1) has the value of 1076 ... The parameter keepalive has the value KEEPALIVE = 600 but it does not seem to stop the increase of the value in (1). This information opens the way to workarounds - decrease the number of requests from the LVM load balancer, - increase the ulimit -n - restart the basex -server before the number in one reaches ulimt -n. But we really would prefer solutions to the increase of the figure in (1) The almost identical settings of the load balancer do work for a MySQL-Database without any problems. And without the load balancer an instance of the basex runs since April 10 without hitting ulimit -n = 1024. Any hints on this are very welcome! With kind regards, Andreas > I just searched our mailing list, cause there was something about too many open files some time ago: > >> On typical Linux installations, die open filedescriptor limit is 1024: >> >> $ grep 'open files' /proc/self/limits >> Max open files 1024 1024 files >> >> In Java, if a file-based object (FileWriter, FileReader, etc.) is not closed, the underlying file descriptor is not closed. Bug detectors (FindBugs) check for that intraprocedurally. I recommend you run both BaseX as well as this application through it. >> >> If an open file-based object becomes unreachable, the finalizer will eventually close it - but it's possible to run out of open file descriptors simply due to unreachable, but not yet finalized objects. (Of course, if the object is leaked, it won't be closed ever.) >> >> In Linux, use 'ps' to find out the pid of the Java JVM process, then do a ls -l /proc/<pid>/fds to see which file descriptors the process in question has open; or use the 'lsof' command. >> >> - Godmar > Probably that helps to investigate the issue. > > -- Nexoma GmbH Theodorus Weg 7 59755 Arnsberg Tel. + 49 (0) 52 51 1613-0 Fax + 49 (0) 52 51 1613-99 mailto:andreas.rulle@nexoma.de Geschäftsführer: Guido Sauerland Sitz der Gesellschaft: Arnsberg Registergericht: Arnsberg, HRB 9365

6 11

XML file with xmlns attribute and XPATH
by Philippe Rathé 26 May '12

26 May '12

Hello there, I just want to do a simple xpath expression but I can't get it works when I use this XML document <?xml version="1.0" encoding='utf-8'?> <feed xmlns="http://example.org" xmlns:abc="http://abc.example.org"> <title>Projects</title> </feed> The expression "exquery //title" does not return anything from the CLI once I open the database containing only that file. But it works with this file <?xml version="1.0" encoding='utf-8'?> <feed xmlns="http://example.org"> <title>Projects</title> </feed> It looks like having both xmlns and xmlns:abc namespace attribute creates a problem. My real use case is atompub feed document which the 2 first lines looks like this: <?xml version="1.0" encoding='utf-8'?> <feed xmlns="http://www.w3.org/2005/Atom" xmlns:app="http://www.w3.org/2007/app"> Any idea? I'm using a 7.2.2 snapshot, thanks.

3 6

whitespace handling
by Godmar Back 25 May '12

25 May '12

Hi, I'm confused about whitespace. I've written an XQuery that returns an XML fragment based on some computation on an underlying XML document. The returned XML contains insignificant whitespace, which then adversely affects my program. I'm not sure if I'm causing it or BaseX, or even what the rules are. My XQuery contains code such as this one (I'm using color to emphasize the relevant parts). declare function local:buildSubTree($id) { let $feed := doc($doc_name)/atom:feed let $parent := $feed/atom:entry[atom:id=$id] let $children := data($parent//libx:entry/@src) return if (fn:count($children) = 0) then () else for $child_id in $children return if (local:isInThisFeed($feed, $child_id)) then <node id='{$child_id}' type='{local:getType($feed, $child_id)}' title='{local:getTitle($feed, $child_id)}'>{ local:buildSubTree($child_id) }</node> else if (functx:is-absolute-uri($child_id)) then <node id='{$child_id}' title='Unpublished Entry' type='external' /> else <node id='{$child_id}' title='Unpublished Entry' type='import' /> }; but the result contains fragments such as this, which is an 2-space indented formatting of the XML, with insignificant whitespace added: <node id="180" title="Put Google Book results into Addison" type="libapp"> <node id="181" title="Search Google Books while searching Addison" type="module"/> <node id="182" title="Display HTML notifications via an embedded panel using jGrowl" type="module"/> </node> If I say <node>{ ... }</node> in the XQuery etc., should there be insignificant whitespace in the response? Thank you for any help/pointers. (Note that I have considered simply stripping insignificant whitespace, but I do not like this solution since for some queries, I'd like to preserve it, whereas for others I absolutely cannot have insignificant whitespace since I'm performing traversals of the resulting XML DOM Tree.) Thank you! - Godmar

2 1

file:list - a question
by Hans-Juergen Rennau 24 May '12

24 May '12

Hello, a question about the function file:list. The documenation says: "The returned paths are relative to the provided path." My understanding is that if the path is a/b/c and this dir contains a file fool.xml, the returned path would be "fool.xml", rather thatn "a/b/c/fool.xml". What confuses me is that I observe two different behaviours in two different BaseX versions, once with and once without path: version 7.2 with path, version 7.2.2 without path. (The OS also differs in the environments, but I suppose this is irrelevant.) My question: is the output without path (more generally: with a path relative to the stated dir) the correct one and will be produced in all future versions? Thank you. Kind regards, Hans-Juergen

2 1

Managing databases via XQuery
by Andy Bunce 24 May '12

24 May '12

I sometimes find myself, in code, wanting to create a database, or more specifically test if a database exists and create it if does not. The database module appears not have functions to create (or delete databases). Is this a deliberate design decision? I can do this via the command interface, although test and create is a bit messy; I can do it via the REST interface, and I think via the WebDAV interface, but not, it seems, via XQuery directly. Adding additional functions to the database module would be a solution, or maybe a util:command() that would execute any BaseX command? /Andy

3 4

RESTXQ and parameters in the media-type
by Philippe Rathé 22 May '12

22 May '12

Hello, I would like to know if the parameters in the media-type should be in the "produces" annotation or elsewhere. Because I can't get it to work with the media-type of an atom feed member. %rest:produces("application/atom+xml;type=entry") The valid syntax can be found in rfc2616 section 3.7 Media Types: http://tools.ietf.org/html/rfc2616#section-3.7 In the RESTXQ wiki page, we can see the media type of an atome feed %rest:produces("application/atom+xml") But I did not found anything about using a parameter in the media type. Thanks.

2 3

Open Source-based Business
by Yu, Yiqing 22 May '12

22 May '12

Hi open source software developer, I am a research assistant at Institute for Information Systems and New Media, University of Munich. Driven by the interest in open source software, I want to investigate open source-based business by connecting the decision-making theory to the software development practice. Based on this idea, I designed an online survey "OS-based business": http://www.unipark.de/uc/osb/ Through "advanced search" and your profile at Github I find you may fit into the survey's target group. I gladly invite you to it and look forward to your participation. Some points in the survey may inspire your research or business ideas. :) I would deeply appreciate your support! Best Regards, Yours, Yiqing Yu Institute for Information Systems and New Media Ludwig-Maximilians-Universität München www.wim.bwl.lmu.de<http://www.wim.bwl.lmu.de/>

1 0

Disable or control query caching
by Manuel Bernhardt 22 May '12

22 May '12

Hello, we're getting an apparent deadlock (followed by a GC overhead limit exceeded) on one machine, when starting some processing on a collection of over 800 000 records. Going after it with YourKit yields the following: application-akka.actor.default-dispatcher-110 <--- Frozen for at least 3m 7s org.basex.server.Query.cache(InputStream) org.basex.server.ClientQuery.cache() org.basex.server.Query.more() eu.delving.basex.client.Implicits$RichClientQuery.hasNext() scala.collection.Iterator$$anon$19.hasNext() scala.collection.Iterator$$anon$29.hasNext() scala.collection.Iterator$class.foreach(Iterator, Function1) scala.collection.Iterator$$anon$29.foreach(Function1) core.processing.CollectionProcessor$$anonfun$process$2.apply(ClientSession) core.processing.CollectionProcessor$$anonfun$process$2.apply(Object) core.storage.BaseXStorage$$anonfun$withSession$1.apply(ClientSession) core.storage.BaseXStorage$$anonfun$withSession$1.apply(Object) eu.delving.basex.client.BaseX$$anonfun$withSession$1.apply(ClientSession) eu.delving.basex.client.BaseX$$anonfun$withSession$1.apply(Object) eu.delving.basex.client.BaseX.withSession(Function1) eu.delving.basex.client.BaseX.withSession(String, Function1) core.storage.BaseXStorage$.withSession(Collection, Function1) core.processing.CollectionProcessor.process(Function0, Function1, Function1, Function3) core.processing.DataSetCollectionProcessor$.process(DataSet) actors.Processor$$anonfun$receive$1.apply(Object)<2 recursive calls> akka.actor.Actor$class.apply(Actor, Object) actors.Processor.apply(Object) akka.actor.ActorCell.invoke(Envelope) akka.dispatch.Mailbox.processMailbox(int, long) akka.dispatch.Mailbox.run() akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec() akka.jsr166y.ForkJoinTask.doExec() akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinTask) akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool$WorkQueue) akka.jsr166y.ForkJoinWorkerThread.run() In the server logs I can observe: 09:10:45.129 [192.168.1.214:47530]: dimcon____geheugen-van-nederland QUERY(3) for $i in /record[@version = 0] order by $i/system/index return $i OK 0.06 ms 09:10:45.129 [192.168.1.214:47530]: dimcon____geheugen-van-nederland QUERY(3) OK 0.03 ms 09:13:23.155 [192.168.1.214:47530]: dimcon____geheugen-van-nederland ITER(3) Error: Connection reset 09:13:23.155 [192.168.1.214:47530]: dimcon____geheugen-van-nederland LOGOUT admin OK I looked up the code, and it looks as though the whole (?) query is cached in memory upon retrieval. Given the DB is over 1.2 GB in size, our client server has a hard time (it only has 1,5 GB of Xmx). Is there any preferred way of dealing with this? What I am going to do for the moment, I think, is to override or intercept the creation of the ClientQuery and make an implementation that has a different caching strategy. Another approach may probably be to limit the query output and implement some custom iteration behavior - but if it that can be handled directly at the query level, I think it would make things easier. Manuel

2 6

db:add and ADD logic
by Fabrice Etanchaud 22 May '12

22 May '12

Dear all at BaseX, I don't understand the ADD/db:add behaviour when adding two documents with the same name : previous document is not replaced, but copied, and I did not find a way to correct, and delete only one of the documents, because they all have the same name... So, thanks to the db:replace function, I wrote a updating function which emulates desired behaviour (that adds or replaces the document if it already exists in the database). But if I try to insert the same document twice with this function (in the same updating query), I still get two documents in the database, because function is 'updating' and because of the pending update list... Are db:XXX functions really 'updating' ? Is there a way to delete only one of the duplicated documents ? Thank you again for your fantastic job. Regards, Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1