April 2016 - BaseX-Talk - mailman.uni-konstanz.de

Querying Basex from a web interface
by Bram Vanroy | KU Leuven 22 May '16

22 May '16

Hello all I have been reading through many emails on this list and I've learnt a lot about how Basex works and how others use it. A month or so back I have sent an email myself to this list concerning caching. Even though I have some more questions about that, I will leave that for another time. Today I am concerned about retrieving chunked input from Basex. (Question also found on StackOverflow, with a nice bounty! :) http://stackoverflow.com/questions/36675388/efficient-and-user-friendly-way- to-present-slow-loading-results) Case at hand: we use Basex to query a 50 million tokens corpus. We also make this available to other users through a website. The thing is that this is slow. For our own projects that's no problem, we dive straight into the back-end and run a search command from terminal and let the query run for all the time it needs. However, for users it is paramount that they get a quick response. At the moment it is taking too long. I don't blame BaseX. We love BaseX and are astounded by its efficiency and optimisations! However, we want to deliver the best user-experience to our users. We call a new session from PHP, wait to receive the results, do some post-processing and then load the result page. As said, this takes too much time. We've been looking into some solutions. The best one that I think should be possible, is returning chunks of the results. Do you know those websites that allow you to see results but only, like, 20 per page? I think something similar is appropriate. When a user has searched for a pattern, we only show the 20 or so first results just so they can get an idea of the results they'd find. Then, when they click a button, we should query for the twenty next results which are then appended to the list (JavaScript solution I guess), and so on. Until all results have been found. Additionally, I will also provide a button from which users can download all results in a text file. This is allowed to take a longer time. The main thing is that users should get early feedback and results on their query. Now the question is if something like this is possible in an efficient manner in BaseX. Can you form a query that only finds the 20 first results, and then the following 20 and so on - and is this faster than searching everything at once? In other words, when I am searching for the results 120-140 (after having pushed the button a couple of times), is BaseX smart enough to skip the search space it has already done to find the 120 previous hits? If so, that would be great. Could you help me on my way, with some PHP/XQuery code that is suitable? I also highly encourage you to participate on StackOverflow. As I said, I am offering a 200 bounty - for the people who are interested in Internet fame. :) Thank you for your time Kind regards Bram Vanroy http://bramvanroy.be

2 8

Re: [basex-talk] Runtime Exception
by Christian Grün 04 May '16

04 May '16

Dear Mansi, you could try to run the INSPECT command on the affected database, or all databases, in order to find out if your database has gone corrupt. Did you repeatedly come across this error? Best, Christian Am 25.04.2016 16:45 schrieb "Mansi Sheth" <mansi.sheth(a)gmail.com>: > > Hello, > > My current BaseXDB is at 920GB, with ~230 databases... I run jetty server visa basexhttp script with giving it explicit 30GB of RAM. While trying to access a query, thru REST api via XQUERY, I get below error. > > HTTP/1.1 400 Bad Request^M > Content-Type: text/plain;charset=UTF-8^M > Content-Length: 4207^M > Server: Jetty(8.1.16.v20140903)^M > ^M > Improper use? Potential bug? Your feedback is welcome: > Contact: basex-talk(a)mailman.uni-konstanz.de > Version: BaseX 8.2.3 > Java: Oracle Corporation, 1.7.0_95 > OS: Linux, amd64 > Stack Trace: > java.lang.RuntimeException: Data Access out of bounds: > - pre value: 126882320 > - #used blocks: 495643 > - #total locks: 495643 > - access: 495642 (495643 > 495642] > at org.basex.util.Util.notExpected(Util.java:60) > at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:458) > at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:148) > at org.basex.data.Data.kind(Data.java:306) > at org.basex.query.value.node.DBNode.<init>(DBNode.java:51) > at org.basex.query.value.seq.DBNodeSeq.itemAt(DBNodeSeq.java:68) > at org.basex.query.value.seq.DBNodeSeq.itemAt(DBNodeSeq.java:22) > at org.basex.query.value.seq.Seq$1.next(Seq.java:77) > at org.basex.query.expr.path.IterPath$1.next(IterPath.java:58) > at org.basex.query.expr.path.IterPath$1.next(IterPath.java:36) > at org.basex.query.MainModule$1.next(MainModule.java:114) > at org.basex.query.func.StandardFunc.cache(StandardFunc.java:384) > at org.basex.query.func.xquery.XQueryEval.eval(XQueryEval.java:129) > at org.basex.query.func.xquery.XQueryEval.eval(XQueryEval.java:59) > at org.basex.query.func.xquery.XQueryEval.value(XQueryEval.java:49) > at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:77) > at org.basex.query.QueryContext.value(QueryContext.java:421) > at org.basex.query.expr.gflwor.Let$LetEval.next(Let.java:187) > at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:95) > at org.basex.query.MainModule$1.next(MainModule.java:114) > at org.basex.core.cmd.AQuery.query(AQuery.java:91) > at org.basex.core.cmd.XQuery.run(XQuery.java:22) > at org.basex.core.Command.run(Command.java:398) > at org.basex.http.rest.RESTCmd.run(RESTCmd.java:99) > at org.basex.http.rest.RESTQuery.query(RESTQuery.java:74) > at org.basex.http.rest.RESTRun.run0(RESTRun.java:41) > at org.basex.http.rest.RESTCmd.run(RESTCmd.java:65) > at org.basex.core.Command.run(Command.java:398) > at org.basex.core.Command.execute(Command.java:100) > at org.basex.core.Command.execute(Command.java:123) > at org.basex.http.rest.RESTServlet.run(RESTServlet.java:22) > at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) > at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) > at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) > at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) > at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > > > -- > - Mansi

2 6

Strange restxq path error "assigned to several functions"
by Maximilian Gärber 02 May '16

02 May '16

Hi, suddenly, I get a path error [1] for functions that worked just minutes ago. When renaming the functions again it sometimes resolves the issue. I have CACHERESTXQ = false But this "feels" just like a caching issue. Appeared while on 8.4.1, but same on 8.4.3 Any ideas? [1] Problem accessing /test/import-convert/19.2.xlsx. Reason: Stopped at /webapp/test/api/import.xqm, 55/10: [bxerr:BASX0003] Path "/test/import-convert/{$file}" assigned to several functions: /webapp/test/api/import.xqm, 55/10 /webapp/test/api/import.xqm, 55/10

2 6

xquery:eval - static context?
by Hans-Juergen Rennau 30 Apr '16

30 Apr '16

Dear BaseX team, a question concerning the static context used by xquery:eval. It seems to me that the static context is not inherited from the location of the evaluating call. Therefore arises the question how to call user-defined functions within code evaluated by xquery:eval. Please consider this example: Attempt #1:=========================================================== declare namespace f="http://www.foxpath.org/xquery-functions"; declare function f:num2($number) {2 * $number}; let $inlineFunctionText := "function($number) {f:num2($number)}" let $f := xquery:eval($inlineFunctionText) return $f(1) =========================================================== Executing this, an error is raised: [XPST0081] No namespace declared for 'f:num2'. Adding a namespace declaration does not help: Attempt #2:=========================================================== declare namespace f="http://www.foxpath.org/xquery-functions"; declare function f:num2($number) {2 * $number}; let $inlineFunctionText := "declare namespace f='http://www.foxpath.org/xquery-functions'; function($number) {f:num2($number)}" let $f := xquery:eval($inlineFunctionText) return $f(1) =========================================================== Now I get this error: [XPST0017] Unknown function: Q{http://www.foxpath.org/xquery-functions}num2. The only (?) way out is pass in the function a parameter: Attempt #3:=========================================================== declare namespace f="http://www.foxpath.org/xquery-functions"; declare function f:num2($number) {2 * $number}; let $inlineFunctionText := "declare namespace f='http://www.foxpath.org/xquery-functions'; declare variable $num2 external; function($number) {$num2($number)}" let $context := map{'num2': f:num2#1} let $f := xquery:eval($inlineFunctionText, $context) return $f(1) ===========================================================Result: 2Ok! Fortunately, the passed in function does inherit its own static context (as it should, per spec), so that it may call other functions themselves not known to the inline function expression: Attempt #4:=========================================================== declare namespace f="http://www.foxpath.org/xquery-functions"; declare function f:num2($number) {2 * $number + f:num3($number)}; declare function f:num3($number) {3 * $number}; let $inlineFunctionText := "declare namespace f='http://www.foxpath.org/xquery-functions'; declare variable $num2 external; function($number) {$num2($number)}" let $context := map{'num2': f:num2#1} let $f := xquery:eval($inlineFunctionText, $context) return $f(1) ===========================================================Result: 5Okissimo! So my question is whether it is by design or by bug that xquery:eval "forgets" the static context in place where the invocation of xquery:eval occurs. And I emphasize the great practical relevance of the question as it impacts the possibility of using within the evaluated expression user-defined functions. Cheers,Hans-Jürgen

1 0

basexhttp on OpenShift
by Mike Cobo 29 Apr '16

29 Apr '16

Hi all I'm trying to migrate my docker based BaseX app to OpenShift Enterprise and after building and deploying successfully the container it can not start due to the following strange permission problem: 1 /srv/.basex could not be written.2 java.nio.file.AccessDeniedException: /srv/.basex3 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)4 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)5 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)6 at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)7 at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:430)8 at java.nio.file.Files.newOutputStream(Files.java:172)9 at java.nio.file.Files.write(Files.java:3092)10 at org.basex.io.IOFile.write(IOFile.java:221)11 at org.basex.util.options.Options.write(Options.java:143)12 at org.basex.util.options.Options.read(Options.java:696)13 at org.basex.util.options.Options.<init>(Options.java:95)14 at org.basex.core.StaticOptions.<init>(StaticOptions.java:115)15 at org.basex.core.Context.<init>(Context.java:78)16 at org.basex.core.Context.<init>(Context.java:70)17 at org.basex.http.HTTPContext.init(HTTPContext.java:424)18 at org.basex.BaseXHTTP.<init>(BaseXHTTP.java:69)19 at org.basex.BaseXHTTP.main(BaseXHTTP.java:52)20 /srv/.basex: writing new configuration file.21 DEBUG: true22 Creating Database...23 . 4.61 ms (18501 KB)24 Creating /srv/BaseXWeb/WEB-INF/web.xml25 java.nio.file.NoSuchFileException: /srv/BaseXWeb/WEB-INF/web.xml26 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)27 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)28 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)29 at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)30 at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:430)31 at java.nio.file.Files.newOutputStream(Files.java:172)32 at java.nio.file.Files.write(Files.java:3092)33 at org.basex.io.IOFile.write(IOFile.java:221)34 at org.basex.BaseXHTTP.locate(BaseXHTTP.java:264)35 at org.basex.BaseXHTTP.initJetty(BaseXHTTP.java:219)36 at org.basex.BaseXHTTP.<init>(BaseXHTTP.java:75)37 at org.basex.BaseXHTTP.main(BaseXHTTP.java:52)38 /srv/BaseXWeb/WEB-INF/web.xml In my Docker file I simply copy some files (the same errors occurs when the last line in the Docker file is omitted, so there's no difference if the .basex file is added or not): FROM basex/basexhttp:latestMAINTAINER MyName <my.name(a)domain.com>VOLUME ["/srv/BaseXData", "/srv/BaseXWeb"]COPY lib/*.jar /usr/src/basex/basex-core/lib/COPY data/users.xml /srv/BaseXData/COPY web/test.xq /srv/BaseXWeb/COPY .basex /srv/.basex I'm not sure if the problem is OpenShift related or if it is based on the way the basex server is started... Any ideas? Thx.Mike

1 0

Non-updating Expressions: copy/modify/return. Multiple for-clauses inside modify
by Günter Dunz-Wolff 29 Apr '16

29 Apr '16

Hi all, hi Christian, for my queries I have to modify some legacy xml-documents. The copy/modify/return-expression works fine with one for-clause. But I have to modify different elements. I can't figure out, how to chain the for-clauses. Working Example: copy $doc := $document modify ( let $first_elements := $doc//element-name for $element in $first_elements let $target := $element/@target let $htm_anchor := substring-after($target, '#') return replace value of node $element/@target with 'xyz' || $htm_anchor ) return $doc This is working for one for-clause. But I have to do a lot more with different elements for example: let $second_element := $doc//element-name-second for $ele in $second_elements let $... return ... The returning document should contain all modifications. How can I chain the for-clauses inside modify? Thanks for a hint in advance. Günter

2 2

Re: [basex-talk] Loading XML file as an InputStream in BaseX
by Christian Grün 29 Apr '16

29 Apr '16

Hi Isuru, I have attached a little example to this mail (in which the result will be streamed as well). Hope this helps Christian On Fri, Apr 29, 2016 at 6:23 AM, Isuru Ranawaka <isurur(a)wso2.com> wrote: > Hi Christian, > > We are in the process of evaluating BaseX as a XPath engine that can be > used inside our products. But samples that we have seen are only for > evaluating XQuery against a XML Document. Do you have any sample or way > that we can evaluate XPath against a InputStream. Because we cannot convert > incoming stream in to a document and read it again it will have huge > overhead. It will be great if you can provide sample code or reference that > can be refer to. > > thanks > > On Thu, Apr 28, 2016 at 11:11 PM, Christian Grün < > christian.gruen(a)gmail.com> wrote: > >> Hi Pubudu, >> >> Streaming may be a non-trivial task. Before thinking about pipelining >> input data, I would first be interested if data transmission will really be >> a potential bottleneck. Have you already done som profiling? >> >> Cheers, >> Christian >> >> >> >> On Thu, Apr 28, 2016 at 10:45 AM, Pubudu Fernando <pubuduf(a)wso2.com> >> wrote: >> >>> Hi, >>> >>> I was going through the sample file given at [1]. Here, the XML file is >>> taken from a file. Is there any way of giving the XML as a stream instead? >>> The requirement is to use XPath expressions on an XML payload that comes >>> with an HTTP request (the payload will be available to consume, as an >>> InputStream) to check whether the payload contains some given text/data. >>> >>> I also had a look at [2]. Is this approach valid only for common types >>> like integer, string etc? >>> >>> [1] - >>> https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o… >>> [2] - >>> https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o… >>> >>> Thanks >>> -- >>> *Pubudu Fernando* >>> Software Engineer - Intern >>> WSO2 Inc. >>> m: +94 77 888 2543 >>> >>> <https://lk.linkedin.com/in/pubuduf> <http://blog.pubudu.xyz/> >>> <https://github.com/pubudu91> >>> <http://stackoverflow.com/users/4329912/pubudu> >>> >>> >> > > > -- > Best Regards > Isuru Ranawaka > M: +94714629880 > Blog : http://isurur.blogspot.com/ >

1 0

Loading XML file as an InputStream in BaseX
by Pubudu Fernando 28 Apr '16

28 Apr '16

Hi, I was going through the sample file given at [1]. Here, the XML file is taken from a file. Is there any way of giving the XML as a stream instead? The requirement is to use XPath expressions on an XML payload that comes with an HTTP request (the payload will be available to consume, as an InputStream) to check whether the payload contains some given text/data. I also had a look at [2]. Is this approach valid only for common types like integer, string etc? [1] - https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o… [2] - https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o… Thanks -- *Pubudu Fernando* Software Engineer - Intern WSO2 Inc. m: +94 77 888 2543 <https://lk.linkedin.com/in/pubuduf> <http://blog.pubudu.xyz/> <https://github.com/pubudu91> <http://stackoverflow.com/users/4329912/pubudu>

2 1

function-lookup with "db:replace" (8.4 -> 8.4.3)
by Wiemer, Sebastian 27 Apr '16

27 Apr '16

2 1

Print local time
by Mike Cobo 27 Apr '16

27 Apr '16

Hi all I want to print the current date and time on a HTML page using XQuery's format-dateTime function. The problem is that I always get the GMT time which currently is 2 hours behind of Central Europe Time. This is what my code looks like: <p>Stand vom {format-dateTime(current-dateTime(), '[D01].[M01].[Y0001] [H01]:[m01]:[s01]', 'de')} Uhr</p> And this is what I get when local time is 15:21:34: Stand vom 27.04.2016 13:21:34 Uhr Is it possible to get the local time using format-dateTime? Thanks for any help! Mike

2 2