- BaseX-Talk - mailman.uni-konstanz.de

what is the best way to add/replace a lot of documents ?
by Fabrice ETANCHAUD 12 Nov '12

12 Nov '12

Dear all at basex, I am wondering what could be the most efficient way to add/replace a lot of documents (several thousands) at the same time. I tried db:add/db:replace but this gives heap memory overflow because of the pending list... Will I have better results with a .bxs with add/replace commands (and autoflush deactivated) ? Did any other user find his way in this use case ? Best regards, Fabrice

3 2

Function will return node(), but db:output(node()) returns 'Attribute xmlns redefined'
by France Baril 12 Nov '12

12 Nov '12

Hi, My search function used to return $html correctly *let* *$html* := <html *xmlns*=*"http://www.w3.org/1999/xhtml"*> <head><link *href*=*"*{*$cssfile*}*"* *rel*=*"stylesheet"* *type*=* "text/css"*/><title>Search help</title> <script *type*=*"text/javascript"* *xmlns*=*" http://www.w3.org/1999/xhtml"*>{*app:get-google-analytics-script*()/*text* ()}</script> </head> <body> <div> (…) <div>{*$result-list*}</div> </div></body></html> *return $html* I am now trying to keep records of users most popular queries in my DB, so I made the function an updating function and adapted the return value: *let* *$html* := <html *xmlns*=*"http://www.w3.org/1999/xhtml"*> <head><link *href*=*"*{*$cssfile*}*"* *rel*=*"stylesheet"* *type*=* "text/css"*/><title>Search help</title> <script *type*=*"text/javascript"* *xmlns*=*" http://www.w3.org/1999/xhtml"*>{*app:get-google-analytics-script*()/*text* ()}</script> </head> <body> <div> (…) <div>{*$result-list*}</div> </div></body></html> *return *(*feedback:analyze-user-search-and-result*(*$keywords*, * $result-ids*, *$this-menu*, *$application*, *$carrier*, *$country*, *$lang*, *$model*, *$os*), *db:output*(*$html*)) I got this error: error on line 30 at column 25: Attribute xmlns redefined After lots of research, I figured that when using db:output, the top element of $result-list is interpreted as coming from a different namespace. Adding xmlns="http://www.w3.org/1999/xhtml" to that top element resolved my issue, but I still don't get why this is happening, can you explain? More specifically, here are definitions of $result-lists and what they produce: let $result-list := <div/> would work when used in the 1st code sample, but not the second (updating function). let $result-list := <div xmlns:="http://www.w3.org/1999/xhtml"/> resolves the issue for the second (updating function). Thanks, France -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com (514) 572-0341

2 4

Database corruption
by Antoni Mylka 12 Nov '12

12 Nov '12

Hi, We're developing an application that uses BaseX. In our tests, the application works properly for a couple of hours, makes queries to BaseX adds and removes data. Then suddenly some BaseX request will return the HTTP 500 response. From that point on all queries that make modifications in the database end in an error and the entire system grinds to a screeching halt. One of the experiments ran for four days, then we examined the access log to see that only the first 6 hours were OK, after that all BaseX queries would end in error. I'm going to investigate and will try to come up with as detailed a description of the issue as possible. For now the things I know are: 1. In our experiment, the application would make about 80000 successful queries over a period of 6 hours. Then it would get the first 500 error. 2. Afterwards it "seems" that the queries which don't modify anything end with 200, while all the modification queries end with a 500 error. That's to be confirmed. 3. BaseX 7.2.1 runs in a war deployed on Tomcat 7.0.32 on a linux system. 4. The access logs don't show anything interesting. 5. The catalina.out that shows the Tomcat standard output contains hundreds of exceptions of the following kind: Exception in thread "Thread-78197" java.lang.RuntimeException: Data Access out of bounds [pre:1422612, indexSize:5540, access:5540 > 5539] at org.basex.util.Util.notexpected(Util.java:55) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:478) at org.basex.io.random.TableDiskAccess.read5(TableDiskAccess.java:211) at org.basex.data.Data.textOff(Data.java:432) at org.basex.data.DiskData.text(DiskData.java:228) at org.basex.index.value.DiskValues.get(DiskValues.java:295) at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:133) at org.basex.data.DiskData$1.run(DiskData.java:461) The thread name is different every time and the numbers in square brackets are also different. I don't see any patterns (yet). 6. BaseX is accessed from a single thread in a single application. AFAIK there are no concurrent requests, though that is to be confirmed. 7. The GC log shows that basex stays below 100M of used heap, which is well below the allotted Xmx setting of 2G. 8. We did three experiments and the same effect has been observed in all three, albeit after different periods of time: (59K queries and 4:25 hours, 37K queries and 2:56 hours, 76.8K queries and 6 hours). Things I intend to do: 1. Set up proper logging for BaseX, so that these errors don't end up in catalina.out, but in the proper log file. 2. Catch the first offending query, see if there is anything unusual about it. It certainly doesn't differ much from the successful queries. It simply updates the BaseX state to keep it in sync with an external data source. All the update queries are generated by the same template and there have been hundreds of successful ones before the first error occurs. 3. Manipulate the log settings, to see if it's possible to get something more meaningful out of them. 4. Try newer versions: 7.3 and the current trunk. I've seen emails similar about "Data Access out of bounds" but they don't seem relevant. As I said, I'll try to come up with more information, but if the error or the stack trace looks familiar - I'd be grateful for pointers. Thanks in advance, -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business

2 1

Compilation issue
by Antoni Mylka 12 Nov '12

12 Nov '12

Hi, I tried to clone and build the basex maven module today and I stumbled upon two issues. 1. When JavaCC is called, it modifies the RegExParser.java. The modified version contains the following lines: } catch(final ParseException pe) { throw Err.REGERR.thrw(ii, pe.getMessage()); } catch(final TokenMgrError err) { throw Err.REGERR.thrw(ii, err.getMessage()); } this causes a compilation error because the Err enum does not seem to contain the REGERR constant. 2. In German.lang the h_revert_file key had to be renamed to h_reopen_file. Otherwise the LangTest would fail. Am I doing something wrong? Are these issues real? -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business

2 1

running xquery file with -c parameter returns unknown command: for
by bryan rasmussen 12 Nov '12

12 Nov '12

When I use -c to run a file and the file is empty_titles.bsx with the following XQUERY for $doc in db:open("dkcontent") where $doc//TITEL[. =""] return <doc>{$doc/*/@ID}</doc> returns fine. when the file is empty_titles.xq with the following: for $doc in db:open("dkcontent") where $doc//TITEL[. =""] return <doc>{$doc/*/@ID}</doc> I get Stopped at line 1, column 3: Unknown command: for. Try HELP. This is on Windows using basexclient. Thanks, Bryan Rasmussen

2 1

Re: [basex-talk] basex RUN local script when talking to Remote server
by Christian Grün 12 Nov '12

12 Nov '12

Hi Bryan (cc @ basex-talk), thanks for the confirmation. Yes, certain combinations of command-line arguments along with "-o" caused a buggy behavior. As usual, I've updated the latest snapshot containing the fix [1]. By the way, please note that all arguments will be evaluated from left to right in BaseX. This way, you can use a single BaseX call to write your results to several files, as e.g. shown here: basex -ofile1.txt -q1 -ofile2.txt -c"xquery 1 to 10" While this behavior may seem surprising at first glance, it has proven to be very helpful in our everyday's use. I tried to make this a little bit more explicit in our documentation [2]. Hope this helps, Christian [1] http://files.basex.org/releases/latest/ [2] http://docs.basex.org/wiki/Startup_Options

1 0

basex RUN local script when talking to Remote server
by bryan rasmussen 12 Nov '12

12 Nov '12

Hi, I have started the basexclient with a -n and -parameter to point to my server, what I would like to do is to execute scripts that I have laying around locally by doing something like RUN name_of_local_script.bsx and not get back a response that it cannot find name_of_local_script.bsx in the folder /home/account that has basex installation... Thanks, Bryan Rasmussen

2 3

UPDINDEX performance problem
by The Trainspotter 11 Nov '12

11 Nov '12

Hi, Just thought I'd share a problem I've got and to see if I'm solving it in the most efficient way. One of the XML documents I process contains various attributes which contain dates in non xs:date forat, e.g. "2010/1/20", "2010/12/1". In order to use the various date functions in XQuery I first need to bulk change all of these attribute values so they're in xs:date format. What I've found is that if I set the UPDINDEX property before I create the database, the XQuery function I have to change the date formats takes an age to complete. If I don't set it then the function completes pretty quickly. I need to use the UPDINDEX setting as I do a lot of updates and queries and I don't want to manually have to keep the indexes up to date. The solution I have at the moment is to create two databases, the first doesn't have UPDINDEX set, I load the document, do the date conversion, then export to a temporary file. I then create a second database from the temporary file, this time with UPDINDEX set. Is there a better/more efficient way of doing this? My date conversion function is this: declare updating function ts:convertToXsDate() { for $e in /*:NML/*:COLLECTION/*:ENTRY/@MODIFIED_DATE union /*:NML/*:COLLECTION/*:ENTRY/INFO/@RELEASE_DATE union /*:NML/*:COLLECTION/*:ENTRY/INFO/@LAST_PLAYED union /*:NML/*:COLLECTION/*:ENTRY/INFO/@IMPORT_DATE let $dateParts:= tokenize( data($e), "/") let $year:= $dateParts[1] let $month:= if( string-length($dateParts[2]) = 1) then concat("0",$dateParts[2]) else $dateParts[2] let $day:= if( string-length($dateParts[3]) = 1) then concat("0",$dateParts[3]) else $dateParts[3] return try { replace value of node $e with xs:date(concat($year,"-",$month,"-",$day)) } catch FORG0001 { () } }; Thanks in advance, Gary

2 3

Documents added by db:add(...) not visible for other existing database connections?
by Arjan van Bentem 11 Nov '12

11 Nov '12

Hi all, I noticed that TCP/IP clients are not seeing new documents that are created using db:add(...) in another client: Once a query has been executed, subsequent queries using the very same database connection do not see/return new documents that were meanwhile created using another connection. When creating some (different) document using one's own existing connection, then everything seems to be in-sync for a moment. But right after that, any new documents are again not seen. Updates made in other clients are visible just fine. So, the documents that are returned are up-to-date. It's just that not all documents are retrieved. I can see that in the query plans, the size in <DBNodeSeq size="..."> does not change, and is lower than the size that clients with a newer connection would show. Should it be okay to keep a connection open? Or is the expected behaviour to need a new connection to see new documents? Details below. Thanks, Arjan. 1. Start the server and a client in debugging mode: ./basexserver -d -S ./basexclient -U admin -P admin -d -x 2. Create and open a database: create database db1 open db1 3. Keep the client open, and make another client insert some documents. Like every second for about 2 minutes, each time using a new connection: for i in {1..120}; \ do doc="<test><a>`uuidgen`</a></test>"; \ echo $i, $doc; \ ./basexclient -U admin -P admin "db:add('db1', '$doc', 'dummy')"; \ sleep 1; \ done 4. While Step 3 is running, back in the first client, execute: xquery count(/test) The log in data/.logs shows: REQUEST XQUERY count(/test) OK Query executed in 25.74 ms. 223.77 ms So far, so good. 5. Repeat Step 4 to see the result NOT change, while the loop from Step 3 is still adding new documents. Logs show a faster result, but otherwise are the same. Also the query plans are always the same. (Typically: the size in <DBNodeSeq size="..."> does not change.) REQUEST XQUERY count(/test) OK Query executed in 5.48 ms. 7.29 ms 6. Using "close" followed by "open db1" gets one a more up-to-date value for "count". Also, using the connection to add some random other type of document, updates the value for "count(/test)" as well: xquery db:add('db1', '<another-test><xyz>123</xyz></another-test>', 'dummy') 7. It's not just "count" that does not change its value. Also, one can only query documents up to that count as returned by Step 4. Like if Step 4 returned "15", then the following would return 5 results, not 10: xquery (/test)[position()=10 to 20] Also, the following would only return the count as returned earlier: xquery /test Like noted, UPDATES made in other clients are visible just fine: xquery replace value of node /another-test/xyz with '456' xquery for $n in /test/a return replace value of node $n with fn:generate-id($n) Tested in both 7.3 and 7.5-20120929.201251 from http://files.basex.org/releases/latest/, in Oracle Java 7 on Mac OS X 10.8.2. I first noticed that my running application did not see new documents created by a test script. But the command line tools show the same issue. Any clue?

2 2

Crash when opening a BaseX database
by Rémi Jannel 10 Nov '12

10 Nov '12

Hi! After crashing my computer while editing a BaseX database, BaseX now returns an error when trying to access the database .I was editing the database through a web application (in Javascript) communicating with BaseX through the PHP wrapper and not with BaseX GUI. My system is Ubuntu 12.04 in a VirtualBox for the development on a Win7 64bits host system. All the other BaseX databases are working properly but when I'm trying to open the one I need, an error is returned. I put the message at the end of the email. I checked on my filesystem, the files corresponding to the database are still present in the BaseX data directory if you think it could be useful to debug what happened I could provide it. Is there something wrong on the BaseX version I'm using that could explain the data being corrupted? And is there a way for me to recover the database? The error message: Potential bug? Improper use? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.0.2 Java: Sun Microsystems Inc., 1.6.0_24 OS: Linux, i386 Stack Trace: java.lang.RuntimeException: Not expected: Data Access out of bounds [pre:1078, indexSize:9, access:9 > 8]. org.basex.util.Util.notexpected(Util.java:67) org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:394) org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:107) org.basex.data.Data.kind(Data.java:319) org.basex.query.item.DBNode$4.next(DBNode.java:317) org.basex.query.path.IterStep$1.next(IterStep.java:46) org.basex.query.path.IterStep$1.next(IterStep.java:33) org.basex.query.path.IterPath$1.next(IterPath.java:68) org.basex.query.path.IterPath$1.next(IterPath.java:40) org.basex.core.cmd.AQuery.query(AQuery.java:102) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:328) org.basex.core.Command.exec(Command.java:309) org.basex.core.Command.execute(Command.java:77) org.basex.server.ClientListener.run(ClientListener.java:200) Thanks, Remi Jannel

2 3