September 2013 - BaseX-Talk - mailman.uni-konstanz.de

JSON or JSONML output formatting
by Zachary Pride 16 Jul '14

16 Jul '14

Hello, I've been playing with BaseX lately and all-in-all am very impressed. Something I'm wrestling with however is the JSON output options. I've got various web applications that would really be simplified by receiving JSON straight away for the obvious scripting benefits. The JSON method has been giving me a lot of grief, stipulating rules I'm not sure how to follow. right now I have an element that "is typed as string and cannot be nested." I've been able to get JSONML to output but the JSON is this morass of nested Arrays... Attributes seem to get transformed into Objects but elements/tags and their values are pushed into Arrays for some rationale - which I suppose is on purpose. My question is how can I get Objects for simple key:value properties from <tag>value</tag> 's. Are there options I'm not utilizing? Would it be feasible to transform the XML output to some middle form first - setting it up to become Objects, before serializing it to JSON? Thanks! Zach Pride Associate Software Engineer Ultra Electronics, ProLogic 1000 Green River Drive, Ste. 201 Fairmont, WV 26554 zachary.pride(a)ultra-prologic.com Tel: +1 304 333 2680 ext.308 www.ultra-prologic.com This e-mail from Ultra Electronics, ProLogic and any attachments to it are confidential to the intended recipient and may also be privileged or controlled by U.S. export control laws and regulations. No technical data, information or other items provided by Ultra Electronics, ProLogic in connection with this email shall be shared or provided to any foreign persons, entities, or subsidiaries without the expressed written authorization of Ultra Electronic, ProLogic. If you have received it in error please notify the sender and delete it from your system. If you are not the intended recipient you must not copy it or use it for any purpose nor disclose or distribute its contents to any other person. All communications may be subject to interception or monitoring for operational and/or security purposes. Please rely on your own virus checking as the sender cannot accept any liability for any damage arising from any bug or virus infection. This e-mail from Ultra Electronics, ProLogic and any attachments to it are confidential to the intended recipient and may also be privileged or controlled by U.S. export control laws and regulations. No technical data, information or other items provided by Ultra Electronics, ProLogic in connection with this email shall be shared or provided to any foreign persons, entities, or subsidiaries without the expressed written authorization of Ultra Electronic, ProLogic. If you have received it in error please notify the sender and delete it from your system. If you are not the intended recipient you must not copy it or use it for any purpose nor disclose or distribute its contents to any other person. All communications may be subject to interception or monitoring for operational and/or security purposes. Please rely on your own virus checking as the sender cannot accept any liability for any damage arising from any bug or virus infection. Please consider the environment before printing this email.

4 4

Dynamic Module Loading
by Jean-philippe Magué 19 May '14

19 May '14

Hello, Exist-db has an util:import-module<http://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xq…>function that allows to dynamically import an XQuery module at run time. I haven't found anything equivalent in BaseX. Is that right ? I have also tried to see how to implement such a function directly in Java, but as far as I understand the module parsing process, functions prefix are checked at compile time. That would compromise any attempt to write such a function. Am I right ? Best, Jean-Philippe -- Jean-Philippe Magué Maître de Conférences en Humanités Numériques Responsable de l'Atelier des Humanités Numériques : ahn.ens-lyon.fr Ecole Normale Supérieure de Lyon +33 (0)4 26 23 38 08 -- "Using computers will therefore lead us to a more profound and systematic knowledge of human expression; in principle, it can help us to be more humanistic than before." Busa, 1980.

6 16

Full Text Module - ft:search function
by John Best 25 Oct '13

25 Oct '13

Dear Christian, The syntax of the ft:search is - *ft:search($db as xs:string, $terms as item()*, $options as item()) as text()** In the $options part, I have seen various options like - Any Word, All Words, Phrase (for Exact Phrase), fuzzy and wildcards. These are really very good features. I am using them extensively. But one thing, I think, is missing and that is the Window/Distance Options for proximity search. Can these options be implemented.. ?? -- Have a nice day JBest

3 5

db:list($db) returns raw files, but not db:list($db, $path)
by France Baril 01 Oct '13

01 Oct '13

I'm currently on 7.7. The second function returns files from folders and subfolders at $path, but omits the folders that contain only raw files. -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com (514) 572-0341

2 1

BaseX http running single threaded?
by Joe Templeman 30 Sep '13

30 Sep '13

Hi all, I'm having trouble with the running a restxq API, set up as either a stand alone app (using basexhttp) or running the servlet in jetty. When I make a request, the server will lazily load in the requested content using a small java library that I've built which can take quite a while, and unfortunately it completely blocks waiting for this request to complete. This means that if I make a request for content which is already loaded, which should return in a couple of milliseconds, it instead waits for the downloading of the first request to complete before returning. I'm sure it's something simple that I'm missing in the config somewhere, but I have the threadpool parameters set up in my jetty.xml config file but it doesn't seem to make any difference. Is this expected behavior? I can't imagine any HTTP server would ever be designed to be single threaded... Any help is welcome, I've been trying to figure this out for days now. Thanks, Joe

3 5

Keep index in main memory-performance
by Aravanis, G. 30 Sep '13

30 Sep '13

Hi community, I have created a DB of around 7 GB documents in BaseX (100 first records of Medline) with full text index option enabled (index size 1 GB). I test it locally on my laptop. The problem is that I observe great variations in the free text query times from 8s up to couple of minutes. Main memory is underused (allocated 2 GB) and the processor has quite low utilization. I observe high usage of the hard disk. I have read that the whole db could be loaded in the main memory, but this is not an option for an 100 GB db collection that I would like to have. My question is if there is a possibility to load only the index in the main memory in order to accelerate the process of searching. Any other performance tips are very welcome. Thank you, George Aravanis

2 4

Re: [basex-talk] Set Operator Examples
by Arto Viitanen 26 Sep '13

26 Sep '13

Question was about set operations in BaseX. I had to calculate set difference on a sequence of values. XQuery has operator except, but it works only on nodes. I guess normal code for this is using distinct-values: declare function local:difference-pred($a, $b) { distinct-values($a[not(.=$b)]) }; I used it on big (thousands of values) sets, and it was slow. Then I tried on maps using BaseX map-module : declare function local:difference-map($a, $b) { let $m1 := map:new(for $i in $a return map:entry($i, true())) let $m2 := map:new(for $i in $b return map:entry($i, false())) let $m3 := map:new(($m1, $m2)) return for $i in map:keys($m3) return if ($m3($i)) then $i else () }; Then we found another solution, with only one map: declare function local:difference-map-2($a, $b) { let $m2 := map:new(for $i in $b return map:entry($i, true())) return for $i in $a return if($m2($i)) then () else $i }; When trying them at same sequences: let $a :=for $i in (1 to 100000) return if (random:double() < 0.01) then () else string($i) let $b := for $i in $a return if (random:double() < 0.45) then () else $i return (count($a), count($b), count(prof:time( local:difference-pred($a, $b), true(), 'pred ')), count(prof:time( local:difference-map($a, $b), true(), 'map ')), count(prof:time( local:difference-map($a, $b), true(), 'map2 '))) This gives times (on BaseX 7.6 running on OpenSUSE on VMWare virtual machine on a PC). pred: 80468.68ms map: 261.23 ms map2: 253.77 ms That is: map2 is 317 times faster than distinct-values version. I did not measure memory usage. Also on different size sequences, each one of the functions can be fastest! -- Arto Viitanen Finland

4 7

Issue with linebreak using REST since 7.7.1
by Yoann Maingon 25 Sep '13

25 Sep '13

Hi, I've got a simple node move PHP script (sorry for my terrible code, I'm an electronician!) 1. // get the node 2. $xquery = "XQUERY //noteitem[@id='$movedID']"; 3. $saveNode = $session->execute($xquery); 4. 5. // delete the node 6. $xquery = "XQUERY delete node //noteitem[@id='$movedID']"; 7. $content = $session->execute($xquery); 8. // insert the node 9. if ($nextID == 'last') { 10. $xquery = "XQUERY insert node ($saveNode) as last into //meeting[@id='$meetingID']/items " ; 11. $content = $session->execute($xquery); 12. } else { 13. echo $saveNode; 14. $xquery = "XQUERY insert node ($saveNode) before //meeting[@id='$meetingID']/items/noteitem[@id='$nextID'] " ; 15. $content = $session->execute($xquery); 16. } 17. $contentFull = "<MN>$content</MN>"; 18. echo $contentFull; If I look into the content that is saved, deleted and inserted I have the following: <noteitem status="new" commentsCounter="0" created_on="Tue, 24 Sep 13 12:09:29 +0000" modified_on="Tue, 24 Sep 13 12:09:29 +0000" id="MnNo524180f9587c1" genid="MnNo524180f9587c1" owner="1" type="question"> <notecontent>...</notecontent> </noteitem> It worked fine until the latest update. Where apparently the query breaks with the line break. It creates an error saying that the closing </noteitem> tag is missing. Just got back to 7.6 and it works well. any idea? Yoann Maingon CEO - mydatalinx +33664324966

2 1

feature request -- GUI database creation overall progress
by Graydon Saunders 25 Sep '13

25 Sep '13

Hi! Right now, creating a database in the GUI gives a per-file progress, which is accurate but not informative. Any chance of getting, in addition to the per-file progress, the overall progress of creating the database? Thanks! Graydon

2 1

empty tags
by Lars Johnsen 24 Sep '13

24 Sep '13

Hello BaseX developers! When using BaseX scripting to generate HTML, I got a problem with the representation of empty nodes. For example, my code has something like this in it <div> { ... some code...} </div> if ...some code... returns nothing, the div is empty as is labelled <div/>. This causes trouble in HTML, since <div/> is treated as if it was the start tag <div>, and will therefore nest over following divs. Is there a way for BaseX to output <div></div> instead? There is an ad hoc solution which checks if ...some code... is empty and inserts a blank in those cases, preventing the div from collapsing into one tag, but was hoping for a more general solution. All the best, Lars G Johnsen National Library of Norway

2 1