BaseX-Talk

basex-talk@mailman.uni-konstanz.de

1 participants
5245 discussions

slow processing
by Mahlow Cerstin 02 Oct '12

02 Oct '12

Hi, I'm trying to understand why my Basex application is slow. I have a database looking like this: <collection> <entry time="2012-03-04T17:43:29"> <node>4119300</node> <query>[text() contains text ('Bank' ftand 'fallen') using stemming using language "de" distance at most 6 words ordered]</query> <person>marcel</person> <phraseme>Ad0032</phraseme> <selected>no</selected> </entry> <entry time="2012-03-04T17:43:29"> <node>11150403</node> <query>[text() contains text ('Bank' ftand 'fallen') using stemming using language "de" distance at most 6 words ordered]</query> <person>marcel</person> <phraseme>Ad0032</phraseme> <selected>no</selected> </entry> <entry time="2012-03-04T17:43:29"> <node>17335179</node> <query>[text() contains text ('Bank' ftand 'fallen') using stemming using language "de" distance at most 6 words ordered]</query> <person>marcel</person> <phraseme>Ad0032</phraseme> <selected>yes</selected> </entry> </collection> It consists of 97500 entries stored in the database "collect", one third has "yes" as value for <selected>, the other two third have "no". The number of entries will probably double over time. I use a CGI script to produce a HTML page first listing the total number of "yes" entries and the total number of distinct phrasemes, and then listing all entries sorted by <phraseme> where <selected> is "yes" in a table with 4 columns (phraseme, distinct persons, number of entries with this phraseme, link to another CGI-Perl script). Additionally, after the table I show the last timestamp for an entry with <selected> "yes". I use this for controlling purposes, to track progress of the use of the actual Basex search application. I put the relevant CGI code at the bottom. It is not that complex, but it takes 80 to 90 seconds. Which is much to slow! Skipping the timestamp information does not improve the speed. Do you have an idea how to improve this? Is the slow processing due to badly constructed XQueries, due to rendering as HTML table, due to server issues (I have a virtual server, but I don't know who else is using it for what)? my $session = Session->new("localhost", 1984, "admin", "admin"); $session->execute("open collect"); my $evidencecount = $session->execute("xquery let \$results := //selected[text() = 'yes'] return <b>{count(\$results)}</b>"); my @phrasemes = sort split(/\s+/, $session->execute("xquery distinct-values(//entry/phraseme/text())")); $session->close; my $phrasemecount = $#phrasemes + 1; print "<p> <b>$phrasemecount</b> accessed phrasemes with a total of $evidencecount hits</p>"; print "<table>"; print "<tr><th>Phraseme-ID</th> <th>Person</th><th>Count</th></tr>"; my $query =<<EOF; for \$phraseme in distinct-values(//entry/phraseme) let \$nodes := //phraseme[text() = \$phraseme] let \$count := count(\$nodes[../selected[text() = "yes"]]) let \$person := distinct-values(\$nodes/../person) order by \$phraseme return <tr><td>({\$phraseme})</td> <td>{\$person}</td> <td>{\$count}</td> <td><a href="basex-show-phraseme.pl?phraseme={\$phraseme}">show</a></td></tr> EOF my $viewsession = Session->new("localhost", 1984, "admin", "admin"); $viewsession->execute("open collect"); my $xquery = $viewsession->query($query); print $xquery->execute(); $xquery->close(); print "</table>"; # display last timestamp my $timequery = <<EOF; let \$i := //entry/\@time order by \$i/\@time ascending return <p>Last access: {data(\$i[last()])}</p> EOF my $xtimequery = $viewsession->query($timequery); print $xtimequery->execute(); $xtimequery->close(); $viewsession->close(); -- Dr. phil. Cerstin Mahlow Universität Basel Deutsches Seminar Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow(a)unibas.ch Web: http://www.oldphras.net

3 6

Silent installation of BaseX on Windows7
by Christian Steinherr 28 Sep '12

28 Sep '12

Dear BaseX Users, i just wanted to ask, if anyone tried to do a silent installation. I'm in charge of computer lab at the university and i was asked to install basex on all of the win7-pcs. Is there a hidden option, where all nessescary parameters can added at the beginning? Often Programms come with an option for silent or unattended installation. If you know such, please tell me, it would help me a lot. Kind regards christian

2 1

Keeping track of XSD versions? And multiple/single huge documents, or many smaller documents?
by Arjan van Bentem 28 Sep '12

28 Sep '12

Hi all, We're planning to use an XML database for a model that we know will change often over time. Like today a <User> might just have a <Name> and <Address>, but in the future we might also require <Telephone>. We will also be updating the data itself a lot, using XQuery Update through the xqj.net drivers. I have read [1] that the number of distinct namespace URIs in BaseX is limited to 256. We were thinking to keep keep track of "releases" of our XSDs through versioning of namespaces, but then 256 seems too low a number. - Any smart ideas about this? Also, I wonder if it would be better to use many small documents, or use a single document that hold all the nodes. Like: <Users> with many <User> nodes, or just many <User> documents. And many <Review> documents, or a single <Reviews> holding many <Review> nodes. Or even one big <Data> document to hold both <Users> and <Reviews>. One reason to use many documents is because, as described above, we will have different versions/releases, which seems easier to maintain in multiple documents. I've seen Christian's research [2], but that does not cover updates. So: - Does anyone know of real life examples of one approach, or the other? - Would updating a small document be faster than updating a node in a larger document? - Would it matter if we update multiple documents, like both a <User> and a <Review>, in a single XQJ statement? Also, one of the developers of eXist-db once wrote [3]: > [...] there are certainly some practical considerations. Storing a very > large document can block the db more than storing many small ones. > It requires a single transaction (and sufficient disk space for the > transaction log). > > [...] I know other users have to deal with much larger documents (many > gigabytes), but if you have a choice, I would definitely recommend to > split your data in smaller chunks, which are easier to handle. - Does anyone know if the same applies to BaseX? Any (other) input appreciated! Thanks, Arjan. [1] http://docs.basex.org/wiki/Statistics [2] http://kops.ub.uni-konstanz.de/bitstream/handle/urn:nbn:de:bsz:352-opus-127… [3] http://exist-open.markmail.org/thread/byuzpoj7cyfmlqdo

1 0

Re: [basex-talk] BUG# Multiple Thread try to execute some Query
by Samuel Lacmene 28 Sep '12

28 Sep '12

3 2

User rights: does using db:add really need the GLOBAL write permission?
by Arjan van Bentem 27 Sep '12

27 Sep '12

Hi all, In 7.3, I am using db:add (through a XQJ XQPreparedExpression) to add documents to a database. For a BaseX user to do that, I need to grant that database user the GLOBAL write permissions. With just LOCAL write permissions, I get: javax.xml.xquery.XQQueryException: [BASX0001]: WRITE permission required. Is this the expected behaviour? This is troublesome, as the GLOBAL write permission also gives that user read and write permissions on ALL other databases, unless these permissions are explicitly revoked. So, as a workaround, I can grant all user GLOBAL write permissions, and then explicitly deny all users LOCAL permissions to all existing databases that are not theirs. (Reading is not a problem with just local permissions. And as an aside: using "-i" on the command line even seems to require global CREATE permissions. But I don't really use that.) Any thoughts? Details below. Arjan. # 1. Create database and a user, with password "secret": ./basexclient -U admin -P admin > create db db1 Database 'db1' created in 11.05 ms. > create user user1 5ebe2294ecd0e0f08eab7690d2a6ee69 User 'user1' created. > grant write on db1 to user1 WRITE granted to 'user1' on 'db1'. > show users on db1 Username Read Write --------------------- user1 X X 1 Users. # 2. Let's this user access their own database. # Odd, using the command line "-i" even needs CREATE permissions? ./basexclient -U user1 -P secret -i db1 CREATE permission needed. # 3. Let's ignore that for now; using "open" works just fine: ./basexclient -U user1 -P secret > open db1 Database 'db1' was opened in 0.91 ms. > xquery db:add("db1", "<test><a>created by user1</a></test>", "test-doc-1") Stopped at line 1, column 67: [BASX0001] WRITE permission required. # 4. Okay, let's give the user the GLOBAL write permissions ./basexclient -U admin -P admin > grant write to user1 WRITE granted to 'user1'. > show users Username Read Write Create Admin ------------------------------------- admin X X X X user1 X X 2 Users. # 5. All fine for user1 now: ./basexclient -U user1 -P secret > xquery db:add("db1", "<test><a>created by user1</a></test>", "test-doc-1") Query executed in 6.18 ms. # 6. Let's create another user, with their own database, and the GLOBAL write as well: ./basexclient -U admin -P admin > create db db2 Database 'db2' created in 5.09 ms. > create user user2 5ebe2294ecd0e0f08eab7690d2a6ee69 User 'user2' created. > grant write to user2 WRITE granted to 'user2'. > grant write on db2 to user2 WRITE granted to 'user2' on 'db2'. > show users Username Read Write Create Admin ------------------------------------- admin X X X X user1 X X user2 X X 3 Users. > show users on db1 Username Read Write --------------------- user1 X X 1 Users. > show users on db2 Username Read Write --------------------- user2 X X 1 Users. # 7. And prove that user2 can now access db1, due to the global WRITE permissions: > open db1 Database 'db1' was opened in 1.9 ms. > xquery db:add("db1", "<test><a>created by user2 in db1</a></test>", "test-doc-2") Query executed in 1.27 ms. > xquery / <test> <a>created by user1</a> </test><test> <a>created by user2 in db1</a> </test> Query executed in 0.72 ms. # 8. That's bad. Let's explicitly revoke permissions from user2: > grant none on db1 to user2 NONE granted to 'user2' on 'db1'. > show users on db1 Username Read Write --------------------- user1 X X user2 2 Users. # 9. And have that user try again: ./basexclient -U user2 -P secret > open db1 READ permission needed. > xquery db:add("db1", "<test><a>created by user2 in db1</a></test>", "test-doc-3") Stopped at line 1, column 74: [BXDB0002] READ permission needed. # That's more like it. But: what when I forget that one day...? http://docs.basex.org/wiki/User_Management http://docs.basex.org/wiki/Database_Module#Updates

2 1

Adding a XML and fulltext index is gone
by Abhishek Jain 27 Sep '12

27 Sep '12

Hi Fellas, We are researching Basex for our upcoming solution to one of our customers. We are required to search and add XML documents to the data set. Just to brief you on the amount of data we receive around 8 million XML message per customer for a month. These set of data needs to be quickly searched and at the same time to expand it on receiving messages. I have created a fulltext index on the database. But as soon as I do insert a data item the fulltext index is gone and my searches are slowed down. How shall I proceed forward on this thread? Please help !! Regards, Abhishek

2 1

Re: [basex-talk] BUG# Multiple Thread try to execute some Query
by Samuel Lacmene 26 Sep '12

26 Sep '12

2 1

Re: [basex-talk] BUG# Multiple Thread try to execute some Query
by Samuel Lacmene 26 Sep '12

26 Sep '12

2 2

Re: [basex-talk] BaseX-Talk Digest, Vol 33, Issue 28
by Seenivasan Gunabalan 26 Sep '12

26 Sep '12

On 09/26/2012 12:00 PM, basex-talk-request(a)mailman.uni-konstanz.de wrote: > Send BaseX-Talk mailing list submissions to > basex-talk(a)mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request(a)mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner(a)mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Re: (How to) build application with BaseX (Christian Gr?n) > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk Hi all, we generally stick to open-source, so we use . Eclipse w/XQDT + basexgui with eclipse + xqdt, its easy for us to deploy the application and maintain version controlling system. XQDT is very good but it is in its beta stage. Sometimes syntax highlighting wont work. Also the auto-complete and parameter suggestion features are not up to the mark. we use basexgui at those time and also to validate the snippets of big code block. Regards, Seenivasan (How to) build application with BaseX

1 0

(How to) build application with BaseX
by Christian Grün 25 Sep '12

25 Sep '12

Hi Andreas, hi everyone, > Since XQDT works under Eclipse 3.7 and offers parentheses matching [...] that's an important point of course. I would generally interested what tools you (all) prefer to develop applications with BaseX, XQuery and XML. As you may know, the embedded BaseX editor can be used to process XQuery, XML and any other types of text documents. It know various powerful hotkeys [1] that are not documented in the GUI. It provides direct feedback on parsing errors in the edited and imported modules. The latest snapshot [2] also parses XML documents and command scripts and, since this weekend, offers a much more powerful facility for incrementally searching and replacing strings (incl. support for regular expressions). Enough of the self-adulation… We're well aware that it would be a waste of time if we seriously tried to cope with existing text editors and IDEs, such as.. • Eclipse w/XQDT • oXygen • vim, Emacs, etc. ..so we’d like to know what are your favorite editors for writing XQuery expressions and applications, and what convinces, or deters you, mostly from using the built-in editor or the alternatives? Thanks in advance for your opinion, Christian [1] http://docs.basex.org/wiki/Shortcuts#Editor_Shortcuts [2] http://files.basex.org/releases/latest/

4 7

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk