November 2015 - BaseX-Talk - mailman.uni-konstanz.de

Assigning incrementing number to 100ks of elements
by Chuck Bearden 24 Nov '15

24 Nov '15

BaseX 8.3.1, installed on Linux today from zip archive. I have about 137,000 target elements, each having an average of 550 descendant elements, and each taking about 45k of space on disk. The target elements each have an @uuid value, and I'd like to extract each @uuid value and assign it a unique number from an incrementing sequence of integers. I'd like to be able to start the sequence at an arbitrary value. I tried a recursive function I borrowed and modified from a StackOverflow post: declare function local:loop($seq, $count) { if(empty($seq)) then () else let $prod := $seq[1], $count := $count + 1 return ( <result> <id>{$count}</id> <uuid>{data($prod)}</uuid> </result>, local:loop(tail($seq), $count) ) }; and calling it like so: return <results>{ local:loop($pubs, 0) }</results> where $prod is a sequence of all the @uuid attributes (not the values, and not the whole elements). My query fails with the error: [bxerr:BASX0005] Stack Overflow: Try tail recursion? I'm using -Xmx6g and -Xss4m as flags for Java in the basexgui script. Is there some way to get BaseX to use tail recursion? Or is there another approach to doing what I want? Many thanks, Chuck

2 1

issue with stack trace in query-info window
by Rob Stapper 24 Nov '15

24 Nov '15

Hi Christian, I found that in case of a runtime error the links in the stack trace in the Query info window to the different locations in the program don't work anymore. I use of course the latest snapshot: BaseX 8.4 beta 3b6f6e0 Regards, Rob --- Dit e-mailbericht is gecontroleerd op virussen met Avast antivirussoftware. https://www.avast.com/antivirus

2 1

basex performance for high request number
by Filippo Gaudenzi 23 Nov '15

23 Nov '15

Hi all, I’m running some scripts to solve a NP-problem in order to check trees compatibility and i’m using Basex as a database for my hierarchy structure, indeed for each path/each node i run a query on the basexserver. This brings to have for a tree with 7 paths a 7! permutations and therefore at least 7!*4=21.000 queries (the query below) in a lapse of 5minutes. If i use the db in my localhost environment i see that basexserver goes over 150% of cpu usage. I was wondering how to increase the performance without causing the server crash. basex is running on a 2,6 GHz Intel Core i5 with 16gb of Ram, OS X 10.10.5 basex version is 8.2 Here the code i use: public class BaseXOntologyManager{ private final int PORT; private final String PASSWORD; private final String USERNAME; private final String ADDRESS; private final String DBNAME; /** * Constructor of the Basex ontology Manager * @param address The address of the DB * @param port The port to use to connect to the DB * @param username The username to use to connect to the DB * @param password The password to use to connect to the DB * @param dbName The db Name */ public BaseXOntologyManager(String address, int port, String username, String password, String dbName){//,boolean connected) { this.ADDRESS = address; this.PORT = port; this.USERNAME = username; this.PASSWORD = password; this.DBNAME = dbName; } private BaseXClient getOpenConnection() { try { BaseXClient session = new BaseXClient(this.ADDRESS, this.PORT, this.USERNAME, this.PASSWORD); return session; } catch (IOException e) { return null; } } /** * Return the subClasses of the element based on the ontology in the DB, * if the direct is true returns only the parents otherwise also the descendants * @param keyName The name of the Ontology to use * @param element the element to search in the param * @param direct true if you want only the parents * @return A list of subClasses of the ontology * @throws Exception when element doesn't belong to Ontology */ public ArrayList<String> getSubClasses(String keyName, String element, boolean direct) throws Exception { BaseXClient client = this.getOpenConnection(); if (client != null) { if(!checkExistence(element)) throw new Exception("notFound"+element); ArrayList<String> result = new ArrayList<String>(); String input = null; if (direct == true) { input = "for $n in db:open('" + this.DBNAME +"', '/" + keyName + "')//*[parent::" + element + "] return $n/name()"; } else { input = "for $n in db:open('" + this.DBNAME +"', '/" + keyName + "')//*[ancestor::" + element + "] return $n/name()"; } try { final BaseXClient.Query query = client.query(input); while (query.more()) { result.add(query.next()); } query.close(); result.add(element); return result; } catch (IOException e) { return null; } finally { try { client.close(); } catch (Exception e) { // TODO Auto-generated catch block e.printStackTrace(); } } }else{ System.out.println("CLIENT NULL"); } return null; } Thanks, Filippo

2 2

issue with the latest snapshot
by Rob Stapper 23 Nov '15

23 Nov '15

Gute Morgen, After installing the latest Basex snaphot (8.4) this morning, the new version couldn't resolved function calls to function that are located in imported modules while compiling. I get an error messages: "Unknown function '...'; similar: '...' ." After installing the previous version (8.3) everything worked fine again. I'm using windows 7 and basex is installed on my L:-drive. Hope this is enough information. Rob Stapper --- Dit e-mailbericht is gecontroleerd op virussen met Avast antivirussoftware. https://www.avast.com/antivirus

2 12

Curious "Stack Overflow: Try tail recursion?" error message
by Andy Bunce 20 Nov '15

20 Nov '15

Hi, Using a fresh install on 8.3.1 and running basexgui.bat on windows. I install this expath package https://github.com/expkg-zone58/ex-xparse/releases/download/v0.0.5/ex-xpars… Then, paste the following code into an editor window.. ------------------------ import module namespace xp="http://expath.org/ns/xparse"; "http://www.xqueryfunctions.com/xq/functx-1.0-doc-2007-01.xq" !unparsed-text(.) !xp:parse(.,map{"lang":"xquery"}) ----------------------- The result is the error... Stack Overflow: Try tail recursion? The curious part is.. if I keep retrying by clicking the execute button - after around 10 tries it starts working and seems to continue to work every time after that within that GUI instance. I have reproduced this on another pc. /Andy

2 2

Re: [basex-talk] Counting Occurences
by Christian Grün 20 Nov '15

20 Nov '15

Hi Ana (cc to the list), > I've tried using the "group by" clause, however, it only returns to me the > list of the present provinces, without the frequence, or, if I use the > "count" function, in only returns 1 (Since every curriculum on the > collection only contains a single occurence the province) Did you combine group by and count (as shown in the Wiki)? How does your current query look like? Christian > > Do you have any idea of how could I list the frequence of each province? > > Thank you for your help. > > 2015-11-20 15:17 GMT-02:00 Christian Grün <christian.gruen(a)gmail.com>: >> >> Hi Ana, >> >> The "group by" clause will help you here [1]. Hope this helps; feel >> free to ask if you need more examples. >> >> Christian >> >> [1] http://docs.basex.org/wiki/XQuery_3.0#group_by >> >> >> On Fri, Nov 20, 2015 at 6:06 PM, Ana Victória Ladeira >> <ladeira.ana(a)gmail.com> wrote: >> > Good afternoon! >> > >> > I have a collection of XML files in which all files contain the >> > attribute >> > //ENDERECO-PROFISISONAL/@UF >> > >> > In which UF is the province in which the person to whom the file makes >> > reference to lives. >> > >> > I want to develop a query that returns to me the number of occurences of >> > each province in the collection. I've tried using count and >> > distinct-values, >> > but it always return the value "1" for me. >> > >> > The query I'm trying to use is the following: >> > for $x in >> > >> > collection('Teste500')/CURRICULO-VITAE/DADOS-GERAIS/ENDERECO/ENDERECO-PROFISSIONAL >> > let $y := count(distinct-values($x/@UF)) >> > return $y >> > >> > The desired result would be, in a collection of documents in which 5 >> > people >> > live in SP, 3 in DF and 9 in AM would be the following: >> > >> > SP: 5 >> > DF: 3 >> > AM:9 >> > >> > Can somebody please help me with that? I'm still struggling with xQuery. >> > >> > > >

2 1

Counting Occurences
by Ana Victória Ladeira 20 Nov '15

20 Nov '15

Good afternoon! I have a collection of XML files in which all files contain the attribute //ENDERECO-PROFISISONAL/@UF In which UF is the province in which the person to whom the file makes reference to lives. I want to develop a query that returns to me the number of occurences of each province in the collection. I've tried using count and distinct-values, but it always return the value "1" for me. The query I'm trying to use is the following: for $x in collection('Teste500')/CURRICULO-VITAE/DADOS-GERAIS/ENDERECO/ENDERECO-PROFISSIONAL let $y := count(distinct-values($x/@UF)) return $y The desired result would be, in a collection of documents in which 5 people live in SP, 3 in DF and 9 in AM would be the following: SP: 5 DF: 3 AM:9 Can somebody please help me with that? I'm still struggling with xQuery.

2 1

Applicative Cache
by Marco Lettere 20 Nov '15

20 Nov '15

Hi all, I went though the emails talking about mainmemory databases and related but I've not been able to find an answer to this loud thinking ... Would it be possible to create an in memory database with a RESTXQ call and keep it accessible for other RESTXQ invocations as if it was a disk db throughout the lifecycle of a basexhttp server? Application level cacheing of (possibly distributed) dbs or management of transient volatile data items like tokens are the scenarios I have in mind. Just wondering whether this makes any sense. Whether it's feasible or whether someone already did this with other means. Thanks a lot! Marco.

3 6

Bash command
by meumapple 19 Nov '15

19 Nov '15

Hi all, I am using proc:system for a bash command but I cannot run it. It looks like opennlp postagger model.bin < file.txt. I suspect that the problem is the less-than sign, which is converted into < Is there any way to keep the less-than sign and pass it to the bash properly? Thanks. Joseph

4 8

How to improve performance of join query
by Chuck Bearden 18 Nov '15

18 Nov '15

Hi all, I'm joining elements from two sets of documents in a single database. There are many elements (six figures) in each set. The query doesn't finish after several minutes. My suspicion is that the indices aren't being used, except to limit the fingerprint 'content' elements to those with "Person" as the value of 'contentFamily'. Perhaps my data and query demand some form of optimization that hasn't been implemented yet, but perhaps there is something I could do to fix the query. I'm using BaseX 8.3.0. The query: ------------------------- xquery version "3.0"; declare namespace fingerprint="http://atira.dk/schemas/pure4/wsdl/template/fingerprint/current"; declare namespace concept="http://atira.dk/schemas/pure4/wsdl/template/concept/current"; declare namespace core="http://atira.dk/schemas/pure4/model/core/current"; declare namespace xsi="http://www.w3.org/2001/XMLSchema-instance"; declare namespace fincur="http://atira.dk/schemas/pure4/model/template/fingerprint/current"; declare namespace concur="http://atira.dk/schemas/pure4/model/template/concept/current"; declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method "csv"; declare option output:csv "header=yes, separator=comma"; let $coll := collection('pure_20151109') let $concept_contents := $coll/concept:GetConceptResponse/core:result/core:content let $fingerprint_contents := $coll/fingerprint:GetFingerprintResponse/core:result/core:content for $fin_con in $fingerprint_contents[fincur:contentFamily/text()="Person"] /fincur:rankedConcepts/fincur:rankedConcept let $concept_uuid := $fin_con/fincur:conceptUuid let $rank := $fin_con/fincur:rank let $weighted_rank := $fin_con/fincur:weightedRank let $person_uuid := $fin_con/ancestor::core:content[1]/@uuid let $concept_name := $concept_contents[@uuid=$concept_uuid]/concur:name/core:localizedString return <csv> <record> <person_uuid>{data($person_uuid)}</person_uuid> <concept_uuid>{$concept_uuid/text()}</concept_uuid> <rank>{$rank/text()}</rank> <weighted_rank>{$weighted_rank/text()}</weighted_rank> <concept_name>{$concept_name/text()}</concept_name> </record> </csv> ------------------------- The query info from the query, which I stopped after 3 minutes: ------------------------- Error: Interrupted. Compiling: - pre-evaluating collection("pure_20151109") - inlining $coll_0 - inlining $fingerprint_contents_2 - applying text index for "Person" - inlining $rank_5 - inlining $weighted_rank_6 - inlining $person_uuid_7 - inlining $concept_name_8 Query: xquery version "3.0"; declare namespace fingerprint="http://atira.dk/schemas/pure4/wsdl/template/fingerprint/current"; declare namespace concept="http://atira.dk/schemas/pure4/wsdl/template/concept/current"; declare namespace core="http://atira.dk/schemas/pure4/model/core/current"; declare namespace xsi="http://www.w3.org/2001/XMLSchema-instance"; declare namespace fincur="http://atira.dk/schemas/pure4/model/template/fingerprint/current"; declare namespace concur="http://atira.dk/schemas/pure4/model/template/concept/current"; declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method "csv"; declare option output:csv "header=yes, separator=comma"; let $coll := collection('pure_20151109') let $concept_contents := $coll/concept:GetConceptResponse/core:result/core:content let $fingerprint_contents := $coll/fingerprint:GetFingerprintResponse/core:result/core:content for $fin_con in $fingerprint_contents[fincur:contentFamily/text()="Person"] /fincur:rankedConcepts/fincur:rankedConcept let $concept_uuid := $fin_con/fincur:conceptUuid let $rank := $fin_con/fincur:rank let $weighted_rank := $fin_con/fincur:weightedRank let $person_uuid := $fin_con/ancestor::core:content[1]/@uuid let $concept_name := $concept_contents[@uuid=$concept_uuid]/concur:name/core:localizedString return <csv> <record> <person_uuid>{data($person_uuid)}</person_uuid> <concept_uuid>{$concept_uuid/text()}</concept_uuid> <rank>{$rank/text()}</rank> <weighted_rank>{$weighted_rank/text()}</weighted_rank> <concept_name>{$concept_name/text()}</concept_name> </record> </csv> Optimized Query: let $concept_contents_1 := (db:open-pre("pure_20151109",0), ...)/concept:GetConceptResponse/core:result/core:content for $fin_con_3 in db:text("pure_20151109", "Person")/parent::fincur:contentFamily/parent::core:content[parent::core:result/parent::fingerprint:GetFingerprintResponse/parent::document-node()]/fincur:rankedConcepts/fincur:rankedConcept let $concept_uuid_4 := $fin_con_3/fincur:conceptUuid return element csv { (element record { (element person_uuid { (data($fin_con_3/ancestor::core:content[position() = 1]/@uuid)) }, element concept_uuid { ($concept_uuid_4/text()) }, element rank { ($fin_con_3/fincur:rank/text()) }, element weighted_rank { ($fin_con_3/fincur:weightedRank/text()) }, element concept_name { (($concept_contents_1)[(@uuid = $concept_uuid_4)]/concur:name/core:localizedString/text()) }) }) } Query plan: <QueryPlan compiled="true"> <GFLWOR> <Let> <Var name="$concept_contents" id="1"/> <IterPath> <DBNodeSeq size="51261"> <DBNode name="pure_20151109" pre="0"/> <DBNode name="pure_20151109" pre="1603"/> <DBNode name="pure_20151109" pre="2781"/> <DBNode name="pure_20151109" pre="5227"/> <DBNode name="pure_20151109" pre="7212"/> </DBNodeSeq> <IterStep axis="child" test="concept:GetConceptResponse"/> <IterStep axis="child" test="core:result"/> <IterStep axis="child" test="core:content"/> </IterPath> </Let> <For> <Var name="$fin_con" id="3"/> <CachedPath> <ValueAccess data="pure_20151109" type="TEXT"> <Str value="Person" type="xs:string"/> </ValueAccess> <IterStep axis="parent" test="fincur:contentFamily"/> <IterStep axis="parent" test="core:content"> <CachedPath> <IterStep axis="parent" test="core:result"/> <IterStep axis="parent" test="fingerprint:GetFingerprintResponse"/> <IterStep axis="parent" test="document-node()"/> </CachedPath> </IterStep> <IterStep axis="child" test="fincur:rankedConcepts"/> <IterStep axis="child" test="fincur:rankedConcept"/> </CachedPath> </For> <Let> <Var name="$concept_uuid" id="4"/> <IterPath> <VarRef> <Var name="$fin_con" id="3"/> </VarRef> <IterStep axis="child" test="fincur:conceptUuid"/> </IterPath> </Let> <CElem> <QNm value="csv" type="xs:QName"/> <CElem> <QNm value="record" type="xs:QName"/> <CElem> <QNm value="person_uuid" type="xs:QName"/> <FnData name="data([items])"> <CachedPath> <VarRef> <Var name="$fin_con" id="3"/> </VarRef> <IterPosStep axis="ancestor" test="core:content"> <Pos min="1" max="1"/> </IterPosStep> <IterStep axis="attribute" test="uuid"/> </CachedPath> </FnData> </CElem> <CElem> <QNm value="concept_uuid" type="xs:QName"/> <CachedPath> <VarRef> <Var name="$concept_uuid" id="4"/> </VarRef> <IterStep axis="child" test="text()"/> </CachedPath> </CElem> <CElem> <QNm value="rank" type="xs:QName"/> <IterPath> <VarRef> <Var name="$fin_con" id="3"/> </VarRef> <IterStep axis="child" test="fincur:rank"/> <IterStep axis="child" test="text()"/> </IterPath> </CElem> <CElem> <QNm value="weighted_rank" type="xs:QName"/> <IterPath> <VarRef> <Var name="$fin_con" id="3"/> </VarRef> <IterStep axis="child" test="fincur:weightedRank"/> <IterStep axis="child" test="text()"/> </IterPath> </CElem> <CElem> <QNm value="concept_name" type="xs:QName"/> <CachedPath> <IterFilter> <VarRef> <Var name="$concept_contents" id="1"/> </VarRef> <CmpG op="="> <CachedPath> <IterStep axis="attribute" test="uuid"/> </CachedPath> <VarRef> <Var name="$concept_uuid" id="4"/> </VarRef> </CmpG> </IterFilter> <IterStep axis="child" test="concur:name"/> <IterStep axis="child" test="core:localizedString"/> <IterStep axis="child" test="text()"/> </CachedPath> </CElem> </CElem> </CElem> </GFLWOR> </QueryPlan> ------------------------- Thanks in advance for any advice! Chuck

2 2