- BaseX-Talk - mailman.uni-konstanz.de

question about the Java API
by m4058＠aol.com 08 May '13

08 May '13

I would like to know if I can just upload a document to an existing collection or database. Attached is the addExample.java Thanks Mark Shortt

2 1

Improvements. On-fly in-memory temporary index
by Alexander Shpack 07 May '13

07 May '13

Hi Folks! I had one task with csv parsing and transforming. Main file contains approximately 100k of nodes. Transformation process should be in memory only, and I thought, that in-memory temporary index would be useful for this type of tasks. Useful scenario copy $clone := $csvContent modify ( tmp:optimize($clone), ... ) return $clone Main goal - increase performance with huge document fragments, modified and changed on-fly. Is it possible?

2 1

BaseX Capacity
by Rajabrata Chaudhuri 06 May '13

06 May '13

Hello, First I'd like to thank you guys for all your great work on BaseX. I am fairly familiar with XML DBs and have done a significant amount of development on top of Mark Logic. I would like to ask some questions about capacity and scalability. I have reviewed the documentation and see that the biggest store is for SDMX @ approximately 8000 GB. So I am just trying to understand what this means better and would appreciate any of your expert advice for my questions below: 1. Is the expectation that you can query against 8 TB of XML data efficiently? 2. My requirements will be to query across probably 24 TB of XML data. Do you guys feel this is possible? 3. What is the method to scale horizontally and vertically? I.E. Would I be adding more servers, or starting more instances, etc.? 4. How does high availability work? I.E. Can I have multiple active-active nodes, or should it be active-passive, etc.? Any help anyone can render is greatly appreciated. Thanks Raj

3 5

Enhanced Locking in BaseX: Status, "How Do You Use It?" Survey and "Wish List"
by Jens Erat 06 May '13

06 May '13

Dear BaseX-Community, # Status On the Improvements Over the last months we implemented huge enhancements on locking in BaseX. Writing transactions do not block the whole system any more; parallel reading and writing transactions are possible. For the 7.7 release (currently still in beta) we did further improvements like downgrading of locks before applying the pending update list to the database (which is IO-intensive) and more fine granular locks for commands, for example `COPY` only needs a read lock on the source database (which is great for backing up the system without need for stopping it). Nonetheless, estimating which locks are needed is a hard job in such powerful languages like XQuery. Databases can be opened at arbitrary positions in the code, and they can even be dependent on the content of other databases or external resources - it is not always possible to determine them at compile time, which leads to unnecessarily locked databases. # How Do You Use BaseX, What Are Your Locking Requirements? There are definitely more ways to enhance locking in BaseX. We developed the basic infrastructure for further enhancement (like MVCC) and already achieved a noticeable improvement by applying two phase locking, but for knowing how to go on, we'd like to hear your thoughts and needs. Possible ways to go would be "the classic database route" by using MVCC or heading straight to the opposite direction by evaluating optimistic locking or similar "non-locking" protocols. All of these have advantages and drawbacks, I'd be glad to receive your input on a few questions to determine some common use cases: - What is your application doing? - Which BaseX APIs do you use? - How do typical queries look like? If you cannot share them, a description following this guide would suffice: - How often will that query be run? Especially I'm asking you to differentiate between writing and updating queries, and how many databases you will access. - Complexity (using functions, modules, nested loops, lines of code) - Where do function calls opening databases and/or collections occur? - Do you use administrative functions (creating/... databases) inside XQuery? - Do you use / require `xquery:eval(…)`? - Do you access external resources like files or using HTTP requests? Do you need concurrency control for these? - Some enhancements could require transactions to be restarted. This could lead to HTTP-Requests / … to be executed again (we cannot roll back resources outside BaseX). Is this a problem for your application? Would it help if you were notified over the API that the query was restarted? (There will always be an alternative without restarted transactions!) - Any other comments you have with respect to locking and transactions * * * You can post information on that on the mailing list or send me a private [mail] (address below) if you do not share these information publicly. I will develop some use cases for my thesis out of your replies, and we will make decisions based on them how to go on with further improvements. If you cannot / do not want to answer all of these, we're happy on every input you give us. If your reply contains information not to be shared, please give me a hint and it will not be shared beyond the core BaseX team - but we will nonetheless include it in our future plans. Kind regards from Lake Constance, Germany, Jens Erat -- Jens Erat [phone]: tel:+49-151-56961126 [mail]: mailto:email@jenserat.de [jabber]: xmpp:jabber@jenserat.de [web]: http://www.jenserat.de PGP: 350E D9B6 9ADC 2DED F5F2 8549 CBC2 613C D745 722B

1 0

Salam Sahajetra
by Widiyana Samudra 05 May '13

05 May '13

Assalamualaikum Wr.Wb... Sebelum dan sesudahnya saya ingin memperkenalkan diri saya,nama saya WIDIYANA SAMUDRA berasal dari Indonesia dan bekerja di London (United Kingdom), ingin berkenalan dengan anda di sana. Saya ingin menawarkan satu peluang bisnis yang begitu bagus,dan bagi Anda yang berminat dengan bisnis ... Inilah kesempatan Anda,kapan lagi kalau bukan sekarang bukan.??? Produk ini di namakan (MULITE CLEANSER) kegunaannya untuk mencuci barangan yang sangat berharga sekali seperti batu intan permata yg masih mentah. Dan pada waktu yang sama perusahaan di tempat saya bekerja membutuhkan MULITE CLEANSER dimana mineral tersbut tersedia dari operator yg di Indonesia. Jadi bagi anda yang berminat dengan bisnis ini, saya ingin Anda menjadi Agent untuk menjual produk tersebut ke pada perusahaan tempat saya bekerja, (Anda membeli produk tersebut dari operator yang di Indonesia terlebih dahulu dengan harga 500USD(Lima Juta Rupiah) per karton dan dijualnya kembali ke pada perusahaan di tempat saya bekerja dengan harga 1,500USD(Lima belas Juta per karton) Yang paling penting Niat dan ikhlas untuk mencari rezeki semata-mata karena Allah. Insya Allah, Allah akan membantu kita untuk mencari rezeki yang halal dan berkat,saya ingin Anda buat keputusan yang bijak, Anda bisa merubahnya .. Insya Allah ' Apa bila ada berminat atau ada pertanyaan, silahkan balas email terus ke saya berdasarkan alamat di bawah atau silahkan berikan number supaya bisa di hubungi dan saya akan menerangkannya lebih lanjut di telepon: Insya Allah saya mencoba membantu Anda. Semoga ikatan silaturahmi sesama kita diberkati Allah. Selamat berkenalan. widiyana26(a)yahoo.com Yang benar... WIDIYANA SAMUDRA

1 0

Problem with XQuery Update replace
by Ludovic Kuty 02 May '13

02 May '13

Dear list, I have an XML document in the database which contains datas that has the form below. The attribute "site" represents the location from where the captured data originates, the element "num" contains the number of the measure (the identifier. it is the same as a primary key), "value" contains the value (xs:token even if no WS are allowed inside of it) and "time" the time of the acquisition (xs:dateTime). <datas site="xxx"> <data> <num>5</num> <value>700</value> <time>2013-05-01T21:43:13</time> </data> ... </datas> I want to update specific values at a given time. I usually have 300 or 400 values to update. That means, changing the content of child elements "value" and "time" but not "num" of many "data" elements. I wrote an XQuery script named update.xq that I placed in the Web repository. It will receive the site, the time, the numbers of the values to be updated (PK) and of course the values themselves as variables with a POST HTTP request. I do not use much typing as pretty much everything is xs:string. I guess it could be better but my problem is not there. declare variable $nums as xs:string external; declare variable $values as xs:string external; declare variable $time as xs:dateTime external; declare variable $site as xs:token external; let $n := tokenize($nums,','), $v := tokenize($values,',') return for $x at $i in $n return (replace value of node /datas[@site=$site]/data[num=$x]/value with $v[$i], replace value of node /datas[@site=$site]/data[num=$x]/time with $time) The problem is that sometimes it works and sometimes it gives me an HTTP 400 error with body : Stopped at update.xq, 9/73: [XUTY0008] Single element, text, attribute, comment or pi expected as replace target. I know the replace target has to have the expected format and I think it has. I don't understand why I get this. I though it was due to too long stuff so I divided the list of datas to update into sublists of size 50. Now, the problem appears sometimes but not everytime. When I use another XQuery script test.xq to display what was received, everything looks ok. The test script is : declare variable $nums as xs:string external; declare variable $values as xs:string external; declare variable $time as xs:dateTime external; declare variable $site as xs:token external; let $n := tokenize($nums,','), $v := tokenize($values,',') return for $x at $i in $n return ($x,$v[$i]) For example, this update is ok: <?xml version="1.0"?> <run xmlns="http://basex.org/rest"> <text>update.xq</text> <variable name="site" value="xxx"/> <variable name="time" value="2013-05-02T09:58:38"/> <variable name="nums" value="439,375,128,426,362,243,115,413,349,230,102,400,336,272,217,153,387,259,204,76,438,374,255,127,425,361,114,412,229,101,399,335,271,216,152,386,258,203,75,11,437,373,254,126,424,360,113,411,228,100"/> <variable name="values" value="1515.200,408,true,973.806,3.892,false,false,0.000,0.0,6,true,430.400,false,2,327.670,true,410,true,45.410,true,0.992,408,false,false,575.060,3.545,false,0.995,87.270,false,436.800,false,94.000,45.410,false,410,false,14.135,true,true,2152.800,0,true,true,4.924,3.639,false,0.992,87.070,true" /> </run> But this one is not : <?xml version="1.0"?> <run xmlns="http://basex.org/rest"> <text>update.xq</text> <variable name="site" value="xxx"/> <variable name="time" value="2013-05-02T09:58:38"/> <variable name="nums" value="36,398,334,215,385,202,74,436,372,253,125,423,359,240,112,442,410,378,227,131,99,429,397,365,333,214,118,54,416,384,352,288,233,201,105,73,9,435,403,371,339,275,220,156,124,422,390,358,294,239"/> <variable name="values" value="true,428.400,false,45.059,410,12.710,true,2152.800,0,false,false,2.437,-0.101,false,true,4907.327,0.994,409,0.000,false,true,1012.437,448.800,0.000,false,62.260,false,true,304.600,410,0.0,58,87.249,13.180,true,true,true,2152.800,454.200,0,false,2,757,true,true,0.326,0.000,0.0,41,false" /> </run> I think those documents are similar and I shouldn't see a difference. Either it works for both of them or it doesn't. Any idea to point me in the right direction is welcome. Thank in advance, Ludovic Kuty

2 4

parse against different DTDs?
by Graydon Saunders 30 Apr '13

30 Apr '13

Hi! I'm having a tough time finding where I can use commands to replicate the ticky-box behavior of the GUI to say "use the Java parser" and "parse against the DTD". What I want to be able to do is parse a given document against three different DTDs, validating it each time. It needs to parse against the specific DTD because these DTDs have fixed attributes, and if I don't parse the documents on load, the fixed attribute won't be there and validation will fail. Thanks! Graydon

2 1

xquery:eval yields NullPointerException
by Marco Lettere 29 Apr '13

29 Apr '13

Hi all, on the way to writing a small DSL I stumbled upon the following BaseX 7.6 issue. I'm not able to explain to myself why the first two expressions of the following xquery module return correctly whereas the last yields NullPointerException (as shown in the appended stacktrace). What I'm mtrying to do is to wrap up some modification code into userfriendly function calls that can be used in mini xquery scripts to formalize xml workflow. Thank you for any help, Marco. declare variable $globals := map { 'change' := local:change#1, 'generate' := local:gen#0 }; declare function local:gen(){ <b/> }; declare function local:change($doc){ copy $newdoc := $doc modify( insert node <a/> as last into $newdoc ) return $newdoc }; ( xquery:eval("copy $newdoc := <b/> modify( insert node <a/> as last into $newdoc ) return $newdoc",$globals), xquery:eval("$generate()", $globals), xquery:eval("$change($doc)", map:new( ($globals, map { "doc" := <b/>}))) ) The stacktrace and query plan: Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.6 Java: Oracle Corporation, 1.7.0_15 OS: Linux, i386 Stack Trace: java.lang.NullPointerException org.basex.query.up.expr.Transform.value(Transform.java:68) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.func.UserFunc.value(UserFunc.java:175) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:68) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:104) org.basex.query.func.DynamicFunc.value(DynamicFunc.java:56) org.basex.query.QueryContext.value(QueryContext.java:296) org.basex.query.QueryContext.value(QueryContext.java:254) org.basex.query.func.FNXQuery.eval(FNXQuery.java:95) org.basex.query.func.FNXQuery.eval(FNXQuery.java:71) org.basex.query.func.FNXQuery.iter(FNXQuery.java:37) org.basex.query.QueryContext.iter(QueryContext.java:285) org.basex.query.expr.List$1.next(List.java:100) org.basex.query.QueryContext.execute(QueryContext.java:524) org.basex.query.QueryProcessor.execute(QueryProcessor.java:96) org.basex.core.cmd.AQuery.query(AQuery.java:77) org.basex.core.cmd.XQuery.run(XQuery.java:22) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.gui.GUI.exec(GUI.java:397) org.basex.gui.GUI$7.run(GUI.java:349) Compiling: - pre-evaluating map { "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } } Optimized Query: declare function local:change($doc) { copy $newdoc as node() := $doc modify insert node element a { () } into $newdoc return $newdoc }; declare function local:gen() { element b { () } }; (xquery:eval("copy $newdoc := $generate() modify( insert node <a/> as last into $newdoc ) return $newdoc", map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }), xquery:eval("$generate()", map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }), xquery:eval("$change($doc)", map:new((map{ "change":=function($1) as item()* { local:change(($1)) }, "generate":=function() as item()* { local:gen(()) } }, map { "doc":=element b { () } })))) Query plan: <QueryPlan> <UserFuncs> <UserFunc name="local:change" arg0="doc"> <Transform> <Let var="$newdoc as node()"> <VarRef> <Var name="$doc" id="2"/> </VarRef> </Let> <Insert> <VarRef> <Var name="$newdoc as node()" id="3"/> </VarRef> <CElem> <QNm value="a" type="xs:QName"/> </CElem> </Insert> <VarRef> <Var name="$newdoc as node()" id="3"/> </VarRef> </Transform> </UserFunc> <UserFunc name="local:gen"> <CElem> <QNm value="b" type="xs:QName"/> </CElem> </UserFunc> </UserFuncs> <VarStack> <Var name="$globals" id="0"/> </VarStack> <List> <FNXQuery name="eval(string[,bindings])"> <Str value="copy $newdoc := $generate()
 modify(
 insert node <a/> as last into $newdoc 
 )
 return $newdoc" type="xs:string"/> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> </FNXQuery> <FNXQuery name="eval(string[,bindings])"> <Str value="$generate()" type="xs:string"/> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> </FNXQuery> <FNXQuery name="eval(string[,bindings])"> <Str value="$change($doc)" type="xs:string"/> <FNMap name="new([maps[,coll]])"> <List> <Map size="2"> <Str value="change" type="xs:string"/> <FuncItem type="function(item()*) as item()*"> <Var name="$1" id="1"/> <BaseFuncCall name="local:change(($1))"> <VarRef> <Var name="$1" id="1"/> </VarRef> </BaseFuncCall> </FuncItem> <Str value="generate" type="xs:string"/> <FuncItem type="function() as item()*"> <BaseFuncCall name="local:gen(())"/> </FuncItem> </Map> <LitMap> <Str value="doc" type="xs:string"/> <CElem> <QNm value="b" type="xs:QName"/> </CElem> </LitMap> </List> </FNMap> </FNXQuery> </List> </QueryPlan>

2 2

validation that returns node identifiers?
by Graydon Saunders 26 Apr '13

26 Apr '13

Hi! Using BaseX77-20130417.002917 though I don't think that's relevant to the question. This may well be an inherent property of the library being used to do validation, but right now, when I use validate:xsd-info, I get error messages that look like: Error: 25:59: cvc-complex-type.2.4.a: Invalid content was found starting with element 'link'. One of '{bold, footnote, italic, superscript}' is expected. These identify the location of the error by line and column number, which is great for code and less than helpful for XML content, since any format-and-indent getting applied to the XML file destroys its usefulness. Is is possible to have the error return a node identifier of some kind, instead? /root/bucket[2]/block[4]/para[3]/emphasis[4] sort of thing? Thanks! Graydon

2 1

Bypassing file locking in BaseX
by Scott Martens 25 Apr '13

25 Apr '13

Hi all, I have an application that's embedding BaseX in Java and generates all xqueries internally. These queries are *always* read-only. The problem I'm running into is that file locking prevents more than one user from reading the same file at the same time - something that BaseX 7.2 didn't do, but BaseX 7.6 does. I've tried using the client-server approach to solve the problem, and it works, sort-of. But it has a number of other problems for the conditions that it's being run under, and the embedded solution was faster and more robust. It looks to me like the BaseX standalone command line app and basexgui just ignore file-locking, and as long as I'm just using read-only queries, I have no trouble issuing two queries at once on the same database. Is this true? And if so, how is it done? Because this app is doing only read-queries, it's a lot better a solution for me than client-server, but I can't figure out where in the source code the database is being opened and how it's being done in a way that bypasses locks. Can anyone give me pointers? -- Scott Martens

3 3