- BaseX-Talk - mailman.uni-konstanz.de

Fwd: Re: How to retrieve HTTP request parameters and use them in XQuery with BaseX
by Andreas Weiler 31 Dec '10

31 Dec '10

Forwarding to the mailing-list... -------- Original-Nachricht -------- Betreff: Re: [basex-talk] How to retrieve HTTP request parameters and use them in XQuery with BaseX Datum: Thu, 30 Dec 2010 13:59:57 +0000 Von: software developer <computer.software.developer(a)gmail.com> An: Andreas Weiler <andreas.weiler(a)uni-konstanz.de> Well, then the problem changes to 'how to pass dynamic parameters from an HTTP client to the java class in question - java class is running on a different machine' - that bridge was to be the HTTP request parameters - what else can serve to pass dynamic parameters to the xquery? I invoke BaseX via jax-rx and xquery is stored on server-side (cannot put that in the URL itself) - xquery is invoked using run param to jax-rx invocation. Your solution has xquery passing the param to the java class, in my case an external HTTP client is to send the parameters...so, how would the java class receive the HTTP request param or alternatively, how would the xquery receive request param directly via jax-rx? On Thu, Dec 30, 2010 at 1:18 PM, Andreas Weiler <andreas.weiler(a)uni-konstanz.de <mailto:andreas.weiler@uni-konstanz.de>> wrote: Hi, passing parameters to xquery dynamically can be done with the xquery java bindings. So the parameter comes directly from your java class.

3 3

Cannot execute xxforms:get-request-parameter in XPL - XSLT processor
by software developer 31 Dec '10

31 Dec '10

Hi, I get the error: XPath syntax error at ... in {...get-request-parameter('quer...}: Cannot find a matching 1-argument function named { http://orbeon.org/oxf/xml/xforms}get-request-parameter(). *Note: external function calls have been disabled* when I attempt to execute a pipeline with: <p:processor name="oxf:xslt"> <p:input name="config"> <xsl:stylesheet version="2.0"> <xsl:template match="/"> <TargetURL> <xsl:variable name="location" select="/Configuration/XMLDB/Location/text()"/> <xsl:variable name="name" select="/Configuration/XMLDB/Name/text()"/> <xsl:variable name="query" select="xxforms:get-request-parameter('query')"/> <xsl:value-of select="fn:concat($location,'/',$name,'?',$query)"/> </TargetURL> </xsl:template> </xsl:stylesheet> </p:input> <p:input name="data" href="#configuration"/> <p:output name="data" id="Target"/> </p:processor> Is XPL not the correct location to retrieve the HTTP request params (should I be doing it in page-flow.xml instead?)

1 1

Multiple producers/writers for one collection and concurrency
by Jan Vlčinský (CAD) 31 Dec '10

31 Dec '10

Hi This topic was mentioned few times before, but it is key aspect of my application, so I will try anyway. *Question*: What is recommended approach for application, where exists multiple document producers which shall store/modify documents in one, share collection. *Background* - Having *feeds*, which might deliver some new messages. - *Multiple preprocessing processes* - preprocessing the messages before storing them in collection (Python, celery and RabbitMQ to be ready for processing bigger amount of incoming messages in parallel) - Having *single process for write access to my BaseX collection*(arranging this is not very easy for me so far) - There are some read only queries for stored messages, there are no problems with this as it shall be easy to run multiple read-only queries concurrently. *Assumptions* If I am wrong here, please, correct me. - one collection can be written/modified only by one process at given moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?) - read only queries are easy as they can be run concurrently without blocking/rejecting *Current approach* 1. Multiple preprocessing processes are concurrently sending documents into single basex-storage queue in RabbitMQ 2. There is single one-process worker, consuming documents from queue basex-storage which does all the work with adding documents for given collection. *Wishes* Managing single write process adds quite complexity to whole solution. It would be nice, if I can write to single collection by multiple processes at once and could expect success. Any comments are welcome Best regards Jan

2 4

Big recursive XQueries
by Charles F. Munat 31 Dec '10

31 Dec '10

I have a BaseX backed website which stores all the pieces of the page in the DB. There is a significant amount of processing to be done. Pretty much any piece of the page can have effective or expiration dates (so that page parts can come and go on schedule). Some pages have dynamic elements, such as a Twitter feed. The page is build from objects which are wrapped in frames (for adding temporal and styling parameters), which are inserted into columns, which go into rows, which make up a page (which itself is wrapped in a template). So there is a lot of recursion. I do the recursion using local functions. It works well, but the page-building query is getting enormous, most of which is all these functions to insert parts of the page, often recursively. I am wondering what the best practice would be for this. I'm working in Scala, and I could break the process up into repeated passes hitting the db with a different query for each, but it seems to me one big query will almost certainly be faster. But including all the local functions in the query makes it huge. Is there some way to compile and load the functions once on startup, and then just run a small XQuery that calls functions which call functions, etc.? Any ideas (or resources) for ways to optimize the query? Everything gets reused, so there is very little nesting -- mostly references are passed. For example: <pages> <page> <id>1</id> <rows> <row id="2"/> <row id="3"/> </rows> </page> </pages> <rows> <row> <id>2</id> <columns> <column width="18"> <frames> <frame id="4"/> <frame id="5"/> </frames> </column> <column width="6"> <frames> <frame id="6/> </frames> </column> </columns> </row> </rows> <frames> <frame> <id>4</id> <contents> <content id="7"/> </contents> </frame> </frames> <contents> <content type="TEXT"/> <id>7</id> <body> <p>Some text here.</p> </body> </content> <content type="WIDGET"/> <id>7</id>  </content> </contents> Thanks! Chas.

6 11

How to retrieve HTTP request parameters and use them in XQuery with BaseX
by software developer 30 Dec '10

30 Dec '10

Hi, eXist DB offers this facility via HTTP related functions: http://en.wikibooks.org/wiki/XQuery/Checking_for_Required_Parameters What I want to achieve is pass parameters to XQuery dynamically (not necessarily by using HTTP request parameters, although that appears to be one way of achieving this). Is that possible in BaseX; if yes, how? Thanks.

2 1

Re: [basex-talk] Database creation from a directory tree
by Andreas Weiler 29 Dec '10

29 Dec '10

Hi David, the syntax of the create command changed between the releases of 6.1 and 6.3.4. In release 6.1 you have to use: create db C:\Users\David\Documents\OpenSong\songpacks\Songs songs Kind regards, Andreas Am 28.12.10 18:00, schrieb David leigh: > > Here are my command line results from 6.3.4: > > C:\Users\David\Documents\OpenSong\OpenSongSearch\BaseX>java -cp > BaseX.jar org.basex.BaseXClient -Uadmin -Padmin > > BaseX 6.3.5 [Client] > > Try "help" to get more information. > > > drop db songs > > Database 'songs' was dropped. > > > create db song > > Database 'song' created in 1683.61 ms. > > > info index > > Tags > > - Structure: Hash > > - Entries: 0 > > Attributes > > - Structure: Hash > > - Entries: 0 > > Text Index > > - Structure: Binary tree > > - Size: 4 Bytes > > - Entries: 0 > > Attribute Index > > - Structure: Binary tree > > - Size: 4 Bytes > > - Entries: 0 > > Full-Text Index > > - Not available > > Path Summary > > doc() 1x > > > info table > > Stopped at line 1, column 10: > > Syntax: INFO ([DATABASE|INDEX|STORAGE]) > > Show information on current database. > > Shows information on the currently opened database: > > - no argument: show global information > > - DATABASE: shows database information > > - INDEX: shows index information > > - STORAGE [start end] | [query]: show internal database table > > > info storage > > PRE DIS SIZ ATS NS KIND CONTENT > > ------------------------------------- > > 0 1 1 1 0 DOC song > > > info > > General Information > > Database Path: C:\Users\David\Documents\OpenSong\OpenSongSearch\BaseX\data > > Used Main Memory: 895 KB > > Database Creation > > Whitespace Chopping: ON > > Entity Parsing: OFF > > Indexes > > Path Summary: ON > > Text Index: ON > > Attribute Index: ON > > Full-Text Index: OFF > > > > > So, for 6.3.4, it creates from the command line, but it's empty. The > root of that path has NO files in it. The 5861 files are in all the > subdirectories. > > Then, when I try the same syntax with 6.1, I get the following results: > > > create db songs C:\Users\David\Documents\OpenSong\songpacks\Songs > > Stopped at line 1, column 17: > > Syntax: CREATE [DB|COLL|FS|INDEX|USER] [...] > > Create database, index or user. > > Creates a new database, index or user: > > - DB [path] [name?]: > > creates the database [name] for the file or directory [path] > > - COLL [name]: > > creates an empty database [name] > > - INDEX [TEXT|ATTRIBUTE|FULLTEXT|PATH]: > > creates the specified index > > - FS [path] [name] ([mountpoint] [backingstore]): > > creates filesystem database [name] for [path] > > (using [mountpoint] and [backingstore] if FUSE is activated) > > - USER [name] [password?]: > > creates the specified user > > Summary: > > 6.1 GUI -- works as I want it > > 6.1 command line -- can't get it to create > > 6.3.4 GUI -- creates but with only 14 entries > > 6.3.4 command line -- creates, but it's empty > > I don't mind staying with 6.1, but I DO want to do everything with the > command line. > > Is the ADD syntax the same, in that I can simply specify the path and > it will recurse the whole directory tree? > > Thanks!!! > > David > > DavidEmailSig <mailto:david@leighweb.com> > > *From:*Andreas Weiler [mailto:andreas.weiler@uni-konstanz.de] > *Sent:* Tuesday, December 28, 2010 01:29 PM > *To:* david(a)leighweb.com > *Cc:* basex-talk(a)mailman.uni-konstanz.de > *Subject:* Re: [basex-talk] Database creation from a directory tree > > Additional question: That was in the GUI. I'd like to do the same > thing using the command line with the .JAR file, but somewhere my > syntax is not right. Additionally, all the XML files have NO > EXTENSION so I don't know how to specify that either in the > command line. So, any help in that area would also be appreciated. > > > Could you provide your syntax? > > The syntax should be: > create db dbname path/to/folder > > The missing extension shouldnt be a problem, i just tested it. > > > In the GUI in 6.1, I can create a database from a whole directory > hierarchy of 5,681 individual XML files. When I point to the same > source file directory and using the 6.3.4 GUI (which I installed with > a pre-release version of the installer), it only processes 14 files. > No error message (that I saw) was produced. The behavior that I'm > wanting is what 6.1 does. Glad to help in solving the problem. > > We will check for modifications between the releases. As quick > workaround you could try to create an empty database and then add the > documents with the Database/Add documents menu. > > Kind regards, > Andreas > > Am 28.12.10 12:55, schrieb David leigh: > > I've just installed 6.3.4 and I've noticed a difference from version 6.1. > > In the GUI in 6.1, I can create a database from a whole directory > hierarchy of 5,681 individual XML files. When I point to the same > source file directory and using the 6.3.4 GUI (which I installed with > a pre-release version of the installer), it only processes 14 files. > No error message (that I saw) was produced. The behavior that I'm > wanting is what 6.1 does. Glad to help in solving the problem. > > Additional question: That was in the GUI. I'd like to do the same > thing using the command line with the .JAR file, but somewhere my > syntax is not right. Additionally, all the XML files have NO > EXTENSION so I don't know how to specify that either in the command > line. So, any help in that area would also be appreciated. > > Thank you, > > David > > DavidEmailSig <mailto:david@leighweb.com> > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de <mailto:BaseX-Talk@mailman.uni-konstanz.de> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

2 1

Database creation from a directory tree
by David leigh 28 Dec '10

28 Dec '10

I've just installed 6.3.4 and I've noticed a difference from version 6.1. In the GUI in 6.1, I can create a database from a whole directory hierarchy of 5,681 individual XML files. When I point to the same source file directory and using the 6.3.4 GUI (which I installed with a pre-release version of the installer), it only processes 14 files. No error message (that I saw) was produced. The behavior that I'm wanting is what 6.1 does. Glad to help in solving the problem. Additional question: That was in the GUI. I'd like to do the same thing using the command line with the .JAR file, but somewhere my syntax is not right. Additionally, all the XML files have NO EXTENSION so I don't know how to specify that either in the command line. So, any help in that area would also be appreciated. Thank you, David <mailto:david@leighweb.com> DavidEmailSig

2 1

Complete control from the command line
by David leigh 25 Dec '10

25 Dec '10

Hi, I'm new to basex and NOT a java person at all. I'm creating an application where a basex database is used to search for song lyrics. I'm using the autoit3 scripting language in Windows and interfacing to basex using the basex61.jar on the command line with the output file. I'm starting the server and then using the client for the queries. So far so good. The performance is amazing in both loading the database from thousands of individual XML documents and in querying the database. Quite pleased. My desire is to have an app where ALL the components reside in one directory or its subdirectories. I also want to be able to create and recreate the basex database programmatically using the CREATE on the command line. I understand that there exists a basex config file where certain parameters, like the data store location, can be set, but that the location of this file is assumed/fixed.i.e. I can't just put it anywhere I want it (like in the single directory I want this all to reside in). It seems that there are perhaps actual java commands that I could use when I launch the server and/or client that would allow me either to specify the location of the basex config file OR simply pass the parameters that I want to pass for specifying the location of the database. Unfortunately, none of the documentation on the site seems to address this level of interaction. Is there another source of more detailed documentation somewhere or could someone please guide me in this? I think it all boils down to how to tell basex, via the command line, where to put the data (for the server) and where it resides (for the client). Thank you very much! <mailto:david@leighweb.com> DavidEmailSig

2 1

Problems with XQuery matching records by compound key
by Jan Vlčinský (CAD) 25 Dec '10

25 Dec '10

Hi I am trying to match records from two different documents. Match key consists of two parts, so I simply build up a string from first and second match key and separate it by a hypen Here are source documents *Events.xml* <EVENTS>** * *<MSG id=*"{e8e85fdd-04d8-43ea-8090-e72337c3a30d}"* location=*"Prague"* event=*"Christmass"*/>** * *<MSG id=*"{7c94b2fc-b97c-4c09-b694-9993bfb2b93c}"* location=* "Bratislava"* event=*"Easterns"*/>** * *<MSG id=*"{bdbb11da-2c92-425d-9bf5-87c1e96e1bd4}"* location=*"Berlin"* event=*"Berlinale"*/>** </EVENTS>** * * *Reports.xml* <REPORTS>** * *<MSG id=*"{abcd}"* location=*"Prague"* event=*"Christmass"*/>** * *<MSG id=*"{efgh}"* location=*"Bratislava"* event=*"Easterns"*/>** * *<MSG id=*"{ijkl}"* location=*"Paris"* event=*"FassionFest"*/>** * *<MSG id=*"{mnqr}"* location=*"Moscow"* event=*"BalletFestival"*/>** </REPORTS> I am trying to find for each report all events, which are having the same location and event attributes So I build intermediate variables, containing pre-calculated keys And finally I try to find related events *match.xq* (:* *Construct* *matching* *keys* **for** *events* *:)** *let** **$event-msg-keys** *:=* * * *(*for** **$event** **in** */EVENTS/MSG** * **return** * * *<EVENT-MSG id="{$event/@id}" key="{concat($event/@location, '-', $event/@event)}"/>** * *)** * * (:* *Construct* *matching* *keys* **for** *reports* *:)** * * *let** **$report-msg-keys** *:=* * * *(*for** **$report** **in** */REPORTS/MSG* * * **return** * * *<REPORT-MSG id="{$report/@id}" key="{concat($report/@location, '-', $report/@event)}"/>** * *)** * * (:* *Find* *all* *events,* *related* **to** *each* *report:)** *for** **$report-msg** **in** **$report-msg-keys*** *let** **$report-msg-id** *:=* **$report-msg*/@id** *let** **$report-msg-key** *:=* **$report-msg*/@key** * * *let** **$related-events** *:=* * * *(*for** **$event-msg** **in** **$event-msg-keys*** * **where** **$event-msg*/@key* *=* **$report-msg-key*** * **return** **$event-msg*)** * * *return*** <REPORT-MSG id="{$report-msg-id}" key="{$report-msg-key}">** * *<RELATED-EVENTS>{$related-events}</RELATED-EVENTS></REPORT-MSG> However, resulting dokument does not show any related events <REPORT-MSG id=*"{abcd}"* key=*"Prague-Christmass"*>** * *<RELATED-EVENTS/>** </REPORT-MSG>** <REPORT-MSG id=*"{efgh}"* key=*"Bratislava-Easterns"*>** * *<RELATED-EVENTS/>** </REPORT-MSG>** <REPORT-MSG id=*"{ijkl}"* key=*"Paris-FassionFest"*>** * *<RELATED-EVENTS/>** </REPORT-MSG>** <REPORT-MSG id=*"{mnqr}"* key=*"Moscow-BalletFestival"*>** * *<RELATED-EVENTS/>** </REPORT-MSG> *Versions of BaseX* All the time I worked with GUI 1. I started testing on BaseX 6.3.4 2. I used a bit more complex data and query 3. After failing I tried on 6.3.1 4. There I was unable to open the collection with complains about Index out of bound Java error 5. Then I dropped the database and created it again. Trying to run the XQuery I got the same error. 6. returned back to 6.3.4 7. rewrote it to shorted sample, shown above I wonder, what I am doing wrong. Any advice? With best regards Jan Vlčinský

4 4

Use with Google App Engine
by Min Idzelis 20 Dec '10

20 Dec '10

I realize that BaseX can't be used as-is on the Google App Engine. The question I have is how much work would it be to get it up and running on the App Engine's datastore API instead of whatever you are using for indexing/storage. There is no access to a filesystem. The datastore does have a concept of hierarchical (parent-child) entities. In the datastore, the only operations that are supported are: get/put by key, inequality (<, >, =) filters, and a way to limit results that have a common ancestor (key). Would this be enough to support BaseX? Thanks, Min

4 8