October 2014 - BaseX-Talk - mailman.uni-konstanz.de

JSON results
by Thomas Daly 27 Oct '14

27 Oct '14

Hello, I'm trying to construct a query that will return the title and ID from a set of records, but instead of XML results I'd prefer JSON. I'd like the results to be in the following format: [ ['id': 'value1', 'title': 'Title1'], ['id': 'value2', 'title': 'Title2'], ['id': 'value3', 'title': 'Title3'] ] I understand I'd need to use the JSON module, but I'm having trouble getting the results into the right format. This is what I have so far: XQUERY for $poem in collection("lk") let $id := fn:string($poem/poem/@uuid) let $title := $poem/poem/title/text() order by $title return (<id>{$id}</id>, <title>{$title}</title>) This returns the XML, but does anyone know how to convert into the desired JSON format? Many thanks

2 1

Cursor keys
by Thomas Daly 27 Oct '14

27 Oct '14

Hello, I'm using the command line interface of BaseX 7.3. I notice the cursor keys don't behave in the normal way - instead of moving the cursor around they print strange characters. Is there something wrong with my terminal configuration or is this a shortcoming of the software?

2 1

Reg : Group by with the top Records
by Chandrasekhar 25 Oct '14

25 Oct '14

Dear Team I am using the group by for getting top n number of record like below mentioned query (for $areaData in collection('mycollection') let $areaName := $areaData/Contry group by $areaName return $areaData )[position() lt 100] In this case i am getting first top 100 records and then it is grouping on those records.Can you please suggest me how can we get the top 100 records based on the Country name Please find the attachment for the sample data.Please do the need full on this/ Thanks&Regards Chandra Sekhar The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited.

2 1

XML Oxygen search: 404 Not found when the file exists, or worse 500 error.
by France Baril 24 Oct '14

24 Oct '14

Hi, We've been having this persistent issue with our webDAV connection to XML Oxygen. How to reproduce: 1. Create a DB with 200/300 documents. 2. Create a webDAV connection to the DB within an XML Oxygen project. 3. Right click the DB from the project > Find/Replace in files, and type a keyword to search for for all files *.xml 4. Run the search 3 to 5 times. Until you notice the error. The error doesn't always occur, but I'd be surprised if you didn't see it after 3 to 5 try. Example 500 error: System ID: http://admin@localhost :8984/webdav/en-us/topics/widget-bookmark.xml Severity: error Description: 500 Internal Server Error for: http://localhost:8984/webdav/en-us/topics/widget-bookmark.xml In the case of a 500 error, you need to restart the server to be able to open or search files from the DB again. Can anything be done to improve the situation? Regards, France -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 5

Re: [basex-talk] serializing an xquery flwor in csv
by Dirk Kirsten 23 Oct '14

23 Oct '14

Hi Emmanuelle, please don't forget to include the mailing list in your mails, so others can also benefit from the questions (and other people might answer you faster or better than I can). The CSV serialization works just on the element name as far as I can remember. You could simply rename the node, e.g. let $destinataire := $creation/persName[@type='destinataire'] update rename node . as "pers1" For help with XQuery Update, oyu might also want to take a look at the docs: http://docs.basex.org/wiki/Update Cheers, Dirk On 10/23/2014 03:12 PM, Emmanuelle Morlock wrote: > Hi again > there's something strange happening with the csv serialization of my query > > Two of my variables have the same element but are distinguished by their > attribute > > declare option output:method "csv"; > declare option output:csv "header=yes, separator=semicolon"; > for $lettre in db:open('myBD')//TEI > where $lettre//opener//addrLine[not(contains(.,'Adresse ?'))] > let $creation := $lettre//profileDesc/creation > let $titre := $lettre//teiHeader//title > let $date := $lettre//profileDesc//date > let $destinataire := $creation/persName[@type='destinataire'] > let $expediteur := $creation/persName[@type='expediteur'] > return > <csv> > <record>data({$date}) data({$expediteur} data({$titre} > data({$destinataire}) </record> > </csv> > > the result creates one field 'persName' with two values (separated by a > comma), whereas I wanted 2 fields > > date;persName;title > 01/01/2000;Pierre Martin,Jean Dupond;Title > > what shoud I do to avoid that and get as a result something like : > > date;persName1;title:persName2 > 01/01/2000;Pierre Martin;Title;Jean Dupond > > > thanks > emmanuelle > > > Le 23/10/2014 14:31, Dirk Kirsten a écrit : > > Hello Emmanuelle, > > > > You're very welcome to ask, even if its basic (or looks basic to you). > > We all started with XQuery at one point in time... > > > > In this case, it is even simpler than you expected I guess. As you > > want to serialize the output as csv, you don't even need the > > serialize() function. You can simply set in the prolog of your XQuery > > > > |declare| |option| |output:method ||"csv"||; > > > > | > > and your output will be automatically converted from XML to CSV. There > > is an example for CSV at ||http://docs.basex.org/wiki/Serialization, > > you can also define options in the prolog. > > > > Also, I noticed that you do $lettre//profileDesc/creation quite often. > > // is > > a quite costly operation, so you might want to put this into a > > separate variable which you then use - The optimizer might do that for > > you, but it is also shorter and cleaner to write. > > > > Cheers, > > Dirk > > > > On Thu, Oct 23, 2014 at 12:24 PM, Emmanuelle Morlock > > <emmanuelle.morlock(a)mom.fr <mailto:emmanuelle.morlock@mom.fr>> wrote: > > > > Hi all, > > > > I'm trying to do some xquery with the GUI in a standalone install. > > As I'm beginning with xquery, I rely a lot on the examples given > > in the doc. > > But I don't manage to be successful with > > http://docs.basex.org/wiki/CSV_Module for what I need. > > > > I'd like to get a result formatted in csv (separator : ';' or > > whatever) so tha I can directly import in a spreadsheet the > > results exported file. > > > > Here's the begining of my query : > > > > for $lettre in db:open('myDB')//TEI > > where $lettre//opener//addrLine[not(contains(.,'Adresse ?'))] > > let $title := $lettre//teiHeader//title > > let $destinataire := > > $lettre//profileDesc/creation/persName[@type='destinataire'] > > let $expediteur := > > $lettre//profileDesc/creation/persName[@type='expediteur'] > > let $date := $lettre//profileDesc//date > > let $locName := > > $lettre//profileDesc/creation/location/placeName[@type='PlaceExp'] > > let $locGeo := $lettre//profileDesc/creation/location/geo > > > > then I guess I should have something like : > > - a variable for the option... is let $options := > > {'separator':';'} correct ? > > something to put all the variable as a result let $fields := ... ? > > then > > return csv:serialize ($fields, $options) > > > > ??? > > thanks for helping even if this sounds too basic ! > > > > best > > > > emmanuelle > > > > -- Emmanuelle Morlock > > IGE CNRS - soutien aux projets d'édition TEI - humanités numériques > > UMR 5189 HISoMA > > http://www.hisoma.mom.fr/ > > 04 72 71 58 40 > > 06 85 84 69 16 > > @emma_morlock > > > > > > > > > > -- > > Dirk Kirsten, BaseX GmbH, http://basex.org <http://basex.org/> > > |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > > |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: > > | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle > > `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22 > > -- Dirk Kirsten, BaseX GmbH, http://basex.org |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22

1 0

serializing an xquery flwor in csv
by Emmanuelle Morlock 23 Oct '14

23 Oct '14

Hi all, I'm trying to do some xquery with the GUI in a standalone install. As I'm beginning with xquery, I rely a lot on the examples given in the doc. But I don't manage to be successful with http://docs.basex.org/wiki/CSV_Module for what I need. I'd like to get a result formatted in csv (separator : ';' or whatever) so tha I can directly import in a spreadsheet the results exported file. Here's the begining of my query : for $lettre in db:open('myDB')//TEI where $lettre//opener//addrLine[not(contains(.,'Adresse ?'))] let $title := $lettre//teiHeader//title let $destinataire := $lettre//profileDesc/creation/persName[@type='destinataire'] let $expediteur := $lettre//profileDesc/creation/persName[@type='expediteur'] let $date := $lettre//profileDesc//date let $locName := $lettre//profileDesc/creation/location/placeName[@type='PlaceExp'] let $locGeo := $lettre//profileDesc/creation/location/geo then I guess I should have something like : - a variable for the option... is let $options := {'separator':';'} correct ? something to put all the variable as a result let $fields := ... ? then return csv:serialize ($fields, $options) ??? thanks for helping even if this sounds too basic ! best emmanuelle -- Emmanuelle Morlock IGE CNRS - soutien aux projets d'édition TEI - humanités numériques UMR 5189 HISoMA http://www.hisoma.mom.fr/ 04 72 71 58 40 06 85 84 69 16 @emma_morlock

2 1

Re: [basex-talk] Architecture Question
by Christian Grün 23 Oct '14

23 Oct '14

Hi Mansi, Good to know! Keep us updated. Christian On Wed, Oct 22, 2014 at 8:13 PM, Mansi Sheth <mansi.sheth(a)gmail.com> wrote: > Christian, > > Actually, I am all set. I will just query using python or Angular or > whatever and do any data manipulation there and use XQUERY just to query and > no further processing. > > Btw, some initial very informal statistics: > > Took, 22 min to import ~2.1k documents with indexing as well. > ~2 min for query to return back. > > Impressive !!! > > Machine specs: 16 GB RAM, 2.7 GHz, i7 processor MacBook Pro. > > I am waiting on my colleague to get me some more production data, which > gives me access to some 10k XML files. Will keep you posted. > > - Mansi > > On Wed, Oct 22, 2014 at 12:04 PM, Mansi Sheth <mansi.sheth(a)gmail.com> wrote: >> >> Christian, >> >> Thanks for all your responses. It truly helps a lot. >> >> re: Importing data into databases: I realized, for the extent of this POC, >> I will just count no of docs in each database (currently programmed to be >> 50) and keep creating new databases. Structure of data is same, but its >> nested in nature. Like a folder can have folder, which can have file etc. >> Usually, it won't be more than 4 levels deep. Thats a good tip, to guess no >> of nodes based on byte size. I guess, for time being I will move on, with >> just storing 50 docs per DB. >> >> re: terabytes of data. Well, I am planning on using ~6 months worth of >> data for any analysis and discarding data prior to that (leaving it around >> in backups). Obviously, would be going some cloud route for such resources, >> will see how much budget I can manage to get :) Am very positive about this. >> So, no its not only a theoretical assumption as far as I can see. >> >> re: Currently, I am looking into querying these databases. I am exploring >> REST for it. From documentation, it seems our only option is supporting >> these queries (on server side) using XQUERY or RestXQ, no Java/Python ? I am >> well versed with XPATH and XSLT, gearing up towards XQUERY now. But, would >> be a little easier (just my personal preference :)) to manipulate data in >> Java/Python before serving it back to client. Is there any such facility ? >> Something like: >> >> "http://localhost:8984/rest?run=getData.java" >> >> similarly for python ? >> >> - Mansi >> >> Some preliminary statistics: Imported 2050 XML documents in 22 min >> (including indexing on attributes). >> >> On Sun, Oct 19, 2014 at 6:14 PM, Christian Grün >> <christian.gruen(a)gmail.com> wrote: >>> >>> Hi Mansi, >>> >>> > Is there some book/resource you can point me to, which helps better >>> > visualize NXD ? >>> >>> sorry for letting you wait. If you want to know more about native XML >>> databases, I recommend you to have a closer look at various articles >>> in our Wiki (e. g. [1,2]). It will also be helpful if you get into the >>> basics of XQuery [3]. >>> >>> Have you tried to realize some of the hints I gave in my previous mails? >>> >>> > I am trying to distribute data across multiple databases. I can't >>> > distribute >>> > based on day, as there could very well be situation, where single day's >>> > data >>> > could more than capacity of BaseX DB. >>> >>> If 2 billion XML nodes per day are not enough, you will probably need >>> to create more than one database per day. Via the "info db" command, >>> you see how many nodes are currently stored in a database, but there >>> is no cheap solution to find out the number of nodes of an incoming >>> document, because XML documents can be very heterogeneous. Some >>> questions back: >>> >>> * Do you have some more information on the data you want to store? >>> * Are all documents similar or do they vary greatly? If the documents >>> are somewhat similar, you can usually estimate the number of nodes by >>> looking at the byte size. >>> * Do you know that you will really need to store lots of terabytes of >>> XML data, or it is more like a theoretical assumption? >>> >>> Christian >>> >>> [1] http://docs.basex.org/wiki/Database >>> [2] http://docs.basex.org/wiki/Table_of_Contents >>> [3] http://docs.basex.org/wiki/Xquery >> >> >> >> >> -- >> - Mansi > > > > > -- > - Mansi

1 0

locking problem
by Lizzi, Vincent 23 Oct '14

23 Oct '14

This is an update, and further problem, on a problem I posted here a few months ago. I've written a program to import XML documents into BaseX in one database, and populate a second database with a small amount of metadata about each document. This works but crashes every so often, and I have to restore the databases from backup and restart the import. To get through a large set of documents I now have the program create a backup (using the CREATE BACKUP command) after every 1000 documents imported. Since my earlier post, I found that the connection between the client and server was occasionally being dropped. I've switched to opening and closing a connection for each document instead of using one long-running connection. The client and server are running on the same machine. I'm using the latest version of BaseX (BaseX80-20141013.213520.zip). Here is the error message reported by my program when it crashes. The error occurs on the Query.execute() method of org.basex.examples.api.BaseXClient.java. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.0 beta a91f5c2 Java: Oracle Corporation, 1.7.0_67 OS: Windows 7, amd64 Stack Trace: java.nio.channels.OverlappingFileLockException at sun.nio.ch.SharedFileLockTable.checkList(Unknown Source) at sun.nio.ch.SharedFileLockTable.add(Unknown Source) at sun.nio.ch.FileChannelImpl.tryLock(Unknown Source) at org.basex.io.random.TableDiskAccess.lock(TableDiskAccess.java:139) at org.basex.io.random.TableDiskAccess.<init>(TableDiskAccess.java:87) at org.basex.data.DiskData.init(DiskData.java:119) at org.basex.data.DiskData.<init>(DiskData.java:80) at org.basex.core.cmd.Open.open(Open.java:70) at org.basex.core.cmd.Open.run(Open.java:36) at org.basex.core.Command.run(Command.java:360) at org.basex.core.Command.execute(Command.java:94) at org.basex.server.ClientListener.run(ClientListener.java:145) The server instance reports the error with this stack trace: C:\Programs\basex\basex\bin>basexhttp BaseX 8.0 beta a91f5c2 [Server] Server was started (port: 1984) HTTP Server was started (port: 8984) java.io.FileNotFoundException: C:\Programs\basex\basex\data\article_xml_meta\doc.basex (The requested operation cannot be performed on a file with a user-mapped section open ) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(Unknown Source) at java.io.FileOutputStream.<init>(Unknown Source) at org.basex.io.out.DataOutput.<init>(DataOutput.java:47) at org.basex.io.out.DataOutput.<init>(DataOutput.java:36) at org.basex.index.resource.Docs.write(Docs.java:73) at org.basex.index.resource.Resources.write(Resources.java:51) at org.basex.data.DiskData.write(DiskData.java:141) at org.basex.data.DiskData.flush(DiskData.java:249) at org.basex.data.DiskData.finishUpdate(DiskData.java:239) at org.basex.query.up.ContextModifier.apply(ContextModifier.java:124) at org.basex.query.up.Updates.apply(Updates.java:129) at org.basex.query.QueryContext.iter(QueryContext.java:351) at org.basex.query.QueryProcessor.iter(QueryProcessor.java:80) at org.basex.server.ServerQuery.execute(ServerQuery.java:128) at org.basex.server.ClientListener.query(ClientListener.java:492) at org.basex.server.ClientListener.run(ClientListener.java:109) Here is the update statement where the error occurs, along with a bit of the surrounding Java. BaseXClient.Query queryIndex = client.query("declare variable $indexDb external; declare variable $hash external; declare variable $database external; declare variable $path external; let $p := <properties><database>{$database}</database><path>{$path}</path><sha1>{$hash}</sha1><loadedDate>{current-dateTime()}</loadedDate></properties> return db:replace('" + indexDatabaseName + "', $path, $p)"); try { queryIndex.bind("$indexDb", indexDatabaseName); queryIndex.bind("$database", managedDatabaseName); queryIndex.bind("$path", path); queryIndex.bind("$hash", hash); queryIndex.execute(); } finally { queryIndex.close(); } Previously this code used a variable for the first parameter of db:replace() but I changed it to "hard code" the database name into the query after reading about transactions and locking in the wiki. This change doesn't appear to have helped. To put this in context, the above update is run after the XML document is imported using BaseXClient.replace(). The next thing that happens in the program is an XQuery db:replace to add more properties for the document to the metadata database. I've added a Thread.sleep(300), to create a small delay between loading each document and this has helped the program to import more documents before crashing. Adjusting the delay doesn't seem to completely stop the crashes though. I'm not sure if this is a bug or improper use. Any help would be appreciated. Thanks, Vincent

2 2

Architecture Question
by Mansi Sheth 22 Oct '14

22 Oct '14

Hello, I have been going thru and comparing different Native XML Databases and so far I am liking BaseX. However, there are still a few questions unanswered, before I make a final choice: 1. I have 1000s of XML files (each between 50MB-400MB) and this is going to grow exponentially (~200 / per day). So, my question is how scalable is BaseX ? Can I configure it to use data from my external HDD, in my initial prototype ? 2. I plan to heavily use XPATH, for data retrieval. Does BaseX, use any multi-processing, multi-threading to speed up search ? Any concurrent processing ? 3. Can I do some post-processing on searched and retrieved data ? Like sorting, unique elements etc ? - Mansi

2 11

Using TcpCatcher as a proxy with Basex Client
by Arto Viitanen 20 Oct '14

20 Oct '14

I am debugging a Basex client built in C#. The client is based on the Git Hub example https://github.com/BaseXdb/basex/tree/master/basex-api/src/main/c%23. The problem is, every now and then the client drops the session. It might be, that the client drops the connection if the server takes too much time to answer (or something). So I need some way to emulate slow connection or connection dropping packages. I downloaded TcpCatcher (http://www.tcpcatcher.org/) and set it as a proxy (client talks to 21984; TcpCatcher catches packages and send them to BaseX listening to 11984). But It seems that BaseX does not respond. I took BasexClient.cs and Example.cs from the github and tried with them. I get three packages; one coming from the server and two going to client; but after that nothing. The Example prints "Access Denied". I guess this comes from: socket = new TcpClient(host, port); stream = socket.GetStream(); ehost = host; string ts = Receive(); Send(username); Send(MD5(MD5(pw) + ts)); if (stream.ReadByte() != 0) { throw new IOException("Access denied."); } that is: client gets ts (timestamp?), sends username and password with timestamp; but receives nothing. So how can I use TcpCatcher (or other proxy program) with Basex client? (Basex 7.7; windows 7) Arto Viitanen Microteam Oy

3 3