June 2016 - BaseX-Talk - mailman.uni-konstanz.de

How to use REST API to add xml document to database
by Adil Hasan 14 Jun '16

14 Jun '16

Hello, Thank you for your work in creating this database management system! I have a question. I have an XML document that is created in an script. I would like to store the xml document in the xml database. The xml file is created in memory and I'm using python and the REST API to store the data. I can't figure out how to add an xml document to an existing database. I tried a POST request with the data being: <command xmlns="http://basex.org/rest"> <text>open test2; add to test21.xml <![CDATA[<greeting>hello3</greeting>]]></text> </command> That fails telling me I cannot do multiple commands at one time. So, I thought I could add 2 command statements with the POST request: <command xmlns="http://basex.org/rest"> <text>open test2</text> </command> <command xmlns="http://basex.org/rest"> <text>add to test21.xml <![CDATA[<greeting>hello3</greeting>]]></text> </command> But, that also doesn't work. Does anyone have a suggestion as to how to add an xml document to an existing database (ideally the xml document is not in a file)? many thanks for your help, adil

2 2

Re: [basex-talk] Nonstandard CSV in RESTXQ
by Christian Grün 13 Jun '16

13 Jun '16

Did you specify "text/comma-separated-values" as Content-Type? Am 12.06.2016 8:05 nachm. schrieb "Alexander Shpack" <shadowkin(a)gmail.com>: > Thank you, Christian! > > But it doesn't work as expected > > declare > %rest:path("/upload") > %rest:POST("{$csv}") > %input:csv("header=false,encoding=CP1251,separator=semicolon") > %output:method("html") > function page:uploadFile($csv as document-node()) > { > $csv > }; > > > [XPTY0004] Cannot cast xs:string to document-node(): "?????????? ?.... > > > On Sun, Jun 12, 2016 at 6:48 PM, Christian Grün <christian.gruen(a)gmail.com > > wrote: > >> Hi Alex, >> >> The following annotation should do what you want: >> >> %input:csv("header=true,encoding=CP1252") >> >> You were completely right, the existing documentation was a bit >> scrappy. I have slightly rewritten it (see [1] and outgoing links). >> >> Hope thie helps, >> Christian >> >> [1] http://docs.basex.org/wiki/RESTXQ#Input_options >> >> >> >> >> On Sun, Jun 12, 2016 at 4:03 PM, Alexander Shpack <shadowkin(a)gmail.com> >> wrote: >> > Hi, >> > >> > I have nonstandard CSV file with semicolon instead of comma as a >> separator >> > and cp1251 encoded. Documentation is not clear in this case, and the >> sample >> > doesn't work properly >> > >> > declare >> > %rest:path("/store.csv") >> > %rest:POST("{$csv}") >> > %input:csv("header=true") >> > function page:store-csv($csv as document-node()) >> > { >> > "Number of rows: " || count($csv/csv/record) >> > }; >> > >> > Even if I changed the encoding manually the next code does nothing >> > %input:csv("separator=semicolon") >> > and returns text instead of document-node() >> > >> > Could you check it? >> > >> > It would be nice if you add additional encoding parameter to csv module. >> > >> > Thanks! >> > > > > -- > s0rr0w >

3 16

Nonstandard CSV in RESTXQ
by Alexander Shpack 12 Jun '16

12 Jun '16

Hi, I have nonstandard CSV file with semicolon instead of comma as a separator and cp1251 encoded. Documentation is not clear in this case, and the sample doesn't work properly declare %rest:path("/store.csv") %rest:POST("{$csv}") %input:csv("header=true") function page:store-csv($csv as document-node()) { "Number of rows: " || count($csv/csv/record) }; Even if I changed the encoding manually the next code does nothing %input:csv("separator=semicolon") and returns text instead of document-node() Could you check it? It would be nice if you add additional encoding parameter to csv module. Thanks!

2 1

Re: [basex-talk] remotely access
by Henning Phan 11 Jun '16

11 Jun '16

hi,I failed to include the mail list the ports you gave me are different 9884 != 8984, could that be the problem? Date: Sat, 11 Jun 2016 20:46:02 +0000 From: med_khr(a)yahoo.fr To: henningphan(a)hotmail.com Subject: Re: [basex-talk] remotely access yes i have added a "NAT" in my router with 9884 portand the ip of server B is accessible outside from anywhere In the php file in servers Ai have wrote: $session = new Session("The_public_IP", 8984, "admin", "admin"); I'm wondering why it shows me this error. De : Henning Phan <henningphan(a)hotmail.com> À : Mohamed kharrat <med_khr(a)yahoo.fr> Envoyé le : Samedi 11 juin 2016 21h42 Objet : RE: [basex-talk] remotely access Hi,A common thing people miss is to ensure that the server B is remotely accessible. Is server B behind a router then you might have to do port forwarding. Another issue is that the client uses the wrong port.Hope that helpsDate: Sat, 11 Jun 2016 20:29:06 +0000From: med_khr(a)yahoo.frTo: christian.gruen(a)gmail.comCC: basex-talk(a)mailman.uni-konstanz.deSubject: Re: [basex-talk] remotely accessI have made a BaseX php client in a server Aand running BaseX server in server B in the php file containing the xquery query and the connection stringwhen i execute it, the browser shows me: "cannot communicate with server"Any idea? De : Christian Grün <christian.gruen(a)gmail.com> À : Mohamed kharrat <med_khr(a)yahoo.fr> Cc : BaseX Talk <basex-talk(a)mailman.uni-konstanz.de> Envoyé le : Samedi 11 juin 2016 20h29 Objet : Re: [basex-talk] remotely access > to execute remotely xquery queries, we have to run BaseX server and> BaseXclient in the same time?The BaseX server needs to be run on your remote machine, and you’llaccess it via your local BaseX client instance. But I’m pretty sure Igot you wrong? If yes, please try a bit harder ;)

2 1

remotely access
by Mohamed kharrat 11 Jun '16

11 Jun '16

Hi,to execute remotely xquery queries, we have to run BaseX server and BaseXclient in the same time? Best regards

2 2

Self-knowledge in xquery
by France Baril 10 Jun '16

10 Jun '16

When I'm in xquery and I need to reference something through the rest interface. Can I know from xquery my host/port or domain, or do I have to leave that information for myself somewhere in a config file. Case: - I keep an indexable version my .xsd in a DB so I can query it to provide guidance to users in the GUI. - Validation won't work with my .xsd because it won't resolve the includes if I just open the main .xsd file. - I keep a binary copy of my .xsd in a different DB so I can reference it though rest and therefore validate content, but I need to know my base url. So far I've been leaving that information in a config file that I query, but as I have multiple BaseX instances in implies that I enter my host:port in .basex and then again in my indexable config file. I'm trying to reduce duplication here. Thanks, France -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

3 3

Optimal size for large databases
by Lizzi, Vincent 09 Jun '16

09 Jun '16

I am importing a large (~3 million) set of XML documents to BaseX and am running into some problems as the databases grow beyond a few gigabytes. I am using BaseX 8.4.4, and have increased the memory available to BaseX to 12288m (12 Gb) using -Xmx (the machine has 20 Gb total). The documents are being stored in several databases that range in size from 300 Mb up to 25 Gb. My question is: what is an optimal maximum database size (in gigabytes)? I am hoping for some general advice. I appreciate that the answer can vary depending various factors. The problems that I've encountered with the larger databases are: 1. Running OPTIMIZE or OPTIMIZE ALL on the larger databases results in an out of memory error. I have switched to running CREATE INDEX to separately create text, attribute, token, and fulltext indexes, and found that creating these indexes separately produces fewer out of memory errors. I would like to be able to use OPTIMIZE ALL because over time some documents will be removed from the databases and the documentation indicates that optimize all will remove stale information from the indexes. 2. The command scripts that run CREATE INDEX or OPTIMIZE (ALL) seem to tie up the machine for a long time, maybe due to heavy disk access. 3. As the database grows in size the rate at which documents are added slows down. I have been measuring the number of documents imported, and observed rates over 100 documents per minute, and typical rates are around 60 - 30 documents per minute. As the database grows over a few gigabytes the speed slows to around 20 documents per minute. This is not much of a problem because when I see the rate slow down I can start a new database. Unfortunately I have been recording the number of documents, not the database size. In case this information is useful, my project is structured as follows: * There is 1 central index database which records for each document the BaseX database name and path where a document is stored, and some metadata that we use to identify or locate documents. * There are multiple content databases to store the actual documents. These content databases are organized by DTD and time period. * Each insert is done using the BaseX REST API. A BaseX HTTP server instance is running to receive documents, and a basex instance is running from the command line to locate and provide documents. Each insert is done by a POST that includes data and an updating query which adds (using db:replace) to the central index database and a document database in one "transaction". This helps to make the import resilient to duplicate documents, and to any problem that can prevent a single from document being added, and allows the process to continue if interrupted. I will probably need to re-organize the content databases so that each database is only a few gigabytes in size. Does anyone have advice on what would be a good maximum size for each database? Thanks, Vincent Vincent M. Lizzi - Electronic Production Manager Taylor & Francis Group 530 Walnut St., Suite 850, Philadelphia, PA 19106 E-Mail: vincent.lizzi(a)taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Phone: 215-606-4221 Fax: 215-207-0047 Web: http://www.tandfonline.com/ Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler."

2 3

Balisage 2016
by C. M. Sperberg-McQueen 03 Jun '16

03 Jun '16

Readers of this list may be interested in the Balisage conference, which focuses on applications of markup. This year's program includes several talks on projects which use XQuery (often in the form of BaseX). I hope to see other XQuery enthusiasts at Balisage this year! Balisage: The Markup Conference 2016 Program Now Available http://www.balisage.net/2016/Program.html Balisage: where serious markup practitioners and theoreticians meet every August. The 2016 program includes papers discussing reducing ambiguity in linked-open-data annotations, the visualization of XSLT execution patterns, automatic recognition of grant- and funding-related information in scientific papers, construction of an interactive interface to assist cybersecurity analysts, rules for graceful extension and customization of standard vocabularies, case studies of agile schema development, a report on XML encoding of subtitles for video, an extension of XPath to file systems, handling soft hyphens in historical texts, an automated validity checker for formatted pages, one no-angle-brackets editing interface for scholars of German family names and another for scholars of Roman legal history, and a survey of non-XML markup such as Markdown. XML In, Web Out: A one-day Symposium on the sub rosa XML that powers an increasing number of websites will be held on Monday, August 1. http://balisage.net/XML-In-Web-Out/ If you are interested in open information, reusable documents, and vendor and application independence, then you need descriptive markup, and Balisage is the conference you should attend. Balisage brings together document architects, librarians, archivists, computer scientists, XML practitioners, XSLT and XQuery programmers, implementers of XSLT and XQuery engines and other markup-related software, Topic-Map enthusiasts, semantic-Web evangelists, standards developers, academics, industrial researchers, government and NGO staff, industrial developers, practitioners, consultants, and the world's greatest concentration of markup theorists. Some participants are busy designing replacements for XML while other still use SGML (and know why they do). Discussion is open, candid, and unashamedly technical. Balisage 2016 Program: http://www.balisage.net/2016/Program.html Symposium Program: http://balisage.net/XML-In-Web-Out/symposiumProgram.html ====================================================================== Balisage: The Markup Conference 2016 mailto:info@balisage.net August 2-5, 2016 http://www.balisage.net Preconference Symposium: August 1, 2016 +1 301 315 9631 ====================================================================== -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

1 0

Re: [basex-talk] Bug report in handling fn:format-number in basex 8.4.4
by Michael Stoop 03 Jun '16

03 Jun '16

Good morning, First of all thank you for your great work on basex. I've been phasing out Zorba in favour of BaseX for excellent large file support and all around better documentation and support. However, I'm running into some issues in formatting numbers in basex 8.4.4 using fn:format-number (using http://www.w3schools.com/xsl/func_formatnumber.asp) To describe this behavior I wrote a simple xquery: let $result := <rows> <row input="0.3576" format="0.##" expected="0.36">{fn:format-number(0.3576, "0.##")}</row> <row input="-0.3576" format="0.##" expected="-0.36">{fn:format-number(-0.3576, "0.##")}</row> <row input="-0.3576" format="0.###" expected="-0.358">{fn:format-number(-0.3576, "0.###")}</row> <row input="-0.3576" format="0.####" expected="-0.3576">{fn:format-number(-0.3576, "0.####")}</row> <row input="-0.3576" format="0.00" expected="-0.36">{fn:format-number(-0.3576, "0.00")}</row> <row input="0.000" format="0.##" expected="0.00">{fn:format-number(0.000, "0.##")}</row> <row input="0" format="0.00" expected="0.00">{fn:format-number(0, "0.00")}</row> <row input="0.3" format="0.####" expected="0.3">{fn:format-number(0.3, "0.####")}</row> <row input="0.3" format="0.0000" expected="0.3000">{fn:format-number(0.3, "0.0000")}</row> <row input="17" format="0.##" expected="17">{fn:format-number(17, "0.##")}</row> <row input="17" format="0.00" expected="17.00">{fn:format-number(17, "0.00")}</row> <row input="17.4555" format="0.##" expected="17.46">{fn:format-number(17.4555, "0.##")}</row> <row input="-17.4555" format="0.##" expected="-17.46">{fn:format-number(-17.4555, "0.##")}</row> </rows> return $result The result of this is as follows: <rows> <row input="0.3576" format="0.##" expected="0.36">.36</row> <row input="-0.3576" format="0.##" expected="-0.36">-.36</row> <row input="-0.3576" format="0.###" expected="-0.358">-.358</row> <row input="-0.3576" format="0.####" expected="-0.3576">-.3576</row> <row input="-0.3576" format="0.00" expected="-0.36">-.36</row> <row input="0.000" format="0.##" expected="0.00">0</row> <row input="0" format="0.00" expected="0.00">0.00</row> <row input="0.3" format="0.####" expected="0.3">.3</row> <row input="0.3" format="0.0000" expected="0.3000">.3000</row> <row input="17" format="0.##" expected="17">17</row> <row input="17" format="0.00" expected="17.00">17.00</row> <row input="17.4555" format="0.##" expected="17.46">17.46</row> <row input="-17.4555" format="0.##" expected="-17.46">-17.46</row> </rows> Unfortunately two things seem to go wrong here: · Zero in the picture string is ignored for negative numbers. · The zero before decimal is ignored for amounts between zero and one (but does occur for 0.0000 for example). I hope someone can take a look at this and provide a fix. I tried messing around with string replace functions where I would just replace -. with -0. and prefix the result for a decimal point with a zero, but this really isn't present and requires a workaround for something that is very common in my code. Using the latest basex (8.4.4) on Windows Server 2012 under Java 8 (64 bit, last update just now: update 91). Thanks in advance and best regards, Michaël Stoop

2 2

Replacing documents in remote database
by Asxetos 02 Jun '16

02 Jun '16

Hey all, I am trying to connect to a remote database and replace its files. I am using db:replace() inside client:query(), but I don't get how I am supposed to pass $input as the new data for the file in the database. Here a quick code snippet I put together: http://sprunge.us/jdaZ ( or http://sprunge.us/OZHj?xquery ) If there's a better method for doing what I am trying, I'm fine with replacing my whole method, even if that is using some sort of REST API and http requests. Just give me some pointers ;) Thanks!

2 4