- BaseX-Talk - mailman.uni-konstanz.de

Nonstandard CSV in RESTXQ
by Alexander Shpack 12 Jun '16

12 Jun '16

Hi, I have nonstandard CSV file with semicolon instead of comma as a separator and cp1251 encoded. Documentation is not clear in this case, and the sample doesn't work properly declare %rest:path("/store.csv") %rest:POST("{$csv}") %input:csv("header=true") function page:store-csv($csv as document-node()) { "Number of rows: " || count($csv/csv/record) }; Even if I changed the encoding manually the next code does nothing %input:csv("separator=semicolon") and returns text instead of document-node() Could you check it? It would be nice if you add additional encoding parameter to csv module. Thanks!

2 1

Re: [basex-talk] remotely access
by Henning Phan 11 Jun '16

11 Jun '16

hi,I failed to include the mail list the ports you gave me are different 9884 != 8984, could that be the problem? Date: Sat, 11 Jun 2016 20:46:02 +0000 From: med_khr(a)yahoo.fr To: henningphan(a)hotmail.com Subject: Re: [basex-talk] remotely access yes i have added a "NAT" in my router with 9884 portand the ip of server B is accessible outside from anywhere In the php file in servers Ai have wrote: $session = new Session("The_public_IP", 8984, "admin", "admin"); I'm wondering why it shows me this error. De : Henning Phan <henningphan(a)hotmail.com> À : Mohamed kharrat <med_khr(a)yahoo.fr> Envoyé le : Samedi 11 juin 2016 21h42 Objet : RE: [basex-talk] remotely access Hi,A common thing people miss is to ensure that the server B is remotely accessible. Is server B behind a router then you might have to do port forwarding. Another issue is that the client uses the wrong port.Hope that helpsDate: Sat, 11 Jun 2016 20:29:06 +0000From: med_khr(a)yahoo.frTo: christian.gruen(a)gmail.comCC: basex-talk(a)mailman.uni-konstanz.deSubject: Re: [basex-talk] remotely accessI have made a BaseX php client in a server Aand running BaseX server in server B in the php file containing the xquery query and the connection stringwhen i execute it, the browser shows me: "cannot communicate with server"Any idea? De : Christian Grün <christian.gruen(a)gmail.com> À : Mohamed kharrat <med_khr(a)yahoo.fr> Cc : BaseX Talk <basex-talk(a)mailman.uni-konstanz.de> Envoyé le : Samedi 11 juin 2016 20h29 Objet : Re: [basex-talk] remotely access > to execute remotely xquery queries, we have to run BaseX server and> BaseXclient in the same time?The BaseX server needs to be run on your remote machine, and you’llaccess it via your local BaseX client instance. But I’m pretty sure Igot you wrong? If yes, please try a bit harder ;)

2 1

remotely access
by Mohamed kharrat 11 Jun '16

11 Jun '16

Hi,to execute remotely xquery queries, we have to run BaseX server and BaseXclient in the same time? Best regards

2 2

Self-knowledge in xquery
by France Baril 11 Jun '16

11 Jun '16

When I'm in xquery and I need to reference something through the rest interface. Can I know from xquery my host/port or domain, or do I have to leave that information for myself somewhere in a config file. Case: - I keep an indexable version my .xsd in a DB so I can query it to provide guidance to users in the GUI. - Validation won't work with my .xsd because it won't resolve the includes if I just open the main .xsd file. - I keep a binary copy of my .xsd in a different DB so I can reference it though rest and therefore validate content, but I need to know my base url. So far I've been leaving that information in a config file that I query, but as I have multiple BaseX instances in implies that I enter my host:port in .basex and then again in my indexable config file. I'm trying to reduce duplication here. Thanks, France -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

3 3

Optimal size for large databases
by Lizzi, Vincent 09 Jun '16

09 Jun '16

I am importing a large (~3 million) set of XML documents to BaseX and am running into some problems as the databases grow beyond a few gigabytes. I am using BaseX 8.4.4, and have increased the memory available to BaseX to 12288m (12 Gb) using -Xmx (the machine has 20 Gb total). The documents are being stored in several databases that range in size from 300 Mb up to 25 Gb. My question is: what is an optimal maximum database size (in gigabytes)? I am hoping for some general advice. I appreciate that the answer can vary depending various factors. The problems that I've encountered with the larger databases are: 1. Running OPTIMIZE or OPTIMIZE ALL on the larger databases results in an out of memory error. I have switched to running CREATE INDEX to separately create text, attribute, token, and fulltext indexes, and found that creating these indexes separately produces fewer out of memory errors. I would like to be able to use OPTIMIZE ALL because over time some documents will be removed from the databases and the documentation indicates that optimize all will remove stale information from the indexes. 2. The command scripts that run CREATE INDEX or OPTIMIZE (ALL) seem to tie up the machine for a long time, maybe due to heavy disk access. 3. As the database grows in size the rate at which documents are added slows down. I have been measuring the number of documents imported, and observed rates over 100 documents per minute, and typical rates are around 60 - 30 documents per minute. As the database grows over a few gigabytes the speed slows to around 20 documents per minute. This is not much of a problem because when I see the rate slow down I can start a new database. Unfortunately I have been recording the number of documents, not the database size. In case this information is useful, my project is structured as follows: * There is 1 central index database which records for each document the BaseX database name and path where a document is stored, and some metadata that we use to identify or locate documents. * There are multiple content databases to store the actual documents. These content databases are organized by DTD and time period. * Each insert is done using the BaseX REST API. A BaseX HTTP server instance is running to receive documents, and a basex instance is running from the command line to locate and provide documents. Each insert is done by a POST that includes data and an updating query which adds (using db:replace) to the central index database and a document database in one "transaction". This helps to make the import resilient to duplicate documents, and to any problem that can prevent a single from document being added, and allows the process to continue if interrupted. I will probably need to re-organize the content databases so that each database is only a few gigabytes in size. Does anyone have advice on what would be a good maximum size for each database? Thanks, Vincent Vincent M. Lizzi - Electronic Production Manager Taylor & Francis Group 530 Walnut St., Suite 850, Philadelphia, PA 19106 E-Mail: vincent.lizzi(a)taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Phone: 215-606-4221 Fax: 215-207-0047 Web: http://www.tandfonline.com/ Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler."

2 3

Balisage 2016
by C. M. Sperberg-McQueen 03 Jun '16

03 Jun '16

Readers of this list may be interested in the Balisage conference, which focuses on applications of markup. This year's program includes several talks on projects which use XQuery (often in the form of BaseX). I hope to see other XQuery enthusiasts at Balisage this year! Balisage: The Markup Conference 2016 Program Now Available http://www.balisage.net/2016/Program.html Balisage: where serious markup practitioners and theoreticians meet every August. The 2016 program includes papers discussing reducing ambiguity in linked-open-data annotations, the visualization of XSLT execution patterns, automatic recognition of grant- and funding-related information in scientific papers, construction of an interactive interface to assist cybersecurity analysts, rules for graceful extension and customization of standard vocabularies, case studies of agile schema development, a report on XML encoding of subtitles for video, an extension of XPath to file systems, handling soft hyphens in historical texts, an automated validity checker for formatted pages, one no-angle-brackets editing interface for scholars of German family names and another for scholars of Roman legal history, and a survey of non-XML markup such as Markdown. XML In, Web Out: A one-day Symposium on the sub rosa XML that powers an increasing number of websites will be held on Monday, August 1. http://balisage.net/XML-In-Web-Out/ If you are interested in open information, reusable documents, and vendor and application independence, then you need descriptive markup, and Balisage is the conference you should attend. Balisage brings together document architects, librarians, archivists, computer scientists, XML practitioners, XSLT and XQuery programmers, implementers of XSLT and XQuery engines and other markup-related software, Topic-Map enthusiasts, semantic-Web evangelists, standards developers, academics, industrial researchers, government and NGO staff, industrial developers, practitioners, consultants, and the world's greatest concentration of markup theorists. Some participants are busy designing replacements for XML while other still use SGML (and know why they do). Discussion is open, candid, and unashamedly technical. Balisage 2016 Program: http://www.balisage.net/2016/Program.html Symposium Program: http://balisage.net/XML-In-Web-Out/symposiumProgram.html ====================================================================== Balisage: The Markup Conference 2016 mailto:info@balisage.net August 2-5, 2016 http://www.balisage.net Preconference Symposium: August 1, 2016 +1 301 315 9631 ====================================================================== -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

1 0

Re: [basex-talk] Bug report in handling fn:format-number in basex 8.4.4
by Michael Stoop 03 Jun '16

03 Jun '16

Good morning, First of all thank you for your great work on basex. I've been phasing out Zorba in favour of BaseX for excellent large file support and all around better documentation and support. However, I'm running into some issues in formatting numbers in basex 8.4.4 using fn:format-number (using http://www.w3schools.com/xsl/func_formatnumber.asp) To describe this behavior I wrote a simple xquery: let $result := <rows> <row input="0.3576" format="0.##" expected="0.36">{fn:format-number(0.3576, "0.##")}</row> <row input="-0.3576" format="0.##" expected="-0.36">{fn:format-number(-0.3576, "0.##")}</row> <row input="-0.3576" format="0.###" expected="-0.358">{fn:format-number(-0.3576, "0.###")}</row> <row input="-0.3576" format="0.####" expected="-0.3576">{fn:format-number(-0.3576, "0.####")}</row> <row input="-0.3576" format="0.00" expected="-0.36">{fn:format-number(-0.3576, "0.00")}</row> <row input="0.000" format="0.##" expected="0.00">{fn:format-number(0.000, "0.##")}</row> <row input="0" format="0.00" expected="0.00">{fn:format-number(0, "0.00")}</row> <row input="0.3" format="0.####" expected="0.3">{fn:format-number(0.3, "0.####")}</row> <row input="0.3" format="0.0000" expected="0.3000">{fn:format-number(0.3, "0.0000")}</row> <row input="17" format="0.##" expected="17">{fn:format-number(17, "0.##")}</row> <row input="17" format="0.00" expected="17.00">{fn:format-number(17, "0.00")}</row> <row input="17.4555" format="0.##" expected="17.46">{fn:format-number(17.4555, "0.##")}</row> <row input="-17.4555" format="0.##" expected="-17.46">{fn:format-number(-17.4555, "0.##")}</row> </rows> return $result The result of this is as follows: <rows> <row input="0.3576" format="0.##" expected="0.36">.36</row> <row input="-0.3576" format="0.##" expected="-0.36">-.36</row> <row input="-0.3576" format="0.###" expected="-0.358">-.358</row> <row input="-0.3576" format="0.####" expected="-0.3576">-.3576</row> <row input="-0.3576" format="0.00" expected="-0.36">-.36</row> <row input="0.000" format="0.##" expected="0.00">0</row> <row input="0" format="0.00" expected="0.00">0.00</row> <row input="0.3" format="0.####" expected="0.3">.3</row> <row input="0.3" format="0.0000" expected="0.3000">.3000</row> <row input="17" format="0.##" expected="17">17</row> <row input="17" format="0.00" expected="17.00">17.00</row> <row input="17.4555" format="0.##" expected="17.46">17.46</row> <row input="-17.4555" format="0.##" expected="-17.46">-17.46</row> </rows> Unfortunately two things seem to go wrong here: · Zero in the picture string is ignored for negative numbers. · The zero before decimal is ignored for amounts between zero and one (but does occur for 0.0000 for example). I hope someone can take a look at this and provide a fix. I tried messing around with string replace functions where I would just replace -. with -0. and prefix the result for a decimal point with a zero, but this really isn't present and requires a workaround for something that is very common in my code. Using the latest basex (8.4.4) on Windows Server 2012 under Java 8 (64 bit, last update just now: update 91). Thanks in advance and best regards, Michaël Stoop

2 2

Replacing documents in remote database
by Asxetos 02 Jun '16

02 Jun '16

Hey all, I am trying to connect to a remote database and replace its files. I am using db:replace() inside client:query(), but I don't get how I am supposed to pass $input as the new data for the file in the database. Here a quick code snippet I put together: http://sprunge.us/jdaZ ( or http://sprunge.us/OZHj?xquery ) If there's a better method for doing what I am trying, I'm fine with replacing my whole method, even if that is using some sort of REST API and http requests. Just give me some pointers ;) Thanks!

2 4

Static files not accessed ?
by Maud Ingarao 31 May '16

31 May '16

Hello dear friends Since we upgraded to 8.3 or 8.4, we have strange behaviours with static files. For example, this file cannot be accessed when called from another xsl (include or import) as shown in the logs : [GET] http://rey.huma-num.fr/inventaire/Rey17470124; Stopped at /sites/papiersmmrey/resource/basex/basex.8.4.3/webapp/synopsx/mappings/htmlWrapping.xqm, 168/33: [bxerr:BXSL0001] Error on line 7 column 78 of tei2html-fiche.xsl: XTSE0165: I/O error reported by XML parser processing http://rey.huma-num.fr/synopsx/files/xsl/tei2html.xsl: Server returned HTTP response code: 502 for URL: http://rey.huma-num.fr/synopsx/files/xsl/tei2html.xsl Stack Trace: - /sites/papiersmmrey/resource/basex/basex.8.4.3/webapp/synopsx/mappings/htmlWrapping.xqm, 78/47 - /sites/papiersmmrey/resource/basex/basex.8.4.3/webapp/synopsx/models/synopsx.xqm, 168/49 - /sites/papiersmmrey/resource/basex/basex.8.4.3/webapp/synopsx/workspace/rey/_restxq/rey.xqm, 443/43 So is it called in the other xsl : <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0" xmlns="http://www.w3.org/1999/xhtml" xmlns:xs="http://www.w3.org/2001/XMLSchema" exclude-result-prefixes="xs"> <xsl:output method="xml" encoding="UTF-8" indent="no" omit-xml-declaration="yes"/> <xsl:import href="http://rey.huma-num.fr/synopsx/files/xsl/tei2html.xsl"/> [...] And still it is available via a direct GET request : http://rey.huma-num.fr/synopsx/files/xsl/tei2html.xsl Notice that the error code is 502 and not 404. We changed our code as discussed in a previous mail (see below), it seemed to work a little, but now we have this new problem showing... Thanks ! Maud Coordonnées ------------------------------------------------------------------------ /« Ne travaillez jamais »/ Maud Ingarao IHRIM - UMR 5317 Institut d’histoire des représentations et des idées dans les modernités Ecole Normale Supérieure de Lyon - Site Descartes - Bureau R322 15 Parvis René Descartes - BP7000 - 69342 Lyon CEDEX 07 +33 4 37 37 65 79 - maud.ingarao(a)ens-lyon.fr /Je suis absente le vendredi - Out of office on Fridays/ http://ihrim.ens-lyon.fr/ http://institutdesanti.ens-lyon.fr/ http://ahn.ens-lyon.fr/ Hi Maud, Sorry for the delay, I’ll soon give you an answer on your other e-mails! > Response : > HTTP ERROR 400 > Problem accessing /synopsx/files/synopsx.js. Reason: > > Stopped at/resource/basex/webapp/synopsx/_restxq/files.xqm, 38/10: > [SEPM0017] Value of 'method' must be one of > (xml,xhtml,html,json,csv,text,adaptive,basex). With 8.4, the "raw" output method has become obsolete. Your code should work if you remove the following line: <output:method value='raw'/> Here is yet another solution for the synopsx.files:file() function, which is a bit more compact: function synopsx.files:file($file as xs:string) as item()+ { let $path := $G:FILES || $file return ( web:response-header(map { 'media-type': web:content-type($path) }), file:read-binary($path) ) }; Hope this helps, feel free to ask more questions, Christian

2 5

Replacing node sets in a large file
by Michael Sanborn 31 May '16

31 May '16

I need to perform a transformation that would be simple in XSLT, but the input is a file about 250 MBs in size. I'm wondering whether XQuery and BaseX in particular would be the most efficient way of doing it. I'm new to XQuery, and I've come up with a couple of ways to do this, but they turn out to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping to find out the proper way of doing this. The input consists of 2 sections. There are about 3600 page elements with this structure: <page> [misc page content...] <list> <unit> [misc unit content 1...] <partinfo> <number>54321</number> <manuf>A321</manuf> </partinfo> <partinfo> <number>12345</number> <manuf>B123</manuf> </partinfo> [misc unit content 2...] </unit> [multiple units...] </list> </page> Each unit can have 1 or 2 partinfo elements. The other section has about 82000 part elements like this: <part> <partinfo> <number>54321</number> <manuf>A321</manuf> </partinfo> [misc part content 1] </part> [...] <part> <partinfo> <number>12345</number> <manuf>B123</manuf> </partinfo> [misc part content 2] </part> I want to replace each unit/partinfo with the correpsonding part, like this: <page> [misc page content...] <list> <unit> [misc unit content 1...] <part> <partinfo> <number>54321</number> <manuf>A321</manuf> </partinfo> [misc part content 1] </part> <part> <partinfo> <number>12345</number> <manuf>B123</manuf> </partinfo> [misc part content 2] </part> [misc unit content 2...] </unit> [multiple units...] </list> </page> Is BaseX a good tool for this task? If so, how does one go about it? Finally, it would help to be able to output each page element in a separate file. Would it be better to have BaseX do this, or to output the whole database and chunk it with another tool? Thanks, Michael

3 15