September 2018 - BaseX-Talk - mailman.uni-konstanz.de

comparing basex and csv datasets
by ＠pyschny.de 10 Sep '18

10 Sep '18

Dear basex and xquery experts, I have to solve the following problem: On a daily basis I get data packages in csv format (approx.. 20.000 records) I imported the 1st package to basex-DB and want to import further changed or new records from the daily deliveries. I tried several solutions but cannot get a solution which is performant enough to be executed every day. While importing complete packages takes a few seconds – comparing against records in the database requires hours. So I decided to use the index variant which is described by Christian Grün. My core code looks as follows: let $db = db:open(‘mydb’) let $ndx = db:open(‘mydb-index’) let $xml := csv:parse($csv, map { "header": true(), "separator": ';' }) let $xml := ( for $x in subsequence($xml//record,2) count $c return if( local:recordUpdated( $x, ‘mydb’, $ndx)) then ( element {'record'} { attribute {'recno'} {$c}, $x/child::*} ) else () ) (: Aufnahme der Differenzen in die DB :) db:replace( $dbname, $tpalow||"/"||$seqno, $xml) declare function local:recordUpdated( $rec, $dbname, $ndx as item()*) as xs:boolean { let $id := $ndx//vp[@ptn=$rec/_14_TPA-Key_VP/text() and @tarif=$rec/_28_TPA-Key_Tarif/text()]/id return if( $id) then ( let $oldrec := db:open-id($dbname, $id)/.. return not(deep-equal($oldrec/child::*, $rec/child::*)) ) else true() }; Both databases ‘mydb’ and ‘mydb-index’ are created with attribute and text indexes. After each run the index is newly created to point to the newest records for the given key values. Can anybody point out why this process takes more than a second per each single record to compare? Thanks for your time

1 0

Example xquery needed to write output to an local xml file
by Marco Roling 06 Sep '18

06 Sep '18

Dear all, I am new on the block here, and trying to find my way around BaseX, which seems to meet my needs for my research on digital museum collection data analysis. I have a very large collection of xml files (> 600000 files) that I am querying, and I want to write the returned result immediately in a file. The following xquery (see below) is not working, which probably comes as no surprise to you all. But to get me started, I would like to have a simple example that works. Any example would do. Hope anyone can help me out with this. Thanks very much, best, Marco. ===== declare variable $data := collection(PRODRMCollectionItems); declare variable $output := 'C:\DATAQUERY\BaseX\Q001\outputPROD'; (:This is where all the work is happening:) declare function local:process-data ($nodes as item()*) as element (root) { <RMCollectionItemsDATA created="{current-dateTime()}"> { for $a in $nodes//artObjectGetResponse/artObject[objectNumber eq 'NG-NM-1395-A'] return <item>{$a}</item> } </RMCollectionItemsDATA> }; (:here we store stuff:) let $filename := concat($output, "marco.xml") return file:write($filename, local:process-data($data)) =====

4 5

diacritics sensitive not working
by Ron Katriel 03 Sep '18

03 Sep '18

Hi, The following from your website (docs.basex.org/wiki/Full-Text) appears to be syntactically incorrect "'Äpfel' will not be found..." contains text "Apfel" diacritics sensitive In the BaseX GUI the keyword diacritics is underlined in red and the following error is reported Unexpected end of query: 'diacritic sens...'. This happens in version 8.6.4 and also the latest (9.0.2). Thanks, Ron *Ron Katriel, Ph.D. *| Principal Data Scientist | Medidata Solutions <http://www.mdsol.com/> 350 Hudson Street, 7th Floor, New York, NY 10014 rkatriel(a)mdsol.com <tbrophy(a)mdsol.com> | direct: +1 201 337 3622 <//201%20337%203622> | mobile: +1 201 675 5598 <//mobile:%20+1%20201%20675%205598> | main: +1 212 918 1800 <//+1%20212%20918%201800>

4 15

BaseX insert/delete node performance
by BIRKNER Michael 01 Sep '18

01 Sep '18

Hello, I asked this question in StackOverflow concerning some performance problems I experienced when inserting nodes into a BaseX database: https://stackoverflow.com/questions/51595210/basex-inserting-nodes-performa… I already made some progress, especially when it comes to querying all data I need for the updates. I work a lot with the indexes now. But I still have problems with inserting - and also deleting - nodes. It doesn't matter if I insert/delete nodes via a Java program or in the editor of the BaseX GUI: Both is quite slow. Inserting just one node in the GUI with an XQuery like this one takes up to 3 seconds: insert node <related_record><title>Test title</title><author>Joe Lastname</author></related_record> into db:open-id('Database_Name', 7947561) Deleting a node with the following command takes up to 7 seconds: delete node db:open-id('Database_Name', 88085737) The problem is that in my case, I have to do about 150000 inserts and deletes, so it would take too much time. Maybe my database is just too big to be performant? Or some settings are wrong? I'm very new to BaseX (and XML databases in general) so maybe there are just some errors I don't see. I also give you some information on my database that I copied from the info screen of the BaseX GUI: Database Properties NAME: Database_Name SIZE: 2568 MB NODES: 135607105 DOCUMENTS: 1 BINARIES: 0 TIMESTAMP: 2018-08-07T07:05:56.000Z UPTODATE: true Resource Properties INPUTPATH: /path/to/file.xml INPUTSIZE: 1774 MB INPUTDATE: 2018-07-24T14:32:58.000Z Indexes TEXTINDEX: true ATTRINDEX: true TOKENINDEX: false FTINDEX: false TEXTINCLUDE: ATTRINCLUDE: TOKENINCLUDE: FTINCLUDE: LANGUAGE: English STEMMING: false CASESENS: false DIACRITICS: false STOPWORDS: UPDINDEX: true AUTOOPTIMIZE: false MAXCATS: 100 MAXLEN: 96 SPLITSIZE: 0 Best regards, Michael Beachten Sie, dass Sie uns ab sofort unter einer geänderten Rufnummer erreichen. Bitte speichern Sie gleich Ihren Kontakt zur AK Wien ein unter 501 65 1, gefolgt von der gewohnten Durchwahl. Dieses Mail ist ausschließlich für die Verwendung durch die/den darin genannten AdressatInnen bestimmt und kann vertrauliche bzw rechtlich geschützte Informationen enthalten, deren Verwendung ohne Genehmigung durch den/ die AbsenderIn rechtswidrig sein kann. Falls Sie dieses Mail irrtümlich erhalten haben, informieren Sie uns bitte und löschen Sie die Nachricht. UID: ATU 16209706 I https://wien.arbeiterkammer.at/Datenschutz_(DSGVO).html

2 9