May 2017 - BaseX-Talk - mailman.uni-konstanz.de

Preparing BaseX 8.5.4 (please don't forget Issue #1341)
by Jan Dölle 20 Feb '18

20 Feb '18

Hi, we noticed you prepare release 8.5.4. Please don't forget to fix issue: SQL with BaseX 8.5.3 #1341 Thx Jan -- Jan Dölle E-Mail: basex-list(a)id1.de Telefon +49-69-244502-0 Home: www.id1.de Information Design One AG, Baseler Straße 10, 60329 Frankfurt am Main Registereintrag: Amtsgericht Frankfurt am Main, HRB 52596 Vorstand: Robert Peters, Aufsichtsrat: Christian Hecht (Vorsitz)

4 5

standalone vs GUI character parsing
by Bridger Dyson-Smith 13 Jun '17

13 Jun '17

Hi all - I'm able to create a database with the GUI from an XML document that contains an invalid character (U+0000) -- I guess BaseX does some character scrubbing, which is awesome :). When I try creating a database with the same input XML document from the commandline (using the basex executable), I get an error message. Here are my commands in standalone mode: BaseX 8.5.3 [Standalone] Try 'help' to get more information. > create db test-bad-char /usr/home/bridger/src/another-test.xml "/usr/home/bridger/src/another-test.xml" (Line 4): An invalid XML character (Unicode: 0xb) was found in the element content of the document. > xquery db:create("test-bad-char", "/usr/home/bridger/src/another-test.xml", "/usr/home/bridger/src/", map { "chop": false() }) Stopped at ., 1/10: [FODC0002] "/usr/home/bridger/src/another-test.xml" (Line 4): An invalid XML character (Unicode: 0xb) was found in the element content of the document. > This isn't a surprise (I'm more surprised (pleasantly) by the GUI behavior); is there a way to apply the same "scrubbing" from the GUI in standalone mode? I'm sure there is, but I'm not able to figure out which option to apply. As always, thank you for your time and trouble, and thanks for BaseX. Best, Bridger

3 17

Combining modules into a single file
by George Sofianos 02 Jun '17

02 Jun '17

Hi, and thanks for your awesome engine again! I wonder if there is an easy and complete way of combining xquery files into a single file. To give an example, I have about 16-17 xquery files, some are only 50-100 lines but some can be up to 1500 lines. Because of multiple reasons, (legacy web application behaviour that makes it hard to deploy updates, network file system that doesn't always work), it is preferable to have one single file to execute on the XQuery engine. I know it can be done with ant for example, but it's not very easy to cover all cases, so maybe someone has a better idea :) I don't mind implementing something in a different language, as long as I know it will be doable. Thanks, George

2 2

Create user doesn’t work if DBPATH folder does not exist
by Simon Chatelain 31 May '17

31 May '17

Hello all, I just faced an issue in my application using BaseX 8.6.4 when trying to create a user. If the folder that will contain the databases (DBPATH in .basex configuration file) does not exist when you try to create a user, the creation of the “users.xml” file fails. To reproduce, make sure that the folder defined by DBPATH does not exist, and execute: final Context ctx = new Context(); try (final Session session = new LocalSession(ctx);) { session.execute(new CreateUser("USERNAME", "USERPASSWORD")); } Then nothing happens. The users.xml file is not created. While executing this: final Context ctx = new Context(); try (final Session session = new LocalSession(ctx);) { session.execute(new CreateDB("TESTDB")); session.execute(new CreateUser("USERNAME", "USERPASSWORD")); } Works as expected as the CreateDB() command will create the missing folder. Is it on purpose or was it overlooked? Best regards Simon Chatelain

2 1

Evaluating XPath within custom QueryModule
by Johannes Echterhoff 29 May '17

29 May '17

Hello, I am writing a custom QueryModule that can be used to parse information from a given XML element. The module has a function like the following: public ParsedType parse (Object arg) throws QueryException { if (arg instanceof ANode) { return utils.parse((ANode) arg); } else if (arg instanceof BXElem) { return utils.parse((Element) arg); } else { ... } } It turned out that parsing requires additional information that can be found either on the XML element itself or - more importantly for my question - in one of its ancestors. The XPath expression to determine the information looks like this: <context XML element>/ancestor-or-self::*[@attributeX or nsX:SomeElement/*/@attributeX][1]. I can evaluate this expression in the XQuery before calling method "parse" from my custom query module, and add the value of attributeX as an argument. To do so, I actually have a function to determine the value of attributeX: declare function local:determineAttributeX($elmt as element()) { let $elementWithAttributeX := $elmt/ancestor-or-self::*[@attributeX or nsX:SomeElement/*/@attributeX][1] return if (empty($elementWithAttributeX)) then () else if($elementWithAttributeX/@attributeX) then $ elementWithAttributeX/data(@attributeX) else $ elementWithAttributeX/nsX:SomeElement/*/data(@attributeX) }; However, I wonder if it is possible to evaluate the expression also within the query module itself, i.e. when method "parse" is called. Is there a way to do so? That would reduce the complexity of the XQuery that calls "parse" (and the value of "attributeX" is always of interest when "parse" is called, so ideally "parse" can determine the value itself). I searched the internet (the archive of this mailing list as well as stackoverflow) for quite some time but could not find an answer to this question. Any help would therefore be greatly appreciated. Best regards, Johannes Echterhoff

2 6

Running BaseX on a parallel cluster
by Kristian Kankainen 25 May '17

25 May '17

Hi all! Is there any way to make BaseX run in parallel on a cluster? Through school I have access to several clusters and I got interested in trying out if BaseX can take advantage of parallel computing. I know my question is vague. To try to be more specific - can I compile BaseX with some extra parameters? Is there anything written about this topic? For the background information I got interested in trying out running parallel XQuery because from somewhere (that I can't find in the mailing list archive) I read that XQuery falls under the "dataflow" programming paradigm [1] and as such should be parallelizable out-of-the-box. Also being a functional language with no side-effects makes automatic parallelization easier. I am not a computer scientist and/but I am aware it isn't that easy as I might make it sound. Cheers Kristian K [1] https://en.wikipedia.org/wiki/Dataflow_programming

2 5

Profiling queries
by Iwan Briquemont 25 May '17

25 May '17

Hello, I was wondering if there are ways to access how much time is spent in parts of a query/program, to find out inefficient functions or expressions? Something like the total time spent in functions would be quite useful. I saw the Profiling module, but it requires you to change the code and it also won't aggregate results when you run it in a loop. It's still doable if you add labels and then do some parsing on stderr to aggregate the results yourself, but it's quite cumbersome. Some thoughts as a side note: A nice way to avoid having to change the code to do profiling (or other debugging like breakpoints) would be to change or add hooks into the AST of the query. That way you could for example add a prof:time() around all the function bodies, with the function name as label. And to help interpret the results of debugging functions like prof:time or trace, it would be nice to have a way to format them and output them as a stream of items (or even multiple streams). Then you could create XQuery scripts that consume the streams to aggregate the information in a human readable way. Best regards, Iwan

2 3

Approximating ML's Reverse Queries Mechanism?
by Eliot Kimber 23 May '17

23 May '17

I’m working on a project that currently depends on MarkLogic’s reverse query mechanism. This is a feature whereby you store documents that contain MarkLogic-specific queries (queries in their cts namespace). These get indexed in some way (no idea how the index works). You can then search for these stored queries in the context of specific nodes and ML will return those query documents that would match the specified nodes. Their driving use case is alerting-type applications where when new docs get added you see what queries they apply to and then use those queries to do something. My use case is classification: given a node to be classified, find all queries that match it and from the query get the classification details (preferred term, variant forms, associated taxonomy, etc.). This process definitely depends on MarkLogic’s full-text search features, for example to match any form of a non-preferred term to a full-text search that would match it. I have a large corpus to classify (approximately 45 million objects at current count). The processing is inherently parallelizable so I’m looking at non-ML options that would allow us to scale to the limits of our hardware budget. Even if each node was less efficient than ML we would be able to implement massive throughput for this classification operation. In theory we could scale to one processor per object if budget were unlimited (it is not unlimited), so even a brute-force solution would perform well at larger scales. So I guess I have two questions really: 1. Can anyone share ways they use BaseX for doing classification in general? I’ve so far just been focused on analyzing the current system to find performance bottlenecks so I haven’t yet had a chance to think through the classification process in general, but there must be well-understood strategies. I suspect that one could build the equivalent of ML’s reverse query index in BaseX. 2. Is there a direct equivalent to ML’s reverse query facility in BaseX or an obvious route to building one? Thanks, Eliot -- Eliot Kimber http://contrext.com

1 0

Querying documents by a full path pattern
by Jacob Borisov 19 May '17

19 May '17

Hello. Is it possible to select documents in a database by a path pattern? For example, there is a database storing following documents: dir1/a.xml dir1/b.xml dir2/a.xml dir2/b.xml I want to select documents named 'b.xml' only. I guess something like db:open('dbname', '*/b.xml')/* but it returns empty result. Thanks!

2 1

BaseX 8.6.4
by Christian Grün 18 May '17

18 May '17

Dear all, Another patch release of BaseX release is available. It includes one critical fix for fn:tail. If you use this function, an update is definitely recommended. Cheers, Christian

1 0