- BaseX-Talk - mailman.uni-konstanz.de

default concatenation of strings? BaseX 10.5
by Patrick Durusau 05 Apr '23

05 Apr '23

Greetings! I'm converting Hebrew text, word by word, into code points, which is returned as: 1493 1463 1497 1468 1463 1513 1473 1456 1499 1468 1461 1448 1501 1500 1464 1489 1464 1436 1503 .... (the file is quite long) What I expect is described at: https://www.w3.org/TR/xslt-xquery-serialization/#sequence-normalization "If the |item-separator| serialization parameter is absent, then for each subsequence of adjacent strings in /S_2 /, copy a single string to the new sequence equal to the values of the strings in the subsequence concatenated in order, each separated by a single space." I maybe very wrong but shouldn't that render the strings as?: 1493 1463 1497 1468 1463 1513 1473 1456 1499 1468 1461 1448 1501 and, 1500 1464 1489 1464 1436 1503 I've tried using replace($a, "\n", " ") but it complains that $a is a sequence, which it is. Then I tried: for $char in $a return ($char, " ") Now I get: 1493 1463 1497 etc. I saw the new line settings under serialization but there didn't appear to be any way to defeat them altogether. Thanks! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

2 2

Re: [basex-talk] False Positives
by Owen Ambur 05 Apr '23

05 Apr '23

Good to hear from you, Jorge. Thanks not only for the pointer but also for all the work you did on your StratML prototype, as documented on GitHub. It helped get us to this point and I'll look forward to any further contributions you may be able to make. Naval, in the reference Jorge cites, here's the text that appears to be relevant: By default, unless the languages codes ja, ar, ko, th, or zh are specified, a tokenizer for Western texts is used: Whitespaces are interpreted as token delimiters. The following is contrary to the intent of the StratML query service: Since the logical flow of the text is not interrupted by the child elements, you will typically want to search across elements, so that the above paragraph would match a search for “real text”. For more examples, see XQuery and XPath Full Text 1.0 Use Cases. The query service SHOULD respect each element as being distinct. The purpose of the service is to enable discrete querying of the elements of the schema, and within each element whitepace should be treated as a delimiter. However, this guidance is confusing: To enable this kind of searches, it is recommendable to: Keep whitespace stripping turned off when importing XML documents. This can be done by ensuring that STRIPWS is disabled. This can also be done in the GUI if a new database is created (Database → New… → Parsing → Strip Whitespaces). The first two sentences seem to suggest that whitespaces will be maintained while the third indicates they would be removed. While this may not be the most important next step to be taken to improve and enhance https://search.aboutthem.info/, it might be one of the easiest. Owen Amburhttps://www.linkedin.com/in/owenambur/ On Wednesday, April 5, 2023 at 03:28:57 AM EDT, <jorge(a)vionta.net> wrote: Hi Owen, You may check the full text configuration cappabilities https://docs.basex.org/wiki/Full-Text like possitional filters and Fuzzy Quering. It may be a bug, but I would exclude configuration at first. I can see that you are making good progresses, and love that you have taken the basex option. I think that you are on the right path. Love to see progresses. Kind regards. On 08/03/2023 17:31, Owen Ambur wrote: Christian, do you know if this has been identified as a bug in BaseX's full-text query capability and, if so, if there are any plans to do anything about it? If memory serves me correctly, I subscribed to the BaseX listserv for awhile to try to enlist a developer(s) for a StratML-enabled query service, like the one on which Naval is now working for me for hosting at https://aboutthem.info/. When the query service is in relatively good shape, I may wish to resubscribe to the listserv to announce it there as well as on LinkedIn and perhaps elsewhere. However, do you think it might be worthwhile to raise this issue on the listserv in the meantime? Owen Ambur https://www.linkedin.com/in/owenambur/ On Tuesday, March 7, 2023 at 03:16:49 PM EST, Naval Sarda <nsarda(a)epicomm.net> wrote: Hi Owen. The inbuild search provided by BaseX is combining the text from next file and then searching. So if the line ends with word "end." and next line starts with "less", it will match search criteria "endless" This is false positive matching. There is nothing much we can do about it as replacing with custom search will be slow. Naval On 07/03/23 6:38 am, Owen Ambur wrote: What can we do about it? Owen Ambur https://www.linkedin.com/in/owenambur/ On Monday, March 6, 2023 at 07:16:17 PM EST, Naval Sarda <nsarda(a)epicomm.net> wrote: Please see below -------- Forwarded Message -------- | Subject: | Re: Fwd: False Positives | | Date: | Mon, 6 Mar 2023 21:38:43 +0530 | | From: | Sudarshana <sudarshanac(a)epicomm.net> | | To: | Naval Sarda <nsarda(a)epicomm.net>, jitendras(a)epicomm.net | Owen, This was known issue we were informed you. In fulltext search, if there is any space character like (tab, space or new line) is present then it is coming in result. In file APQC.xml, Board of Governors of the Federal Reserve System is one organization and Bombardier Aerospace Inc. is next adjacent organization. So Board of Governors of the Federal Reserve System Bombardier Aerospace Inc. highlighted keyword is considering as tembom . So those files are coming in result. -Sudarshana On 3/6/2023 10:18 AM, Naval Sarda wrote: Get Outlook for iOS From: Owen Ambur <owen.ambur(a)verizon.net> Sent: Monday, March 6, 2023 6:35 AM To: Naval Sarda <nsarda(a)epicomm.net> Cc: abouttheminfoplan(a)googlegroups.com <abouttheminfoplan(a)googlegroups.com> Subject: False Positives Naval, Ken Holman's LinkedIn posting about his health issue prompted me to query to confirm that Project TEMBO's about statement is in the StratML collection. However, the full-text query also revealed a couple apparently false positives, as shown in the screen shot below. They are: https://stratml.us/docs/APQC.xml https://stratml.us/docs/DOSAID2022.xml Since the latter is tangentially related in terms of foreign aid, it might be logical for an AI-enhanced query service to reveal it as such. However, ours isn't that "intelligent," is it? What do you suppose might account for the false positives? This isn't the first time I've encountered them. Owen Ambur https://www.linkedin.com/in/owenambur/ -- Thanks & Regards Sudarshana

1 0

Constructing "resolved" DITA Map in XQuery: How to Avoid High Memory usage?
by Eliot Kimber 05 Apr '23

05 Apr '23

I’m implementing a feature of DITA which involves pulling together all the DITA maps and submaps linked from a root map so that you can then process them as a single unit in order to then construct “key spaces”, which are defined by the topicrefs contained in the maps and which depend on both the structural hierarchy defined by the tree of maps and submaps and on the markup details of both the maps and the topicref elements. It’s a challenging bit of data processing. In other contexts where I’ve implemented this processing I start by creating a “resolved map” using a relatively simple transform, resulting in a single XML document with all the stuff needed to then construct the DITA key space. With the resolved map, the logic to construct the key space is a relatively simple three-phase process. My naïve attempt to do this in BaseX using the normal typeswitch approach to implement an identity transform worked at a small scale, but for our real DITA maps, which have 10s of 1000s of elements, the process quickly exhausts the 2GB of RAM allocated to BaseX GUI. The reason I’m doing the transform in XQuery and not just using Saxon via the XSLT module is because I need to annotate the resulting resolved map with the database node IDs of each element so I can then capture those details in the final key space, which I’m storing as XML in another database—the constructed key space acts as an index where the input is a context element/key name pair and the result is the topicref element that defines the key, from which I can then get the resource associated with that topicref (i.e., the topic it references or a string it defines or whatever it might be). So my first question is: Is there a general technique for doing this kind of identity transform that won’t blow up the memory? I suspect the answer is “no” but figured I’d ask. Or is it possible to apply a Saxon transform to content pulled from the BaseX database and have access to the node IDs? I didn’t immediately see a way that you could do that. There must be a pretty sharp separation between BaseX and Saxon here, but again, maybe I missed something? If the answer is “no” I can work out a more sophisticated way to build the initial data from which the key space is ultimately constructed by walking the map tree and populating XQuery maps or something, but I was hoping to keep my simple code that just operates on the resolved map. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 3

XQuery inconsistent result related to overflow
by Shuxin Li 05 Apr '23

05 Apr '23

Hi, I'm Shuxin. Recently I came across this test case in which BaseX returned an incorrect result. Currently I could not confirm it as a bug and is only a corner case which might not be of importance, but there are some interesting behaviors which I'm also curious about. Give XML document <A1 id="1"><B1 id="2"></B1><B1 id="3"></B1></A1> and XPath query //A1[((count(./*) idiv 3) * -1763118392 * 2144097893) != 1] BaseX return empty result set while node A1 should be returned. This should be related to overflow since making either number smaller BaseX will then return correct result and therefore could be considered not a bug. But interestingly, this only occurs when the operand given after idiv (in this case 3) is greater than 2, which result is 0. If changed to (count(./*) idiv 2) or (count(./*) idiv 1), the operand is larger but correct results are still returned. Simply substituting the whole (count(./*) idiv 3) to constant either 0, 1, 2 all returns correct results. This indicates that BaseX should have the ability of handling this expression, but is somehow affected. The version of BaseX I produced this on is the latest development commit e607ecc. Still, this is just a corner case which might not require a fix. I submit this report just for your reference. Thank you very much! Best Regards, Shuxin Li 2023.4.5

2 1

BaseX HTTP service goes down due to Qualys Agent
by ykhabins＠bellsouth.net 04 Apr '23

04 Apr '23

Hello, We are using BaseX 10.5 via its HTTP service in a corporate environment. We have an automated Qualys Agent that does a vulnerability scan of that server with the BaseX. Qualys Agent scan process includes web sites related tests such as Cross-Site Scripting, SQL Injection, etc. The rapid nature of the Qualys Agent requests effectively gives us a DoS attack on the eclipse.jetty.server. It cannot process so many requests and goes down. In the meantime, our solution is to restart BaseX HTTP service manually via basexhttp.bat. Question: is it possible to somehow configure the eclipse.jetty.server so it will be able to sustain the Qualys Agent vulnerability scan? Regards, Yitzhak Khabinsky

3 12

Problem with using XInclude in BaseX
by Virgile Reignier 03 Apr '23

03 Apr '23

Bonjour, Je me permets de vous contacter dans le cadre de mon doctorat en histoire médiévale. Je souhaiterai en effet organiser le dépouillement de mes archives à l'aide d'une base de données en XML en utilisant le logiciel BaseX, mais je suis face à une difficulté que je n'arrive pas à surmonter dans son utilisation. Cette difficulté est liée à l'utilisation de la balise <xi:include>. J'ai tenté de produire un exemple minimaliste avec deux fichiers dans un même répertoire pour voir ce qui ne marche pas : file music.xml : <musique xmlns:xi="http://www.w3.org/2001/XInclude"> <titre>Smoke on the water</titre> <artiste>Deep Purple</artiste> <xi:include href="label.xml"/> </musique> file label.xml : <label>Purple Records</label> Alors que ces deux fichiers sont décrits comme valides par Oxygen, l'importation du fichier music.xml avec BaseX (que ce soit avec l'interface DBA ou l'interface GUI) génère systématiquement l'erreur suivante : "An include with href 'label.xml'failed, and no fallback element was found". Je peux en revanche importer le fichier dans la base de données via Oxygen et l'utilisation du WebDAV de BaseX, mais le fichier est alors indiqué comme étant de type "binary" et est absolument inemployable pour réaliser des requêtes. J'ai pourtant tenté de réaliser la requête suivante au sein de ma DB dans BaseX GUI, ce qui n'a pas renvoyé d'erreurs : declare namespace xi = "http://www.w3.org/2001/XInclude"; xi:include J'ai également tenté de charger ces mêmes fichiers dans eXist DB, et l'inclusion s'est réalisée sans problèmes. Je ne comprends donc pas ce qui ne va pas avec l'utilisation de XInclude dans BaseX. Est-ce qu'il y a un problème avec la résolution des URI ? Je travaille sous Windows 10 (j'ai aussi fait des tests sur Ubuntu et obtenu le même résultat), BaseX 10.4 et Java 1.8.0_361. Est-ce que c'est quelque chose qui vous est déjà arrivé ? J'ai essayé de chercher dans les archives mail de basex-talk et trouvé quelques problèmes similaires, mais qui ont été résolus par l'emploi de BaseX GUI. Or chez moi cela marche aussi mal avec GUI qu'avec DBA. Je peux transférer mon travail sur eXist DB, mais j'avoue m'être assez familiarisé avec BaseX et souhaiterai rester sur ce logiciel tant que possible. Je suis donc preneur de toutes vos propositions. En vous remerciant par avance pour vos réponses, Bien cordialement, Virgile Reignier Hello, I am contacting you as part of my PhD in medieval history. I would like to organise the analysis of my archives with the help of an XML database using the BaseX software, but I am encountering a difficulty that I cannot overcome in its use. This difficulty is related to the use of the <xi:include> tag. I tried to produce a minimalist example with two files in the same directory to see what goes wrong: file music.xml : <musique xmlns:xi="http://www.w3.org/2001/XInclude"> <titre>Smoke on the water</titre> <artiste>Deep Purple</artiste> <xi:include href="label.xml"/> </musique> file label.xml : <label>Purple Records</label> While both files are described as valid by Oxygen, importing the music.xml file with BaseX (either with the DBA interface or the GUI interface) consistently generates the following error: "An include with href 'label.xml'failed, and no fallback element was found". I can import the file into the database via Oxygen and the use of BaseX WebDAV, but the file is then indicated as being of type "binary" and is absolutely unusable for queries. However, I tried to perform the following query within my DB in BaseX GUI, which did not return any errors: declare namespace xi = "http://www.w3.org/2001/XInclude"; xi:include I also tried to load these same files into eXist DB, and the inclusion was successful. So I don't understand what is wrong with using XInclude in BaseX. Is there a problem with URI resolution? I'm working on Windows 10 (I also tested on Ubuntu and got the same result), BaseX 10.4 and Java 1.8.0_361. Is this something that has happened to you before? I tried to search the basex-talk mail archives and found some similar problems, but they were solved by using BaseX GUI. However, in my case it works just as badly with GUI as with DBA. I can transfer my work to eXist DB, but I have become quite familiar with BaseX and would like to stay with it as long as possible. I am therefore interested in any suggestions you may have. Thank you in advance for your answers, Virgile Reignier

4 16

Potential bug in GUI, Unit Tests
by Zimmel, Daniel 01 Apr '23

01 Apr '23

Hi, while I am trying to find my way around the Unit Module, I noticed the following unexpected behavior. 0. Using the test example from https://docs.basex.org/wiki/Unit_Module#Query in the GUI 1. changes that are not saved to disk do not get respected - as opposed to other queries 2. pressing the default button for running queries executes the most recent query which is open in another tab While 1 might not be a bug , 2 feels like one (a query fires that is not displayed on the current tab) BaseX 10.3 Bye, Daniel

2 3

Automated testing for XML databases
by Shuxin Li 31 Mar '23

31 Mar '23

Greetings! I'm Shuxin, an undergraduate CS student, who is currently an intern at NUS Test lab. I’ve been working on an automated testing tool for XML processors and databases. My goal is to find bugs in important systems, in order to improve these systems’ reliability. In the past few days, I submitted a few bug reports on Github (Github id: Twilight-Shuxin). Thanks for having already fixed these! I would like to ask whether you find these reports useful and would also appreciate future reports if I find more bugs? Any feedback for us is highly appreciated. We also plan to open-source this tool for public usage after it's fully realized, and truly hope that it might be of help to the community. Thanks! Best Regards, Shuxin Li

5 6

Database Query in RESTXQ function
by Alexander Krasnogolowy 29 Mar '23

29 Mar '23

Hi, given the following RESTXQ function: declare %rest:path("/myrest/{$db}") %rest:GET %output:method("xhtml") %output:omit-xml-declaration("no") %output:doctype-public("-//W3C//DTD XHTML 1.0 Transitional//EN") %output:doctype-system("http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd") function page:html($db as xs:string) { <html xmlns="http://www.w3.org/1999/xhtml"> <body> Database creation date: {db:info("NB")/databaseproperties/timestamp/text()}<br/> </body> </html> }; The website returns "Database creation date: " (without a timestamp) If I put that into a function: declare function page:getDBTimestamp($db as xs:string) { db:info($db)/databaseproperties/timestamp/text() }; and call it in the page: <body> Database creation date: {page:getDBTimestamp($db)}<br/> </body> then I can see the timestamp. Can somebody explain to my why it makes a difference and why I cannot query the database directly from the RESTXQ function? Alex

4 6

collection('i_do_not_exist')/users/user[@name=()] is () even if i_do_not_exist does not exit
by Omar Siam 23 Mar '23

23 Mar '23

Hi, Either I found an interesting new XQuery feature or a bug: collection('i_do_not_exist')/some/path[@attr=()] retruns the empty sequence whereas collection('i_do_not_exist')/some/path throws an error err:FODC0002 so does collection('i_do_not_exist')/some/path[@attr="i_am_an_attr"] Am I missing something? Best regards, -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

2 3