BaseX-Talk

basex-talk@mailman.uni-konstanz.de

5 participants
5247 discussions

Performance Question
by Thomas Kaltofen 13 Dec '12

13 Dec '12

Hi, I have a question regarding performance, because my database shows a somehow strange behavior. I have a database with around 3000 documents, the size of each document does not really matter, because I tried it with small and big documents and there is no real difference (total size of test database is around 40 MB). I am connecting to the database through the client/server interface to execute a query over all documents (all documents have the same structure). The first time the query is executed after the server has been started takes a little longer, subsequent executions of the same query are very fast (around 250 ms). Now I leave the server running for several hours without touching the database at all (e.g. over the night). If I now execute the same query again (the database is still running but was idle for several hours), the execution takes very long (several minutes), but only for the first execution after the idle time, subsequent calls to the query are fast again (around 250 ms). This behavior always happens whenever the database was idle for several hours. What could be the cause for this delay? What I tried already: - Run the database on another computer -> same behavior - Try out different JRE versions (6, 7) -> same behavior - Try out different BaseX versions (7.3, latest 7.5 snapshot) -> same behavior - Start the VM with different options regarding performance & garbage collection -> same behavior - Use smaller or larger documents in the database -> same behavior - Execute a different query -> same behavior I am really running out of ideas and this seems to be independent of the size of the individual documents. What I noticed is that the described lag after the idle phase increases with the amount of documents in the database. Any help and suggestions are appreciated. Thank you! Best regards, Thomas P.S. When (approx.) do you plan to release the next version of BaseX?

5 11

BaseX and Saxon8
by NORFARADILLA WAHID 13 Dec '12

13 Dec '12

Dear sir, I am currently using BaseX to support my research project. Unfortunately when I include Saxon8 into the class-path(for my additional feature) I keep receiving the following error: java.lang.NoClassDefFoundError: net/sf/saxon/om/Name Can you help me? p/s: BaseX 7.1.2 beta if that matter Thanks, Dilla

2 1

Fwd: BaseX Server
by Christian Grün 13 Dec '12

13 Dec '12

Hi all, attached is some interesting information from Raymond Meester on how to get BaseX running as Windows service. Everyone’s feedback is welcome, Christian ---------- Forwarded message ---------- From: Raymond Meester <raymondmeester(a)gmail.com> Hi Christian, A small update on implementing basex. Still working on the issue that BaseXServer suddenly isn't available more (started as BaseXServer -S on a Windows 2008 r2 server). I tried to monitor the process with procmon, unfortunaly due to a leak procmon flooded the page file of the server. So I had to go for another approach. As our administrator are familiar with windows services I tried to install basex as windows service. - First I tried the Java launcher Janel (http://www.redskaper.com/), but I could get it working. Don't know why, but it seemed to be a Janel issue. - Secondly I tried another Java launcher (http://winrun4j.sourceforge.net/). With this launcher I was able to got a working executable. I attached the ini file I used. However to use it as a windows service it needs a java method to work with services (see "Service example" on their webpage). - Last I used the working exe and this manual http://marckean.wordpress.com/2009/05/24/run-shoutcast-as-a-windows-service/ ((I attached the tools which the manual uses: instrv.exe and srvany.exe. (I changed the extension to .txt to get them through the email scanner. I believe they are part of the tool "resource hacker" http://www.angusj.com/resourcehacker/) And now I got BaseX running as a windows service. So of course I would like to see a native way to install Basex as Windows Service, but I thought I let you know how I manage this. Also when there is a better way (not changing the registry directly), please put it on the wiki. Regards, Raymond

1 0

Re: [basex-talk] Problems DST
by Randy reed 12 Dec '12

12 Dec '12

I think I have found my problem. I post this here in case it helps anyone else. The java date constructor is initialized with a long, my attempts use a literal are what caused it to fail. I cast my literal as a long xs:long("133554732") and it worked, though not perfectly because epoch is in seconds and java is milliseconds. Thus I had to multiply my literal by 1000 first and then cast it as a long let $fdate: =133554732*1000 let $edat := dt:new(xs:long($fdate)) I then can access the TimeZone.IsDaylightTime property Thanks to Christian for his help on this. Randy On Wed, Dec 12, 2012 at 1:13 AM, Randy reed <randywreed(a)gmail.com> wrote: > Hi Christian, > > Thanks for your response. I think that fn:implicit-timezone() would work for > current dates but I'm not sure if it would work for dates in the past since > it doesn't take any arguements. > > I need some way that it can take a date in utc and adjust to the current > timezone if its during that time period (i.e. we are currently in standard > time) or adjust it if it was in the past during daylight savings time. If > there was something that could read a utc date and then indicate whether it > was during the daylight savings time period or not I could make the time > adjustment. > > In java there is a inDaylightTime property but it requires a date and > timezone object. and it errors when I try to invoke it "invalid arguments" > when it tries to initialize the constructor.. > > declare namespace tz = "java:java.util.TimeZone"; > declare namespace dt = "java:java.util.Date"; > > let $fdate := dt:new(1347425558) > let $tmz := tz:getDefault() > let $day := tz:inDaylightTime($tmz, $fdate) > return $day > > Any help you could provide would be much appreciated. I am not a Java > programmer so I might not understand something here. > > Randy > > > On Dec 11, 2012 6:18 PM, "Christian Grün" <christian.gruen(a)gmail.com> wrote: >> >> Hi Randy, >> >> fn:implicit-timezone() may come in handy here. It returns your current >> local timezone: >> >> http://www.xqueryfunctions.com/xq/fn_implicit-timezone.html >> >> Best, >> Christian >> ___________________________ >> >> On Wed, Dec 12, 2012 at 12:13 AM, Randy reed <randywreed(a)gmail.com> wrote: >> > Hi All, >> > >> > I wrote a xquery in Basex that created an xml file that calculated >> > student grades. Points are dependent on students doing an activity in >> > a window which ends at midnight on sunday each week. The date/time of >> > the activities are stored in the source file in UTC. However, to know >> > if they met the deadline, I need it in local time. I do this by >> > creating a function that translates epoch to a regulate dateTime >> > variable. But I see that when we left daylight savings time, things >> > were thrown off by an hour. The command I use is: >> > >> > adjust-dateTime-to-timezone((xs:dateTime("1970-01-01T00:00:00-00:00") >> > + xs:dayTimeDuration(concat("PT", $v, >> > "S"))),xs:dayTimeDuration('-PT4H')) >> > >> > clearly the problem is the last part -PT4H, because dates that fall >> > after the switch back to standard time from daylight savings time >> > should be -PT5H. Is there a better way of doing this? Is there another >> > command that will give me a localized date that takes into account >> > daylight savings time issues? I tried to set a java calendar but I >> > could not make it work. >> > >> > Any help would be gratefully appreciated. >> > Thanks in advance >> > Randy >> > >> > -- >> > Dr. Randall Reed >> > Assistant Professor, Dept. of Philosophy and Religion >> > Appalachian State University >> > _______________________________________________ >> > BaseX-Talk mailing list >> > BaseX-Talk(a)mailman.uni-konstanz.de >> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk -- Dr. Randall Reed Assistant Professor, Dept. of Philosophy and Religion Appalachian State University

1 0

Re: [basex-talk] BaseX in Farsi?
by Masoumeh Seydi Gheranghiyeh 12 Dec '12

12 Dec '12

Hi Christian That would be a great idea. I am really interested in. Supporting Persian language makes support of the other right-to-left languages, like Arabic and Hebrew, also possible. As you've said, the first step is to find out how much effort is needed to add the right-to-left format to the editor. I will take a look at it to investigate how to do it and highly appreciate any suggestion. Regards, Masoumeh On Monday, December 10, 2012 22:30 CET, Christian Grün <christian.gruen(a)gmail.com> wrote: > Hi Masoumeh, > > yet another idea I got: as you may know, BaseX is available in various > languages. Is Farsi your native language? And would you be interested > in translating BaseX to your language? It may be, however, that BaseX > may get into troubles when writing text from right to left. > Particularly, it's the text editor that's very limited in this > regard.. > > Thanks in advance, > Christian > > PS: just in case: please forward your reply on basex-talk back to the list..

1 0

Problems DST
by Randy reed 11 Dec '12

11 Dec '12

Hi All, I wrote a xquery in Basex that created an xml file that calculated student grades. Points are dependent on students doing an activity in a window which ends at midnight on sunday each week. The date/time of the activities are stored in the source file in UTC. However, to know if they met the deadline, I need it in local time. I do this by creating a function that translates epoch to a regulate dateTime variable. But I see that when we left daylight savings time, things were thrown off by an hour. The command I use is: adjust-dateTime-to-timezone((xs:dateTime("1970-01-01T00:00:00-00:00") + xs:dayTimeDuration(concat("PT", $v, "S"))),xs:dayTimeDuration('-PT4H')) clearly the problem is the last part -PT4H, because dates that fall after the switch back to standard time from daylight savings time should be -PT5H. Is there a better way of doing this? Is there another command that will give me a localized date that takes into account daylight savings time issues? I tried to set a java calendar but I could not make it work. Any help would be gratefully appreciated. Thanks in advance Randy -- Dr. Randall Reed Assistant Professor, Dept. of Philosophy and Religion Appalachian State University

2 1

Re: [basex-talk] Unable to integrate with oxygen to run xquery
by Christian Grün 11 Dec '12

11 Dec '12

Dear Ionela, thanks a lot for your support and your feedback. I decided to also pass this on to our BaseX mailing list; hope it’s ok. All the best, Christian ____________________________________ On Tue, Dec 11, 2012 at 8:52 PM, oXygen XML Editor Support <support(a)oxygenxml.com> wrote: > Hello, > > I just wanted to let you know that this problem has been resolved in the > latest maintenance build of Oxygen 14.1, 2012121012 (released on December > 11th): >> >> Database/XQJ: BaseX via XQJ no longer worked because of conflicting jar >> libraries. > > > You can download it from our web site: > http://www.oxygenxml.com/download.html > > The list of bug-fixes can be found here: > http://www.oxygenxml.com/build_history.html#2012121012 > > You can follow the release/build RSS feed here: > http://www.oxygenxml.com/rssBuildID.xml > > To upgrade: > - For Windows and Linux you can install the new build in the same folder as > the previous installation, it will automatically upgrade it. > Before you upgrade, if you have added files or made changes to any of the > files from the Oxygen installation folder(especially the frameworks folder), > you may want to create a backup of them because they will be overwritten > during the upgrade procedure. Custom frameworks will be preserved but we > recommend backing them up anyway, just to be safe. > - For Mac OS X you will have to install in a different folder and copy any > files you may have changed from the old folder (if any) to the new folder. > The Oxygen preferences will be preserved since they are located elsewhere > (user home folder). > > Let us know if you encounter further problems with this new build. > > Regards, > Ionela > > -- > Ionela Istodor > > oXygen XML Editor and Author Support > > Tel: +1-650-352-1250 ext.301 > > Fax: +40-251-461482 > support(a)oxygenxml.com > http://www.oxygenxml.com > > > --------------------------------------------------------------------------- > This message (including any attachments) contains confidential information > and is intended only for the individual(s) named. If you are not the named > addressee you should not disseminate, distribute or copy this e-mail. Please > notify the sender immediately by e-mail if you have received this e-mail by > mistake and delete this e-mail from your system. Any quotation contained > herein has a validity of 30 days, unless otherwise specified. > > > On 11/26/2012 2:04 PM, Alex Muir wrote: >> >> That works on my side... >> >> Thanks a tonne! >> Cheers >> >> On Mon, Nov 26, 2012 at 5:13 AM, oXygen XML Editor Support >> <support(a)oxygenxml.com <mailto:support@oxygenxml.com>> wrote: >> >> Hi, >> >> We've identified the issue as being class loader related. Oxygen >> comes bundled with an xml-apis.jar which contains javax.xml >> interfaces slightly different from the ones present in the JRE. The >> main problem regards class javax.xml.stream.XMLInputFactory from >> xml-apis.jar. First of all it requires >> com.ctc.wstx.stax.WstxInputFactory (for which you must add >> org.apache.servicemix.bundles.woodstox-3.2.7_1.jar) and it's missing >> method newFactory(). The API basex-xqj-1.2.3.jar is compatible with >> it because it uses newInstance() instead of newFactory(). So if you >> use basex-xqj-1.2.3.jar in the connection and also add >> org.apache.servicemix.bundles.woodstox-3.2.7_1.jar in Oxygen lib, >> the connection will begin to work. >> >> But there should be no need for you to add >> org.apache.servicemix.bundles.woodstox-3.2.7_1.jar so I've attached >> an xml-apis.jar with these interfaces removed. Just replace the >> xml-apis.jar from {oXygenInstallDir}/lib with this new version and >> the BaseX connection will function with just basex-xqj-1.2.3.jar, >> xqj-api-1.0.jar and xqj2-0.1.0.jar added in the connection. No need >> to add org.apache.servicemix.bundles.woodstox-3.2.7_1.jar either. >> >> I've also added an issue to fix this issue on our side in the next >> Oxygen build. >> >> Thank you for following this through! >> Alex >> -- >> Alex Jitianu >> <oXygen/> XML Editor, Schema Editor and XSLT Editor/Debugger >> http://www.oxygenxml.com >> >> >> On 26-Nov-12 5:21 AM, Alex Muir wrote: >>> >>> Hi, >>> >>> I decided to try using basex.exe on windows 7 on a virtual machine >>> with java 7. I get the same results given a fresh download and >>> install of java, oxygen and xqj. I pasted some screen shots so >>> perhaps you'll see the error I'm making or get a better idea of >>> that is not working. When I started the server windows 7 asked >>> permission for the firewall which I gave. I started the server >>> from gui and as well closing the gui and starting from the start >>> menu basex server (start) and both gave the same result. >>> >>> There must be some difference between yours and mine system or >>> some setup thing I'm doing wrong. >>> >>> On linux, I also tried adjusting firewall settings on fedora >>> although this had no affect. >>> >>> If you want to do some remote desktop interaction to check out my >>> system, your highly welcome. >>> >>> Regards >>> Alex >>> >>> >>> On Fri, Nov 23, 2012 at 10:06 PM, Alex Muir <alex.g.muir(a)gmail.com >>> <mailto:alex.g.muir@gmail.com>> wrote: >>> >>> >>> >>> On Fri, Nov 23, 2012 at 9:12 PM, Christian Grün <cg(a)basex.org >>> <mailto:cg@basex.org>> wrote: >>> >>> > Also I wonder why I was getting this error in oxygen? >>> >>> At which stage does this error occur? >>> >>> >>> When running the xquery I would get that error message >>> initially, though it seemed to go away randomly.. the >>> following error is the more common problem. *"E [BaseX7.3] >>> Could not initialize class net.xqj.basex.bin.m"* >>> >>> >>> >>> > Is the woodstox-core-asl-4.1.4.jar required in anyway >>> and if so how? >>> >>> I can’t find this jar in my environment; where did you get >>> it from? >>> >>> >>> >>> Well based on the error message I thought this jar was needed >>> given a search online although I really don't know why I was >>> getting that >>> error message or what to do about it nor if that jar is the >>> required although seemed likely when searching for that error >>> online. >>> >>> From here http://wiki.fasterxml.com/WoodstoxDownload >>> >>> >>> -- >>> - >>> >>> Alex G. Muir >>> Software Engineering Consultant >>> Linkedin Profile : >>> http://ca.linkedin.com/pub/alex-muir/36/ab7/125 >>> Love African Kora Music? Take a moment to listen to Gambia's - >>> Amadu Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com >>> <http://www.bafila.bandcamp.com> Your support keeps Africa's >>> >>> griot tradition alive... Cheers! >>> >>> >>> >>> >>> -- >>> - >>> >>> Alex G. Muir >>> Software Engineering Consultant >>> Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 >>> Love African Kora Music? Take a moment to listen to Gambia's - >>> Amadu Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com >>> <http://www.bafila.bandcamp.com> Your support keeps Africa's griot >>> tradition alive... Cheers! >>> >> >> >> >> >> -- >> - >> >> Alex G. Muir >> Software Engineering Consultant >> Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 >> Love African Kora Music? Take a moment to listen to Gambia's - Amadu >> Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com >> <http://www.bafila.bandcamp.com> Your support keeps Africa's griot >> tradition alive... Cheers! >> >

1 0

Are command scripts optimized before execution? / Optimization of my query
by Ben Companjen 10 Dec '12

10 Dec '12

Hi all, I was wondering whether command scripts are optimized before they are executed. I would think so, because when I had made an error in the, say, sixth command in my script, it was caught before the first command was executed. But more importantly for my application: it seems that combinations of "CLOSE; OPEN <same DB as just closed>" are skipped. I created an XQuery to copy 680000 records from my XML DB to MySQL. These records are MODS, found inside record elements at XPath /file/record. The database is about 2.5 GB and eats even more of my main memory when I try to copy all records in one run. I found out that doing subsequences of 50000 to 10000 records speeds up the process significantly and the server uses a lot less memory. For instance, when I start processing records from 400001, BaseX apparently doesn't keep the first 400000 in memory. But just doing 50000 records in a row, shifting the begin parameter of the subsequence and going on, eventually slows the process from tens to more than 100 records per second to less than one per second, probably because memory consumption goes up to all available memory (I assigned 3 GB to the JVM at startup of the server). Therefore I tried to free some memory by closing the database, re-opening it and then resume processing. But I don't see the memory usage reduction that I hoped for. >From my XQuery mods2sql5.xq: <namespace declarations> declare variable $begin as xs:integer external := 1; declare variable $length as xs:integer external := 50000; <function declarations: prepare statement return sql:execute-prepared(...)> (: main process :) for $record at $rodb in subsequence(/file/record[mods:mods][@id != ""], $begin, $length) <call functions to process records> return <functions' output as sequence> >From my command script mods2sql.bxs: SET QUERYINFO true OPEN monly2 SET BINDINGS $begin=1,$length=50000 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq SET BINDINGS $begin=50001 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq CLOSE OPEN monly2 SET BINDINGS $begin=100001 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq SET BINDINGS $begin=150001 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq SET BINDINGS $begin=200001 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq CLOSE OPEN monly2 ... Are my observations about optimizations of command scripts correct? Is there any way to speed up my process? Thanks for any advice! Regards, Ben

2 1

UPDINDEX fails; command basexserver does not work
by Mahlow Cerstin 10 Dec '12

10 Dec '12

Hi, I just encountered two problems: I'm using a collection created wit UPDINDEX TRUE using BaseX75-20121114. When I tested it, everything worked fine, the index was recreated each time I stored or deleted something. However, today one of the users really worked with the whole webapplication and after half an hour (searching two collections, storing some new nodes and updating some nodes), she got a server time out. In the GUI I could see, that the value of INDEX for "up-to-date" was "false". Which obviously caused the time out. But what could be the reason the index wasn't up to date anymore? How can I make sure the option is working as intended? Anyway, I decided to get the latest snapshot BaseX75-20121204 and re-create the collection with again setting UPDINDEX TRUE. For the webapplication I have a Linux server: Linux version 2.6.18-274.el5 (mockbuild(a)builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Fri Jul 22 04:43:29 EDT 2011 with java version "1.6.0_07" Java(TM) SE Runtime Environment (build 1.6.0_07-b06) Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode) For development, I use a Mac with OSX 10.6.8. with java version "1.6.0_37" Java(TM) SE Runtime Environment (build 1.6.0_37-b06-434-10M3909) Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01-434, mixed mode) So I installed the latest snapshot on both and then tried to stop and restart the server. Which worked as usually fine on the Mac, but on the Linux server, when calling basexserver stop nothing happens. It looks like the script is waiting for some input -- the process does not stop. The old BaseX server seems to be running somehow; I cannot start the server either. Is this a known issue with the latest version? How can I force basexserver to stop and then restart it again? Best regards Cerstin -- Dr. phil. Cerstin Mahlow Universität Basel Deutsches Seminar Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow(a)unibas.ch Web: http://www.oldphras.net

3 5

Re: [basex-talk] suggestions for visualisations etc
by pw＠themail.co.uk 09 Dec '12

09 Dec '12

HI Johannes I was wondering if there were any libraries that would consume JSON-ML and to which I could feed the output from the parameterised RESTXQ API. Do you know of any? Cheers Peter > > > >---- Original Message ---- >From: Johannes.Lichtenberger(a)uni-konstanz.de >To: basex-talk(a)mailman.uni-konstanz.de >Subject: Re: [basex-talk] suggestions for visualisations etc >Date: Sun, 09 Dec 2012 21:22:19 +0100 > >>On 12/09/2012 08:47 PM, pw(a)themail.co.uk wrote: >>> Hello List >>> >>> I am testing the use of BaseX to deliver modest amounts of XML >data >>> through the RESTXQ API. See http://www.semantechs.co.uk for the >>> illustration. >>> >>> I am looking for suggestions for client side graphics packages >that >>> can consume the json-ml provide by BaseX >>> >>> I am also interested in any other suggestions for improving >>> utility/performance >> >>Hello, >> >>what you probably need is a shared presentation model. I wouldn't >use >>any human readable presentation, but a binary protocol to exchange >data, >>for instance the needed node pages to draw the treemap or any other >>visualization. >> >>The "real" model resides on the server, but the presentation model >is >>shared, thus for instance the server knows immediately when you push >a >>button or change any other GUI component state and the server reacts >and >>sends a response. Thus the visualizations are located at the client, >but >>for instance queries are executed on the server and only the results >are >>sent to the client. But as suggested you probably need the Java >POJOs >>(for instance I think ANode and so on). >> >>At least that's something on my radar (for another project, but >should >>be pretty much the same for each Swing/JavaFX Application) and >that's >>how I would do it. >> >>kind regards >>Johannes >> >>_______________________________________________ >>BaseX-Talk mailing list >>BaseX-Talk(a)mailman.uni-konstanz.de >>https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk