- BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] Abstürze basex/DeepFS bei vielen Dateien
by Christian Grün 01 Jul '10

01 Jul '10

Hallo Markus, im aktuellen DeepFS-Code kenne ich mich nicht bis ins Detail aus. Ich leite Deine Mail mal an Bastian L. und Alex weiter; vielleicht haben sie ein paar Ideen? Grüße, Christian ___________________________ 2010/6/17 Markus Roth <markus.roth(a)uni-konstanz.de>: > Hallo Christian, > > ich bin derzeit ein wenig am Rumspielen mit DeepFS und es tritt > kontinuierlich ein Fehler auf (immer zum gleichen Zeitpunkt): too many open > files. > Wenn ich das Limit für offene file handles hochsetze per ulimit -n (standard > in der bash sind 1024), dann kommt der Fehler später, aber trotzdem. > Siehe dazu weiter unten. > Warum sollte baseX aber so viele offene file handles haben? Mit dem > Lesen/Schreiben zusammen sollten es ja nicht so arg viele sein. Oder werden > u.U. (da die Traversierungs-Fkt. wohl rekursiv sein dürfte) je nach Tiefe > des aktuellen Pfades die Ascendants offen gelassen? Da der Fehler in der > Konsole mehrfach ausgegeben wird bis zum Abbruch: gibt es mehrere parallele > "crawler"/Indexer? > Inwiefern die Volltext-Indexierung bzw. inkludieren der Textdaten Einfluß > hat habe ich noch nicht betrachtet (also die folgenden Daten/Fehler sind mit > den Standardvoreinstellungen von DeepFS OHNE Volltext-Indexierung und > Inklusion von Text entstanden). > > > Gruß, > Markus > > > Linux (CentOS 5.4 final) || basex 6.1 & 6.1.5 -> too many open files > ca. 115k Dateien > XML-Nodes: ulimit: > 1,74Mio 1024 > 1,75Mio 2048 > 1,77Mio 4096 > 1,91Mio 16384 > 3,38Mio 200000 > > Windows XP SP3 hatte mit ca. 73k Dateien keine Probleme (konnte auch nicht > rausfinden wie hoch da das OFH ist) > > > Auszug aus der Fehlermeldung: > > java.io.FileNotFoundException: /root/BaseXData/DeepFS/tbli.basex (Too many > open files) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.<init>(FileOutputStream.java:209) > at java.io.FileOutputStream.<init>(FileOutputStream.java:160) > at org.basex.io.DataOutput.<init>(DataOutput.java:49) > at org.basex.io.DataOutput.<init>(DataOutput.java:38) > at org.basex.io.TableDiskAccess.flush(TableDiskAccess.java:77) > at org.basex.data.DiskData.flush(DiskData.java:122) > at org.basex.data.DiskData.cls(DiskData.java:134) > at org.basex.data.Data.close(Data.java:128) > at org.basex.core.proc.Close.close(Close.java:46) > at org.basex.core.proc.Close.run(Close.java:29) > at org.basex.core.Proc.run(Proc.java:139) > at org.basex.core.Proc.run(Proc.java:163) > at org.basex.core.proc.Open.run(Open.java:30) > at org.basex.core.Proc.run(Proc.java:139) > at org.basex.core.Proc.run(Proc.java:163) > at org.basex.core.proc.CreateFS.run(CreateFS.java:54) > at org.basex.core.Proc.run(Proc.java:139) > at org.basex.core.Proc.exec(Proc.java:109) > at org.basex.core.Proc.exec(Proc.java:123) > at org.basex.gui.GUICommands$49.run(GUICommands.java:931) > /root/.basexwin could not be written. > >

2 1

basex
by evan＠wingonwing.com 28 Jun '10

28 Jun '10

Hello, I would like to do some development with basex. I downloaded the project and am looking for a book or documentation to print out. What sort of references do you suggest. I am looking for more documentation than I found on the main page documentation from your website. Thanks, Evan

2 1

Reminder about your invitation from sayali Kulkarni
by sayali Kulkarni (LinkedIn Invitations) 28 Jun '10

28 Jun '10

LinkedIn ------------ This is a reminder that on June 21, sayali Kulkarni sent you an invitation to become part of his or her professional network at LinkedIn. Follow this link to accept sayali Kulkarni's invitation. https://www.linkedin.com/e/87foxj-gaz2pgqu-5s/doi/1401339739/ZCnLLJ4a/gir_8… Signing up is free and takes less than a minute. On June 21, sayali Kulkarni wrote: > To: [basex-talk(a)mailman.uni-konstanz.de] > From: sayali Kulkarni [sayalikulkarni7(a)gmail.com] > Subject: sayali Kulkarni wants to stay in touch on LinkedIn > > I'd like to add you to my professional network on LinkedIn. > > - sayali Kulkarni The only way to get access to sayali Kulkarni's professional network on LinkedIn is through the following link: https://www.linkedin.com/e/87foxj-gaz2pgqu-5s/doi/1401339739/ZCnLLJ4a/gir_8… You can remove yourself from sayali Kulkarni's network at any time. -------------- ------ (c) 2010, LinkedIn Corporation

1 0

Small project based on BaseX
by iorulezz＠gmail.com 27 Jun '10

27 Jun '10

Hello all, we are two master students at the university of Utrecht. We have a DB Architecture class that includes a small project (15%*7.5 ECTS= 1.125 ECTS). So, BaseX looks like a good chance to work on such a project. Our idea so far is based on some benchmarking experiments and small summary on how it works (maybe its relation with Monet) including any experiments we run (5-6 pages). Do you have anything better for us to do (maybe something you can get advantage of yourselves) or any guidelines/hints/whatever? Take into account that out time limit is approximately a week (full-time) and that we cannot do more work that what deserves 2.25ECTS for the 2 of us. I look forward to an answer (so we can decide and start working -the deadline is 9/7-. Best regards, Antonis

1 0

Small project based on BaseX
by iorulezz＠gmail.com 27 Jun '10

27 Jun '10

Hello all, we are two master students at the university of Utrecht. We have a DB Architecture class that includes a small project (15%*7.5 ECTS= 1.125 ECTS). So, BaseX looks like a good chance to work on such a project. Our idea so far is based on some benchmarking experiments and small summary on how it works (maybe its relation with Monet) including any experiments we run (5-6 pages). Do you have anything better for us to do (maybe something you can get advantage of yourselves) or any guidelines/hints/whatever? Take into account that out time limit is approximately a week (full-time) and that we cannot do more work that what deserves 2.25ECTS for the 2 of us. I look forward to an answer (so we can decide and start working -the deadline is 9/7-. Best regards, Antonis

1 0

Small project based on BaseX
by iorulezz＠gmail.com 27 Jun '10

27 Jun '10

Hello all, we are two master students at the university of Utrecht. We have a DB Architecture class that includes a small project (15%*7.5 ECTS= 1.125 ECTS). So, BaseX looks like a good chance to work on such a project. Our idea so far is based on some benchmarking experiments and small summary on how it works (maybe its relation with Monet) including any experiments we run (5-6 pages). Do you have anything better for us to do (maybe something you can get advantage of yourselves) or any guidelines/hints/whatever? Take into account that out time limit is approximately a week (full-time) and that we cannot do more work that what deserves 2.25ECTS for the 2 of us. I look forward to an answer (so we can decide and start working -the deadline is 9/7-. Best regards, Antonis

1 0

Small project based on BaseX
by iorulezz＠gmail.com 27 Jun '10

27 Jun '10

Hello all, we are two master students at the university of Utrecht. We have a DB Architecture class that includes a small project (15%*7.5 ECTS= 1.125 ECTS). So, BaseX looks like a good chance to work on such a project. Our idea so far is based on some benchmarking experiments and small summary on how it works (maybe its relation with Monet) including any experiments we run (5-6 pages). Do you have anything better for us to do (maybe something you can get advantage of yourselves) or any guidelines/hints/whatever? Take into account that out time limit is approximately a week (full-time) and that we cannot do more work that what deserves 2.25ECTS for the 2 of us. I look forward to an answer (so we can decide and start working -the deadline is 9/7-. Best regards, Antonis

1 0

Small project based on BaseX
by iorulezz＠gmail.com 27 Jun '10

27 Jun '10

Hello all, we are two master students at the university of Utrecht. We have a DB Architecture class that includes a small project (15%*7.5 ECTS= 1.125 ECTS). So, BaseX looks like a good chance to work on such a project. Our idea so far is based on some benchmarking experiments and small summary on how it works (maybe its relation with Monet) including any experiments we run (5-6 pages). Do you have anything better for us to do (maybe something you can get advantage of yourselves) or any guidelines/hints/whatever? Take into account that out time limit is approximately a week (full-time) and that we cannot do more work that what deserves 2.25ECTS for the 2 of us. I look forward to an answer (so we can decide and start working -the deadline is 9/7-. Best regards, Antonis

1 0

ADMIN permission needed for basex:read(...)
by Sandra Maria Silcot 27 Jun '10

27 Jun '10

Fellow basex users, Test case: (as 'admin' user:) run file1.xq, containing: let $test := basex:read('/tmp/222.readme') return <test>{$test}</test> Returns, as expected: <test>readme from basex </test> But when run as any other user WITHOUT admin privilige (from the log): RUN /tmp/222 Error: ADMIN permission needed. 7.75 ms Behaviour is the same if the file is copied to be inside the basex home directory e.g. basex:read('222.readme'). I would like to be able to read files in basex queries as a non-admin user. But perhaps the admin restriction on this behaviour is by design? Cheers, Sandra

2 1

possible bug
by szabolcs kurdi 26 Jun '10

26 Jun '10

Hi! I'm not 100% sure if it is a bug in BaseX or it's just me, but maybe it's better to post it. The error below happens after importing lots of xml files (~100mb) and then calling Optimize - on Arch linux, inside Virtualbox. The same process finishes without problems on Windows XP. 0 [SwingWorker-pool-3-thread-1] ERROR com.rosamez.jdvdkat.DbManager - Could not optimize the db. Possible bug? Feedback is welcome: info(a)basex.org BaseX 6.1.7: java.lang.ArrayIndexOutOfBoundsException: 0 org.basex.util.Num.size(Num.java:84) org.basex.index.ValueBuilder.writeWithNum(ValueBuilder.java:176) org.basex.index.ValueBuilder.merge(ValueBuilder.java:125) org.basex.index.ValueBuilder.build(ValueBuilder.java:75) org.basex.index.ValueBuilder.build(ValueBuilder.java:1) org.basex.core.proc.ACreate.index(ACreate.java:116) org.basex.core.proc.ACreate.index(ACreate.java:91) org.basex.core.proc.Optimize.stats(Optimize.java:90) org.basex.core.proc.Optimize.run(Optimize.java:35) org.basex.core.Proc.run(Proc.java:140) org.basex.core.Proc.exec(Proc.java:110) org.basex.core.Proc.execute(Proc.java:67) com.rosamez.jdvdkat.DbManager.optimize(DbManager.java:240) com.rosamez.jdvdkat.AddAllFilesWorker.doInBackground(AddAllFilesWorker.java:34) com.rosamez.jdvdkat.AddAllFilesWorker.doInBackground(AddAllFilesWorker.java:14) javax.swing.SwingWorker$1.call(SwingWorker.java:277) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) javax.swing.SwingWorker.run(SwingWorker.java:316) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:619) Thanks! Szabolcs

2 4