Fwd: Re: Database creation from a directory tree - BaseX-Talk - mailman.uni-konstanz.de

31 Dec 2010

      Forwarding to mailing-list...maybe interesting for others.
-------- Original-Nachricht --------
Betreff: 	Re: [basex-talk] Database creation from a directory tree
Datum: 	Fri, 31 Dec 2010 16:41:05 +0100
Von: 	Andreas Weiler andreas.weiler@uni-konstanz.de
An: 	david@leighweb.com
Hi David,
thanks a lot for your tests.
I couldnt fix the gui bug yet...but i can help you with command line...
There is a createfilter option...it can be set with:
set createfilter *
so all files in the folder are recognized for creating a database.
I hope this works for now,
Happy new year,
Andreas
Am 31.12.10 15:13, schrieb David leigh:
...
Andreas,
I just did an interesting test (using the version that the windows 
installer installs: 6.3.4).
I copied my directory tree of files and in the new copy, I gave all my 
data files an extension of "xml" (previously they had NO extension).  
NOW, both with the command line and the GUI, the full database is 
created without any problem.  To double check, I went back to the old 
directory with no extensions and created a new database using a filter 
of "*.*" and only 14 documents were created in the database (instead 
of 5000+).
Conclusions:
1.In the command line you must have input documents with an extension 
of "xml" since there is no visible way to provide a filter.
2.The GUI interfaces differently than the command line with the basex 
core since it can receive a filter as well as a path in the database 
creation process
3.The GUI filter functionality in 6.3.4 is different than in 6.1...for 
me it is "broken" vs. how it works in 6.1
So, I think for my project to work completely, I need new 
functionality: a file name filter that works correctly on the CREATE 
DB statement in the command line interface.  Does this sound correct 
to you?
Unfortunately, I don't have control over the extension names of my 
input files as they are created by another piece of software.   Until 
I can do a create database in the command line, I'll have to do manual 
gui-based (6.1 version) refreshes of the data or some sort of long 
copy/rename process in the background.
Thanks!
David
DavidEmailSig mailto:david@leighweb.com
*From:*Andreas Weiler [mailto:andreas.weiler@uni-konstanz.de]
*Sent:* Wednesday, December 29, 2010 11:26 AM
*To:* david@leighweb.com; BaseX-Talk@mailman.uni-konstanz.de
*Subject:* Re: [basex-talk] Database creation from a directory tree
Hi David,
the syntax of the create command changed between the releases of 6.1 
and 6.3.4.
In release 6.1 you have to use:
create db C:\Users\David\Documents\OpenSong\songpacks\Songs songs
Kind regards,
Andreas
Am 28.12.10 18:00, schrieb David leigh:
Here are my command line results from 6.3.4:
C:\Users\David\Documents\OpenSong\OpenSongSearch\BaseX>java -cp 
BaseX.jar org.basex.BaseXClient -Uadmin -Padmin
BaseX 6.3.5 [Client]
Try "help" to get more information.
...
drop db songs
Database 'songs' was dropped.
...
create db song
Database 'song' created in 1683.61 ms.
...
info index
Tags

Structure: Hash

Entries: 0

Attributes

Structure: Hash

Entries: 0

Text Index

Structure: Binary tree

Size: 4 Bytes

Entries: 0

Attribute Index

Structure: Binary tree

Size: 4 Bytes

Entries: 0

Full-Text Index

Not available

Path Summary
doc() 1x
...
info table
Stopped at line 1, column 10:
Syntax: INFO ([DATABASE|INDEX|STORAGE])
Show information on current database.
Shows information on the currently opened database:

no argument: show global information

DATABASE: shows database information

INDEX: shows index information

STORAGE [start end] | [query]: show internal database table

...
info storage
PRE  DIS  SIZ  ATS  NS  KIND  CONTENT

0    1    1    1   0  DOC   song
...
info
General Information
Database Path: C:\Users\David\Documents\OpenSong\OpenSongSearch\BaseX\data
Used Main Memory: 895 KB
Database Creation
Whitespace Chopping: ON
Entity Parsing: OFF
Indexes
Path Summary: ON
Text Index: ON
Attribute Index: ON
Full-Text Index: OFF
...
So, for 6.3.4, it creates from the command line, but it's empty.  The 
root of that path has NO files in it.  The 5861 files are in all the 
subdirectories.
Then, when I try the same syntax with 6.1, I get the following results:
...
create db songs C:\Users\David\Documents\OpenSong\songpacks\Songs
Stopped at line 1, column 17:
Syntax: CREATE [DB|COLL|FS|INDEX|USER] [...]
Create database, index or user.
Creates a new database, index or user:

DB [path] [name?]:
creates the database [name] for the file or directory [path]

COLL [name]:
creates an empty database [name]

INDEX [TEXT|ATTRIBUTE|FULLTEXT|PATH]:
creates the specified index

FS [path] [name] ([mountpoint] [backingstore]):
creates filesystem database [name] for [path]
(using [mountpoint] and [backingstore] if FUSE is activated)

USER [name] [password?]:
creates the specified user

Summary:
6.1 GUI -- works as I want it
6.1 command line -- can't get it to create
6.3.4 GUI -- creates but with only 14 entries
6.3.4 command line -- creates, but it's empty
I don't mind staying with 6.1, but I DO want to do everything with the 
command line.
Is the ADD syntax the same, in that I can simply specify the path and 
it will recurse the whole directory tree?
Thanks!!!
David
DavidEmailSig mailto:david@leighweb.com
*From:*Andreas Weiler [mailto:andreas.weiler@uni-konstanz.de]
*Sent:* Tuesday, December 28, 2010 01:29 PM
*To:* david@leighweb.com mailto:david@leighweb.com
*Cc:* basex-talk@mailman.uni-konstanz.de 
mailto:basex-talk@mailman.uni-konstanz.de
*Subject:* Re: [basex-talk] Database creation from a directory tree
Additional question: That was in the GUI.  I'd like to do the same
thing using the command line with the .JAR file, but somewhere my
syntax is not right.  Additionally, all the XML files have NO
EXTENSION so I don't know how to specify that either in the
command line.  So, any help in that area would also be appreciated.

Could you provide your syntax?
The syntax should be:
create db dbname path/to/folder
The missing extension shouldnt be a problem, i just tested it.
In the GUI in 6.1, I can create a database from a whole directory 
hierarchy of 5,681 individual XML files.  When I point to the same 
source file directory and using the 6.3.4 GUI (which I installed with 
a pre-release version of the installer), it only processes 14 files.  
No error message (that I saw) was produced.  The behavior that I'm 
wanting is what 6.1 does.   Glad to help in solving the problem.
We will check for modifications between the releases. As quick 
workaround you could try to create an empty database and then add the 
documents with the Database/Add documents menu.
Kind regards,
Andreas
Am 28.12.10 12:55, schrieb David leigh:
I've just installed 6.3.4 and I've noticed a difference from version 6.1.
In the GUI in 6.1, I can create a database from a whole directory 
hierarchy of 5,681 individual XML files.  When I point to the same 
source file directory and using the 6.3.4 GUI (which I installed with 
a pre-release version of the installer), it only processes 14 files.  
No error message (that I saw) was produced.  The behavior that I'm 
wanting is what 6.1 does.   Glad to help in solving the problem.
Additional question: That was in the GUI.  I'd like to do the same 
thing using the command line with the .JAR file, but somewhere my 
syntax is not right.  Additionally, all the XML files have NO 
EXTENSION so I don't know how to specify that either in the command 
line.  So, any help in that area would also be appreciated.
Thank you,
David
DavidEmailSig mailto:david@leighweb.com

BaseX-Talk mailing list
BaseX-Talk@mailman.uni-konstanz.de  mailto:BaseX-Talk@mailman.uni-konstanz.de
https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk