Hi.
Although I was trying implements of Japanese Lexer to 6.8 beta, I have noticed that the result of a ft:mark() function shifts. 55 line of https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/query/u...
final FTLexer lex = new FTLexer(ftopt).sc().init(nd.atom());
FTLexer will generate the tokenizer of the default language instead of an original language, if it does not perform in this way.
How is it? In code which I am correcting, ft:mark() operated normally by carrying out like this.
thanks.
-- Toshio HIRAI
Toshio HIRAI,
thanks for developing the Japanese Lexer for BaseX! I agree that the code snippet in question seems to lack a reference to the original fulltext options (it could be passed on to the DataFTBuilder constructor as well). I can have a second look at this.
Have you already been successful with 6.7.1, or is it the first time you're adapting the GUI visualization to your tokenizer?
I've added a GitHub issue on that..
https://github.com/BaseXdb/basex/issues/208
..and I'll keep you updated, Christian ___________________________
On Mon, Sep 26, 2011 at 9:41 AM, Toshio HIRAI toshio.hirai@gmail.com wrote:
Hi. Although I was trying implements of Japanese Lexer to 6.8 beta, I have noticed that the result of a ft:mark() function shifts. 55 line of https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/query/u...
final FTLexer lex = new FTLexer(ftopt).sc().init(nd.atom()); FTLexer will generate the tokenizer of the default language instead of an original language, if it does not perform in this way. How is it? In code which I am correcting, ft:mark() operated normally by carrying out like this. thanks. -- Toshio HIRAI
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Toshio HIRAI