multi-language full-text indexing
Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g. CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( (path_a)/PART_A, (path_b)/ PART_B,… ) Kind regards, Goetz -----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing
It is desirable to have documents indexed by locale-specific parts, e.g.
I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture. As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). Hope this helps, Christian [1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg04837.htm... [2] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06089.htm...
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH (
(path_a)/LOCALIZED_PART_A[@LANG=$lang],
(path_b)/LOCALIZED_PART_B[@LG=$lang],…
) FOR LANGUAGE $lang IN (
BG,
DN,
DE WITH STOPWORDS filepath_de WITH STEM = YES,
EN WITH STOPWORDS filepath_en,
FR, …
) [USING language_code_map]
and then to write full-text retrieval queries with a clause such as ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller and full-text retrieval therefore much faster. The language codes would be mapped somehow to standard values recognized by BaseX in the language_code_map file.
Are there any efforts towards such a feature?
Reminds me of an old GitHub issue.. I have added a link to your request: https://github.com/BaseXdb/basex/issues/59. On Wed, Apr 22, 2015 at 11:35 AM, Goetz Heller <heller@hellerim.de> wrote:
Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g.
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( (path_a)/PART_A, (path_b)/ PART_B,… )
Kind regards,
Goetz
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing
It is desirable to have documents indexed by locale-specific parts, e.g.
I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture.
As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]).
Hope this helps, Christian
[1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg04837.htm... [2] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06089.htm...
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH (
(path_a)/LOCALIZED_PART_A[@LANG=$lang],
(path_b)/LOCALIZED_PART_B[@LG=$lang],…
) FOR LANGUAGE $lang IN (
BG,
DN,
DE WITH STOPWORDS filepath_de WITH STEM = YES,
EN WITH STOPWORDS filepath_en,
FR, …
) [USING language_code_map]
and then to write full-text retrieval queries with a clause such as ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller and full-text retrieval therefore much faster. The language codes would be mapped somehow to standard values recognized by BaseX in the language_code_map file.
Are there any efforts towards such a feature?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi, I just want to say that for the dictionary that I used BaseX for, having a multi-lingual full text would have been very nice. Bar that a partial index based on certain rules the user supplies would have also been nice. For instance, being able to distinguish between á, a, and ā in a word. In early Irish textual criticism, length marks are often added by text editors with a macron to denote a long vowel that has been idenified by the editor but is not in the original text. Being able to say: "build an index with á and a but not ā" would be helpful. I would suggest as a first pass building the index by using xml:lang attributes to determine what stemmer to use, etc. If the document has supplied them, you could use them to build the indices differently based on it. All the best, Chris On Wed, Apr 22, 2015 at 11:35:48AM +0200, Goetz Heller wrote:
Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g.
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( (path_a)/PART_A, (path_b)/ PART_B,… )
Kind regards,
Goetz
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing
It is desirable to have documents indexed by locale-specific parts, e.g.
I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture.
As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]).
Hope this helps, Christian
[1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg04837.htm... [2] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06089.htm...
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH (
(path_a)/LOCALIZED_PART_A[@LANG=$lang],
(path_b)/LOCALIZED_PART_B[@LG=$lang],…
) FOR LANGUAGE $lang IN (
BG,
DN,
DE WITH STOPWORDS filepath_de WITH STEM = YES,
EN WITH STOPWORDS filepath_en,
FR, …
) [USING language_code_map]
and then to write full-text retrieval queries with a clause such as ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller and full-text retrieval therefore much faster. The language codes would be mapped somehow to standard values recognized by BaseX in the language_code_map file.
Are there any efforts towards such a feature?
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iF4EAREIAAYFAlU4EIcACgkQDjE+CSbP7HrWyAEAsW698gwkbKrtEb9Vkv7S1aV/ r5YyIp/UKv9k8gYxqboA/0+oesCxM6K2dZGEfIolUJG+x3vhxMfocY+QGpwjtLhy =Zq8Q -----END PGP SIGNATURE-----
Chris, Thanks for your feedback. Yes, I see that there is a lot of demand for a more customizable full-text index. Did you already try to build some additional index databases, based on the rules you were listing here? It's not as comfortable as a tightly coupled full-text index, but the more use case I get to hear of, the more I wonder if we could at all manage to satisfy everyone's needs.. Cheers, Christian
Wed, Apr 22, 2015 at 11:20 PM, Chris Yocum <cyocum@gmail.com> wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
I just want to say that for the dictionary that I used BaseX for, having a multi-lingual full text would have been very nice. Bar that a partial index based on certain rules the user supplies would have also been nice. For instance, being able to distinguish between á, a, and ā in a word. In early Irish textual criticism, length marks are often added by text editors with a macron to denote a long vowel that has been idenified by the editor but is not in the original text. Being able to say: "build an index with á and a but not ā" would be helpful.
I would suggest as a first pass building the index by using xml:lang attributes to determine what stemmer to use, etc. If the document has supplied them, you could use them to build the indices differently based on it.
All the best, Chris
On Wed, Apr 22, 2015 at 11:35:48AM +0200, Goetz Heller wrote:
Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g.
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( (path_a)/PART_A, (path_b)/ PART_B,… )
Kind regards,
Goetz
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing
It is desirable to have documents indexed by locale-specific parts, e.g.
I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture.
As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]).
Hope this helps, Christian
[1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg04837.htm... [2] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06089.htm...
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH (
(path_a)/LOCALIZED_PART_A[@LANG=$lang],
(path_b)/LOCALIZED_PART_B[@LG=$lang],…
) FOR LANGUAGE $lang IN (
BG,
DN,
DE WITH STOPWORDS filepath_de WITH STEM = YES,
EN WITH STOPWORDS filepath_en,
FR, …
) [USING language_code_map]
and then to write full-text retrieval queries with a clause such as ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller and full-text retrieval therefore much faster. The language codes would be mapped somehow to standard values recognized by BaseX in the language_code_map file.
Are there any efforts towards such a feature?
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iF4EAREIAAYFAlU4EIcACgkQDjE+CSbP7HrWyAEAsW698gwkbKrtEb9Vkv7S1aV/ r5YyIp/UKv9k8gYxqboA/0+oesCxM6K2dZGEfIolUJG+x3vhxMfocY+QGpwjtLhy =Zq8Q -----END PGP SIGNATURE-----
Hi Christian, No problem. I am always happy to help. I did not try that as I did not have the time to implement something like that for the project. Also, on my machine and on the vps we were using at least, it was fast enough for most uses. Also, I told the user about the problem as "using diacritics" was an "all or nothing" setting and there was not much I could do to make it more granular. They were content with that but I thought as the subject came up that I would mention something. Please let me know if you need any more information. All the best, Chris On Wed, Apr 22, 2015 at 10:24 PM, Christian Grün <christian.gruen@gmail.com> wrote:
Chris,
Thanks for your feedback.
Yes, I see that there is a lot of demand for a more customizable full-text index. Did you already try to build some additional index databases, based on the rules you were listing here? It's not as comfortable as a tightly coupled full-text index, but the more use case I get to hear of, the more I wonder if we could at all manage to satisfy everyone's needs..
Cheers, Christian
Wed, Apr 22, 2015 at 11:20 PM, Chris Yocum <cyocum@gmail.com> wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
I just want to say that for the dictionary that I used BaseX for, having a multi-lingual full text would have been very nice. Bar that a partial index based on certain rules the user supplies would have also been nice. For instance, being able to distinguish between á, a, and ā in a word. In early Irish textual criticism, length marks are often added by text editors with a macron to denote a long vowel that has been idenified by the editor but is not in the original text. Being able to say: "build an index with á and a but not ā" would be helpful.
I would suggest as a first pass building the index by using xml:lang attributes to determine what stemmer to use, etc. If the document has supplied them, you could use them to build the indices differently based on it.
All the best, Chris
On Wed, Apr 22, 2015 at 11:35:48AM +0200, Goetz Heller wrote:
Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g.
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( (path_a)/PART_A, (path_b)/ PART_B,… )
Kind regards,
Goetz
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing
It is desirable to have documents indexed by locale-specific parts, e.g.
I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture.
As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]).
Hope this helps, Christian
[1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg04837.htm... [2] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06089.htm...
CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH (
(path_a)/LOCALIZED_PART_A[@LANG=$lang],
(path_b)/LOCALIZED_PART_B[@LG=$lang],…
) FOR LANGUAGE $lang IN (
BG,
DN,
DE WITH STOPWORDS filepath_de WITH STEM = YES,
EN WITH STOPWORDS filepath_en,
FR, …
) [USING language_code_map]
and then to write full-text retrieval queries with a clause such as ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller and full-text retrieval therefore much faster. The language codes would be mapped somehow to standard values recognized by BaseX in the language_code_map file.
Are there any efforts towards such a feature?
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iF4EAREIAAYFAlU4EIcACgkQDjE+CSbP7HrWyAEAsW698gwkbKrtEb9Vkv7S1aV/ r5YyIp/UKv9k8gYxqboA/0+oesCxM6K2dZGEfIolUJG+x3vhxMfocY+QGpwjtLhy =Zq8Q -----END PGP SIGNATURE-----
participants (4)
-
Chris Yocum -
Christian Grün -
Christopher Yocum
-
Goetz Heller