Hi,
I encountered a peculiar error with a query using a stopwords file in the context of a full text search. The query joins two XML databases: CT.gov (containing 86635503 nodes) and the 2015 MeSH dictionary (containing 12064461 nodes). I am debugging using CT.gov trial NCT00303472, hardcoded in the ‘where' clause of the following query:
let $trees := db:open('MeSH')/DescriptorRecordSet/DescriptorRecord for $article in db:open('CTGov')/clinical_study where $article/id_info/nct_id = 'NCT00303472' let $mesh := $article/condition_browse/mesh_term let $tn1 := $trees[DescriptorName/String contains text { $mesh }] let $tn2 := $trees[DescriptorName/String contains text { $mesh } using stop words at "/Volumes/Extra/Documents/Standards/MeSH/stopwords.txt"]/TreeNumberList/TreeNumber return <match> { $article/id_info/nct_id, $mesh, $tn2 } </match>
When the return clause contains the variable $tn2 (i.e., using stopwords - as shown above) a Java NullPointerException is generated (see the stack trace below). However, when only $tn1 is returned there is no problem (the code for $tn2 is removed by the optimizer).
The issue is related to a specific stopword (“syndrome”). When the stopword is removed from the file the exception does not occur. Surprisingly, when the stopword is in uppercase (“Syndrome”) the issue does not occur - even though the target MeSH term in this CT.gov trial is in uppercase, that is
<mesh_term>Syndrome</mesh_term>
Am I doing something wrong, or is this a real bug in BaseX? If the former, please suggest a workaround as I would like to filter out generic MeSH terms that match the stopwords before any further processing (I removed a lot of code from the above query to make it easier to debug).
Thanks, Ron
Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 8.2 Java: Oracle Corporation, 1.8.0_20 OS: Mac OS X, x86_64 Stack Trace: java.lang.NullPointerException at org.basex.query.expr.ft.FTWords$1.next(FTWords.java:166) at org.basex.query.expr.ft.FTIndexAccess$1.next(FTIndexAccess.java:48) at org.basex.query.expr.ft.FTIndexAccess$1.next(FTIndexAccess.java:45) at org.basex.query.iter.Iter.value(Iter.java:53) at org.basex.query.expr.ParseExpr.value(ParseExpr.java:67) at org.basex.query.QueryContext.value(QueryContext.java:421) at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:41) at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:22) at org.basex.query.QueryContext.iter(QueryContext.java:410) at org.basex.query.expr.List$1.next(List.java:133) at org.basex.query.expr.constr.Constr.add(Constr.java:70) at org.basex.query.expr.constr.CElem.item(CElem.java:92) at org.basex.query.expr.constr.CElem.item(CElem.java:23) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:43) at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99) at org.basex.query.MainModule$1.next(MainModule.java:114) at org.basex.query.QueryContext.cache(QueryContext.java:660) at org.basex.query.QueryProcessor.cache(QueryProcessor.java:103) at org.basex.core.cmd.AQuery.query(AQuery.java:83) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:398) at org.basex.core.Command.execute(Command.java:100) at org.basex.gui.GUI.exec(GUI.java:472) at org.basex.gui.GUI.access$400(GUI.java:43) at org.basex.gui.GUI$7.run(GUI.java:412) Compiling: - pre-evaluating db:open("MeSH") - pre-evaluating db:open("CTGov") - inlining $trees_0 - applying full-text index for { $mesh_2 } using language 'English' - applying full-text index for { $mesh_2 } using language 'English' - inlining $tn2_4 - removing variable $tn1_3 - applying text index for "NCT00303472" - rewriting where clause(s) Query: let $trees := db:open('MeSH')/DescriptorRecordSet/DescriptorRecord for $article in db:open('CTGov')/clinical_study where $article/id_info/nct_id = 'NCT00303472' let $mesh := $article/condition_browse/mesh_term let $tn1 := $trees[DescriptorName/String contains text { $mesh }] let $tn2 := $trees[DescriptorName/String contains text { $mesh } using stop words at "/Volumes/Extra/Documents/Standards/MeSH/stopwords.txt"]/TreeNumberList/TreeNumber return <match> { $article/id_info/nct_id, $mesh, $tn2 } </match> Optimized Query: for $article_1 in db:text("CTGov", "NCT00303472")/parent::*:nct_id/parent::*:id_info/parent::*:clinical_study let $mesh_2 := $article_1/*:condition_browse/*:mesh_term return element match { (($article_1/*:id_info/*:nct_id, $mesh_2, ft:search("MeSH", { $mesh_2 } using language 'English')/parent::*:String/parent::*:DescriptorName/parent::*:DescriptorRecord/TreeNumberList/TreeNumber)) } Query plan: <QueryPlan compiled="true"> <GFLWOR> <For> <Var name="$article" id="1"/> <IterPath> <ValueAccess data="CTGov" type="TEXT" name="*:nct_id"> <Str value="NCT00303472" type="xs:string"/> </ValueAccess> <IterStep axis="parent" test="*:id_info"/> <IterStep axis="parent" test="*:clinical_study"/> </IterPath> </For> <Let> <Var name="$mesh" id="2"/> <IterPath> <VarRef> <Var name="$article" id="1"/> </VarRef> <IterStep axis="child" test="*:condition_browse"/> <IterStep axis="child" test="*:mesh_term"/> </IterPath> </Let> <CElem> <QNm value="match" type="xs:QName"/> <List> <IterPath> <VarRef> <Var name="$article" id="1"/> </VarRef> <IterStep axis="child" test="*:id_info"/> <IterStep axis="child" test="*:nct_id"/> </IterPath> <VarRef> <Var name="$mesh" id="2"/> </VarRef> <CachedPath> <FTIndexAccess data="MeSH"> <FTWords> <VarRef> <Var name="$mesh" id="2"/> </VarRef> </FTWords> </FTIndexAccess> <IterStep axis="parent" test="*:String"/> <IterStep axis="parent" test="*:DescriptorName"/> <IterStep axis="parent" test="*:DescriptorRecord"/> <IterStep axis="child" test="TreeNumberList"/> <IterStep axis="child" test="TreeNumber"/> </CachedPath> </List> </CElem> </GFLWOR> </QueryPlan>
Ron Katriel, Ph.D. | Senior Data Scientist | Medidata Solutions 350 Hudson Street, 7th Floor, New York, NY 10014 rkatriel@mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800