Optimize database never returns, leaves database in "opened by another process" state
Using Basex 10.7 on Linux. I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk). I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc. Optimization code is: try { if (db:exists($database)) then ( util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``), db:optimize($database, true(), $dbadmin:dbOptimizeOptions) ) else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``) } catch * { util:logToConsole( 'dbadmin:optimizeDatabase', ``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``, 'error') } And the optimization options are: declare variable $dbadmin:dbOptimizeOptions as map(*) := (: Turn on all the indexes :) map { 'attrindex' : true(), 'tokenindex' : true(), 'textindex' : true(), 'ftindex' : true() }; This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering: A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up). Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
…this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished… [1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com> wrote:
Using Basex 10.7 on Linux.
I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk).
I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc.
Optimization code is:
try {
if (db:exists($database))
then
(
util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``),
db:optimize($database, true(), $dbadmin:dbOptimizeOptions)
)
else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``)
} catch * {
util:logToConsole(
'dbadmin:optimizeDatabase',
``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``,
'error')
}
And the optimization options are:
declare variable $dbadmin:dbOptimizeOptions as map(*) :=
(: Turn on all the indexes :)
map {
'attrindex' : true(),
'tokenindex' : true(),
'textindex' : true(),
'ftindex' : true()
};
This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering:
A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up).
Thanks,
Eliot
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
I’ll check out the snapshot. I was starting to dig into the Java code yesterday but was first testing if I needed the optimization at all in this case, which it looks like I don’t (the consumer of the stored docs doesn’t depend on indexing to find things). Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com> Date: Wednesday, February 28, 2024 at 9:36 AM To: Eliot Kimber <eliot.kimber@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state [External Email] ________________________________ …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished…[1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished… [1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> wrote: Using Basex 10.7 on Linux. I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk). I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc. Optimization code is: try { if (db:exists($database)) then ( util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``), db:optimize($database, true(), $dbadmin:dbOptimizeOptions) ) else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``) } catch * { util:logToConsole( 'dbadmin:optimizeDatabase', ``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``, 'error') } And the optimization options are: declare variable $dbadmin:dbOptimizeOptions as map(*) := (: Turn on all the indexes :) map { 'attrindex' : true(), 'tokenindex' : true(), 'textindex' : true(), 'ftindex' : true() }; This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering: A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up). Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
I’m trying to compile the latest code but “mvn clean install” fails on failure to download some dependencies: [INFO] --------------------------< org.basex:basex >--------------------------- [INFO] Building BaseX Core 11.0-SNAPSHOT [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0... [WARNING] The POM for jp.sourceforge.igo:igo:jar:0.4.3 is missing, no dependency information available Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme... [WARNING] The POM for org.apache:lucene-stemmers:jar:3.4.0 is missing, no dependency information available Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0... Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.589 s [INFO] Finished at: 2024-02-29T02:34:11-06:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project basex: Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): Could not find artifact jp.sourceforge.igo:igo:jar:0.4.3 in devsnc-mirror (http://nexus.proxy.devsnc.com/content/groups/public) -> [Help 1] And then: [ERROR] Failed to execute goal on project basex: Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): jp.sourceforge.igo:igo:jar:0.4.3 was not found in http://nexus.proxy.devsnc.com/content/groups/public during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of devsnc-mirror has elapsed or updates are forced -> [Help 1] Where ‘nexus.proxy.devsnc.com’ is our internal Maven proxy. This is after removing my local maven repository cache. Before doing that, I got the “present but not available” failure on these libraries. The jars are present in my original .m2/repository but not in a newly-created repository (having moved the original one out of the way). I’m using maven 3.9.6. As far as I can tell from my research, this is a change in behavior from Maven 2 to Maven 3 where it refuses to use local dependencies that were fetched as a side effect of some other dependency that cannot now be fetched. But I’m not finding any maven options that will work around this failure. Any idea how I can move past this? Or is there a nightly build somewhere I can download? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Wednesday, February 28, 2024 at 10:01 AM To: Christian Grün <christian.gruen@gmail.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state I’ll check out the snapshot. I was starting to dig into the Java code yesterday but was first testing if I needed the optimization at all in this case, which it looks like I don’t (the consumer of the stored docs doesn’t depend on indexing to find things). Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com> Date: Wednesday, February 28, 2024 at 9:36 AM To: Eliot Kimber <eliot.kimber@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state [External Email] ________________________________ …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished…[1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished… [1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> wrote: Using Basex 10.7 on Linux. I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk). I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc. Optimization code is: try { if (db:exists($database)) then ( util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``), db:optimize($database, true(), $dbadmin:dbOptimizeOptions) ) else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``) } catch * { util:logToConsole( 'dbadmin:optimizeDatabase', ``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``, 'error') } And the optimization options are: declare variable $dbadmin:dbOptimizeOptions as map(*) := (: Turn on all the indexes :) map { 'attrindex' : true(), 'tokenindex' : true(), 'textindex' : true(), 'ftindex' : true() }; This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering: A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up). Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Found the latest build at https://files.basex.org/releases/latest/ Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Thursday, February 29, 2024 at 2:45 AM To: Christian Grün <christian.gruen@gmail.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state I’m trying to compile the latest code but “mvn clean install” fails on failure to download some dependencies: [INFO] --------------------------< org.basex:basex >--------------------------- [INFO] Building BaseX Core 11.0-SNAPSHOT [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0... [WARNING] The POM for jp.sourceforge.igo:igo:jar:0.4.3 is missing, no dependency information available Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme... [WARNING] The POM for org.apache:lucene-stemmers:jar:3.4.0 is missing, no dependency information available Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0... Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.589 s [INFO] Finished at: 2024-02-29T02:34:11-06:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project basex: Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): Could not find artifact jp.sourceforge.igo:igo:jar:0.4.3 in devsnc-mirror (http://nexus.proxy.devsnc.com/content/groups/public) -> [Help 1] And then: [ERROR] Failed to execute goal on project basex: Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): jp.sourceforge.igo:igo:jar:0.4.3 was not found in http://nexus.proxy.devsnc.com/content/groups/public during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of devsnc-mirror has elapsed or updates are forced -> [Help 1] Where ‘nexus.proxy.devsnc.com’ is our internal Maven proxy. This is after removing my local maven repository cache. Before doing that, I got the “present but not available” failure on these libraries. The jars are present in my original .m2/repository but not in a newly-created repository (having moved the original one out of the way). I’m using maven 3.9.6. As far as I can tell from my research, this is a change in behavior from Maven 2 to Maven 3 where it refuses to use local dependencies that were fetched as a side effect of some other dependency that cannot now be fetched. But I’m not finding any maven options that will work around this failure. Any idea how I can move past this? Or is there a nightly build somewhere I can download? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Wednesday, February 28, 2024 at 10:01 AM To: Christian Grün <christian.gruen@gmail.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state I’ll check out the snapshot. I was starting to dig into the Java code yesterday but was first testing if I needed the optimization at all in this case, which it looks like I don’t (the consumer of the stored docs doesn’t depend on indexing to find things). Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com> Date: Wednesday, February 28, 2024 at 9:36 AM To: Eliot Kimber <eliot.kimber@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state [External Email] ________________________________ …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished…[1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished… [1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> wrote: Using Basex 10.7 on Linux. I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk). I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc. Optimization code is: try { if (db:exists($database)) then ( util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``), db:optimize($database, true(), $dbadmin:dbOptimizeOptions) ) else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``) } catch * { util:logToConsole( 'dbadmin:optimizeDatabase', ``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``, 'error') } And the optimization options are: declare variable $dbadmin:dbOptimizeOptions as map(*) := (: Turn on all the indexes :) map { 'attrindex' : true(), 'tokenindex' : true(), 'textindex' : true(), 'ftindex' : true() }; This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering: A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up). Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Using the latest build, 11.0 beta 17d8426, the prof:dump() function is reported as an unknown function. What replaces it (or where can I find the V11 docs)? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Thursday, February 29, 2024 at 3:15 AM To: Christian Grün <christian.gruen@gmail.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state Found the latest build at https://files.basex.org/releases/latest/ Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Thursday, February 29, 2024 at 2:45 AM To: Christian Grün <christian.gruen@gmail.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state I’m trying to compile the latest code but “mvn clean install” fails on failure to download some dependencies: [INFO] --------------------------< org.basex:basex >--------------------------- [INFO] Building BaseX Core 11.0-SNAPSHOT [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0... [WARNING] The POM for jp.sourceforge.igo:igo:jar:0.4.3 is missing, no dependency information available Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme... [WARNING] The POM for org.apache:lucene-stemmers:jar:3.4.0 is missing, no dependency information available Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0... Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.589 s [INFO] Finished at: 2024-02-29T02:34:11-06:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project basex: Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): Could not find artifact jp.sourceforge.igo:igo:jar:0.4.3 in devsnc-mirror (http://nexus.proxy.devsnc.com/content/groups/public) -> [Help 1] And then: [ERROR] Failed to execute goal on project basex: Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): jp.sourceforge.igo:igo:jar:0.4.3 was not found in http://nexus.proxy.devsnc.com/content/groups/public during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of devsnc-mirror has elapsed or updates are forced -> [Help 1] Where ‘nexus.proxy.devsnc.com’ is our internal Maven proxy. This is after removing my local maven repository cache. Before doing that, I got the “present but not available” failure on these libraries. The jars are present in my original .m2/repository but not in a newly-created repository (having moved the original one out of the way). I’m using maven 3.9.6. As far as I can tell from my research, this is a change in behavior from Maven 2 to Maven 3 where it refuses to use local dependencies that were fetched as a side effect of some other dependency that cannot now be fetched. But I’m not finding any maven options that will work around this failure. Any idea how I can move past this? Or is there a nightly build somewhere I can download? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Wednesday, February 28, 2024 at 10:01 AM To: Christian Grün <christian.gruen@gmail.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state I’ll check out the snapshot. I was starting to dig into the Java code yesterday but was first testing if I needed the optimization at all in this case, which it looks like I don’t (the consumer of the stored docs doesn’t depend on indexing to find things). Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com> Date: Wednesday, February 28, 2024 at 9:36 AM To: Eliot Kimber <eliot.kimber@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state [External Email] ________________________________ …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished…[1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished… [1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> wrote: Using Basex 10.7 on Linux. I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk). I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc. Optimization code is: try { if (db:exists($database)) then ( util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``), db:optimize($database, true(), $dbadmin:dbOptimizeOptions) ) else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``) } catch * { util:logToConsole( 'dbadmin:optimizeDatabase', ``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``, 'error') } And the optimization options are: declare variable $dbadmin:dbOptimizeOptions as map(*) := (: Turn on all the indexes :) map { 'attrindex' : true(), 'tokenindex' : true(), 'textindex' : true(), 'ftindex' : true() }; This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering: A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up). Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Maybe: https://help.basex.org/main/Profiling_Module /Andy On Thu, 29 Feb 2024 at 09:40, Eliot Kimber <eliot.kimber@servicenow.com> wrote:
Using the latest build, 11.0 beta 17d8426, the prof:dump() function is reported as an unknown function.
What replaces it (or where can I find the V11 docs)?
Thanks,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Eliot Kimber <eliot.kimber@servicenow.com> *Date: *Thursday, February 29, 2024 at 3:15 AM *To: *Christian Grün <christian.gruen@gmail.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state
Found the latest build at https://files.basex.org/releases/latest/
Cheers,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Eliot Kimber <eliot.kimber@servicenow.com> *Date: *Thursday, February 29, 2024 at 2:45 AM *To: *Christian Grün <christian.gruen@gmail.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state
I’m trying to compile the latest code but “mvn clean install” fails on failure to download some dependencies:
[*INFO*] *--------------------------< *org.basex:basex*
---------------------------*
[*INFO*] *Building BaseX Core 11.0-SNAPSHOT*
[*INFO*] from pom.xml
[*INFO*] *--------------------------------[ jar ]---------------------------------*
Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0...
[*WARNING*] The POM for jp.sourceforge.igo:igo:jar:0.4.3 is missing, no dependency information available
Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme...
[*WARNING*] The POM for org.apache:lucene-stemmers:jar:3.4.0 is missing, no dependency information available
Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0...
Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme...
[*INFO*] *------------------------------------------------------------------------*
[*INFO*] *BUILD FAILURE*
[*INFO*] *------------------------------------------------------------------------*
[*INFO*] Total time: 5.589 s
[*INFO*] Finished at: 2024-02-29T02:34:11-06:00
[*INFO*] *------------------------------------------------------------------------*
[*ERROR*] Failed to execute goal on project basex: *Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): Could not find artifact jp.sourceforge.igo:igo:jar:0.4.3 in devsnc-mirror (http://nexus.proxy.devsnc.com/content/groups/public <http://nexus.proxy.devsnc.com/content/groups/public>)* -> *[Help 1]*
And then:
[*ERROR*] Failed to execute goal on project basex: *Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): jp.sourceforge.igo:igo:jar:0.4.3 was not found in http://nexus.proxy.devsnc.com/content/groups/public <http://nexus.proxy.devsnc.com/content/groups/public> during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of devsnc-mirror has elapsed or updates are forced* -> *[Help 1]*
Where ‘nexus.proxy.devsnc.com’ is our internal Maven proxy.
This is after removing my local maven repository cache.
Before doing that, I got the “present but not available” failure on these libraries. The jars are present in my original .m2/repository but not in a newly-created repository (having moved the original one out of the way).
I’m using maven 3.9.6.
As far as I can tell from my research, this is a change in behavior from Maven 2 to Maven 3 where it refuses to use local dependencies that were fetched as a side effect of some other dependency that cannot now be fetched. But I’m not finding any maven options that will work around this failure.
Any idea how I can move past this? Or is there a nightly build somewhere I can download?
Thanks,
Eliot
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Eliot Kimber <eliot.kimber@servicenow.com> *Date: *Wednesday, February 28, 2024 at 10:01 AM *To: *Christian Grün <christian.gruen@gmail.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state
I’ll check out the snapshot.
I was starting to dig into the Java code yesterday but was first testing if I needed the optimization at all in this case, which it looks like I don’t (the consumer of the stored docs doesn’t depend on indexing to find things).
Cheers,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Christian Grün <christian.gruen@gmail.com> *Date: *Wednesday, February 28, 2024 at 9:36 AM *To: *Eliot Kimber <eliot.kimber@servicenow.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state *[External Email]*
------------------------------
…this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished…
[1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9...
On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com> wrote:
Using Basex 10.7 on Linux.
I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk).
I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc.
Optimization code is:
try {
if (db:exists($database))
then
(
util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``),
db:optimize($database, true(), $dbadmin:dbOptimizeOptions)
)
else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``)
} catch * {
util:logToConsole(
'dbadmin:optimizeDatabase',
``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``,
'error')
}
And the optimization options are:
declare variable $dbadmin:dbOptimizeOptions as map(*) :=
(: Turn on all the indexes :)
map {
'attrindex' : true(),
'tokenindex' : true(),
'textindex' : true(),
'ftindex' : true()
};
This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering:
A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up).
Thanks,
Eliot
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
…exactly; thanks, Andy. Time works against us, so we’ve added a link to the new documentation on the start page of docs.basex.org. On Thu, Feb 29, 2024 at 11:15 AM Andy Bunce <bunce.andy@gmail.com> wrote:
Maybe: https://help.basex.org/main/Profiling_Module
/Andy
On Thu, 29 Feb 2024 at 09:40, Eliot Kimber <eliot.kimber@servicenow.com> wrote:
Using the latest build, 11.0 beta 17d8426, the prof:dump() function is reported as an unknown function.
What replaces it (or where can I find the V11 docs)?
Thanks,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Eliot Kimber <eliot.kimber@servicenow.com> *Date: *Thursday, February 29, 2024 at 3:15 AM *To: *Christian Grün <christian.gruen@gmail.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state
Found the latest build at https://files.basex.org/releases/latest/
Cheers,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Eliot Kimber <eliot.kimber@servicenow.com> *Date: *Thursday, February 29, 2024 at 2:45 AM *To: *Christian Grün <christian.gruen@gmail.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state
I’m trying to compile the latest code but “mvn clean install” fails on failure to download some dependencies:
[*INFO*] *--------------------------< *org.basex:basex*
---------------------------*
[*INFO*] *Building BaseX Core 11.0-SNAPSHOT*
[*INFO*] from pom.xml
[*INFO*] *--------------------------------[ jar ]---------------------------------*
Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0...
[*WARNING*] The POM for jp.sourceforge.igo:igo:jar:0.4.3 is missing, no dependency information available
Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme...
[*WARNING*] The POM for org.apache:lucene-stemmers:jar:3.4.0 is missing, no dependency information available
Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/jp/sourceforge/igo/igo/0...
Downloading from devsnc-mirror: http://nexus.proxy.devsnc.com/content/groups/public/org/apache/lucene-stemme...
[*INFO*] *------------------------------------------------------------------------*
[*INFO*] *BUILD FAILURE*
[*INFO*] *------------------------------------------------------------------------*
[*INFO*] Total time: 5.589 s
[*INFO*] Finished at: 2024-02-29T02:34:11-06:00
[*INFO*] *------------------------------------------------------------------------*
[*ERROR*] Failed to execute goal on project basex: *Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): Could not find artifact jp.sourceforge.igo:igo:jar:0.4.3 in devsnc-mirror (http://nexus.proxy.devsnc.com/content/groups/public <http://nexus.proxy.devsnc.com/content/groups/public>)* -> *[Help 1]*
And then:
[*ERROR*] Failed to execute goal on project basex: *Could not resolve dependencies for project org.basex:basex:jar:11.0-SNAPSHOT: The following artifacts could not be resolved: jp.sourceforge.igo:igo:jar:0.4.3 (absent), org.apache:lucene-stemmers:jar:3.4.0 (absent): jp.sourceforge.igo:igo:jar:0.4.3 was not found in http://nexus.proxy.devsnc.com/content/groups/public <http://nexus.proxy.devsnc.com/content/groups/public> during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of devsnc-mirror has elapsed or updates are forced* -> *[Help 1]*
Where ‘nexus.proxy.devsnc.com’ is our internal Maven proxy.
This is after removing my local maven repository cache.
Before doing that, I got the “present but not available” failure on these libraries. The jars are present in my original .m2/repository but not in a newly-created repository (having moved the original one out of the way).
I’m using maven 3.9.6.
As far as I can tell from my research, this is a change in behavior from Maven 2 to Maven 3 where it refuses to use local dependencies that were fetched as a side effect of some other dependency that cannot now be fetched. But I’m not finding any maven options that will work around this failure.
Any idea how I can move past this? Or is there a nightly build somewhere I can download?
Thanks,
Eliot
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Eliot Kimber <eliot.kimber@servicenow.com> *Date: *Wednesday, February 28, 2024 at 10:01 AM *To: *Christian Grün <christian.gruen@gmail.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state
I’ll check out the snapshot.
I was starting to dig into the Java code yesterday but was first testing if I needed the optimization at all in this case, which it looks like I don’t (the consumer of the stored docs doesn’t depend on indexing to find things).
Cheers,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Christian Grün <christian.gruen@gmail.com> *Date: *Wednesday, February 28, 2024 at 9:36 AM *To: *Eliot Kimber <eliot.kimber@servicenow.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state *[External Email]*
------------------------------
…this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished…
[1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9...
On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com> wrote:
Using Basex 10.7 on Linux.
I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk).
I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc.
Optimization code is:
try {
if (db:exists($database))
then
(
util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``),
db:optimize($database, true(), $dbadmin:dbOptimizeOptions)
)
else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``)
} catch * {
util:logToConsole(
'dbadmin:optimizeDatabase',
``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``,
'error')
}
And the optimization options are:
declare variable $dbadmin:dbOptimizeOptions as map(*) :=
(: Turn on all the indexes :)
map {
'attrindex' : true(),
'tokenindex' : true(),
'textindex' : true(),
'ftindex' : true()
};
This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering:
A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up).
Thanks,
Eliot
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
Using the 27-02-2024 build I have confirmed that the optimize database deadlock seems to be resolved. I was able to easily upgrade my code to replace prof:dump() with message() and db:open() with db:get() and everything else seems to be working as it should. I like the new Editor replacement for the old Query feature in the DBA app. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com> Date: Wednesday, February 28, 2024 at 9:36 AM To: Eliot Kimber <eliot.kimber@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state [External Email] ________________________________ …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished…[1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished… [1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> wrote: Using Basex 10.7 on Linux. I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk). I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc. Optimization code is: try { if (db:exists($database)) then ( util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``), db:optimize($database, true(), $dbadmin:dbOptimizeOptions) ) else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``) } catch * { util:logToConsole( 'dbadmin:optimizeDatabase', ``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``, 'error') } And the optimization options are: declare variable $dbadmin:dbOptimizeOptions as map(*) := (: Turn on all the indexes :) map { 'attrindex' : true(), 'tokenindex' : true(), 'textindex' : true(), 'ftindex' : true() }; This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering: A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up). Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Glad to hear it, thanks Eliot. Eliot Kimber <eliot.kimber@servicenow.com> schrieb am Fr., 1. März 2024, 16:21:
Using the 27-02-2024 build I have confirmed that the optimize database deadlock seems to be resolved.
I was able to easily upgrade my code to replace prof:dump() with message() and db:open() with db:get() and everything else seems to be working as it should.
I like the new Editor replacement for the old Query feature in the DBA app.
Cheers,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Christian Grün <christian.gruen@gmail.com> *Date: *Wednesday, February 28, 2024 at 9:36 AM *To: *Eliot Kimber <eliot.kimber@servicenow.com> *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state *[External Email]*
------------------------------
…this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished…
[1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9...
On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com> wrote:
Using Basex 10.7 on Linux.
I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk).
I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc.
Optimization code is:
try {
if (db:exists($database))
then
(
util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``),
db:optimize($database, true(), $dbadmin:dbOptimizeOptions)
)
else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``)
} catch * {
util:logToConsole(
'dbadmin:optimizeDatabase',
``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``,
'error')
}
And the optimization options are:
declare variable $dbadmin:dbOptimizeOptions as map(*) :=
(: Turn on all the indexes :)
map {
'attrindex' : true(),
'tokenindex' : true(),
'textindex' : true(),
'ftindex' : true()
};
This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering:
A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up).
Thanks,
Eliot
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
As reported in a separate thread, I’m still getting the optimization failure for large databases but not smaller ones. This behavior is consistent in my tests. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com> Date: Friday, March 1, 2024 at 9:30 AM To: Eliot Kimber <eliot.kimber@servicenow.com> Cc: BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state [External Email] ________________________________ Glad to hear it, thanks Eliot. Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> schrieb am Fr., 1. März 2024, 16:21: Using the 27-02-2024 build I have confirmed that the optimize database deadlock seems to be resolved. I was able to easily upgrade my code to replace prof:dump() with message() and db:open() with db:get() and everything else seems to be working as it should. I like the new Editor replacement for the old Query feature in the DBA app. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com<mailto:christian.gruen@gmail.com>> Date: Wednesday, February 28, 2024 at 9:36 AM To: Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> Cc: basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> <basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Optimize database never returns, leaves database in "opened by another process" state [External Email] ________________________________ …this one could be related to a bug that was recently fixed in the latest snapshot [1]. About time to get BaseX 11 finished… [1] https://github.com/BaseXdb/basex/commit/45d97f8065615fb734b712bc4c77c39899e9... On Mon, Feb 26, 2024 at 5:25 PM Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> wrote: Using Basex 10.7 on Linux. I’m running a sequence of jobs to update and optimize a set of databases following loading a number of documents created dynamically (as opposed to being read from disk). I’m seeing a new behavior, which is that the optimization step never completes but also doesn’t show any error in the log. The database shows no items and is in the locked by another process state if I try to drop it. This behavior seems to be consistently repeatable with my current code base (I’m working on some code updates, so it’s possible I’ve introduced something that would cause this behavior but I haven’t changed the code that leads up to the failing optimize). The server has plenty of disk space, etc. Optimization code is: try { if (db:exists($database)) then ( util:logToConsole('dbadmin:optimizeDatabase', ``[Optimizing database `{$database}`]``), db:optimize($database, true(), $dbadmin:dbOptimizeOptions) ) else util:logToConsole('dbadmin:optimizeDatabase', ``[Database '`{$database}`' does not exist. Nothing to optimize.]``) } catch * { util:logToConsole( 'dbadmin:optimizeDatabase', ``[Exception optimizing database '`{$database}`': `{$err:code}` - `{$err:description}`]``, 'error') } And the optimization options are: declare variable $dbadmin:dbOptimizeOptions as map(*) := (: Turn on all the indexes :) map { 'attrindex' : true(), 'tokenindex' : true(), 'textindex' : true(), 'ftindex' : true() }; This code has been working fine for a long time and I’ve been running 10.7 for a least a couple of months, so I’m wondering: A) Would would cause this behavior? B) How can I diagnose it short of debugging the Java code (which I can do but it’s non-trivial for me to set up). Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
participants (3)
-
Andy Bunce -
Christian Grün -
Eliot Kimber