Db:token() vs db:attribute(): Attribute is much slower -- why
Using 10.6 on macOS: I have a data file with elements like: <record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579"> One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data). I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry. The database with this data set is indexed with an attribute and a token index. In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token(). That is, I have this code: $records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds! I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path. But using the db:token database is clearly faster by a lot. If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time. I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track. Reporting the raw times returned by db:token(), I get: "lookup by docPath took 0.255696ms" "lookup by docPath took 0.054183ms" "lookup by docPath took 0.211373ms" "lookup by docPath took 0.175889ms" "lookup by docPath took 0.220833ms" "lookup by docPath took 0.156875ms" "lookup by docPath took 0.139612ms" "lookup by docPath took 0.007018ms" "lookup by docPath took 0.179297ms" "lookup by docPath took 0.00964ms" "lookup by docPath took 0.093668ms" Which adds up to about 400ms for 1300 topics looked up. Using db:attribute() I get: "lookup by docPath took 3.794676ms" "lookup by docPath took 0.032425ms" "lookup by docPath took 0.092237ms" "lookup by docPath took 0.097815ms" "lookup by docPath took 1.246617ms" "lookup by docPath took 0.050855ms" "lookup by docPath took 0.065685ms" "lookup by docPath took 0.00511ms" "lookup by docPath took 0.057833ms" Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Tuesday, October 10, 2023 at 6:02 PM To: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Db:token() vs db:attribute(): Attribute is much slower -- why Using 10.6 on macOS: I have a data file with elements like: <record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579"> One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data). I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry. The database with this data set is indexed with an attribute and a token index. In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token(). That is, I have this code: $records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds! I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path. But using the db:token database is clearly faster by a lot. If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time. I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Hi Eliot, If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database. Hope this helps, Christian [1] https://docs.basex.org/wiki/Options#MAXLEN On Wed, Oct 11, 2023 at 1:16 AM Eliot Kimber <eliot.kimber@servicenow.com> wrote:
Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track.
Reporting the raw times returned by db:token(), I get:
"lookup by docPath took 0.255696ms"
"lookup by docPath took 0.054183ms"
"lookup by docPath took 0.211373ms"
"lookup by docPath took 0.175889ms"
"lookup by docPath took 0.220833ms"
"lookup by docPath took 0.156875ms"
"lookup by docPath took 0.139612ms"
"lookup by docPath took 0.007018ms"
"lookup by docPath took 0.179297ms"
"lookup by docPath took 0.00964ms"
"lookup by docPath took 0.093668ms"
Which adds up to about 400ms for 1300 topics looked up.
Using db:attribute() I get:
"lookup by docPath took 3.794676ms"
"lookup by docPath took 0.032425ms"
"lookup by docPath took 0.092237ms"
"lookup by docPath took 0.097815ms"
"lookup by docPath took 1.246617ms"
"lookup by docPath took 0.050855ms"
"lookup by docPath took 0.065685ms"
"lookup by docPath took 0.00511ms"
"lookup by docPath took 0.057833ms"
Cheers,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
*From: *Eliot Kimber <eliot.kimber@servicenow.com> *Date: *Tuesday, October 10, 2023 at 6:02 PM *To: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Db:token() vs db:attribute(): Attribute is much slower -- why
Using 10.6 on macOS:
I have a data file with elements like:
<record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579">
One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data).
I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry.
The database with this data set is indexed with an attribute and a token index.
In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token().
That is, I have this code:
$records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/..
When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds!
I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path.
But using the db:token database is clearly faster by a lot.
If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time.
I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation.
Cheers,
E.
_____________________________________________
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com <https://www.servicenow.com>
LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter <https://twitter.com/servicenow> | YouTube <https://www.youtube.com/user/servicenowinc> | Facebook <https://www.facebook.com/servicenow>
I see, so if I’m indexing on longer strings, such as the URLs in my data sets, I need to set MAXLEN to accommodate. Thanks for the tip. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com> Date: Sunday, October 15, 2023 at 6:41 AM To: Eliot Kimber <eliot.kimber@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Db:token() vs db:attribute(): Attribute is much slower -- why [External Email] ________________________________ Hi Eliot,If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database.Hope this helps,Christian[1] https://docs.basex.org/wiki/Options#MAXLENOn<https://docs.basex.org/wiki/Options#MAXLENOn> i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> Hi Eliot, If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database. Hope this helps, Christian [1] https://docs.basex.org/wiki/Options#MAXLEN<https://docs.basex.org/wiki/Options#MAXLEN> On Wed, Oct 11, 2023 at 1:16 AM Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> wrote: Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track. Reporting the raw times returned by db:token(), I get: "lookup by docPath took 0.255696ms" "lookup by docPath took 0.054183ms" "lookup by docPath took 0.211373ms" "lookup by docPath took 0.175889ms" "lookup by docPath took 0.220833ms" "lookup by docPath took 0.156875ms" "lookup by docPath took 0.139612ms" "lookup by docPath took 0.007018ms" "lookup by docPath took 0.179297ms" "lookup by docPath took 0.00964ms" "lookup by docPath took 0.093668ms" Which adds up to about 400ms for 1300 topics looked up. Using db:attribute() I get: "lookup by docPath took 3.794676ms" "lookup by docPath took 0.032425ms" "lookup by docPath took 0.092237ms" "lookup by docPath took 0.097815ms" "lookup by docPath took 1.246617ms" "lookup by docPath took 0.050855ms" "lookup by docPath took 0.065685ms" "lookup by docPath took 0.00511ms" "lookup by docPath took 0.057833ms" Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> Date: Tuesday, October 10, 2023 at 6:02 PM To: basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> <basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>> Subject: Db:token() vs db:attribute(): Attribute is much slower -- why Using 10.6 on macOS: I have a data file with elements like: <record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579"> One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data). I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry. The database with this data set is indexed with an attribute and a token index. In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token(). That is, I have this code: $records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds! I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path. But using the db:token database is clearly faster by a lot. If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time. I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
I increased MAXLEN to 255 and now I find my lookups are taking a consistent 0.07 seconds. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Sunday, October 15, 2023 at 9:04 AM To: Christian Grün <christian.gruen@gmail.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Db:token() vs db:attribute(): Attribute is much slower -- why I see, so if I’m indexing on longer strings, such as the URLs in my data sets, I need to set MAXLEN to accommodate. Thanks for the tip. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gruen@gmail.com> Date: Sunday, October 15, 2023 at 6:41 AM To: Eliot Kimber <eliot.kimber@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Db:token() vs db:attribute(): Attribute is much slower -- why [External Email] ________________________________ Hi Eliot,If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database.Hope this helps,Christian[1] https://docs.basex.org/wiki/Options#MAXLENOn<https://docs.basex.org/wiki/Options#MAXLENOn> i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.<https://mimecast.com> Hi Eliot, If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database. Hope this helps, Christian [1] https://docs.basex.org/wiki/Options#MAXLEN<https://docs.basex.org/wiki/Options#MAXLEN> On Wed, Oct 11, 2023 at 1:16 AM Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> wrote: Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track. Reporting the raw times returned by db:token(), I get: "lookup by docPath took 0.255696ms" "lookup by docPath took 0.054183ms" "lookup by docPath took 0.211373ms" "lookup by docPath took 0.175889ms" "lookup by docPath took 0.220833ms" "lookup by docPath took 0.156875ms" "lookup by docPath took 0.139612ms" "lookup by docPath took 0.007018ms" "lookup by docPath took 0.179297ms" "lookup by docPath took 0.00964ms" "lookup by docPath took 0.093668ms" Which adds up to about 400ms for 1300 topics looked up. Using db:attribute() I get: "lookup by docPath took 3.794676ms" "lookup by docPath took 0.032425ms" "lookup by docPath took 0.092237ms" "lookup by docPath took 0.097815ms" "lookup by docPath took 1.246617ms" "lookup by docPath took 0.050855ms" "lookup by docPath took 0.065685ms" "lookup by docPath took 0.00511ms" "lookup by docPath took 0.057833ms" Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com<mailto:eliot.kimber@servicenow.com>> Date: Tuesday, October 10, 2023 at 6:02 PM To: basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> <basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>> Subject: Db:token() vs db:attribute(): Attribute is much slower -- why Using 10.6 on macOS: I have a data file with elements like: <record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579"> One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data). I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry. The database with this data set is indexed with an attribute and a token index. In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token(). That is, I have this code: $records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds! I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path. But using the db:token database is clearly faster by a lot. If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time. I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
participants (2)
-
Christian Grün -
Eliot Kimber