Db:token() vs db:attribute(): Attribute is much slower -- why

List overview All Threads
Download

newer

older

RESTXQ Custom Multipart Response

Crash when starting basexgui

Eliot Kimber

10 Oct 2023 10 Oct '23

7:02 p.m.

Using 10.6 on macOS:

I have a data file with elements like:

One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data).

I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry.

The database with this data set is indexed with an attribute and a token index.

In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token().

That is, I have this code:

$records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/..

When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds!

I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path.

But using the db:token database is clearly faster by a lot.

If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time.

I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation.

Cheers,

E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow

Attachments:

attachment.html (text/html — 6.2 KB)

Show replies by date

Eliot Kimber

10 Oct 10 Oct

7:16 p.m.

Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track.

Reporting the raw times returned by db:token(), I get:

"lookup by docPath took 0.255696ms" "lookup by docPath took 0.054183ms" "lookup by docPath took 0.211373ms" "lookup by docPath took 0.175889ms" "lookup by docPath took 0.220833ms" "lookup by docPath took 0.156875ms" "lookup by docPath took 0.139612ms" "lookup by docPath took 0.007018ms" "lookup by docPath took 0.179297ms" "lookup by docPath took 0.00964ms" "lookup by docPath took 0.093668ms"

Which adds up to about 400ms for 1300 topics looked up.

Using db:attribute() I get:

"lookup by docPath took 3.794676ms" "lookup by docPath took 0.032425ms" "lookup by docPath took 0.092237ms" "lookup by docPath took 0.097815ms" "lookup by docPath took 1.246617ms" "lookup by docPath took 0.050855ms" "lookup by docPath took 0.065685ms" "lookup by docPath took 0.00511ms" "lookup by docPath took 0.057833ms"

Cheers,

_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow

From: Eliot Kimber eliot.kimber@servicenow.com Date: Tuesday, October 10, 2023 at 6:02 PM To: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Db:token() vs db:attribute(): Attribute is much slower -- why Using 10.6 on macOS:

I have a data file with elements like:

One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data).

I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry.

The database with this data set is indexed with an attribute and a token index.

In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token().

That is, I have this code:

$records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/..

When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds!

But using the db:token database is clearly faster by a lot.

Cheers,

Christian Grün

15 Oct 15 Oct

7:40 a.m.

Hi Eliot,

If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database.

Hope this helps, Christian

[1] https://docs.basex.org/wiki/Options#MAXLEN

On Wed, Oct 11, 2023 at 1:16 AM Eliot Kimber eliot.kimber@servicenow.com wrote:

...

Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track.

Reporting the raw times returned by db:token(), I get:

"lookup by docPath took 0.255696ms"

"lookup by docPath took 0.054183ms"

"lookup by docPath took 0.211373ms"

"lookup by docPath took 0.175889ms"

"lookup by docPath took 0.220833ms"

"lookup by docPath took 0.156875ms"

"lookup by docPath took 0.139612ms"

"lookup by docPath took 0.007018ms"

"lookup by docPath took 0.179297ms"

"lookup by docPath took 0.00964ms"

"lookup by docPath took 0.093668ms"

Which adds up to about 400ms for 1300 topics looked up.

Using db:attribute() I get:

"lookup by docPath took 3.794676ms"

"lookup by docPath took 0.032425ms"

"lookup by docPath took 0.092237ms"

"lookup by docPath took 0.097815ms"

"lookup by docPath took 1.246617ms"

"lookup by docPath took 0.050855ms"

"lookup by docPath took 0.065685ms"

"lookup by docPath took 0.00511ms"

"lookup by docPath took 0.057833ms"

Cheers,

E.

*Eliot Kimber*

Sr Staff Content Engineer

O: 512 554 9368

M: 512 554 9368

servicenow.com https://www.servicenow.com

LinkedIn https://www.linkedin.com/company/servicenow | Twitter https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Facebook https://www.facebook.com/servicenow

*From: *Eliot Kimber eliot.kimber@servicenow.com *Date: *Tuesday, October 10, 2023 at 6:02 PM *To: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Db:token() vs db:attribute(): Attribute is much slower -- why

Using 10.6 on macOS:

I have a data file with elements like:

<record language="en-US" family="pce-test-data" bundlename="bundle-appportman-application-portfolio-management" topicpath="product/application-portfolio-management/concept/application-portfolio-management-landing-page.dita" notviewed="viewed" viewcount="13579">

One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data).

I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry.

The database with this data set is indexed with an attribute and a token index.

In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token().

That is, I have this code:

$records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/..

When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds!

I’m surprised at this result—I would expect the two forms of index to perform with the same speed: the attribute is a single blank-delimited token and I would expect the string match under the covers to be using the same code path.

But using the db:token database is clearly faster by a lot.

If I report the time each lookup uses, the times vary quite a lot, but for the token lookup they tend to be in the 10-60ms range, but for the attribute lookup, many are in the 1 or 2 second range, which explains the much longer run time.

I’m wondering if this result is expected due to implementation details of the attribute index as distinct from the token index or if this indicates some logic bug in the attribute index implementation.

Cheers,

E.

*Eliot Kimber*

Sr Staff Content Engineer

O: 512 554 9368

M: 512 554 9368

servicenow.com https://www.servicenow.com

LinkedIn https://www.linkedin.com/company/servicenow | Twitter https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Facebook https://www.facebook.com/servicenow

Eliot Kimber

10:04 a.m.

I see, so if I’m indexing on longer strings, such as the URLs in my data sets, I need to set MAXLEN to accommodate.

Thanks for the tip.

Cheers,

From: Christian Grün christian.gruen@gmail.com Date: Sunday, October 15, 2023 at 6:41 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Db:token() vs db:attribute(): Attribute is much slower -- why [External Email]

________________________________ Hi Eliot,If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database.Hope this helps,Christian[1] https://docs.basex.org/wiki/Options#MAXLENOn https://docs.basex.org/wiki/Options#MAXLENOn

i This message needs your attention • Someone new is on this email. Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees.https://mimecast.com Hi Eliot,

If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database.

Hope this helps, Christian

[1] https://docs.basex.org/wiki/Options#MAXLEN https://docs.basex.org/wiki/Options#MAXLEN

On Wed, Oct 11, 2023 at 1:16 AM Eliot Kimber <eliot.kimber@servicenow.commailto:eliot.kimber@servicenow.com> wrote: Sorry, my times are off—I was doing a nonsense multiplication of the result of prof:track.

Reporting the raw times returned by db:token(), I get:

Which adds up to about 400ms for 1300 topics looked up.

Using db:attribute() I get:

Cheers,

From: Eliot Kimber <eliot.kimber@servicenow.commailto:eliot.kimber@servicenow.com> Date: Tuesday, October 10, 2023 at 6:02 PM To: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> Subject: Db:token() vs db:attribute(): Attribute is much slower -- why Using 10.6 on macOS:

I have a data file with elements like:

One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data).

I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry.

The database with this data set is indexed with an attribute and a token index.

In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token().

That is, I have this code:

$records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/..

When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds!

But using the db:token database is clearly faster by a lot.

Cheers,

Eliot Kimber

26 Oct 26 Oct

11:11 a.m.

I increased MAXLEN to 255 and now I find my lookups are taking a consistent 0.07 seconds.

Cheers,

From: Eliot Kimber eliot.kimber@servicenow.com Date: Sunday, October 15, 2023 at 9:04 AM To: Christian Grün christian.gruen@gmail.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Db:token() vs db:attribute(): Attribute is much slower -- why I see, so if I’m indexing on longer strings, such as the URLs in my data sets, I need to set MAXLEN to accommodate.

Thanks for the tip.

Cheers,

If the length of a search strings exceeds MAXLEN [1], a sequential search will be performed instead. You can achieve index-based lookups by increasing the limit when creating/optimizing a database.

Hope this helps, Christian

[1] https://docs.basex.org/wiki/Options#MAXLEN https://docs.basex.org/wiki/Options#MAXLEN

Reporting the raw times returned by db:token(), I get:

Which adds up to about 400ms for 1300 topics looked up.

Using db:attribute() I get:

Cheers,

I have a data file with elements like:

One such entry for each topic in our public docs server that is accessed at least once (Adobe page views data).

I generate a report for all the topics used from a root map that includes the page views data for each topic for which there is an entry.

The database with this data set is indexed with an attribute and a token index.

In my testing I’m finding that lookup using db:attribute() on the @topicpath attribute (which is always a single value) is about 10x slower than using db:token().

That is, I have this code:

$records := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/..

When I run this for a set of about 1200 topics, the db:token version takes about 400ms but the db:attribute version takes 9 seconds!

But using the db:token database is clearly faster by a lot.

Cheers,

671

Age (days ago)

687

Last active (days ago)

basex-talk@mailman.uni-konstanz.de

4 comments

2 participants

tags (0)

participants (2)

Christian Grün
Eliot Kimber