Maybe a general question: Is the insertion really a bottleneck in your
scenario? How many data do you want to store in a single database? You
could e.g. store your data in multiple databases, which can then all
be queried by a single XQuery expression.
On Tue, Sep 23, 2014 at 1:50 PM, Gerald de Jong <gerald@delving.eu> wrote:
> The other case I'm testing has five necessary namespaces. :(
>
> 10000: 6462ms
> 20000: 7592ms
> 30000: 8689ms
> 40000: 9417ms
> 50000: 9566ms
> 60000: 10368ms
> 70000: 10963ms
> 80000: 12167ms
>
> Is there any direction you can suggest to look for a workaround?
>
>
> On Tue, Sep 23, 2014 at 1:43 PM, Christian Grün <christian.gruen@gmail.com>
> wrote:
>>
>> > This namespace happens to be unnecessary, but others won't be. I'm so
>> > curious how this can be the thing.
>>
>> Unfortunately, the intricacies of namespaces have been keeping us XML
>> implementers busy for a long time, and the XPath and storage
>> algorithms would be much simpler, if not trivial, without the notion
>> of namespaces. This is why it would take quite a while to explain what
>> are the reasons for that, and as your input document only contains one
>> namespaces, I'm not surprised that you are surprised ;) To put it in a
>> nutshell: it's usually easy to optimize single namespaces issues, but
>> it's difficult to optimize all cases that happen in practice.
>>
>> But I'll keep track of your use case.
>>
>>
>> On Tue, Sep 23, 2014 at 1:30 PM, Gerald de Jong <gerald@delving.eu> wrote:
>> >
>> > On Tue, Sep 23, 2014 at 1:20 PM, Gerald de Jong <gerald@delving.eu>
>> > wrote:
>> >>
>> >> WOW, really... the namespace? Because it's unused, or is it always
>> >> going
>> >> to slow when there are namespaces?
>> >>
>> >> On Tue, Sep 23, 2014 at 1:13 PM, Christian Grün
>> >> <christian.gruen@gmail.com> wrote:
>> >>>
>> >>> Thanks for the document. The declaration of the (unused) namespace in
>> >>> the root element seems to be the cause for the decreasing performance
>> >>> (I noticed that the time for adding documents stays constant after
>> >>> removing the declaration). I'll do some profiling in order to find out
>> >>> if this can be sped up without too much effort (it may take a while,
>> >>> though, because I'll be on leave for a while from tomorrow).
>> >>>
>> >>>
>> >>> On Tue, Sep 23, 2014 at 12:25 PM, Gerald de Jong <gerald@delving.eu>
>> >>> wrote:
>> >>> > I don't know what causes the gradual slowdown. My assumption was
>> >>> > that
>> >>> > it
>> >>> > was the "optimize" which would cause the index to be built, so I
>> >>> > didn't
>> >>> > expect a slowdown at all during "add" calls, especially when
>> >>> > autoflush
>> >>> > is
>> >>> > false.
>> >>> >
>> >>> > I add documents with the following paths:
>> >>> >
>> >>> > /f/f/e/ffe0f5be2aa14e81050f759c8f9c3eb7.xml
>> >>> >
>> >>> > The xml file name is a hash of the contents, and it is placed in a
>> >>> > path
>> >>> > such
>> >>> > that the export spreads out the files nicely into a file system
>> >>> > tree,
>> >>> > rather
>> >>> > than putting a million docs into one directory.
>> >>> >
>> >>> > The document content is nothing special, wrapped in a special tag:
>> >>> >
>> >>> > <narthex xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> > id="20412518"
>> >>> > mod="2014-09-23T11:11:51.007+02:00">
>> >>> > <record>
>> >>> > <priref>20412518</priref>
>> >>> > <current_location>FTA</current_location>
>> >>> > <current_location.type/>
>> >>> > <description>Ingang op de binnenplaats van de
>> >>> > zuidvleugel</description>
>> >>> > <collection>Fotocollectie</collection>
>> >>> > <production.date.start>1925-08-06</production.date.start>
>> >>> > <reproduction.format/>
>> >>> >
>> >>> >
>> >>> >
>> >>> > <reproduction.reference>2186abf4-7108-f9b8-ffbb-902881afe836</reproduction.reference>
>> >>> > <creator.role>Fotograaf</creator.role>
>> >>> > <object_number>9.387</object_number>
>> >>> > <monument.label/>
>> >>> > <monument.zipcode/>
>> >>> > <monument.name>Kasteel Hoensbroek</monument.name>
>> >>> > <monument.record_number>284330</monument.record_number>
>> >>> > <reproduction.date/>
>> >>> > <reproduction.notes>Oude filepath:
>> >>> > 0009\009387.jpg</reproduction.notes>
>> >>> > <reproduction.type/>
>> >>> > <reproduction.creator/>
>> >>> > <rights.type>Copyright</rights.type>
>> >>> > <technique>Neg.zw</technique>
>> >>> > <creator>Scheepens, W.C.L.A.</creator>
>> >>> > <order_number>avh04-2008</order_number>
>> >>> > <input.date>2008-04-01</input.date>
>> >>> > <edit.date>2011-05-03</edit.date>
>> >>> > <edit.date>2008-04-28</edit.date>
>> >>> > <monument.historical_address/>
>> >>> > <content.subject.type value="SUBJECT" option="SUBJECT">
>> >>> > <text language="0">subject</text>
>> >>> > <text language="1">onderwerp</text>
>> >>> > <text language="2">sujet</text>
>> >>> > <text language="3">Thema</text>
>> >>> > <text language="4">موضوع</text>
>> >>> > <text language="6">θέμα</text>
>> >>> > </content.subject.type>
>> >>> > <content.subject.type value="SUBJECT" option="SUBJECT">
>> >>> > <text language="0">subject</text>
>> >>> > <text language="1">onderwerp</text>
>> >>> > <text language="2">sujet</text>
>> >>> > <text language="3">Thema</text>
>> >>> > <text language="4">موضوع</text>
>> >>> > <text language="6">θέμα</text>
>> >>> > </content.subject.type>
>> >>> > <content.subject>Kasteel</content.subject>
>> >>> > <content.subject>Binnenplaats</content.subject>
>> >>> > <monument.province>Limburg</monument.province>
>> >>> > <monument.place>Hoensbroek</monument.place>
>> >>> > <monument.number/>
>> >>> > <monument.county/>
>> >>> > <monument.country>Nederland</monument.country>
>> >>> > <monument.house_number>18</monument.house_number>
>> >>> > <monument.street>Klinkertstraat</monument.street>
>> >>> > <monument.house_number.addition/>
>> >>> > <monument.complex_number/>
>> >>> > <monument.number.x_coordinates/>
>> >>> > <monument.number.y_coordinates/>
>> >>> > <monument.geographical_keyword/>
>> >>> > <monument.complex_number.x_coordinates/>
>> >>> > <monument.complex_number.y_coordinates/>
>> >>> > <creator.date_of_birth/>
>> >>> > <creator.date_of_death/>
>> >>> > <input.name>a.vanhoute</input.name>
>> >>> > <edit.name>RCEadmin</edit.name>
>> >>> > <edit.name>a.vanhoute</edit.name>
>> >>> > <creator.history/>
>> >>> > <record_type value="OBJECT" option="OBJECT">
>> >>> > <text language="0">single object</text>
>> >>> > <text language="2">objet individuel</text>
>> >>> > <text language="3">Einzelnes Objekt</text>
>> >>> > </record_type>
>> >>> > <edit.time>03:10:32</edit.time>
>> >>> > <edit.time>11:17:08</edit.time>
>> >>> > <input.time>09:58:28</input.time>
>> >>> > <input.source>document>photographs</input.source>
>> >>> > <edit.source>collect>photograph</edit.source>
>> >>> > <edit.source>document>photographs</edit.source>
>> >>> > </record>
>> >>> > </narthex>
>> >>> >
>> >>> > On Tue, Sep 23, 2014 at 11:36 AM, Christian Grün
>> >>> > <christian.gruen@gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> > I set up to use the 8.0-SNAPSHOT and used the internal parser as
>> >>> >> > well.
>> >>> >> > In
>> >>> >> > your example you're not really giving much of a challenge to the
>> >>> >> > index,
>> >>> >> > since every doc is just <a/>.
>> >>> >>
>> >>> >> If I get it right, you assume the slowdown is due to the index
>> >>> >> structures?
>> >>> >>
>> >>> >> > With respect to ADD, I'm not seeing a significant performance
>> >>> >> > difference:
>> >>> >>
>> >>> >> Please give us more info on the data you are adding. Could you
>> >>> >> provide
>> >>> >> us with a sample document?
>> >>> >>
>> >>> >>
>> >>> >> > 8.0-SNAPSHOT
>> >>> >> > -------
>> >>> >> > 10000: 9250ms
>> >>> >> > 20000: 7626ms
>> >>> >> > 30000: 7885ms
>> >>> >> > 40000: 8111ms
>> >>> >> > 50000: 8365ms
>> >>> >> > 60000: 8784ms
>> >>> >> > 70000: 9270ms
>> >>> >> > 80000: 9692ms
>> >>> >> > 90000: 10158ms
>> >>> >> > 100000: 10612ms
>> >>> >> > 110000: 11018ms
>> >>> >> > 120000: 11478ms
>> >>> >> > 130000: 11940ms
>> >>> >> > 140000: 12505ms
>> >>> >> > 150000: 13047ms
>> >>> >> > 160000: 13536ms
>> >>> >> > 170000: 14055ms
>> >>> >> > 180000: 14371ms
>> >>> >> > 190000: 14883ms
>> >>> >> > 200000: 15330ms
>> >>> >> > 210000: 15888ms
>> >>> >> > 220000: 16398ms
>> >>> >> > 230000: 16878ms
>> >>> >> > 240000: 17038ms
>> >>> >> > 250000: 17453ms
>> >>> >> > 260000: 17965ms
>> >>> >> > 270000: 18317ms
>> >>> >> > 280000: 18832ms
>> >>> >> > 290000: 19373ms
>> >>> >> > 300000: 19735ms
>> >>> >> > 310000: 20062ms
>> >>> >> > 320000: 20675ms
>> >>> >> > 330000: 21113ms
>> >>> >> > 340000: 21754ms
>> >>> >> > 350000: 22887ms
>> >>> >> > 360000: 22810ms
>> >>> >> > 370000: 22985ms
>> >>> >> > 380000: 23506ms
>> >>> >> > 390000: 23856ms
>> >>> >> > 400000: 24338ms
>> >>> >> >
>> >>> >> > 7.9
>> >>> >> > -----
>> >>> >> > 10000: 8229ms
>> >>> >> > 20000: 7587ms
>> >>> >> > 30000: 7973ms
>> >>> >> > 40000: 8282ms
>> >>> >> > 50000: 8717ms
>> >>> >> > 60000: 9294ms
>> >>> >> > 70000: 10105ms
>> >>> >> > 80000: 10669ms
>> >>> >> > 90000: 11301ms
>> >>> >> > 100000: 11835ms
>> >>> >> > 110000: 12413ms
>> >>> >> > 120000: 13000ms
>> >>> >> > 130000: 13577ms
>> >>> >> > 140000: 14331ms
>> >>> >> > 150000: 14488ms
>> >>> >> > 160000: 15025ms
>> >>> >> > 170000: 15463ms
>> >>> >> > 180000: 15815ms
>> >>> >> > 190000: 16153ms
>> >>> >> > 200000: 16314ms
>> >>> >> > 210000: 16562ms
>> >>> >> > 220000: 17186ms
>> >>> >> > 230000: 17862ms
>> >>> >> > 240000: 18340ms
>> >>> >> > 250000: 18790ms
>> >>> >> > 260000: 19313ms
>> >>> >> > 270000: 19850ms
>> >>> >> > 280000: 20225ms
>> >>> >> > 290000: 20650ms
>> >>> >> > 300000: 21062ms
>> >>> >> > 310000: 21595ms
>> >>> >> > 320000: 22022ms
>> >>> >> > 330000: 22414ms
>> >>> >> > 340000: 22925ms
>> >>> >> > 350000: 23514ms
>> >>> >> > 360000: 23762ms
>> >>> >> > 370000: 24360ms
>> >>> >> > 380000: 25028ms
>> >>> >> > 390000: 25446ms
>> >>> >> > 400000: 25700ms
>> >>> >> >
>> >>> >> > - Gerald de Jong
>> >>> >> >
>> >>> >> >
>> >>> >> > On Thu, Sep 18, 2014 at 6:57 PM, Christian Grün
>> >>> >> > <christian.gruen@gmail.com>
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> > Perhaps you can give me a hint as to why inserts slow down.j
>> >>> >> >> I didn't have time to check out 7.9, but I have done some
>> >>> >> >> testing
>> >>> >> >> with
>> >>> >> >> 8.0, and I didn't notice a real slow-down. This is Java testing
>> >>> >> >> script
>> >>> >> >> (1 mio documents are added in just 17 seconds; I'm using the
>> >>> >> >> internal
>> >>> >> >> BaseX parser to speed up the import):
>> >>> >> >>
>> >>> >> >> Performance p = new Performance();
>> >>> >> >> Context ctx = new Context();
>> >>> >> >>
>> >>> >> >> new CreateDB("db").execute(ctx);
>> >>> >> >> new Set(MainOptions.AUTOFLUSH, false).execute(ctx);
>> >>> >> >> new Set(MainOptions.INTPARSE, true).execute(ctx);
>> >>> >> >> for(int i = 0; i < 1000000; i++) {
>> >>> >> >> new Add("db", "<a/>").execute(ctx);
>> >>> >> >> }
>> >>> >> >> ctx.close();
>> >>> >> >> System.out.println(p);
>> >>> >> >>
>> >>> >> >> Hope this helps,
>> >>> >> >> Christian
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > --
>> >>> >> > Delving BV, Vasteland 8, Rotterdam
>> >>> >> > http://www.delving.eu
>> >>> >> > http://twitter.com/fluxe
>> >>> >> > skype: beautifulcode
>> >>> >> > +31629339805
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Delving BV, Vasteland 8, Rotterdam
>> >>> > http://www.delving.eu
>> >>> > http://twitter.com/fluxe
>> >>> > skype: beautifulcode
>> >>> > +31629339805
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Delving BV, Vasteland 8, Rotterdam
>> >> http://www.delving.eu
>> >> http://twitter.com/fluxe
>> >> skype: beautifulcode
>> >> +31629339805
>> >
>> >
>> >
>> >
>> > --
>> > Delving BV, Vasteland 8, Rotterdam
>> > http://www.delving.eu
>> > http://twitter.com/fluxe
>> > skype: beautifulcode
>> > +31629339805
>
>
>
>
> --
> Delving BV, Vasteland 8, Rotterdam
> http://www.delving.eu
> http://twitter.com/fluxe
> skype: beautifulcode
> +31629339805