Gerald,
I'm glad to tell that the latest snapshot [1] contains some additional
optimizations for adding documents with namespaces. It should now be
irrelevant if your added document has a namespace on top or not.
I'll be offline for some days (and I hope I didn't introduce a bad bug
with the latest commit ;).
Have fun,
Christian
[1] http://files.basex.org/releases/latest
On Tue, Sep 23, 2014 at 1:43 PM, Christian Grün
<christian.gruen@gmail.com> wrote:
>> This namespace happens to be unnecessary, but others won't be. I'm so
>> curious how this can be the thing.
>
> Unfortunately, the intricacies of namespaces have been keeping us XML
> implementers busy for a long time, and the XPath and storage
> algorithms would be much simpler, if not trivial, without the notion
> of namespaces. This is why it would take quite a while to explain what
> are the reasons for that, and as your input document only contains one
> namespaces, I'm not surprised that you are surprised ;) To put it in a
> nutshell: it's usually easy to optimize single namespaces issues, but
> it's difficult to optimize all cases that happen in practice.
>
> But I'll keep track of your use case.
>
>
> On Tue, Sep 23, 2014 at 1:30 PM, Gerald de Jong <gerald@delving.eu> wrote:
>>
>> On Tue, Sep 23, 2014 at 1:20 PM, Gerald de Jong <gerald@delving.eu> wrote:
>>>
>>> WOW, really... the namespace? Because it's unused, or is it always going
>>> to slow when there are namespaces?
>>>
>>> On Tue, Sep 23, 2014 at 1:13 PM, Christian Grün
>>> <christian.gruen@gmail.com> wrote:
>>>>
>>>> Thanks for the document. The declaration of the (unused) namespace in
>>>> the root element seems to be the cause for the decreasing performance
>>>> (I noticed that the time for adding documents stays constant after
>>>> removing the declaration). I'll do some profiling in order to find out
>>>> if this can be sped up without too much effort (it may take a while,
>>>> though, because I'll be on leave for a while from tomorrow).
>>>>
>>>>
>>>> On Tue, Sep 23, 2014 at 12:25 PM, Gerald de Jong <gerald@delving.eu>
>>>> wrote:
>>>> > I don't know what causes the gradual slowdown. My assumption was that
>>>> > it
>>>> > was the "optimize" which would cause the index to be built, so I didn't
>>>> > expect a slowdown at all during "add" calls, especially when autoflush
>>>> > is
>>>> > false.
>>>> >
>>>> > I add documents with the following paths:
>>>> >
>>>> > /f/f/e/ffe0f5be2aa14e81050f759c8f9c3eb7.xml
>>>> >
>>>> > The xml file name is a hash of the contents, and it is placed in a path
>>>> > such
>>>> > that the export spreads out the files nicely into a file system tree,
>>>> > rather
>>>> > than putting a million docs into one directory.
>>>> >
>>>> > The document content is nothing special, wrapped in a special tag:
>>>> >
>>>> > <narthex xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> > id="20412518"
>>>> > mod="2014-09-23T11:11:51.007+02:00">
>>>> > <record>
>>>> > <priref>20412518</priref>
>>>> > <current_location>FTA</current_location>
>>>> > <current_location.type/>
>>>> > <description>Ingang op de binnenplaats van de
>>>> > zuidvleugel</description>
>>>> > <collection>Fotocollectie</collection>
>>>> > <production.date.start>1925-08-06</production.date.start>
>>>> > <reproduction.format/>
>>>> >
>>>> >
>>>> > <reproduction.reference>2186abf4-7108-f9b8-ffbb-902881afe836</reproduction.reference>
>>>> > <creator.role>Fotograaf</creator.role>
>>>> > <object_number>9.387</object_number>
>>>> > <monument.label/>
>>>> > <monument.zipcode/>
>>>> > <monument.name>Kasteel Hoensbroek</monument.name>
>>>> > <monument.record_number>284330</monument.record_number>
>>>> > <reproduction.date/>
>>>> > <reproduction.notes>Oude filepath:
>>>> > 0009\009387.jpg</reproduction.notes>
>>>> > <reproduction.type/>
>>>> > <reproduction.creator/>
>>>> > <rights.type>Copyright</rights.type>
>>>> > <technique>Neg.zw</technique>
>>>> > <creator>Scheepens, W.C.L.A.</creator>
>>>> > <order_number>avh04-2008</order_number>
>>>> > <input.date>2008-04-01</input.date>
>>>> > <edit.date>2011-05-03</edit.date>
>>>> > <edit.date>2008-04-28</edit.date>
>>>> > <monument.historical_address/>
>>>> > <content.subject.type value="SUBJECT" option="SUBJECT">
>>>> > <text language="0">subject</text>
>>>> > <text language="1">onderwerp</text>
>>>> > <text language="2">sujet</text>
>>>> > <text language="3">Thema</text>
>>>> > <text language="4">موضوع</text>
>>>> > <text language="6">θέμα</text>
>>>> > </content.subject.type>
>>>> > <content.subject.type value="SUBJECT" option="SUBJECT">
>>>> > <text language="0">subject</text>
>>>> > <text language="1">onderwerp</text>
>>>> > <text language="2">sujet</text>
>>>> > <text language="3">Thema</text>
>>>> > <text language="4">موضوع</text>
>>>> > <text language="6">θέμα</text>
>>>> > </content.subject.type>
>>>> > <content.subject>Kasteel</content.subject>
>>>> > <content.subject>Binnenplaats</content.subject>
>>>> > <monument.province>Limburg</monument.province>
>>>> > <monument.place>Hoensbroek</monument.place>
>>>> > <monument.number/>
>>>> > <monument.county/>
>>>> > <monument.country>Nederland</monument.country>
>>>> > <monument.house_number>18</monument.house_number>
>>>> > <monument.street>Klinkertstraat</monument.street>
>>>> > <monument.house_number.addition/>
>>>> > <monument.complex_number/>
>>>> > <monument.number.x_coordinates/>
>>>> > <monument.number.y_coordinates/>
>>>> > <monument.geographical_keyword/>
>>>> > <monument.complex_number.x_coordinates/>
>>>> > <monument.complex_number.y_coordinates/>
>>>> > <creator.date_of_birth/>
>>>> > <creator.date_of_death/>
>>>> > <input.name>a.vanhoute</input.name>
>>>> > <edit.name>RCEadmin</edit.name>
>>>> > <edit.name>a.vanhoute</edit.name>
>>>> > <creator.history/>
>>>> > <record_type value="OBJECT" option="OBJECT">
>>>> > <text language="0">single object</text>
>>>> > <text language="2">objet individuel</text>
>>>> > <text language="3">Einzelnes Objekt</text>
>>>> > </record_type>
>>>> > <edit.time>03:10:32</edit.time>
>>>> > <edit.time>11:17:08</edit.time>
>>>> > <input.time>09:58:28</input.time>
>>>> > <input.source>document>photographs</input.source>
>>>> > <edit.source>collect>photograph</edit.source>
>>>> > <edit.source>document>photographs</edit.source>
>>>> > </record>
>>>> > </narthex>
>>>> >
>>>> > On Tue, Sep 23, 2014 at 11:36 AM, Christian Grün
>>>> > <christian.gruen@gmail.com>
>>>> > wrote:
>>>> >>
>>>> >> > I set up to use the 8.0-SNAPSHOT and used the internal parser as
>>>> >> > well.
>>>> >> > In
>>>> >> > your example you're not really giving much of a challenge to the
>>>> >> > index,
>>>> >> > since every doc is just <a/>.
>>>> >>
>>>> >> If I get it right, you assume the slowdown is due to the index
>>>> >> structures?
>>>> >>
>>>> >> > With respect to ADD, I'm not seeing a significant performance
>>>> >> > difference:
>>>> >>
>>>> >> Please give us more info on the data you are adding. Could you provide
>>>> >> us with a sample document?
>>>> >>
>>>> >>
>>>> >> > 8.0-SNAPSHOT
>>>> >> > -------
>>>> >> > 10000: 9250ms
>>>> >> > 20000: 7626ms
>>>> >> > 30000: 7885ms
>>>> >> > 40000: 8111ms
>>>> >> > 50000: 8365ms
>>>> >> > 60000: 8784ms
>>>> >> > 70000: 9270ms
>>>> >> > 80000: 9692ms
>>>> >> > 90000: 10158ms
>>>> >> > 100000: 10612ms
>>>> >> > 110000: 11018ms
>>>> >> > 120000: 11478ms
>>>> >> > 130000: 11940ms
>>>> >> > 140000: 12505ms
>>>> >> > 150000: 13047ms
>>>> >> > 160000: 13536ms
>>>> >> > 170000: 14055ms
>>>> >> > 180000: 14371ms
>>>> >> > 190000: 14883ms
>>>> >> > 200000: 15330ms
>>>> >> > 210000: 15888ms
>>>> >> > 220000: 16398ms
>>>> >> > 230000: 16878ms
>>>> >> > 240000: 17038ms
>>>> >> > 250000: 17453ms
>>>> >> > 260000: 17965ms
>>>> >> > 270000: 18317ms
>>>> >> > 280000: 18832ms
>>>> >> > 290000: 19373ms
>>>> >> > 300000: 19735ms
>>>> >> > 310000: 20062ms
>>>> >> > 320000: 20675ms
>>>> >> > 330000: 21113ms
>>>> >> > 340000: 21754ms
>>>> >> > 350000: 22887ms
>>>> >> > 360000: 22810ms
>>>> >> > 370000: 22985ms
>>>> >> > 380000: 23506ms
>>>> >> > 390000: 23856ms
>>>> >> > 400000: 24338ms
>>>> >> >
>>>> >> > 7.9
>>>> >> > -----
>>>> >> > 10000: 8229ms
>>>> >> > 20000: 7587ms
>>>> >> > 30000: 7973ms
>>>> >> > 40000: 8282ms
>>>> >> > 50000: 8717ms
>>>> >> > 60000: 9294ms
>>>> >> > 70000: 10105ms
>>>> >> > 80000: 10669ms
>>>> >> > 90000: 11301ms
>>>> >> > 100000: 11835ms
>>>> >> > 110000: 12413ms
>>>> >> > 120000: 13000ms
>>>> >> > 130000: 13577ms
>>>> >> > 140000: 14331ms
>>>> >> > 150000: 14488ms
>>>> >> > 160000: 15025ms
>>>> >> > 170000: 15463ms
>>>> >> > 180000: 15815ms
>>>> >> > 190000: 16153ms
>>>> >> > 200000: 16314ms
>>>> >> > 210000: 16562ms
>>>> >> > 220000: 17186ms
>>>> >> > 230000: 17862ms
>>>> >> > 240000: 18340ms
>>>> >> > 250000: 18790ms
>>>> >> > 260000: 19313ms
>>>> >> > 270000: 19850ms
>>>> >> > 280000: 20225ms
>>>> >> > 290000: 20650ms
>>>> >> > 300000: 21062ms
>>>> >> > 310000: 21595ms
>>>> >> > 320000: 22022ms
>>>> >> > 330000: 22414ms
>>>> >> > 340000: 22925ms
>>>> >> > 350000: 23514ms
>>>> >> > 360000: 23762ms
>>>> >> > 370000: 24360ms
>>>> >> > 380000: 25028ms
>>>> >> > 390000: 25446ms
>>>> >> > 400000: 25700ms
>>>> >> >
>>>> >> > - Gerald de Jong
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Sep 18, 2014 at 6:57 PM, Christian Grün
>>>> >> > <christian.gruen@gmail.com>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> > Perhaps you can give me a hint as to why inserts slow down.j
>>>> >> >> I didn't have time to check out 7.9, but I have done some testing
>>>> >> >> with
>>>> >> >> 8.0, and I didn't notice a real slow-down. This is Java testing
>>>> >> >> script
>>>> >> >> (1 mio documents are added in just 17 seconds; I'm using the
>>>> >> >> internal
>>>> >> >> BaseX parser to speed up the import):
>>>> >> >>
>>>> >> >> Performance p = new Performance();
>>>> >> >> Context ctx = new Context();
>>>> >> >>
>>>> >> >> new CreateDB("db").execute(ctx);
>>>> >> >> new Set(MainOptions.AUTOFLUSH, false).execute(ctx);
>>>> >> >> new Set(MainOptions.INTPARSE, true).execute(ctx);
>>>> >> >> for(int i = 0; i < 1000000; i++) {
>>>> >> >> new Add("db", "<a/>").execute(ctx);
>>>> >> >> }
>>>> >> >> ctx.close();
>>>> >> >> System.out.println(p);
>>>> >> >>
>>>> >> >> Hope this helps,
>>>> >> >> Christian
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > Delving BV, Vasteland 8, Rotterdam
>>>> >> > http://www.delving.eu
>>>> >> > http://twitter.com/fluxe
>>>> >> > skype: beautifulcode
>>>> >> > +31629339805
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Delving BV, Vasteland 8, Rotterdam
>>>> > http://www.delving.eu
>>>> > http://twitter.com/fluxe
>>>> > skype: beautifulcode
>>>> > +31629339805
>>>
>>>
>>>
>>>
>>> --
>>> Delving BV, Vasteland 8, Rotterdam
>>> http://www.delving.eu
>>> http://twitter.com/fluxe
>>> skype: beautifulcode
>>> +31629339805
>>
>>
>>
>>
>> --
>> Delving BV, Vasteland 8, Rotterdam
>> http://www.delving.eu
>> http://twitter.com/fluxe
>> skype: beautifulcode
>> +31629339805