Hello,
I've got a collection with about 1.4 million small XML files in it. I can load all these through the GUI with no problems.
However really I need to be loading them, a few at a time, through PHP and have code to do this. Because the process involves add, query, add and so on I have set UPDINDEX to true on the database. In most instances everything works fine but sometimes I get the error below (it's not on particular documents, a second run will give the error in different places).
Once this error has been received it seems the OPTIMIZE doesn't work properly on the database - giving an array out of range error.
I don't understand the stack trace enough to know what the most likely cause is. Can anyone provide me with some guidance?
Many thanks, James
exception 'Exception' with message 'Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.7 Java: Apple Inc., 1.6.0_51 OS: Mac OS X, x86_64 Stack Trace: java.lang.IllegalStateException: Key should not exist at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:79) at org.basex.data.DiskData.indexEnd(DiskData.java:356) at org.basex.data.Data.insert(Data.java:817) at org.basex.core.cmd.Add.run(Add.java:107) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.run(Command.java:117) at org.basex.core.cmd.Replace.run(Replace.java:68) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.exec(Command.java:321) at org.basex.core.Command.execute(Command.java:78) at org.basex.core.Command.execute(Command.java:90) at org.basex.server.ClientListener.execute(ClientListener.java:386) at org.basex.server.ClientListener.replace(ClientListener.java:365) at org.basex.server.ClientListener.run(ClientListener.java:106) ' in /Library/WebServer/Documents/ISIS/BaseXClient.php:101 Stack trace: #0 /Library/WebServer/Documents/ISIS/BaseXClient.php(60): Session->sendCmd(12, 'ISIS_IND_BIO/IW...', '<?xml version="...') #1 /Library/WebServer/Documents/ISIS/uploadBioCli.php(79): Session->replace('ISIS_IND_BIO/IW...', '<?xml version="...') #2 {main}Processing file 358 /Users/jb8748/Documents/Data Auditing/ISIS_IND_BIO/IWF/-9458-.xml
Hi James,
this looks like a potential bug, but it might be tricky to reproduce it with sample data. Have you experienced similar feedback when using Java?
Best, Christian ___________________________
2013/9/9 James Ball basex-talk@jamesball.co.uk:
Hello,
I've got a collection with about 1.4 million small XML files in it. I can load all these through the GUI with no problems.
However really I need to be loading them, a few at a time, through PHP and have code to do this. Because the process involves add, query, add and so on I have set UPDINDEX to true on the database. In most instances everything works fine but sometimes I get the error below (it's not on particular documents, a second run will give the error in different places).
Once this error has been received it seems the OPTIMIZE doesn't work properly on the database - giving an array out of range error.
I don't understand the stack trace enough to know what the most likely cause is. Can anyone provide me with some guidance?
Many thanks, James
exception 'Exception' with message 'Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.7 Java: Apple Inc., 1.6.0_51 OS: Mac OS X, x86_64 Stack Trace: java.lang.IllegalStateException: Key should not exist at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:79) at org.basex.data.DiskData.indexEnd(DiskData.java:356) at org.basex.data.Data.insert(Data.java:817) at org.basex.core.cmd.Add.run(Add.java:107) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.run(Command.java:117) at org.basex.core.cmd.Replace.run(Replace.java:68) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.exec(Command.java:321) at org.basex.core.Command.execute(Command.java:78) at org.basex.core.Command.execute(Command.java:90) at org.basex.server.ClientListener.execute(ClientListener.java:386) at org.basex.server.ClientListener.replace(ClientListener.java:365) at org.basex.server.ClientListener.run(ClientListener.java:106) ' in /Library/WebServer/Documents/ISIS/BaseXClient.php:101 Stack trace: #0 /Library/WebServer/Documents/ISIS/BaseXClient.php(60): Session->sendCmd(12, 'ISIS_IND_BIO/IW...', '<?xml version="...') #1 /Library/WebServer/Documents/ISIS/uploadBioCli.php(79): Session->replace('ISIS_IND_BIO/IW...', '<?xml version="...') #2 {main}Processing file 358 /Users/jb8748/Documents/Data Auditing/ISIS_IND_BIO/IWF/-9458-.xml
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Christian,
Thank you for that information. I haven't tried it with Java (and my Java is a bit rusty at the moment) but I will try it with Java and see what happens. It won't be for at least a few days now.
Whilst I can't share the exact data being used it's of a very regular form. If I find this behaviour is repeatable I will try to generate a test set that replicates it.
I will be in touch.
Thanks, James
On 9 Sep 2013, at 23:09, Christian Grün christian.gruen@gmail.com wrote:
Hi James,
this looks like a potential bug, but it might be tricky to reproduce it with sample data. Have you experienced similar feedback when using Java?
Best, Christian ___________________________
2013/9/9 James Ball basex-talk@jamesball.co.uk:
Hello,
I've got a collection with about 1.4 million small XML files in it. I can load all these through the GUI with no problems.
However really I need to be loading them, a few at a time, through PHP and have code to do this. Because the process involves add, query, add and so on I have set UPDINDEX to true on the database. In most instances everything works fine but sometimes I get the error below (it's not on particular documents, a second run will give the error in different places).
Once this error has been received it seems the OPTIMIZE doesn't work properly on the database - giving an array out of range error.
I don't understand the stack trace enough to know what the most likely cause is. Can anyone provide me with some guidance?
Many thanks, James
exception 'Exception' with message 'Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.7 Java: Apple Inc., 1.6.0_51 OS: Mac OS X, x86_64 Stack Trace: java.lang.IllegalStateException: Key should not exist at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:79) at org.basex.data.DiskData.indexEnd(DiskData.java:356) at org.basex.data.Data.insert(Data.java:817) at org.basex.core.cmd.Add.run(Add.java:107) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.run(Command.java:117) at org.basex.core.cmd.Replace.run(Replace.java:68) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.exec(Command.java:321) at org.basex.core.Command.execute(Command.java:78) at org.basex.core.Command.execute(Command.java:90) at org.basex.server.ClientListener.execute(ClientListener.java:386) at org.basex.server.ClientListener.replace(ClientListener.java:365) at org.basex.server.ClientListener.run(ClientListener.java:106) ' in /Library/WebServer/Documents/ISIS/BaseXClient.php:101 Stack trace: #0 /Library/WebServer/Documents/ISIS/BaseXClient.php(60): Session->sendCmd(12, 'ISIS_IND_BIO/IW...', '<?xml version="...') #1 /Library/WebServer/Documents/ISIS/uploadBioCli.php(79): Session->replace('ISIS_IND_BIO/IW...', '<?xml version="...') #2 {main}Processing file 358 /Users/jb8748/Documents/Data Auditing/ISIS_IND_BIO/IWF/-9458-.xml
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Am Dienstag, 10. September 2013, 10:24:50 schrieb James Ball:
I've got a collection with about 1.4 million small XML files in it. I can load all these through the GUI with no problems.
However really I need to be loading them, a few at a time, through PHP and have code to do this. Because the process involves add, query, add and so on I have set UPDINDEX to true on the database. In most instances everything works fine but sometimes I get the error below (it's not on particular documents, a second run will give the error in different places).
Can you try to reproduce the problem without using PHP, but a BaseX command script [1] instead (e.g. many XQUERY commands each executing a replace with a few documents). If the problem persists, we will be able to eliminate the PHP binding and you will not have to use java ;)
exception 'Exception' with message 'Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.7 Java: Apple Inc., 1.6.0_51 OS: Mac OS X, x86_64 Stack Trace: java.lang.IllegalStateException: Key should not exist
at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValu es.java:79) at org.basex.data.DiskData.indexEnd(DiskData.java:356)
Can you check if the error occurs always at line 356 in DiskData.java? If yes, will it be possible to send us a list of all attribute values from all documents?
The problem is that a key, which has been determined to be not existing in a previous step, actually exists in the index when trying to insert it. I checked the relevant code and I fail the see any obvious error. In my opinion the following can cause the incorrect behavior:
1. the PHP binding somehow manages to update the database in parallel (the locking mechanism should already prevent that) 2. there is a bug in the index update code: possible, although I fail to see it; either reproducible example should be provided or another pair of eyes 3. something else (e.g. our implementation of hash table contains duplicate keys, which is very unlikely).
Best regards, Dimitar
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Dimitar Popov -
James Ball