Dear Sekhar,
Please do the needful.
Sorry again, can't resist: http://www.reddit.com/r/sysadmin/comments/2eaa0b/do_the_needful/
I hope it's not interpreted the wrong way… I guess that Germans use lots of phrases that sound more than quirky to British and American speakers. Feedback is always welcome!
Still, I did the needful, and I extended the documentation for the ADD command to make it more obvious what's going on [1]. When sending the data via an input stream, there is no way to detect in advance that your input may not fit into main memory, but you want to enforce caching of large documents, you can use the ADDCACHE option [2].
There is currently no way to both speed up processing and reduce memory consumption, because we need to be sure that the requested input will be valid before storing it in the database.
I also have a question: how many documents (or how many gigabytes in terms of input documents) do you store in a single database?
All the best, Christian
[1] http://docs.basex.org/wiki/Commands#ADD [2] http://docs.basex.org/wiki/Options#ADDCACHE