Re: [basex-talk] Wasn't there a function, that would walk a website?

Vincent Lizzi wrote:

Hi Andreas, Christian,

Here attached is a module that I wrote a while ago to limit the rate of requests sent to a web server. This module has been useful in accessing APIs where the SLA does not allow more than a certain number requests per minute, and might be useful for this web crawling scenario. Although Cristian's crawler module already has a sleep built in to it.

Nice! I think I'll just lean back and wait for the code to roll in ;-)

Finally a full example of unit testing in the wild! I never really could wrap my mind around it. Also interesting to see the call to the Java environment. These are advanced techniques (though, they look simple enough, for the start, at least), which I yet have to come by.