Rank: Newbie Groups: Member
Joined: 9/15/2010 Posts: 1 Points: 3 Location: Atlanta, GA
|
Are there plans for coming out with a multi threaded version?
I am running on a 12gb Extreme Q9300, which has 4 cores, it looks to just be using one of the core but is doing that at 100% for that individual core. Allowing for more connections on more cores would speed up the crawling for me, am not connection bound.
|
|
|
|
Rank: Administration Groups: Administration
Joined: 1/31/2007 Posts: 590 Points: 396 Location: Chicago, IL
|
The original versions of the Sitemap Generator were multi-threaded, but we removed that functionality because it doesn't help most websites (it actually hurts them). Most websites are running on slower, shared servers and cannot handle a lot of requests, especially concurrent requests. Some customers who then used our product couldn't get their entire sites crawled because the software would "take down" their site. We had to implement the opposite: Pause Time. Pause Time allows you to specify how many seconds to wait between requests. Also, the software is already very fast. It really cannot run any faster. It's waiting on the website to serve up webpages. You could run the software on a Dual core or a Quad Core and the overall timing would be nearly identical. So to answer your question, no, we do not have plans to reimplement multi-threading.
Thanks, KeyLimeTie
|