Rank: Newbie
Groups: Member
Joined: 5/22/2007
Posts: 2
Points: 6
|
When spidering our site we have decided to limit the number spidered pages to avoid duplicate results (same content re-sorted, etc). After pairing down available pages the total count is about 200K. The program spiders the site fine but we get a "memory error" message and the spidered results are not available.
I saw the post in regards to large sites and obviously the spidering results are stored in memory, the system we are running has 2GB ram, should this be a problem? Or is it something else?
|
|
|
|
Rank: Administration
Groups: Administration
Joined: 1/31/2007
Posts: 440
Points: 646
Location: Chicago, IL
|
That's a very large site, and the Sitemap Generator is certainly using a lot of resources.
Can you view the processes to see what's consuming the memory? What else is running?
We regularly spider sites with over 1 or 2 million URLs on a computer that has 4 GB memory.
So 200K URLs with 2GB memory should not be an issue.
|
