What is the limit on the amount of pages this generator can handle? I tried indexing a very large site (over 1 million pages) with another generator and it wouldn't go past 125K pages because the database got too big. I want to buy this software but need to know it can handle a very large site.

Also will this generator split up the sitemaps into 50,000 url files automatically or would I have to do this manually?

Thanks,
PK

pk_synths,

Sorry for not replying sooner.

We have successfully indexed sites with as many as 3 million URLs. We purposely designed the software to run without a database and completely in memory...databases slow the spidering process down dramatically and put a huge amount of stress on the computer processor, memory and hard drive. Because we run in memory, you will need a good amount of memory to index such a huge site. When we spidered the 3 million URLs website, we did it on a computer that has 4 GB RAM. When we checked on it every so often, it never got anywhere near needing the 4GB RAM. In fact, I think it stayed under 2 GB. So on a site with 1 million pages, I think 1 GB might be enough, and 2GB would definitely be enough.

Also, while it's running, you will definitely want to turn on the "Auto Save" preference. In case you lose your internet connection, your power fails, etc., this feature automatically saves the spidering progress every "n" pages (you specify) to the hard drive. When you go to resume the spidering, it will pick up where it left off. I recommend setting that to 500 or 1,000.

Yes, the generator will automatically split the sitemap into multiple sitemaps.

Thanks,
Brian