Hi I have purchased the i architect site map generator some months ago and found it to be one of the best site map generators.

I have run into a problem though since one one of my websites i have built some what a massive website with hundereds of thousands of pages.

I am using a free affiliate store builder script called affilistore http://www.affilistore.com and this is a scripts working in conjunction with a mysql database.
I use merchants data feeds to create webpages and content and i so far am using about 30 data feeds with a total of more than 200,000 at least... probably alot more...products , each producing their own unique webpage per product.
However I when I went to spider the website with Iarchitect it gets upto about 17,500 unique urls quite quickly and then it pretty much seems to jam up

Any suggestions on whats happening here? I have heard that websites that have mysql databases that use scripts to generate content can sometimes be a little complicated to make site maps for and wanted to know if there are any queires or url strings etc that I might need to filter in order to get the site map maker functioning again properly and be able to spider more than 17,500 pages before mucking up?

My site that i am having difficulty with is http://www.online-shop.com.au the starting folder where the affilistore script is used is http://www.online-shop.com.au/catalogues/

Any help or suggestions would be much appreciated.

---

Our Websites: Online shopping * Online shop catalogues * Cell Phone * fishing tackle store

Hi onlineshop,

We have successfully spidered sites with over 4 million pages.
These sites have been PHP, ASP, .NET and Java...and with a MySQL, SQL Server and DB/2 database.
Sometimes the software can seem hung up for a minute (usually less) when it's spidering a very large page and has to check if all of the links have been added to the queue already.
Next time you run it, can you see if the processor/memory is going up and down? If so, the application is working...and not just looping.

Thanks,
iArchitect

Sounds like you're on the right track.
I'm certain the software is working fast...the website must have a ton of links on each page and/or the HTTP Redirects are cuasing issues.
Also, as each page is visited, it has to fully render the HTML before it can move onto the next page.
If you have external HTML/links that need to contact a 3rd party before fully rendering, you're at the mercy of their servers.
Hope this helps.