I have had problems, now overcome, with a site for a domain name that is hosted on Freeparking, but site is on another ISP with redirection pointing to target pages in the redirection options at FreeParking. I assume this is quite common with servers that host names and offer redirection control to your current target location. I now know this is achieved with a simple page generated by FreeParking servers with target site held in a single jumbo frame, a LOCATION statement points to the true target.

It seems the spider doesn't see through this LOCATION statement, so doesn't generate a site map. I assume this is to emulate browser web crawlers, but I suspect that these days the crawlers do get through such code, and all sorts of other http that all the browsers just sail through to display pages to the public.

Could we have an option for spider to see through whatever the web crawlers now cross?

Regards

Rob Young

We will investigate this.