I have a domain name hosted on FreeParking (domain is nidderdalemessiah.org.uk), with redirection to actual web pages on ryoung.myzen.co.uk/nm1/. If I spider redirected name it finds way to redirected root and scans a page at ryoung.myzen.co.uk/nm1, but then spidering stops. It reports that it has found the URL www.domain-parking.net, but I don't have that in any page on my sites. It is a URL belonging to the FreeParking company, but I have never seen it before.

If I spider ryoung.myzen.co.uk/nm1/ directly (with nm1 as first directory to spider because it seems to discard the /nm1/ off the website name I type) then it spiders all pages in the site.

What is going on and how do I overcome it and create site map for my domain name hosted on FreeParking.

Rob

For anyone stuck with similar problem I have found explanation myself. FreeParking (and presumably other domain hosts) uses URL masking to control redirection, and I now understand that this is done with a simple frames page. The target site exists in a single jumbo frame which retrieves your site with a LOCATION statement. There is a hyperlink to your target in the NOFRAMES for browsers that don't do frames.

The spider does not see through this location statement, and so does not crawl your site and build a site map. I asume this is to emulate the action of the search engine crawlers that have a reputation of not getting through a page with frames, but I am not sure if this is still true, most of the search engines now see through most code that a browser can traverse it seems to me.

I have overcome this by setting DNS record to point directly to my target site, and option that you have on FreeParking. Spider now crawls and builds site maps.

Perhaps we need an option to instruct the spider to traverse such location statements, maybe only if they occur on 1st hop if there is a reason for current behaviour of SiteMap Generator.

Hope this helps someone

Rob Young

This feature will be added to v5.0 set to be released 1/23/08.

Thanks,
iArchitect