Welcome Guest Search | Active Topics | Members | Log In | Register

Spider picks up 404 page Options
jfordbevnet
Posted: Tuesday, November 17, 2009 4:03:08 PM
Rank: Newbie
Groups: Member

Joined: 11/17/2009
Posts: 1
Points: 3
Our site has a 404 Page that is served when an invalid URL is entered into the browser:

e.g. http://www.bevnet.com/Aisle-Grocery/b/ref=sv_gro_0?ie=UTF8&node=52128011

Unfortunately, it seems the Sitemap Generator considers this a real URL, and adds these bad URLs to the sitemap. Is there a way of filtering these out?

Sponsor
Posted: Tuesday, November 17, 2009 4:03:08 PM
Get your Sitemap Generator license today! http://www.keylimetie.com/Checkout/Quick-PayPal/
KeyLimeTie
Posted: Tuesday, November 17, 2009 11:01:27 PM
Rank: Administration
Groups: Administration

Joined: 1/31/2007
Posts: 586
Points: 384
Location: Chicago, IL
If the website does not return a 404, the software cannot determine it's a 404.
One thing you can do is let the software spider the entire site, then edit the results, sort by title, and delete all that have the 404 title.
Users browsing this topic
Guest


Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

Main Forum RSS : RSS

None
Powered by Yet Another Forum.net version 1.9.1.2 (NET v2.0) - 9/27/2007
Copyright © 2003-2006 Yet Another Forum.net. All rights reserved.
This page was generated in 0.061 seconds.