Rank: Newbie
Groups: Member
Joined: 8/15/2007
Posts: 2
Points: 6
Location: Toronto, ON
|
What can I do to stop this. I have 25 or so webpages and maybe 200 posts in our forum. Our site http://www.dynamicgreens.comOur forum http://www.dynamicgreens.com/wheatgrassforumHelp! Derek
|
Rank: Administration
Groups: Administration
Joined: 1/31/2007
Posts: 590
Points: 396
Location: Chicago, IL
|
Your forums software appears to pass a lot of parameters around.
I see a lot of pages passing "s" in the querystring. Must be the session ID.
To ignore that, enter "s" in the "Ignore Querystring" field in the preferences.
Once I did that, the total # pages dropped to around 1,200 (I stopped thye spider though).
Now, you need to go through the pages it's finding and decided what you want and what you don't want.
For example, do you want "misc.php" in the sitemap? If not add it to the "Ignore URL Strings".
Same with search.php...I would think that shouldn't be in the sitemap.
And finally, if there are pages you want in the sitemap, check out the querystring parameters.
For example, if there's a "sort" in the parameter, add it to the "Ignore Querystring" list.
You don't want the same page in the sitemap 5 times sorted 5 ways.
Thanks,
Brian
|
Rank: Newbie
Groups: Member
Joined: 8/15/2007
Posts: 2
Points: 6
Location: Toronto, ON
|
Thank you Brian. I am used to really simple sitemap generators and just didn't give this enough thought.
If anyone is curious about vBulletin forum settings, I am using v 3.6.8 and this is what works for me:
Ignore Querystring:
daysprune,do,find,forumdisplay,goto,ltr,order,postcount,pp,s,searchthreadid,sort,t
Ignore URL Strings:
login.php,member.php,misc.php,newreply.php,newthread.php,online.php,printthread,search.php,sendmessage.php,showgroups.php,showpost.php
Ignore Folders:
none
Your requirements may be a little different, but this took the indexing down from (unlimited - not stopping with an 80,000+ count when I ran it overnight) to about 250 pages (which is right on).
Thanks.
Derek
|
Rank: Administration
Groups: Administration
Joined: 1/31/2007
Posts: 590
Points: 396
Location: Chicago, IL
|
Thanks for the great information Derek!
Locking thread as issue has been resolved.
|
