|
|
Rank: Newbie
Groups: Member
Joined: 8/9/2008
Posts: 5
Points: 15
Location: Arizona
|
I read a few places that if I place all my sitemap files in a robots.txt file, they will be autodiscovered. Is still accurate?
If so, perhaps the software could generate this file as part of the process?
-- Thomas
|
|
|
|
|
|
Rank: Administration
Groups: Administration
Joined: 1/31/2007
Posts: 590
Points: 396
Location: Chicago, IL
|
The robots.txt file is primarily used to tell search engines which folders/files to exclude from their spidering/indexing. Read more about "Robots Exclusion Standard" at http://en.wikipedia.org/wiki/Robots.txt
|
|
Rank: Newbie
Groups: Member
Joined: 8/9/2008
Posts: 5
Points: 15
Location: Arizona
|
Are you saying that you do not recommend adding a sitemap entry to the robots.txt file? I'm reading all over the place that it is best practice to do this so I'm confused now. This line is what is recommended to be added within the robots file to "autodiscover" the sitemaps files. Sitemap: http://www.mywebsite.com/sitemap.xmlThomas
|
|
Rank: Administration
Groups: Administration
Joined: 1/31/2007
Posts: 590
Points: 396
Location: Chicago, IL
|
We have always seen this file as a location to put folders/files to disallow access to.
It is not traditional to put "allow" folders/files in here, but this might be a new SEO trick.
|
|
|
Guest |
