Sometimes, we might want search engines to not catalog certain parts of the site, and on occasion even prohibit other SE from the site all together. This really is where a simple, little 2 line text file called robots.txt will come in. Once we've a web site up and running, we should make certain that all visiting search engines can access all the pages we want them to look at. Sometimes, we may want search-engines to not index certain elements of the site, as well as exclude other SE in the site altogether. This is where a simple, little 2 line text file called robots.txt will come in. Robots.txt rests inside your websites main directory (o-n LINUX systems that is your /public_html/ directory), and looks something like the following User-agent * Disallow The initial line controls the robot that will be visiting your site, the next line controls if they are allowed in, or which elements of the site they're perhaps not allowed to see If you like to take care of multiple robots, then basic repeat the aforementioned lines. So an example User-agent googlebot Disallow User-agent askjeeves Disallow / This may enable Goggle (user-agent name GoogleBot) to see every page and index, while at the same time banning Ask Jeeves from the site fully. To discover a reasonably up to date list of robot consumer names this visit http://www.robotstxt.org/wc/active/html/index.html Its still very advisable to place a robots.txt report on your own site, even if you wish to let every software to index every page of your site. It'll end your mistake logs filling up with items from se's wanting to access your robots.txt file that doesnt occur. In the event people require to identify additional resources on [http://people.tribe.net/f0db649b-d838-498b-bb98-bf6280271166/blog clicky], we know about many online libraries people might pursue. Browsing To [https://dribbble.com/tricktea87so home page] probably provides aids you should give to your co-worker. For more information on robots.txt see, the full listing of resources about robots.txt at http://www.websitesecrets101.com/robotstxt-further-reading-resources. [https://foursquare.com/user/122467636/list/making-your-own-personal-facebook-movie The Internet] is a novel resource for more concerning the reason for this viewpoint. Dig up extra info on this affiliated site by visiting [http://www.bookcrossing.com/mybookshelf/companywebsitegemini/ http://youtube.com/user/1orangecountyseo].
There are no comments on this page.
Valid XHTML :: Valid CSS: :: Powered by WikkaWiki