About robots.txt

[TwiLight]

Even online theres no privacy, huh. Anyone knows how to put it? where is the root server directory in the pc? I wanna kill them or just ignore them.:banghead:

 

[enouwee]

Just look under How to create a /robots.txt file.

 

[TwiLight]

Dont have my own website or blogsite. Guess have to live and die with it huh.:banghead: Aargh.

 

[TFlan]

If you don't have a webserver, then you don't need robots.txt.

Why are you referring to robots.txt for online privacy? Robots.txt tells what search engine crawlers should/should not index.

 

[TwiLight]

Aah okey, that clears it. But still...? wondering those online pests. Hmmm....

 

[Samuraid]

Search engine bots cannot spider content from you if you don't have any web space anywhere, so there's nothing about which to worry.

 

[The Helper]

Everything you want to know about robots.txt is in this thread except for the fact that you don't need it. If you want the spiders to go crazy on your site leave it blank but remember those spiders come back every day. If you have stuff you host that you dont want downloaded by robots every day and eating up your bandwidth put those exclusions in your robots.txt files.