#Spiders shouldn't look at the stuff needed under athentication User-agent: * Disallow: /authenticated/ #The Sosospider is a real pain. Disable it. User-agent: Sosospider Disallow: /