# The robots.txt file 

# When a Robot visits a Web site, say http://www.foobar.com/, it firsts checks for http://www.foobar.com/robots.txt. 
# If it can find this document, it will analyze its contents to see if it is allowed to retrieve the document. 
# You can customize the robots.txt file to apply only to specific robots, and to disallow access to specific directories or files.

# Here is a sample robots.txt file that prevents all robots from visiting the entire site

#        User-agent: *    # applies to all robots
#        Disallow: /      # disallow indexing of all pages

# for the moment, allow the robots to index the entire pfm website, this file is in htdocs/robots.txt
         User-agent: *    # applies to all robots
          Disallow:       # disallow indexing of all pages