# This file is used to tell webcrawlers what they should and should not index on your 
# sites.  Each BW website should have one of these files, called robots.txt, in their home 
# directory.  The file must be customized for the site, but it isn't very hard.  The lines 
# starting with "#" are comments, which are ignored by the indexers.  The next few lines 
# are an example.
 
# # robots.txt for http://www.site.com/
#

# User-agent: *
	# The user-agent line allows you to specify that one web indexer (e.g. AltaVista)
	# may look at "x" directory, but another (e.g. WebCrawler) may not.  Since there 
	# isn't any real reason to do this, our sites should just have an * next to the colon 
	# to say that it applies to all searchers.

# Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Disallow: /tmp/ # these will soon disappear
	# The two lines above tell the indexers to ignore these two directories.  The "#" in 
	# the middle of the line allows you to start a comment there to explain that line, 
	# just making a note as to why the directory is disallowed.  

##################################
# Everything that isn't specifically mentioned here will be fair game.  That's all that there 
# is to it.  Any questions, ask me, and I'll get you an answer.  Below is a generic file that 
# you can use as a jumping off point.  Just  replace the entries below with entries for your 
# site, save your changes, and then place the whole file in the home directory of your 
# website, and then you're done.
##################################

# robots.txt for http://www.monsanto.co.uk/

User-agent: *
Disallow: /activityreport/
Disallow: /news/Disabled/

# No other directories are present at this time. Maybe after a restructure?