Omeka and Search Engines

How do I configure the robots.txt file, in the root directory of my website, so that search engines, such as Google and Yahoo, properly index my omeka installation? What directories should be allowed or dis-allowed?

Will a search engine spider the entire database, and its metadata?
My omeka installation directory is: industrial-history.org/archive/