Robots.txt

  Back Help Me!

Hints

Directory Browsing Hints

Overview Directory Browsing is allowed when the web server is misconfigured to show the user the contents of directories on the server. Discovery Methodology Use search engines to look for pages which include "index of" in the title. Additionally attempt to read the robots.txt file and spider the application with a tool such as Burp-Suite, OWASP ZAP, or Nikto, and directories named by search engines. Exploitation Catalog and inspect the folders named in robots.txt and any directories discovered during spidering. Use a tool such as Burp-Intruder to brute-force sub-directory names in the root directory and other discovered directories.

Information Path Hints

Overview Information Disclosure may result when internal information is disclosed to the user-agent (browser). These paths can be used in other attacks. Discovery Methodology Attempt to discover if it is possible to cause errors by injecting all input parameters with characters that are reserved in various contexts. Search web page sources (view source) for internal information disclosure. Search for custom administrative pages and administrative consoles such as phpMyAdmin installations. Exploitation Search pages with and without injection. Use the grep feature of Burp-Suite to seach for inappropriate information. Search for known common administrative consoles such as phpMyAdmin installations, Drupal and Wordpress consoles. Videos Determine HTTP Methods using Netcat: Load the video Determine Server Banners using Netcat, Nikto, and w3af: Load the video Using Nmap to Fingerprint HTTP servers and Web Applications: Load the video Finding Comments and File Metadata using Multiple Techniques: Load the video

Robots.txt Hints

Overview The Robot Exclusion Standard, Robots Exclusion Protocol or robots.txt protocol is a suggestion for "honest" browsers which should prevent web crawlers from accessing the parts of a website listed in the roboos.txt file. Discovery Methodology Browse to the robots.txt file in Mutillidae Exploitation Follow the paths in the robots.txt file Example Robots.txt is located at localhost/mutillidae/robots.txt

Robots.txt

Sites use the robot.txt file in the site root to discourage web crawlers from indexing site content. Robots.txt is a plain text file which can be read by site visitors. In some cases, the robots.txt file will point to sensitive pages or directories. If a sensitive file is placed in robots.txt without proper authorization controls protecting the file, site visitors may discover the contents and browse to the files. More information is available at Robots Exclusion Standard