R: Parse and Test Robots Exclusion Protocol Files and Rules

# Parse and Test Robots Exclusion Protocol Files and Rules

## Documentation for package ‘spiderbar’ version 0.2.5

- DESCRIPTION file

## Help Pages

can_fetch | Test URL paths against a 'robxp' 'robots.txt' object
crawl_delays | Retrieve all agent crawl delay values in a 'robxp' 'robots.txt' object
robxp | Parse a 'robots.txt' file & create a 'robxp' object
sitemaps | Retrieve a character vector of sitemaps from a parsed robots.txt object
spiderbar | Parse and Test Robots Exclusion Protocol Files and Rules