Protego is a pure-Python robots.txt parser with support for modern conventions.