Adds a regex of URLs that should be excluded from the spider scans.