Glossary
robots.txt
A file at a site's root stating which paths automated clients should not fetch.
It is a convention, not an access control. Whether your collection honours it is a policy decision to record, particularly where the data feeds a model or a product.
Related terms
- Know your customer (KYC)
Checks a provider makes on a customer before enabling access.
Test it on your own targets.
Create an account, run the measurement harness against your real workload, and read the numbers before you commit to anything.