Skip to main content

Acceptable Use Policy

Effective Date: January 22, 2026

This Acceptable Use Policy ("AUP") governs your use of DocWeb. Violations may result in account suspension or termination.

Responsible Crawling​

Respect robots.txt​

DocWeb strictly follows robots.txt directives. You should not:

  • Request discovery of sites that explicitly disallow our bot
  • Attempt to circumvent crawling restrictions

Rate Limiting​

DocWeb has built-in rate limiting. You should not:

  • Make excessive repeated discoveries of the same domain
  • Attempt to bypass rate limits
  • Deliberately overload target servers

Crawl-Delay Compliance​

We honor crawl-delay directives. Do not expect faster crawling than the target site allows.

Permitted Use Cases​

Documentation Analysis​

  • Exploring technical documentation for learning
  • Understanding API structures and capabilities
  • Competitive intelligence on publicly available docs

Research​

  • Academic research on information architecture
  • SEO analysis of documentation sites
  • User experience research

Business Use​

  • Onboarding developers to new technologies
  • Creating internal knowledge bases
  • Training materials development

Prohibited Activities​

Illegal Activities​

  • Accessing systems without authorization
  • Violating computer fraud laws
  • Infringing intellectual property

Harmful Use​

  • Scraping personal or sensitive information
  • Building datasets for malicious AI training
  • Competitive harm to scraped sites

Circumvention​

  • Bypassing authentication or paywalls
  • Evading anti-bot measures
  • Manipulating our credit system

Abuse​

  • Creating multiple accounts to evade limits
  • Automated abuse of our API
  • Denial of service attacks

Content Restrictions​

Do not use DocWeb to discover or scrape:

  • Personal data (unless authorized)
  • Financial information
  • Medical records
  • Authentication credentials
  • Proprietary source code
  • Illegal content

Third-Party Rights​

Intellectual Property​

  • Scraped content belongs to original owners
  • Respect copyright and trademark laws
  • Do not redistribute content without rights

Terms of Service​

  • You are responsible for compliance with target site ToS
  • Some sites prohibit automated access
  • Obtain permission when required

Reporting Violations​

If you discover violations of this AUP:

  • Contact us at [email protected]
  • Provide details and evidence
  • We will investigate and take appropriate action

Enforcement​

Violation Response​

Depending on severity, we may:

  1. Issue a warning
  2. Temporarily suspend your account
  3. Permanently terminate your account
  4. Report to law enforcement

Appeals​

If you believe enforcement was in error:

  • Contact us within 30 days
  • Provide explanation and evidence
  • We will review and respond

Changes​

We may update this AUP. Check back periodically for changes.

Questions​

Contact us at [email protected] with questions about permitted use.