Discover URLs on a domain

Crawls a domain to build a sitemap. Returns cached results if available within cache_ttl. Results delivered via webhook and downloadable from results_url. Webhook deliveries include `X-RapidCrawl-Event: sitemap.completed` or `X-RapidCrawl-Event: sitemap.failed` so receivers can dispatch on the header without parsing the body shape.

Authorization

AuthorizationRequiredBearer <token>

API key. Obtain from POST /v1/account/api-keys.

In: header

Request Body

application/jsonRequired

urlRequiredstring

Format: "uri"

strategystring

Default: "standard"Value in: "quick" | "standard" | "deep"

max_depthinteger

Default: 2Minimum: 1Maximum: 5

max_urlsinteger

Default: 10000Minimum: 1Maximum: 100000

processobject | null

Default: null

webhook_urlRequiredstring

Webhook endpoint URL (HTTPS only; http:// rejected with HTTP 422)

Pattern: "^https://"Format: "uri"

webhook_secretstring | null

Default: null

cache_ttlstring

Max 90 days.

Default: "30d"Pattern: "^(0|\\d+[dh])$"

compactboolean

Default: false

metadataobject

Arbitrary key-value metadata. Max 4096 bytes when JSON-serialized.

curl -X POST "https://api.bytekit.com/v1/sitemap" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "http://example.com",
    "strategy": "quick",
    "max_depth": 2,
    "max_urls": 10000,
    "process": null,
    "webhook_url": "http://example.com",
    "webhook_secret": null,
    "cache_ttl": "30d",
    "compact": false,
    "metadata": {}
  }'

Sitemap job accepted (or cached result returned).

{
  "id": "sm_01j9abc123",
  "url": "string",
  "strategy": "string",
  "max_depth": 0,
  "max_urls": 0,
  "status": "pending",
  "total_urls": 0,
  "sources": {},
  "total_bytes": 0,
  "results_url": "string",
  "warnings": [
    "string"
  ],
  "error_code": "string",
  "error_message": "string",
  "compact": true,
  "process": {},
  "metadata": {},
  "created_at": "2019-08-24T14:15:22Z",
  "completed_at": "2019-08-24T14:15:22Z",
  "expires_at": "2019-08-24T14:15:22Z"
}

Discover URLs on a domain

Authorization

Request Body

Response

TypeScript