Bulk scrape URLs

Enqueues multiple URLs for scraping. Results delivered via webhook. Webhook deliveries include `X-RapidCrawl-Event: scrape.completed` or `X-RapidCrawl-Event: scrape.failed` so receivers can dispatch on the header without parsing the body shape.

Authorization

AuthorizationRequiredBearer <token>

API key. Obtain from POST /v1/account/api-keys.

In: header

Request Body

application/jsonRequired

urlsarray<string>

Simple URL list. Provide urls OR items, not both.

itemsarray<object>

Per-item configs. Provide urls OR items, not both.

defaultsobject

webhook_urlRequiredstring

Webhook endpoint URL (HTTPS only; http:// rejected with HTTP 422)

Pattern: "^https://"Format: "uri"

webhook_secretstring

metadataobject

Arbitrary key-value metadata. Max 4096 bytes when JSON-serialized.

curl -X POST "https://api.bytekit.com/v1/scrape/bulk" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "http://example.com"
    ],
    "items": [
      {
        "url": "http://example.com",
        "formats": [
          "rawHtml"
        ],
        "country": "string",
        "cookies": [
          {
            "name": "string",
            "value": "string",
            "domain": "string",
            "path": "string",
            "secure": true,
            "httpOnly": true
          }
        ],
        "headers": {
          "property1": "string",
          "property2": "string"
        },
        "delay_ms": 10000,
        "timeout_ms": 1000,
        "metadata": {}
      }
    ],
    "defaults": {
      "formats": [
        "rawHtml"
      ],
      "country": "US",
      "cookies": [
        {
          "name": "string",
          "value": "string",
          "domain": "string",
          "path": "string",
          "secure": true,
          "httpOnly": true
        }
      ],
      "headers": {
        "property1": "string",
        "property2": "string"
      },
      "delay_ms": 0,
      "timeout_ms": 30000
    },
    "webhook_url": "http://example.com",
    "webhook_secret": "string",
    "metadata": {}
  }'

Bulk scrape job accepted.

{}

Bulk scrape URLs

Authorization

Request Body

Response

TypeScript