Quickstart

Nothing to install. You only need curl and an API key.

Get an API key

Sign up at app.bytekit.com and create an API key from the dashboard. Keys are prefixed sk_live_ for production and sk_test_ for staging.

See Authentication for details on key formats and management.

Make your first scrape

Use the staging endpoint with the test key to try it without affecting production quotas:

#!/usr/bin/env bash
# POST /v1/scrape — scrape a URL and return its content as markdown.
# Usage: RAPIDCRAWL_API_KEY=sk_... bash examples/curl/scrape.sh

set -euo pipefail

: "${RAPIDCRAWL_API_KEY:?RAPIDCRAWL_API_KEY is required}"
BASE_URL="${RAPIDCRAWL_BASE_URL:-https://api-stg.bytekit.com}"

curl -sf -X POST "$BASE_URL/v1/scrape" \
  -H "Authorization: Bearer $RAPIDCRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}' \
  | jq .

A successful response looks like this:

{
  "status": "success",
  "data": {
    "id": "sc_...",
    "url": "https://example.com",
    "content": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "content_length": 1024,
    "metadata": {
      "title": "Example Domain",
      "statusCode": 200
    }
  }
}

content_length is the compressed wire size in bytes (used for bandwidth billing). The id field uses the sc_ prefix — you can poll GET /v1/scrape/{'{id}'} if the scrape was queued asynchronously.

Call production

When you are ready to hit production, swap the base URL and use your live key:

curl -X POST https://api.bytekit.com/v1/scrape \
  -H "Authorization: Bearer sk_live_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Prefer an SDK?

Skip the raw HTTP and use an official client:

• TypeScript SDK — npm install @rapidcrawl/sdk
• Python SDK — pip install rapidcrawl-python

Next steps

• Authentication — key formats, Bearer header, self-serve key management
• Scraping — fast path vs slow path, when each triggers, automatic browser retry
• Errors — error shape, common codes, retry guidance
• Rate Limits — quota model, concurrency slots, rate limit headers
• Monitors — visual change detection with webhook notifications
• API Reference — full endpoint documentation