CLI
Scrape
Scrape URL content
Scrape URL content
scrape › bulk
Bulk scrape operations
| Option | Description |
|---|---|
-h, --help | display help for command |
scrape › bulk › create
Create a bulk scrape job
| Option | Description |
|---|---|
--file <path> | Path to a newline-delimited file. Each line is a plain URL, or a JSON object carrying "url" plus per-item config. A line whose only key is "url" behaves exactly as before (sent as a bare URL string); any line with additional keys switches the payload to the per-item shape. |
--webhook-url <url> | HTTPS webhook URL notified when the job completes |
--webhook-secret <secret> | HMAC secret used to sign the completion webhook (omitted → unsigned) |
--defaults <json> | Defaults applied to every entry, as a JSON object (omitted → none). Unknown keys are rejected by /v1/bulk (422) and ignored by scrape bulk / fetch bulk; fetch bulk additionally rejects browser-only keys (cookies, wait_until, wait_for_selector, delay_ms) with a 400. |
--metadata <json> | Arbitrary metadata echoed on the job envelope, as a JSON object, e.g. '{"run":"7"}' (omitted → none) |
--custom <json> | User payload echoed on every scrape entry, as a JSON object, e.g. '{"job":"1"}' (omitted → none; a per-entry value in --file wins) |
-h, --help | display help for command |
$ bytekit scrape bulk create --file <path> --webhook-url <url> --webhook-secret <secret> --defaults <json> --metadata <json> --custom <json>scrape › bulk › get
Get a bulk scrape job
| Option | Description |
|---|---|
-h, --help | display help for command |
$ bytekit scrape bulk get <id>scrape › create
Create a scrape job
| Option | Description |
|---|---|
--url <url> | URL to scrape |
--format <format> | Output format; repeat or comma-separate for multiple (e.g. --format raw_html --format markdown, or --format raw_html,markdown). Sent as the formats array; omitted → server default. |
--country <cc> | Two-letter country code to route the request through (e.g. us, de) |
--timeout-ms <n> | Per-request timeout in milliseconds |
--markdown-mode <mode> | Markdown extraction mode: article, raw, llm |
--markdown-query <q> | BM25 query string for relevance-ranked markdown filtering (omitted → disabled) |
--cache-ttl <ttl> | Cache freshness: 0, Nh (hours), or Nd (days), e.g. 48h, 2d, 0 (omitted → server default) |
--cookies <json> | Cookies as a JSON array, e.g. '[{"name":"a","value":"b"}]' (omitted → none) |
--headers <json> | Extra request headers as a JSON object, e.g. '{"X-A":"b"}' (omitted → none) |
--delay-ms <n> | Post-load delay in milliseconds before capture |
--events <csv> | Webhook events (comma-separated): queued, completed, failed (omitted → server default) |
--token-encoding <enc> | Tokenizer for --token-budget: cl100k_base, o200k_base (omitted → server default) |
--custom <json> | User payload echoed on the envelope as a JSON object, e.g. '{"job":"1"}' (omitted → none) |
--remove-base64-images | Strip base64 data: images before conversion (server default) |
--no-remove-base64-images | Preserve base64 data: images in the pipeline output |
--markdown-links <mode> | Link rendering: inline, references, none, text |
--markdown-images <mode> | Image retention: inline, references, none, text |
--with-links-summary | Append a Links footer to the markdown output |
--with-images-summary | Append an Images footer to the markdown output |
--markdown-compact | Collapse excessive whitespace in the markdown output |
--markdown-filter-images | Filter low-signal images from the markdown output |
--markdown-include-media | Return rich link/image objects and a tables array |
--markdown-include-warnings | Include markdown-pipeline and tag-filter warnings |
--markdown-include-stats | Include a top-level stats object (chars, tokens, blocks) |
--async | Queue the scrape and return a job envelope immediately (no wait) |
--webhook-url <url> | HTTPS webhook URL notified when an async scrape completes |
--token-budget <n> | Maximum tokens of markdown to return |
--clean-markdown | Post-process markdown with the LLM cleaner (billed 3x) |
--include-tags <csv> | Comma-separated HTML tags to keep (e.g. article,main) |
--exclude-tags <csv> | Comma-separated HTML tags to drop (e.g. nav,footer) |
--mobile | Render with a mobile viewport/user-agent |
--raw | Print the requested format body raw, with no JSON wrapper |
-o, --output <file> | Write the output to the given file path (implies --raw unless --json is set) |
-h, --help | display help for command |
$ bytekit scrape create --url <url> --format <format> --country <cc> --timeout-ms <n> --markdown-mode <mode> --markdown-query <q> --cache-ttl <ttl> --cookies <json> --headers <json> --delay-ms <n> --events <csv> --token-encoding <enc> --custom <json> --remove-base64-images --no-remove-base64-images --markdown-links <mode> --markdown-images <mode> --with-links-summary --with-images-summary --markdown-compact --markdown-filter-images --markdown-include-media --markdown-include-warnings --markdown-include-stats --async --webhook-url <url> --token-budget <n> --clean-markdown --include-tags <csv> --exclude-tags <csv> --mobile --rawscrape › get
Get a scrape job by ID (use --raw/-o/--format to retrieve a completed body)
| Option | Description |
|---|---|
--raw | Print the requested format body raw, with no JSON wrapper |
-o, --output <file> | Write the output to the given file path (implies --raw unless --json is set) |
--format <format> | Which persisted format to emit — only takes effect with --raw/-o; ignored otherwise |
-h, --help | display help for command |
$ bytekit scrape get <id> --raw --format <format>