Scrape

Fetch a URL as raw HTML, clean markdown, or structured content. Accessed via client.scrape.

create

client.scrape.create(opts: ScrapeOpts, requestOptions?: RequestOptions): Promise<ScrapeCreateResponse>

POST /v1/scrape — create a scrape job.

Resolves to the 200 success envelope, or the 202 queued envelope when async: true. Narrow on status before reading the result.

requestOptions carries the CLIENT-side abort budget (requestTimeoutMs) and an optional caller signal; both override the constructor's timeoutMs default. They are a different axis from opts.timeout_ms, which is the SERVER-side scrape budget.

get

client.scrape.get(id: ScrapeId, requestOptions?: RequestOptions): Promise<ScrapeGetResponse>

GET /v1/scrape/{id} — poll a scrape by ID

KNOWN DRIFT (#2426): live staging spot-checks (two independent fetches) found the gateway serializes billing_multiplier as a JSON string (e.g. "0.50") on this GET path, while the OpenAPI spec — and this derived ScrapeGetResponse type — declare it number. POST /v1/scrape's success envelope for the same job serializes the identical field as a proper number, so the drift is specific to the GET path (likely a Postgres NUMERIC column round-tripping as a string on this query path). A consumer doing arithmetic on billing_multiplier after get() (e.g. content_length * billing_multiplier) should coerce with Number(...) defensively rather than trust the static type. This is a gateway/worker serialization bug, not a client bug — the fix belongs upstream in the gateway and is out of this ticket's scope.

Options (ScrapeOpts)

• url (required) — string
• formats — Array<'raw_html' | 'html' | 'markdown' | 'links' | 'images'>
• country — string
• cookies — ScrapeCookie[] — Cookies to set before loading the page. name and value are required (spec Cookie schema).
• headers — Record<string, string>
• delay_ms — number
• timeout_ms — number — SERVER-side scrape budget in ms — how long the gateway may spend on the scrape itself. For the CLIENT-side abort budget (when this SDK gives up on the wire), see RequestOptions.requestTimeoutMs.
• async — boolean
• webhook_url — string
• events — Array<'queued' | 'completed' | 'failed'>
• markdown_mode — MarkdownMode — Markdown processing mode. article=article extraction (default), raw=minimal cleanup, llm=compact LLM-optimised output.
• markdown_query — string — BM25 query string for relevance-ranked content filtering. Omit or leave empty to disable.
• markdown_links — MarkdownLinks — Link rendering style in the markdown output.
• markdown_images — MarkdownImages — Image retention mode in the markdown output. Distinct from markdown_filter_images (which drops low-signal images before conversion).
• with_links_summary — boolean — When true, appends a Links: footer to the markdown output. Automatically enabled when markdown_links is 'references'.
• with_images_summary — boolean — When true, appends an Images: footer to the markdown output.
• markdown_compact — boolean — Collapse excessive whitespace for a more compact output.
• markdown_filter_images — boolean — Filter low-signal images from the markdown output.
• markdown_include_media — boolean — When true, formats.links and formats.images return ScrapeScoredLink[] / ScrapeScoredImage[] (rich objects) instead of string[], and a top-level tables array is included. Only effective when markdown is in formats.
• markdown_include_warnings — boolean — When true, the response includes markdown-pipeline and tag-filter warnings. artifact_unavailable may appear independently when a requested persisted format cannot be rehydrated.
• markdown_include_stats — boolean — When true, the response includes a top-level stats object with ScrapeStats (chars, tokens, blocks). Only effective when markdown is in formats.
• token_budget — number — Maximum token count for the markdown output. Truncates at the last paragraph → sentence → token boundary under the limit. Absent → no truncation.
• token_encoding — TokenEncoding — Tokenizer encoding hint for token_budget. cl100k_base (default) matches GPT-4/4o; o200k_base matches GPT-4o-mini / o-series.
• mobile — boolean — When true, emulate a mobile device (mobile UA + viewport). Default false (desktop). Part of the cache key, so mobile and desktop responses cache independently.
• remove_base64_images — boolean — When true (default), base64-encoded data: image sources are stripped before markdown conversion and from html output. Set false to preserve them. Does not affect raw_html.
• include_tags — string[] — HTML element names (not CSS selectors) to keep. When non-empty, only content within these elements is included in html/markdown output. Applied before exclude_tags. Does not affect raw_html.
• exclude_tags — string[] — HTML element names (not CSS selectors) to strip from html/markdown output. Applied after include_tags. Does not affect raw_html.
• clean_markdown — boolean — When true and markdown is in formats, the converted markdown is sent to Groq for cleaning before return. Graceful degradation on any Groq failure. Billing: 3× credits and bytes applied only when cleaning succeeds.
• cache_ttl — string | 0 — How long a freshly fetched URL may be served from cache. '0'/0 skips the cache read (the result is still written), 'Nh'/'Nd' set a TTL (capped at 168h / 7d). Default '48h'. Honoured on both the synchronous and asynchronous paths; the async success envelope reports the outcome via the cache field.
• custom — Record<string, unknown> — User-supplied JSON payload, echoed back on the success envelope so callers can correlate the response to caller-side state (job IDs, batch metadata). Capped at 4096 UTF-8 bytes after JSON serialization. Does NOT affect cache-key inputs — two requests differing only in custom share the same cache slot.