---
title: "Website Scraper reference"
url: "https://lookvalid.com/docs/api/website-scraper.md"
canonical_url: "https://lookvalid.com/docs/api/website-scraper"
last_updated: "2026-10-22"
type: "reference"
summary: "Request and response reference for Website Scraper: parameters, response fields, credit cost and error codes."
related: ["https://lookvalid.com/docs/errors.md"]
---

# Website Scraper reference

Request and response reference for Website Scraper: parameters, response fields, credit cost and error codes.

Fetches a public web page and returns its title, metadata, readable text and links.

```http
POST https://YOUR_API_HOST/v1/website-scraper
```

## Credits and caching

- Success billing. 10 credits per successful scrape (errors, empty and blocked pages are free). You are charged only when the lookup finds something; a not-found result is free.
- Results are cached per account for 7 days. Repeating the same lookup in that window is free and returns `cache_hit: true`.
- Invalid input, rate limits and errors are never charged.
- Read the [billing rules](https://lookvalid.com/docs/credits-and-billing.md) for the full picture.

## Request

Send a JSON body with `Authorization: Bearer <key>`. The body can be up to 64 KB.

| Field | Type | Required | Notes |
| --- | --- | --- | --- |
| `url` | string | yes | Up to 2048 characters. |
| `format` | string | no | One of `markdown`, `text`, `html`. Default `"markdown"`. |

### Example request

**cURL**

```sh
curl -X POST https://YOUR_API_HOST/v1/website-scraper \
  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","format":"markdown"}'
```

**Node.js**

```js
const response = await fetch("https://YOUR_API_HOST/v1/website-scraper", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.LOOKVALID_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "url": "https://example.com",
    "format": "markdown"
  }),
});

const result = await response.json();
if (!response.ok) throw new Error(`${result.error.code}: ${result.error.fix}`);
console.log(result.found, result.data, result.credits_charged);
```

**Python**

```python
import os
import requests

response = requests.post(
    "https://YOUR_API_HOST/v1/website-scraper",
    headers={"Authorization": f"Bearer {os.environ['LOOKVALID_API_KEY']}"},
    json={
        "url": "https://example.com",
        "format": "markdown",
    },
    timeout=30,
)
result = response.json()
if not response.ok:
    raise RuntimeError(f"{result['error']['code']}: {result['error']['fix']}")
print(result["found"], result["data"], result["credits_charged"])
```

## Response

The status is 200 whether or not anything was found. Check `found`. Every response also carries the `X-Request-Id`, `X-Credits-Charged`, `X-Credits-Remaining` and `X-RateLimit-*` headers.

### Envelope

| Field | Type | Notes |
| --- | --- | --- |
| `request_id` | string | - |
| `product` | "website-scraper" | - |
| `found` | boolean | False means the lookup ran and found nothing; it is free. |
| `reason` | string or null | Why nothing was found; null when found. |
| `sources` | array of string | Where the data came from, including any attribution the licence requires. |
| `credits_charged` | integer | 0 for a cache hit, a not-found result or an error. |
| `credits_remaining` | integer | - |
| `cache_hit` | boolean | True when the account already paid for this lookup within the cache window. |
| `cache` | object | - |

### The `data` object

Present when `found` is true, otherwise `null`.

| Field | Type | Notes |
| --- | --- | --- |
| `final_url` | string | - |
| `status` | integer | - |
| `title` | string or null | - |
| `description` | string or null | - |
| `canonical` | string or null | - |
| `language` | string or null | - |
| `headings` | array of object | - |
| `headings[].level` | integer | Minimum 1. Maximum 6. |
| `headings[].text` | string | - |
| `format` | string | One of `markdown`, `text`, `html`. |
| `content` | string | - |
| `content_truncated` | boolean | - |
| `links` | object | - |
| `links.internal` | array of object | - |
| `links.internal[].url` | string | - |
| `links.internal[].text` | string | - |
| `links.external` | array of object | - |
| `links.external[].url` | string | - |
| `links.external[].text` | string | - |
| `images` | integer | - |
| `word_count` | integer | - |
| `fetched_at` | string | - |

### Captured example (2026-10-18)

A real run of this lookup on 2026-10-18, using the request above, which is a public input. The envelope fields are left out.

```json
{
  "product": "website-scraper",
  "found": true,
  "data": {
    "final_url": "https://example.com/",
    "status": 200,
    "title": "Example Domain",
    "description": null,
    "canonical": null,
    "language": "en",
    "headings": [],
    "format": "markdown",
    "content": "This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
    "content_truncated": false,
    "links": {
      "internal": [],
      "external": []
    },
    "images": 0,
    "word_count": 25,
    "fetched_at": "2026-10-18T02:06:04.110Z"
  },
  "sources": [
    "https://example.com"
  ]
}
```

## Errors

- `400` [`invalid_input`](https://lookvalid.com/docs/errors.md#invalid_input): The request input is invalid.
- `401` [`unauthorized`](https://lookvalid.com/docs/errors.md#unauthorized): The API key is missing, malformed, revoked or unknown.
- `402` [`insufficient_credits`](https://lookvalid.com/docs/errors.md#insufficient_credits): The account does not have enough credits for this lookup. Error code: insufficient_credits. During a trial whose allowance is used up the code is trial_allowance_exhausted: nothing is charged and the customer starts the plan from the billing page.
- `405` [`method_not_allowed`](https://lookvalid.com/docs/errors.md#method_not_allowed): This endpoint does not accept that HTTP method.
- `413` [`payload_too_large`](https://lookvalid.com/docs/errors.md#payload_too_large): The request body is too large.
- `429` [`rate_limited`](https://lookvalid.com/docs/errors.md#rate_limited): Too many requests for this API key.
- `500` [`internal_error`](https://lookvalid.com/docs/errors.md#internal_error): An unexpected error occurred.
- `503` [`provider_unavailable`](https://lookvalid.com/docs/errors.md#provider_unavailable): The data source for this product is unavailable right now.

## Related

- [Errors](https://lookvalid.com/docs/errors.md)