Website Scraper

Fetches a public web page and returns its title, metadata, readable text and links.

POST https://YOUR_API_HOST/v1/website-scraper

Credits and caching

• Success billing. 10 credits per successful scrape (errors, empty and blocked pages are free). You are charged only when the lookup finds something; a not-found result is free.
• Results are cached per account for 7 days. Repeating the same lookup in that window is free and returns cache_hit: true.
• Invalid input, rate limits and errors are never charged.
• Read the billing rules for the full picture.

Request

Send a JSON body with Authorization: Bearer <key>. The body can be up to 64 KB.

Example request

curl -X POST https://YOUR_API_HOST/v1/website-scraper \
  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","format":"markdown"}'

1const response = await fetch("https://YOUR_API_HOST/v1/website-scraper", {
2  method: "POST",
3  headers: {
4    Authorization: `Bearer ${process.env.LOOKVALID_API_KEY}`,
5    "Content-Type": "application/json",
6  },
7  body: JSON.stringify({
8    "url": "https://example.com",
9    "format": "markdown"
10  }),
11});
12​
13const result = await response.json();
14if (!response.ok) throw new Error(`${result.error.code}: ${result.error.fix}`);
15console.log(result.found, result.data, result.credits_charged);

1import os
2import requests
3​
4response = requests.post(
5    "https://YOUR_API_HOST/v1/website-scraper",
6    headers={"Authorization": f"Bearer {os.environ['LOOKVALID_API_KEY']}"},
7    json={
8        "url": "https://example.com",
9        "format": "markdown",
10    },
11    timeout=30,
12)
13result = response.json()
14if not response.ok:
15    raise RuntimeError(f"{result['error']['code']}: {result['error']['fix']}")
16print(result["found"], result["data"], result["credits_charged"])

Response

The status is 200 whether or not anything was found. Check found. Every response also carries the X-Request-Id, X-Credits-Charged, X-Credits-Remaining and X-RateLimit-* headers.

Envelope

The data object

Present when found is true, otherwise null.

Captured example (2026-10-18)

A real run of this lookup on 2026-10-18, using the request above, which is a public input. The envelope fields are left out.

1{
2  "product": "website-scraper",
3  "found": true,
4  "data": {
5    "final_url": "https://example.com/",
6    "status": 200,
7    "title": "Example Domain",
8    "description": null,
9    "canonical": null,
10    "language": "en",
11    "headings": [],
12    "format": "markdown",
13    "content": "This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
14    "content_truncated": false,
15    "links": {
16      "internal": [],
17      "external": []
18    },
19    "images": 0,
20    "word_count": 25,
21    "fetched_at": "2026-10-18T02:06:04.110Z"
22  },
23  "sources": [
24    "https://example.com"
25  ]
26}

Errors

• 400 invalid_input: The request input is invalid.
• 401 unauthorized: The API key is missing, malformed, revoked or unknown.
• 402 insufficient_credits: The account does not have enough credits for this lookup. Error code: insufficient_credits. During a trial whose allowance is used up the code is trial_allowance_exhausted: nothing is charged and the customer starts the plan from the billing page.
• 405 method_not_allowed: This endpoint does not accept that HTTP method.
• 413 payload_too_large: The request body is too large.
• 429 rate_limited: Too many requests for this API key.
• 500 internal_error: An unexpected error occurred.
• 503 provider_unavailable: The data source for this product is unavailable right now.