Website Scraper
Any URL as clean Markdown and metadata, ready for LLM and RAG pipelines.
curl -X POST https://api.1lookup.dev/v1/website-scraper \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/pricing"}'{
"success": true,
"data": {
"url": "https://example.com/pricing",
"final_url": "https://example.com/pricing",
"status_code": 200,
"title": "Pricing | Example",
"description": "Simple plans for clinics of every size. Start free, upgrade when you need more seats.",
"markdown": "# Pricing\n\nSimple plans for clinics of every size.\n\n## Starter\n\n$19 per seat per month. Online booking, reminders and one location.\n\n## Practice\n\n$49 per seat per month. Everything in Starter, plus multiple locations and reporting.\n\n[Contact sales](https://example.com/contact)",
"links": [
{
"text": "Contact sales",
"url": "https://example.com/contact"
},
{
"text": "Documentation",
"url": "https://example.com/docs"
},
{
"text": "Terms",
"url": "https://example.com/terms"
}
],
"metadata": {
"language": "en",
"canonical_url": "https://example.com/pricing",
"og_image": "https://example.com/og/pricing.png",
"word_count": 36
}
},
"meta": {
"credits_used": 10,
"credits_remaining": 999,
"cached": false,
"request_id": "req_8f3a2c1e9b7d"
}
}Authorization
Authorizationstring<bearer>requiredYour API key as a Bearer token. Live keys start with 1lk_live_. Test keys start with 1lk_test_ and return this endpoint's sample response without spending credits. See authentication.
Body
Send the fields as a JSON object with Content-Type: application/json.
urlstring<uri>requiredURL. Example: https://example.com/pricing
freshbooleanSkip the 7-day cache and run a new lookup. Charged as a normal lookup.
referencestringYour own identifier, echoed back in meta and shown in lookup history.
Response
A 200 response carries the result in data and the cost of the call in meta. These are the fields most integrations read.
successbooleantrue when the lookup ran.
data.markdownstringThe main page content as Markdown, with headings, lists and links preserved.
data.titlestringThe page title.
data.descriptionstringThe meta description, when the page sets one.
data.linksarrayLinks found on the page, each with its anchor text and absolute URL.
data.metadataobjectLanguage, canonical URL, social preview image and word count.
data.status_codenumberThe status the page returned, so you can tell a 404 from real content.
data.final_urlstringWhere the request ended up after redirects.
meta.credits_usednumberCredits charged for this call.
meta.credits_remainingnumberYour balance after the call.
meta.cachedbooleantrue when the result came from the 7-day cache.
meta.request_idstringQuote this when you contact support.
Errors
A failed request returns success: false with an error object, and is never charged. The full list is on the errors page.
| HTTP | Code | Meaning |
|---|---|---|
| 401 | unauthorized | The key is missing, malformed or revoked. |
| 402 | insufficient_credits | Your credits are used up and pay as you go is off or at its limit. |
| 422 | invalid_input | A field is missing or malformed. The field property names it. |
| 429 | rate_limited | Too many requests for this key. Wait for the Retry-After interval. |
| 500 | internal_error | Something failed on our side. Safe to retry. |
| 503 | source_unavailable | An upstream source did not answer in time. Safe to retry. |
Billing
10 credits / page. Charged when the lookup runs and returns a result. Errors are never charged, and repeating the same input within 7 days is served from cache at no charge.
Product overview and use cases