Skip to content

Website scraper API: any URL as clean Markdown

Send a URL. We render the page, JavaScript included, and return the content as Markdown with its title, description, links and metadata. Pricing is 10 credits per page.

10 credits / pageIntelligenceCSV bulkMCP
POST/v1/website-scraper
{
  "success": true,
  "data": {
    "url": "https://example.com/pricing",
    "final_url": "https://example.com/pricing",
    "status_code": 200,
    "title": "Pricing | Example",
    "description": "Simple plans for clinics of every size. Start free, upgrade when you need more seats.",
    "markdown": "# Pricing\n\nSimple plans for clinics of every size.\n\n## Starter\n\n$19 per seat per month. Online booking, reminders and one location.\n\n## Practice\n\n$49 per seat per month. Everything in Starter, plus multiple locations and reporting.\n\n[Contact sales](https://example.com/contact)",
    "links": [
      {
        "text": "Contact sales",
        "url": "https://example.com/contact"
      },
      {
        "text": "Documentation",
        "url": "https://example.com/docs"
      },
      {
        "text": "Terms",
        "url": "https://example.com/terms"
      }
    ],
    "metadata": {
      "language": "en",
      "canonical_url": "https://example.com/pricing",
      "og_image": "https://example.com/og/pricing.png",
      "word_count": 36
    }
  },
  "meta": {
    "credits_used": 10,
    "credits_remaining": 999,
    "cached": false,
    "request_id": "req_8f3a2c1e9b7d"
  }
}
[01/Response]

What you get back

The fields most teams branch on. The full response is in the sample on this page and in the API reference.

markdownMarkdownThe main page content as Markdown, with headings, lists and links preserved.
titleTitleThe page title.
descriptionDescriptionThe meta description, when the page sets one.
linksLinksLinks found on the page, each with its anchor text and absolute URL.
metadataMetadataLanguage, canonical URL, social preview image and word count.
status_codeHTTP statusThe status the page returned, so you can tell a 404 from real content.
final_urlFinal URLWhere the request ended up after redirects.
[02/Capabilities]

What Website Scraper does

01

JavaScript rendering

Pages are rendered with JavaScript before content is extracted. Single-page apps and client-rendered sites return the text a visitor sees, not an empty shell.

02

Markdown built for models

Page chrome such as navigation and footers is removed, and what remains is structured Markdown. It costs fewer tokens than raw HTML and keeps the headings your chunker needs.

03

Links and metadata included

Every response carries the page's links with anchor text, plus title, description, canonical URL and language. Use the links to decide what to fetch next without a second parse.

04

One page per call, volume through bulk

The API fetches the URL you send and nothing else, so cost is predictable at 10 credits a page. For many URLs, upload a CSV or use the bulk API and collect the results by webhook.

[03/Try it]

See the response before you sign up

Fill in the request and run it. The dashboard has the same form, plus history, bulk upload and live data.

POST /v1/website-scraper10 credits / page

Preview mode: this page returns a sample response so you can see the shape of the data. Create a key to run it on live data.

Response

Fill in the field and run the lookup to see what comes back.

[04/Integrate]

One POST request

Send JSON with a Bearer key. The same key works for all 79 products.

curl -X POST https://api.1lookup.dev/v1/website-scraper \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/pricing"}'
Endpoint
POST /v1/website-scraper
Base URL
https://api.1lookup.dev
Request
url
Price
10 credits / page
Coverage
Global
Bulk
CSV upload and bulk API
Cache
Repeats within 7 days are free
Also via
Dashboard and MCP server (run_lookup)
[05/Use cases]

What teams use it for

Feed a RAG index

Scrape documentation, help center or policy pages to Markdown, chunk by heading and embed. Re-run on a schedule to keep the index current.

Give an agent web access

Call the scraper from a tool, or through the MCP server, so a model can read a page it found in search results.

Monitor pages for changes

Fetch pricing or terms pages on a schedule with "fresh": true and diff the Markdown against the last version.

Extract structured data with an LLM

Pass the Markdown to a model along with a schema and pull out product specs, job listings or event details without writing selectors for each site.

You are responsible for how you use scraped content. Respect site terms and robots rules, and do not use the scraper on pages that require a login.

[06/Cost]

10 credits / page

Credits come with your plan and work on every product. Requests that fail on our side are never charged. Beyond your plan, extra credits are pay as you go.

See pricing
PlanPer 1,000 creditsPer 1,000 pages
Starter $29/mo$4.83$48.33
Growth $99/mo$3.96$39.60
Pro $299/mo$3.32$33.22
Scale $799/mo$2.66$26.63
[07/FAQ]

Website Scraper questions

What does the website scraper return?

A JSON object with the page as Markdown, the title, the meta description, a list of links and metadata such as canonical URL and language. You do not get raw HTML that you have to clean up yourself.

Does the scraper handle JavaScript-heavy sites?

Yes. Pages are rendered with JavaScript before extraction, so content loaded by client-side frameworks is included.

What does scraping a page cost?

10 credits for each page. One URL is one page, and the scraper does not follow links on its own. The same URL requested again within 7 days is free unless you ask for a fresh copy.

Which pages can it not scrape?

Pages behind a login or paywall, and sites that block automated access, may return little or nothing. Content that only appears after a click or a scroll can also be missing. If a request fails on our side, you are not charged.

Is scraping a website allowed?

That depends on the site and on what you do with the content. Respect each site's terms of service and robots rules, avoid personal data you have no basis to collect, and get advice for your use case. Nothing here is legal advice.

[ Validate ][ Enrich ]
[Get started]

Run Website Scraper on your own data

Starter is $29 a month for 6,000 credits, enough for 600 pages a month.

[ Screen ][ Scrape ]