Show HN: Hikugen – minimalistic LLM-generated web scrapers for structured data

Hey HN! I wanted to share a little library I’ve been working on to leverage AI to get structured data from arbitrary pages. Instead of sending the page’s HTML to an LLM, Hikugen asks it to generate python code to fetch the data and enforces the generated data conforms to a Pydantic schema defined by the user. I’m using this to power yomu (<https://github.com/goncharom/yomu>), a personal email newsletter built from arbitrary websites.

Hikugen’s main features:
* Automatically generates, runs, regenerates and caches the LLM-generated extraction code.
* It uses sqlite to save the current working code for each page so it can be reused across executions.
* It uses OpenRouter (<https://openrouter.ai/>) to call the LLM.
* Hikugen can fetch the page automatically (it can even reuse Netscape-formatted cookies) but you can also just feed it the raw HTML and leverage the rest of its functionalities.

Here’s a snippet of what it looks like:

  from hikugen import HikuExtractor
  from pydantic import BaseModel
  from typing import List
  
  class Article(BaseModel):
      title: str
      author: str
      published_date: str
      content: str
  
  class ArticlePage(BaseModel):
      articles: List[Article]
  
  extractor = HikuExtractor(api_key="your-openrouter-api-key")
  
  result = extractor.extract(
      url="https://example.com/articles",
      schema=ArticlePage
  )
  
  for a in result.articles:
      print(a.title, a.author)

Hikugen is intentionally minimal: it doesn’t attempt website navigation, login flows, headless browsers, or large-scale crawling. Just “given this HTML, extract this structured data”.

A good chunk of this was built with Claude Code (shoutout to Harper’s blog: https://harper.blog/2025/02/16/my-llm-codegen-workflow-atm/.

Would love feedback or ideas—especially from others playing with codegen for scraping tasks.

Comments URL: https://news.ycombinator.com/item?id=45870231

Points: 2

# Comments: 0

Source: github.com