Ares — LLM-Powered Web Scraper

Ares is a Rust library, CLI, and HTTP server that extracts structured data from websites using LLMs and JSON Schemas.

Repository: https://github.com/AndreaBozzo/Ares License: Apache-2.0 | Rust edition: 2024 | MSRV: 1.88+

Pipeline

URL → [ContentCache?] → Fetcher (HTML) → Cleaner (Markdown) → [ExtractionCache?] → Extractor (LLM + JSON Schema) → Hash → Compare → Store

Each stage is a trait, so every component can be swapped or mocked independently. Optional in-memory caches (moka) skip fetch/extraction when content or results are already cached.

Crate Map

Core Traits (ares-core::traits)

pub trait Fetcher: Send + Sync + Clone {
    fn fetch(&self, url: &str) -> impl Future<Output = Result<String, AppError>> + Send;
}

pub trait Cleaner: Send + Sync + Clone {
    fn clean(&self, html: &str) -> Result<String, AppError>;
}

pub trait Extractor: Send + Sync + Clone {
    fn extract(&self, content: &str, schema: &serde_json::Value)
        -> impl Future<Output = Result<serde_json::Value, AppError>> + Send;
}

pub trait ExtractorFactory: Send + Sync + Clone {
    type Extractor: Extractor;
    fn create(&self, model: &str, base_url: &str) -> Result<Self::Extractor, AppError>;
}

pub trait ExtractionStore: Send + Sync + Clone {
    fn save(&self, extraction: &NewExtraction) -> impl Future<Output = Result<Uuid, AppError>> + Send;
    fn get_latest(&self, url: &str, schema_name: &str) -> impl Future<Output = Result<Option<Extraction>, AppError>> + Send;
    fn get_history(&self, url: &str, schema_name: &str, limit: usize, offset: usize) -> impl Future<Output = Result<Vec<Extraction>, AppError>> + Send;
}

JobQueue trait: see ares-core::job_queue — persistent queue with atomic claiming (SELECT FOR UPDATE SKIP LOCKED).

pub trait LinkDiscoverer: Send + Sync + Clone {
    fn discover_links(&self, html: &str, base_url: &str) -> Result<Vec<String>, AppError>;
}

pub trait RobotsChecker: Send + Sync + Clone {
    fn is_allowed(&self, url: &str) -> impl Future<Output = Result<bool, AppError>> + Send;
}

Key Types

Quick Start (Library Usage)

use ares_client::{ReqwestFetcher, HtmdCleaner, OpenAiExtractor};
use ares_core::{ScrapeService, NullStore};

let fetcher = ReqwestFetcher::new()?;
let cleaner = HtmdCleaner::new();
let extractor = OpenAiExtractor::with_base_url(&api_key, "gpt-4o-mini", "https://api.openai.com/v1")?;

let service = ScrapeService::<_, _, _, NullStore>::new(fetcher, cleaner, extractor, "gpt-4o-mini".into());

let schema = serde_json::json!({
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "author": {"type": "string"}
    },
    "required": ["title", "author"]
});

let result = service.scrape("https://example.com/blog", &schema, "blog").await?;
println!("{}", serde_json::to_string_pretty(&result.extracted_data)?);

With persistence, use ScrapeService::with_store(fetcher, cleaner, extractor, store, model).

Reference Guides

Version Notes

• Current version: 0.2.0
• Until crates.io release, use git dependency: ares-core = { git = "https://github.com/AndreaBozzo/Ares" }
• Works with any OpenAI-compatible API (OpenAI, Gemini, etc.)
• Browser support requires feature flag: --features browser
• New in 0.2.0: Web crawling, in-memory caching, output formats (json/jsonl/csv/table/jq), schema validation, 8 built-in schema templates, proxy rotation, User-Agent rotation, browser stealth mode, TLS backend selection