WaterCrawl is a web-based tool that transforms any website into a structured knowledge base, perfect for training LLMs, content analysis, and data-driven applications.
Features
Smart Crawling Control
Fine-tune your crawling scope with advanced controls for depth, domains, and paths, making it ideal for targeted content extraction.
Precise Content Extraction
Allows customizable selectors to extract exactly what's needed, focusing on main content while filtering out unwanted elements like ads and footers.
AI-Powered Processing
Integrates with OpenAI to transform raw HTML into structured, meaningful data automatically.
Extensible Plugin System
Create and integrate custom plugins to extend functionality, enabling data to be processed and transformed exactly as required.
JavaScript Rendering
Capture dynamic content with configurable wait times and JavaScript rendering, and take screenshots in PDF or JPG format.
Open Source Freedom
Built with transparency and collaboration in mind, allowing users to customize, extend, and contribute to the ecosystem.
Playground Interface
Use the interactive playground to test your selectors and extractors.
Scrap Tool
Enables single URL content extraction in various formats such as markdown, HTML, JSON, and screenshots, helping users efficiently gather data from websites.
Crawl Tool
Supports web crawling with configurable URL patterns and can generate image alt text, allowing for more customizable and comprehensive data collection.
OpenAI integration
Allows the use of OpenAI's Large Language Models for extracting information from crawled content, enabling advanced data analysis capabilities.
API Key based authentication
Enhances security by using API key-based authentication for connecting with different plugins and services.
Automated daily page credit reset
Daily resets of page credits for users with active subscriptions using an automated system.
Stripe integration for subscriptions
Manage subscriptions with Stripe, including handling of webhooks, billing cycles, and team plan management.
Playwright integration
Support for dynamic page rendering and JavaScript execution to facilitate advanced interaction with web pages.
Plugin service for crawl management
Allows managing crawl plugins with a dynamic middleware and pipeline loading system.
Multi-platform Docker builds
Supports Docker build process across multiple platforms, optimizing caching and formalizing version tracking.
Data Security
Implements encryption and secure data storage to protect personal information.
Cookie Management
Allows users to manage cookie preferences, providing control over which cookies are accepted.
Payment Processing
All payments are handled securely by third-party payment processors without storing sensitive payment information.