Web Crawler & Data Processing System
Isoxya is a web crawler and data processing system. It can process websites with tens of millions of pages, and extract and transform that data in myriad ways. This allows it to power many different types of software, in many different industries.
Although it can be considered as an SEO web crawler, the potential of Isoxya is far too big to be reduced to one purpose and one industry only. Rather, it defines a plugin system, abstracting away the complexities of running a large-scale web crawler, solving many of the challenges in building such a system in a robust and scalable manner, whilst providing a straightforward, considered interface.
Websites can be checked for SEO, e-commerce data can be extracted, content can be audited, and human language can be analysed—all using the same crawling system. If it’s possible to write a small script to process data from a single webpage, it’s likely possible to use Isoxya to process data from millions of pages, with minimal or no code changes.
Crawling as a Service
You concentrate on your core product; we concentrate on processing the data and streaming it to you.
Multi-computer, designed for close to 24/7 operation, with automated error recovery and backlog queues.
Crawls typically start and begin streaming data within seconds; no ‘crawl finalisation’ stage; analyse data immediately.
Tested with sites with millions of pages; designed to scale to sites with tens of millions of pages.
Supports many-tiny-site workloads; able to process tiny sites end-to-end within seconds, cost-effectively.
Not just an SEO crawler: multi-industry, multi-purpose; Spellchecking, Data Mining, Machine Learning…