Get Started with Zeomine Web Crawler

Course Description

Zeomine Web Crawler

Zeomine Web Crawler (ZWC) is an open source, plugin-based website analysis program that allows you to test websites, extract information for analysis, or conduct simple web scraping.

In this course, you will learn about:

Installing Zeomine Web Crawler and the required Python libraries
Helpful extra programs for developing with ZWC, and reading data generated by ZWC
Writing a configuration YAML file
Introduction to Zeomine components, including initializers, the core object, crawlers, and plugins
Executing ZWC from the command line
Example configurations for broken link checking and extracting word counts

Advanced use of ZWC will be covered in additional courses. please check the Zeomine Web Crawler series for additional courses.

Installation and Intro to Zeomine

Install Python, Libraries, and Software
How to run Zeomine and first run example
Basics of Zeomine Configuration

Zeomine Components

Anatomy of a Zeomine run

Config examples

Crawl every page and check for any broken links
Crawl 100 pages and get H1 and H2 text
Crawl 100 pages and get top 50 words from headings
Fetch and run configuration with RemoteConf, and return data with SpectraReporter

Query Examples

Select data for a particular Zeomine instance