WebSPHINX - A Personal, Customizable Web Crawler

WebSPHINX ( Website-Specific Processors for HTML INformation eXtraction) is a Java class library and interactive development environment for web crawlers. A web crawler (also called a robot or spider) is a program that browses and processes Web pages automatically.

WebSPHINX consists of two parts: the Crawler Workbench and the WebSPHINX class library.

• Heritrix - Internet Archive Crawler
• Web-Harvest
• JSpider - a Web Spider Engine
• JoBo
• Arale - A java web spider
• bitmechanic- spindle
• Arachnid Web Spider Framework
• LARM
• WebLech URL Spider
• Snoics-Reptile