Comments on: WGet all the way

By: Chris Henry

Chris Henry — Sun, 12 Jul 2009 04:21:00 +0000

Nice post. I’ve often thought that wget could be used as part of a simple test framework. Build out the latest to a host, have wget spider it, thereby hitting every page (more reliably than a human test). Then grep through the error logs for errors / warnings / notices etc.

By: Anarchy Angel

Anarchy Angel — Wed, 08 Jul 2009 14:20:53 +0000

nice work y0 imma post this on Blisque!

By: dblackshell

dblackshell — Tue, 07 Jul 2009 19:41:21 +0000

yes, but it was just an example… and egrep is an alias for grep -E, and I used the Perl regex pattern

By: Mark

Mark — Tue, 07 Jul 2009 19:02:47 +0000

Pretty interesting… I’d just point out that you don’t need ‘cat’ to feed your file to grep, you can always ‘grep somepattern file.txt’. Similarly, ‘egrep’ shortens your command a lot:

egrep -o “http://.*$” links.txt | sort | uniq

To get all the links from links.txt.