Simple scraping with wget

From XPUB & Lens-Based wiki

From Roel, a very nice one-liner:

 wget --random-wait -r -p -e robots=off -U mozilla www.somepage.com

The options:

  • --random-wait: Makes wget less likely to be blocked by "bot" detection algorithms that look for regularity in access
  • -r (--recursive): Recursive (ie keep following links)
  • -p (--page-requisites): Download dependent files
  • -e (--execute): Performs the command (in this case robots=off).
  • -U (--user-agent): Sets the user agent string to that of a "known" browser, in this case mozilla.