[aklug] Re: recursive HTTP file downloading...

From: Arthur Corliss <acorliss@nevaeh-linux.org>
Date: Thu Jun 10 2010 - 11:34:48 AKDT

On Thu, 10 Jun 2010, blair parker wrote:

> Ok... Maybe somebody out there can help me with a recursive download
> issue...
>
> The state DOT has a bunch of specs that my wife wants to download:
>
> http://www.dot.state.ak.us/creg/design/highways/Specs/
>
> She wants all of the files, subdirectories included.
>
> I can't seem to get 'wget' to download any of the files listed, and
> 'curl' only downloads files individually. Am I missing something, or is
> there some relatively simple, recursive command to download all of these
> files ?..
>
> Thanks.

:-) Looks like their robots.txt forbids it, which wget obeys. That's where
it's nice to be able to have the source. You can edit wget's source to
ignore robots.txt. Of course, whether or not that's in keeping with proper
web etiquette should weigh in your decision. If they catch you they'll
certainly try to ban you.

         --Arthur Corliss
           Live Free or Die
---------
To unsubscribe, send email to <aklug-request@aklug.org>
with 'unsubscribe' in the message body.
Received on Thu Jun 10 11:34:56 2010

This archive was generated by hypermail 2.1.8 : Thu Jun 10 2010 - 11:34:56 AKDT