[aklug] Re: recursive HTTP file downloading...

On Thu, 10 Jun 2010, Arthur Corliss wrote:

> :-) Looks like their robots.txt forbids it, which wget obeys. That's where
> it's nice to be able to have the source. You can edit wget's source to
> ignore robots.txt. Of course, whether or not that's in keeping with proper
> web etiquette should weigh in your decision. If they catch you they'll
> certainly try to ban you.

I should have mentioned that enabling wget's debug mode explicitly shows
robots.txt forbidding it.

         --Arthur Corliss
           Live Free or Die
---------
To unsubscribe, send email to <aklug-request@aklug.org>
with 'unsubscribe' in the message body.
Received on Thu Jun 10 11:36:29 2010