GNU Wget is capable of traversing parts of the Web (or a single HTTP or FTP server), depth-first following links and directory structure. This is called recursive retrieving, or recursion.
With HTTP URLs, Wget retrieves and parses the HTML from
the given URL, documents, retrieving the files the HTML
document was referring to, through markups like href, or
src. If the freshly downloaded file is also of type
text/html, it will be parsed and followed further.
The maximum depth to which the retrieval may descend is specified with the `-l' option (the default maximum depth is five layers). See section Recursive Retrieval.
When retrieving an FTP URL recursively, Wget will retrieve all
the data from the given directory tree (including the subdirectories up
to the specified depth) on the remote server, creating its mirror image
locally. FTP retrieval is also limited by the depth
parameter.
By default, Wget will create a local directory tree, corresponding to the one found on the remote server.
Recursive retrieving can find a number of applications, the most important of which is mirroring. It is also useful for WWW presentations, and any other opportunities where slow network connections should be bypassed by storing the files locally.
You should be warned that invoking recursion may cause grave overloading on your system, because of the fast exchange of data through the network; all of this may hamper other users' work. The same stands for the foreign server you are mirroring--the more requests it gets in a rows, the greater is its load.
Careless retrieving can also fill your file system unctrollably, which can grind the machine to a halt.
The load can be minimized by lowering the maximum recursion level (`-l') and/or by lowering the number of retries (`-t'). You may also consider using the `-w' option to slow down your requests to the remote servers, as well as the numerous options to narrow the number of followed links (See section Following Links).
Recursive retrieval is a good thing when used properly. Please take all precautions not to wreak havoc through carelessness.
Go to the first, previous, next, last section, table of contents.