Go to the first, previous, next, last section, table of contents.
By default, Wget is very simple to invoke. The basic syntax is:
wget [option]... [URL]...
Wget will simply download all the URLs specified on the command
line. URL is a Uniform Resource Locator, as defined below.
However, you may wish to change some of the default parameters of
Wget. You can do it two ways: permanently, adding the appropriate
command to `.wgetrc' (See section Startup File), or specifying it on
the command line.
URL is an acronym for Uniform Resource Locator. A uniform
resource locator is a compact string representation for a resource
available via the Internet. Wget recognizes the URL syntax as per
RFC1738. This is the most widely used form (square brackets denote
optional parts):
http://host[:port]/directory/file
ftp://host[:port]/directory/file
You can also encode your username and password within a URL:
ftp://user:password@host/path
http://user:password@host/path
Either user or password, or both, may be left out. If you
leave out either the HTTP username or password, no authentication
will be sent. If you leave out the FTP username, `anonymous'
will be used. If you leave out the FTP password, your email
address will be supplied as a default password.(1)
You can encode unsafe characters in a URL as `%xy', xy
being the hexadecimal representation of the character's ASCII
value. Some common unsafe characters include `%' (quoted as
`%25'), `:' (quoted as `%3A'), and `@' (quoted as
`%40'). Refer to RFC1738 for a comprehensive list of unsafe
characters.
Wget also supports the type
feature for FTP URLs. By
default, FTP documents are retrieved in the binary mode (type
`i'), which means that they are downloaded unchanged. Another
useful mode is the `a' (ASCII) mode, which converts the line
delimiters between the different operating systems, and is thus useful
for text files. Here is an example:
ftp://host/directory/file;type=a
Two alternative variants of URL specification are also supported,
because of historical (hysterical?) reasons and their wide-spreadedness.
FTP-only syntax (supported by NcFTP
):
host:/dir/file
HTTP-only syntax (introduced by Netscape
):
host[:port]/dir/file
These two alternative forms are deprecated, and may cease being
supported in the future.
If you do not understand the difference between these notations, or do
not know which one to use, just use the plain ordinary format you use
with your favorite browser, like Lynx
or Netscape
.
Since Wget uses GNU getopts to process its arguments, every option has a
short form and a long form. Long options are more convenient to
remember, but take time to type. You may freely mix different option
styles, or specify options after the command-line arguments. Thus you
may write:
wget -r --tries=10 http://fly.cc.fer.hr/ -o log
The space between the option accepting an argument and the argument may
be omitted. Instead `-o log' you can write `-olog'.
You may put several options that do not require arguments together,
like:
wget -drc URL
This is a complete equivalent of:
wget -d -r -c URL
Since the options can be specified after the arguments, you may
terminate them with `--'. So the following will try to download
URL `-x', reporting failure to `log':
wget -o log -- -x
The options that accept comma-separated lists all respect the convention
that specifying an empty list clears its value. This can be useful to
clear the `.wgetrc' settings. For instance, if your `.wgetrc'
sets exclude_directories
to `/cgi-bin', the following
example will first reset it, and then set it to exclude `/~nobody'
and `/~somebody'. You can also clear the lists in `.wgetrc'
(See section Wgetrc Syntax).
wget -X '' -X /~nobody,/~somebody
- `-V'
-
- `--version'
-
Display the version of Wget.
- `-h'
-
- `--help'
-
Print a help message describing all of Wget's command-line options.
- `-b'
-
- `--background'
-
Go to background immediately after startup. If no output file is
specified via the `-o', output is redirected to `wget-log'.
- `-e command'
-
- `--execute command'
-
Execute command as if it were a part of `.wgetrc'
(See section Startup File). A command thus invoked will be executed
after the commands in `.wgetrc', thus taking precedence over
them.
- `-o logfile'
-
- `--output-file=logfile'
-
Log all messages to logfile. The messages are normally reported
to standard error.
- `-a logfile'
-
- `--append-output=logfile'
-
Append to logfile. This is the same as `-o', only it appends
to logfile instead of overwriting the old log file. If
logfile does not exist, a new file is created.
- `-d'
-
- `--debug'
-
Turn on debug output, meaning various information important to the
developers of Wget if it does not work properly. Your system
administrator may have chosen to compile Wget without debug support, in
which case `-d' will not work. Please note that compiling with
debug support is always safe--Wget compiled with the debug support will
not print any debug info unless requested with `-d'.
See section Reporting Bugs for more information on how to use `-d' for
sending bug reports.
- `-q'
-
- `--quiet'
-
Turn off Wget's output.
- `-v'
-
- `--verbose'
-
Turn on verbose output, with all the available data. The default output
is verbose.
- `-nv'
-
- `--non-verbose'
-
Non-verbose output--turn off verbose without being completely quiet
(use `-q' for that), which means that error messages and basic
information still get printed.
- `-i file'
-
- `--input-file=file'
-
Read URLs from file, in which case no URLs need to be on
the command line. If there are URLs both on the command line and
in an input file, those on the command lines will be the first ones to
be retrieved. The file need not be an HTML document (but no
harm if it is)---it is enough if the URLs are just listed
sequentially.
However, if you specify `--force-html', the document will be
regarded as `html'. In that case you may have problems with
relative links, which you can solve either by adding
<base
href="url">
to the documents or by specifying
`--base=url' on the command line.
- `-F'
-
- `--force-html'
-
When input is read from a file, force it to be treated as an HTML
file. This enables you to retrieve relative links from existing
HTML files on your local disk, by adding
<base
href="url">
to HTML, or using the `--base' command-line
option.
- `-t number'
-
- `--tries=number'
-
Set number of retries to number. Specify 0 or `inf' for
infinite retrying.
- `-O file'
-
- `--output-document=file'
-
The documents will not be written to the appropriate files, but all will
be concatenated together and written to file. If file
already exists, it will be overwritten. If the file is `-',
the documents will be written to standard output. Including this option
automatically sets the number of tries to 1.
- `-nc'
-
- `--no-clobber'
-
Do not clobber existing files when saving to directory hierarchy within
recursive retrieval of several files. This option is extremely
useful when you wish to continue where you left off with retrieval of
many files. If the files have the `.html' or (yuck) `.htm'
suffix, they will be loaded from the local disk, and parsed as if they
have been retrieved from the Web.
- `-c'
-
- `--continue'
-
Continue getting an existing file. This is useful when you want to
finish up the download started by another program, or a previous
instance of Wget. Thus you can write:
wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
If there is a file name `ls-lR.Z' in the current directory, Wget
will assume that it is the first portion of the remote file, and will
require the server to continue the retrieval from an offset equal to the
length of the local file.
Note that you need not specify this option if all you want is Wget to
continue retrieving where it left off when the connection is lost--Wget
does this by default. You need this option only when you want to
continue retrieval of a file already halfway retrieved, saved by another
FTP client, or left by Wget being killed.
Without `-c', the previous example would just begin to download the
remote file to `ls-lR.Z.1'. The `-c' option is also
applicable for HTTP servers that support the Range
header.
- `--dot-style=style'
-
Set the retrieval style to style. Wget traces the retrieval of
each document by printing dots on the screen, each dot representing a
fixed amount of retrieved data. Any number of dots may be separated in
a cluster, to make counting easier. This option allows you to
choose one of the pre-defined styles, determining the number of bytes
represented by a dot, the number of dots in a cluster, and the number of
dots on the line.
With the
default
style each dot represents 1K, there are ten dots
in a cluster and 50 dots in a line. The binary
style has a more
"computer"-like orientation--8K dots, 16-dots clusters and 48 dots
per line (which makes for 384K lines). The mega
style is
suitable for downloading very large files--each dot represents 64K
retrieved, there are eight dots in a cluster, and 48 dots on each line
(so each line contains 3M). The micro
style is exactly the
reverse; it is suitable for downloading small files, with 128-byte dots,
8 dots per cluster, and 48 dots (6K) per line.
- `-N'
-
- `--timestamping'
-
Turn on time-stamping. See section Time-Stamping for details.
- `-S'
-
- `--server-response'
-
Print the headers sent by HTTP servers and responses sent by
FTP servers.
- `--spider'
-
When invoked with this option, Wget will behave as a Web spider,
which means that it will not download the pages, just check that they
are there. You can use it to check your bookmarks, e.g. with:
wget --spider --force-html -i bookmarks.html
This feature needs much more work for Wget to get close to the
functionality of real WWW spiders.
- `-T seconds'
-
- `--timeout=seconds'
-
Set the read timeout to seconds seconds. Whenever a network read
is issued, the file descriptor is checked for a timeout, which could
otherwise leave a pending connection (uninterrupted read). The default
timeout is 900 seconds (fifteen minutes). Setting timeout to 0 will
disable checking for timeouts.
Please do not lower the default timeout value with this option unless
you know what you are doing.
- `-w seconds'
-
- `--wait=seconds'
-
Wait the specified number of seconds between the retrievals. Use of
this option is recommended, as it lightens the server load by making the
requests less frequent. Instead of in seconds, the time can be
specified in minutes using the
m
suffix, in hours using h
suffix, or in days using d
suffix.
Specifying a large value for this option is useful if the network or the
destination host is down, so that Wget can wait long enough to
reasonably expect the network error to be fixed before the retry.
- `-Y on/off'
-
- `--proxy=on/off'
-
Turn proxy support on or off. The proxy is on by default if the
appropriate environmental variable is defined.
- `-Q quota'
-
- `--quota=quota'
-
Specify download quota for automatic retrievals. The value can be
specified in bytes (default), kilobytes (with `k' suffix), or
megabytes (with `m' suffix).
Note that quota will never affect downloading a single file. So if you
specify `wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz', all of the
`ls-lR.gz' will be downloaded. The same goes even when several
URLs are specified on the command-line. However, quota is
respected when retrieving either recursively, or from an input file.
Thus you may safely type `wget -Q2m -i sites'---download will be
aborted when the quota is exceeded.
Setting quota to 0 or to `inf' unlimits the download quota.
- `-nd'
-
- `--no-directories'
-
Do not create a hierarchy of directories when retrieving
recursively. With this option turned on, all files will get saved to the
current directory, without clobbering (if a name shows up more than
once, the filenames will get extensions `.n').
- `-x'
-
- `--force-directories'
-
The opposite of `-nd'---create a hierarchy of directories, even if
one would not have been created otherwise. E.g. `wget -x
http://fly.cc.fer.hr/robots.txt' will save the downloaded file to
`fly.cc.fer.hr/robots.txt'.
- `-nH'
-
- `--no-host-directories'
-
Disable generation of host-prefixed directories. By default, invoking
Wget with `-r http://fly.cc.fer.hr/' will create a structure of
directories beginning with `fly.cc.fer.hr/'. This option disables
such behavior.
- `--cut-dirs=number'
-
Ignore number directory components. This is useful for getting a
fine-grained control over the directory where recursive retrieval will
be saved.
Take, for example, the directory at
`ftp://ftp.xemacs.org/pub/xemacs/'. If you retrieve it with
`-r', it will be saved locally under
`ftp.xemacs.org/pub/xemacs/'. While the `-nH' option can
remove the `ftp.xemacs.org/' part, you are still stuck with
`pub/xemacs'. This is where `--cut-dirs' comes in handy; it
makes Wget not "see" number remote directory components. Here
are several examples of how `--cut-dirs' option works.
No options -> ftp.xemacs.org/pub/xemacs/
-nH -> pub/xemacs/
-nH --cut-dirs=1 -> xemacs/
-nH --cut-dirs=2 -> .
--cut-dirs=1 -> ftp.xemacs.org/xemacs/
...
If you just want to get rid of the directory structure, this option is
similar to a combination of `-nd' and `-P'. However, unlike
`-nd', `--cut-dirs' does not lose with subdirectories--for
instance, with `-nH --cut-dirs=1', a `beta/' subdirectory will
be placed to `xemacs/beta', as one would expect.
- `-P prefix'
-
- `--directory-prefix=prefix'
-
Set directory prefix to prefix. The directory prefix is the
directory where all other files and subdirectories will be saved to,
i.e. the top of the retrieval tree. The default is `.' (the
current directory).
- `--http-user=user'
-
- `--http-passwd=password'
-
Specify the username user and password password on an
HTTP server. According to the type of the challenge, Wget will
encode them using either the
basic
(insecure) or the
digest
authentication scheme.
Another way to specify username and password is in the URL itself
(See section URL Format). For more information about security issues with
Wget, See section Security Considerations.
- `-C on/off'
-
- `--cache=on/off'
-
When set to off, disable server-side cache. In this case, Wget will
send the remote server an appropriate directive (`Pragma:
no-cache') to get the file from the remote service, rather than
returning the cached version. This is especially useful for retrieving
and flushing out-of-date documents on proxy servers.
Caching is allowed by default.
- `--ignore-length'
-
Unfortunately, some HTTP servers (CGI programs, to be more
precise) send out bogus
Content-Length
headers, which makes Wget
go wild, as it thinks not all the document was retrieved. You can spot
this syndrome if Wget retries getting the same document again and again,
each time claiming that the (otherwise normal) connection has closed on
the very same byte.
With this option, Wget will ignore the Content-Length
header--as
if it never existed.
- `--header=additional-header'
-
Define an additional-header to be passed to the HTTP servers.
Headers must contain a `:' preceded by one or more non-blank
characters, and must not contain newlines.
You may define more than one additional header by specifying
`--header' more than once.
wget --header='Accept-Charset: iso-8859-2' \
--header='Accept-Language: hr' \
http://fly.cc.fer.hr/
Specification of an empty string as the header value will clear all
previous user-defined headers.
- `--proxy-user=user'
-
- `--proxy-passwd=password'
-
Specify the username user and password password for
authentication on a proxy server. Wget will encode them using the
basic
authentication scheme.
- `-s'
-
- `--save-headers'
-
Save the headers sent by the HTTP server to the file, preceding the
actual contents, with an empty line as the separator.
- `-U agent-string'
-
- `--user-agent=agent-string'
-
Identify as agent-string to the HTTP server.
The HTTP protocol allows the clients to identify themselves using a
User-Agent
header field. This enables distinguishing the
WWW software, usually for statistical purposes or for tracing of
protocol violations. Wget normally identifies as
`Wget/version', version being the current version
number of Wget.
However, some sites have been known to impose the policy of tailoring
the output according to the User-Agent
-supplied information.
While conceptually this is not such a bad idea, it has been abused by
servers denying information to clients other than Mozilla
or
Microsoft Internet Explorer
. This option allows you to change
the User-Agent
line issued by Wget. Use of this option is
discouraged, unless you really know what you are doing.
NOTE that Netscape Communications Corp. has claimed that false
transmissions of `Mozilla' as the User-Agent
are a copyright
infringement, which will be prosecuted. DO NOT misrepresent
Wget as Mozilla.
- `--retr-symlinks'
-
Retrieve symbolic links on FTP sites as if they were plain files,
i.e. don't just create links locally.
- `-g on/off'
-
- `--glob=on/off'
-
Turn FTP globbing on or off. Globbing means you may use the
shell-like special characters (wildcards), like `*',
`?', `[' and `]' to retrieve more than one file from the
same directory at once, like:
wget ftp://gnjilux.cc.fer.hr/*.msg
By default, globbing will be turned on if the URL contains a
globbing character. This option may be used to turn globbing on or off
permanently.
You may have to quote the URL to protect it from being expanded by
your shell. Globbing makes Wget look for a directory listing, which is
system-specific. This is why it currently works only with Unix FTP
servers (and the ones emulating Unix ls
output).
- `--passive-ftp'
-
Use the passive FTP retrieval scheme, in which the client
initiates the data connection. This is sometimes required for FTP
to work behind firewalls.
- `-r'
-
- `--recursive'
-
Turn on recursive retrieving. See section Recursive Retrieval for more
details.
- `-l depth'
-
- `--level=depth'
-
Specify recursion maximum depth level depth (See section Recursive Retrieval). The default maximum depth is 5.
- `--delete-after'
-
This option tells Wget to delete every single file it downloads,
after having done so. It is useful for pre-fetching popular
pages through proxy, e.g.:
wget -r -nd --delete-after http://whatever.com/~popular/page/
The `-r' option is to retrieve recursively, and `-nd' not to
create directories.
- `-k'
-
- `--convert-links'
-
Convert the non-relative links to relative ones locally. Only the
references to the documents actually downloaded will be converted; the
rest will be left unchanged.
Note that only at the end of the download can Wget know which links have
been downloaded. Because of that, much of the work done by `-k'
will be performed at the end of the downloads.
- `-m'
-
- `--mirror'
-
Turn on options suitable for mirroring. This option turns on recursion
and time-stamping, sets infinite recursion depth and keeps FTP
directory listings. It is currently equivalent to
`-r -N -l inf -nr'.
- `-nr'
-
- `--dont-remove-listing'
-
Don't remove the temporary `.listing' files generated by FTP
retrievals. Normally, these files contain the raw directory listings
received from FTP servers. Not removing them can be useful to
access the full remote file list when running a mirror, or for debugging
purposes.
- `-A acclist --accept acclist'
-
- `-R rejlist --reject rejlist'
-
Specify comma-separated lists of file name suffixes or patterns to
accept or reject (See section Types of Files for more details).
- `-D domain-list'
-
- `--domains=domain-list'
-
Set domains to be accepted and DNS looked-up, where
domain-list is a comma-separated list. Note that it does
not turn on `-H'. This option speeds things up, even if
only one host is spanned (See section Domain Acceptance).
- `--exclude-domains domain-list'
-
Exclude the domains given in a comma-separated domain-list from
DNS-lookup (See section Domain Acceptance).
- `-L'
-
- `--relative'
-
Follow relative links only. Useful for retrieving a specific home page
without any distractions, not even those from the same hosts
(See section Relative Links).
- `--follow-ftp'
-
Follow FTP links from HTML documents. Without this option,
Wget will ignore all the FTP links.
- `-H'
-
- `--span-hosts'
-
Enable spanning across hosts when doing recursive retrieving (See section All Hosts).
- `-I list'
-
- `--include-directories=list'
-
Specify a comma-separated list of directories you wish to follow when
downloading (See section Directory-Based Limits for more details.) Elements
of list may contain wildcards.
- `-X list'
-
- `--exclude-directories=list'
-
Specify a comma-separated list of directories you wish to exclude from
download (See section Directory-Based Limits for more details.) Elements of
list may contain wildcards.
- `-nh'
-
- `--no-host-lookup'
-
Disable the time-consuming DNS lookup of almost all hosts
(See section Host Checking).
- `-np'
-
- `--no-parent'
-
Do not ever ascend to the parent directory when retrieving recursively.
This is a useful option, since it guarantees that only the files
below a certain hierarchy will be downloaded.
See section Directory-Based Limits for more details.
Go to the first, previous, next, last section, table of contents.