wget -- A powerful Download Utility

[sajay]

wget  is a  well known command line utility for downloading.Here are few options used with this utility for complex downloading.

============================================================
1.Download a Full Website Using wget --mirror Option.

wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

--mirror : turn on options suitable for mirroring.
-p : download all files that are necessary to properly display a given HTML page.
--convert-links : after the download, convert the links in document for local viewing.
-P ./LOCAL-DIR : save all the files and directories to the specified directory.
==============================================================
2. .Download Multiple Files / URLs Using Wget -i Option.

First, store all the download files or URLs in a text file.

eg:vi download.txt

https://www.kernel.org/pub/linux/kernel/next/patch-v3.2-next-20120106.bz2
https://www.kernel.org/pub/linux/kernel/next/patch-v4.2-next-20120106.gzip
https://www.kernel.org/pub/linux/kernel/next/patch-v5.2-next-20120106.tar
https://www.kernel.org/pub/linux/kernel/next/patch-v6.2-next-20120106.zip

wget -i download.txt
=================================================================
3.Resume the download using -c option for a stopped or interrupted file download.

 wget -c "Download-url"
 eg: wget -c  https://www.kernel.org/pub/linux/kernel/next/patch-v3.2-next-20120106.bz2
=================================================================
4.Run the Download process in background using -b option.

wget -b "Download-url"
eg: wget -b  https://www.kernel.org/pub/linux/kernel/next/patch-v3.2-next-20120106.bz2

     a. Output will be written to `wget-log'
     b.status of the download can be viewed using  tail -f wget-log
=================================================================
5.Download by masking the user agent as Browser.

 wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" "URL-TO-DOWNLOAD"
 
Some websites can disallow you to download its page by identifying that the user agent is not a browser.
     
eg: wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" https://www.kernel.org/pub/linux/kernel/next/patch-v3.2-next-20120106.bz2

===============================================================================
6.Test Download URL Using wget --spider

  wget --spider "DOWNLOAD-URL"

  eg:wget --spider https://www.kernel.org/pub/linux/kernel/next/patch-v3.2-next-20120106.bz2

   It will ensure wether the link is availabe or broken.
==========================================================================
7.Increase Total Number of Retry Attempts Using wget --tries option

 wget --tries "DOWNLOAD-URL"
 
 If the internet connection has problem, and if the download file is large there is a chance of failures in the download.
 By default wget retries 20 times to make the download successful aslo the tries can be increased using
 
wget --tries=50 "DOWNLOAD-URL"

=================================================================================
8.Limit the download speed using --limit-rate

  wget --limit-rate=(k|m|g -KB,MB,GB)  "Download-url"
 eg: wget --limit-rate=200k  https://www.kernel.org/pub/linux/kernel/next/patch-v3.2-next-20120106.bz2

==================================================================================
9.Download and store the file in a different name

 wget -O "file-name" "Download-url"
 
eg: wget -O test.zip https://www.kernel.org/pub/linux/kernel/next/patch-v3.2-next-20120106.bz2

      The file will be stored to test.zip
==================================================================================

10.Download Only Certain File Types Using wget -r -A Option
eg:
    wget -r -A .pdf http://url  ---Download all .pdf files
    wget -r -A .jpeg http://url   ----Download all .jpeg files

===================================================================================