Coding Mantis

The Magic Of Wget.

Sun 23 June 2013

What if you could download all the “non-downloadable” content on internet ?,

Or What if you could save a  full website on your PC ? , Or What if you could extract all those images from a webpage/website  ? Cool , isn’t it ?

Comrades, Wget is the key to all of your problems ! Written in C , this command line download manager can easily satisfy all your download needs. You can download literally anything using Wget. And the silver lining is , its cross-platform , meaning it can work seamlessly with any OS.

  1. Downloading a single file:
    wget <URL> - Downloading file by above method stores your URL , which may be a webpage or any type of file in your current working directory.

  2. Downloading the file and store with a different name OR location: We use -O to do that. By default wget will pick the filename from the last word after last forward slash.

    • wget -O <Desired file name> <URL>
      Above will download the URL with your desired filename to . (Current working directory.)
    • wget -P <Desired file location> <URL>
      Above will download the URL to your desired location.If the desired location does not exist , then specified directory will be created.
    • If you want to change the file name and store it in a user defined location then you’ll have to be a little more specific, you cannot use -P and -O in a single command , you’ll need to use something like this-
      • wget -O /home/mautbaba/Desktop/ <filename.filetype> <URL> This will download your filename.filetype to your desktop

  3. Specifying Download Speed: While executing wget , by default wget will hijack all of your bandwidth which can be undesirable in certain situations and networks. We use —limit-rate to control download speed.

    • wget --limit-rate=200k <URL>
  4. Continue Incomplete Download: What if your download gets interrupted ? , Well there’s fix for that too, you can always continue your incomplete downloads by adding -c parameter.
    • wget -c <URL>
  5. Background Downloading: For downloading huge files , you can also put wget to download in background.
    • wget -b <URL> -Above will initiate the download and give back BASH to you. You can also see progress of your download using : tail -f wget-log

  6. Download A Full Site: Wget can also download a full website for local viewing. It can make the site completely offline and even convert all the links.

    • wget --mirror -p --convert-links -P ./<DIRECTORY> <URL>
       --mirror       : Turns on the options suitable for mirroring.  
  -p            : Download all files that are necessary to properly display a given HTML page.  
--convert-links : After download convert links in document for local viewing.  
 -P ./<LOCATION>: Save all files and directories to specified directory.

  7. Download multiple files: You can download multiple files (Maybe mp3s, images) at one go using wget, all you need is the URL. First store all download URLs into a text file , lets say its download.txt
    Then, give download.txt as an argument using wget -i

    • wget -i download.txt

  8. Download only certain file types from a webpage/website: Wget can also be used to download all videos, images, PDFs from a website.

    • wget -r -A. <filetype> URL You may also use wget to reject certain filetypes while downloading, the syntax would be :
    • wget --reject=<filetype> URL

  9. FTP Download with wget: You can also download files from FTP using wget.

    • Download files as “Anonymous User” :
      • wget <FTP-URL>
    • Download files with user-name and password authentication :-
      • wget --ftp-user=USERNAME --ftp-password=PASSWORD <FTP-URL>

  10. Increase Total Retry Attempts: By default wget retries 20 times before giving up on a download , you can change it by

    • wget --tries=75 <URL>

Comments