Download Large Files from Remote Servers Using Nginx Docker
Introduction
There are times when we have to download large files such as database backups and huge log files from our servers. We typically use SSH (Secure Shell) or SCP (Secure Copy) in these scenarios. But if the server is in a remote location, these protocols become extremely slow in comparison to downloading in HTTP.
So why not use HTTP for these purposes? The solution is pretty simple: just use a server like Nginx and configure it to serve the files you need. Then simply download the file using aria2 or some other download accelerator from your local machine. However, these situations don’t necessarily arise on a regular basis, so our solution would need to be temporary — something that we could easily remove after use.
Docker to the rescue!
Side note: if you're new to using docker, be sure to check out our Beginner's Guide to docker first.
We’ll first build an Nginx image (check out this discussion about images and containers for more) with our custom configuration file and run it. That way, when we need to download something, we’ll simply start the image and stop it once our download is complete.
As a docker image, this download process also becomes portable; you can use this on any of your servers without touching the existing system.
Prerequisites
The solution requires you to have the following packages (preferrably the latest versions) installed on your server:
- Docker — installation guide
- Docker-compose — installation guide
Nginx configuration file
It is strongly recommended to use SSL (Secure Socket Layer) for encryption if you use this guide on your production servers. Here's an article that might be helpful.
1server {
2 listen 80;
3 server_name localhost;
4 # serve the static files on port 80
5 location /downloads/ {
6 alias /files/;
7 }
8}The above configuration tells Nginx to listen on port 80 and serve our files when a request is made on /downloads/ url.
Docker-compose file
The next step would be to create the docker-compose.yml file. This file tells docker how to run a specific container.
It must be noted that the docker-compose.yml file below assumes that you have created the directories
confandfileswheredocker-compose.ymlresides. Moreover it also assumes that the Nginx configuration file is inside theconfdirectory and the files you want to download are in thefilesdirectory.
1nginx:
2 image: nginx:latest
3 volumes:
4 - "./files:/files"
5 - "./conf:/etc/nginx/conf.d"
6 ports:
7 - "8080:80"The above file tells docker to run a container using the nginx:latest image, mount the directories files and conf from the host machine, and expose port 80 to host machine’s port 8080.
Controlling the container
To start the container for the first time:
1docker-compose up -dTo start the container repeatedly:
1docker-compose startTo stop the container:
1docker-compose stopTo restart the container:
1docker-compose restartTo stop the container and remove the images from your host machine:
1docker-compose downIf you find out that the down command is not working, you probably lack the latest docker-compose.yml and need to upgrade.
After you’ve used the down command, the next time you want to start the container, you’ll have to use the up command to recreate it — in other words, you will have to boot up the container as if starting for the first time.
Downloading your files
Once you’ve started your container, you can start downloading your files. At first you’ll need to put your files in the appropriate location,
I’m assuming you’ve kept the files in the
~/downloadsdirectory. For this step, you need to download a file calleddb.tar.gz
1cp db.tar.gz ~/downloader/files/Then you can simply download your file from the URL below:
1http://your_server_ip:8080/downloads/db.tar.gzI hope this simple, portable solution will meet your needs, especially when it comes to occasionally downloading large content from servers.