There are times when we have to download large files such as database backups and huge log files from our servers. We typically use SSH (Secure Shell) or SCP (Secure Copy) in these scenarios. But if the server is in a remote location, these protocols become extremely slow in comparison to downloading in HTTP.
So why not use HTTP for these purposes? The solution is pretty simple: just use a server like Nginx and configure it to serve the files you need. Then simply download the file using aria2 or some other download accelerator from your local machine. However, these situations don’t necessarily arise on a regular basis, so our solution would need to be temporary — something that we could easily remove after use.
Docker to the rescue!
Side note: if you're new to using docker, be sure to check out our Beginner's Guide to docker first.
We’ll first build an Nginx image (check out this discussion about images and containers for more) with our custom configuration file and run it. That way, when we need to download something, we’ll simply start the image and stop it once our download is complete.
As a docker image, this download process also becomes portable; you can use this on any of your servers without touching the existing system.
The solution requires you to have the following packages (preferrably the latest versions) installed on your server:
It is strongly recommended to use SSL (Secure Socket Layer) for encryption if you use this guide on your production servers. Here's an article that might be helpful.
1server {
2 listen 80;
3 server_name localhost;
4 # serve the static files on port 80
5 location /downloads/ {
6 alias /files/;
7 }
8}
The above configuration tells Nginx to listen on port 80 and serve our files when a request is made on /downloads/ url.
The next step would be to create the docker-compose.yml
file. This file tells docker how to run a specific container.
It must be noted that the docker-compose.yml file below assumes that you have created the directories
conf
andfiles
wheredocker-compose.yml
resides. Moreover it also assumes that the Nginx configuration file is inside theconf
directory and the files you want to download are in thefiles
directory.
1nginx:
2 image: nginx:latest
3 volumes:
4 - "./files:/files"
5 - "./conf:/etc/nginx/conf.d"
6 ports:
7 - "8080:80"
The above file tells docker to run a container using the nginx:latest
image, mount the directories files
and conf
from the host machine, and expose port 80 to host machine’s port 8080.
To start the container for the first time:
1docker-compose up -d
To start the container repeatedly:
1docker-compose start
To stop the container:
1docker-compose stop
To restart the container:
1docker-compose restart
To stop the container and remove the images from your host machine:
1docker-compose down
If you find out that the down
command is not working, you probably lack the latest docker-compose.yml
and need to upgrade.
After you’ve used the down
command, the next time you want to start the container, you’ll have to use the up
command to recreate it — in other words, you will have to boot up the container as if starting for the first time.
Once you’ve started your container, you can start downloading your files. At first you’ll need to put your files in the appropriate location,
I’m assuming you’ve kept the files in the
~/downloads
directory. For this step, you need to download a file calleddb.tar.gz
1cp db.tar.gz ~/downloader/files/
Then you can simply download your file from the URL below:
1http://your_server_ip:8080/downloads/db.tar.gz
I hope this simple, portable solution will meet your needs, especially when it comes to occasionally downloading large content from servers.