Back Up and Restore the Parse Server Database on DigitalOcean Spaces

January 16, 2019

In the previous article, we saw how to set up a Parse Server environment. Today's goal is to add an automatic database backup service for the production environment.

Our database backup will be stored on DigitalOcean Spaces.

Spaces is an object storage service, roughly DigitalOcean's counterpart to Amazon S3. What is interesting is that it works the same way as Amazon S3, which means many libraries that manipulate S3 will also work with Spaces. You can test Spaces and the other DigitalOcean services from DigitalOcean.

This article covers backing up the Parse Server database. In our case it is a MongoDB database, but the system we set up can apply to any environment using a MongoDB database.

How it works

To set up this system, we will use the following libraries and services:

  • Docker: we will create a container whose only goal is to run backups.
  • Boto3: a Python SDK for Amazon Web Services that will let us push backups to Spaces.
  • Cron: to schedule when our backup starts.
  • MongoDump / MongoRestore: to generate and restore our database backup.

First, we will create two Python scripts. These two scripts will use boto3 to interact with Spaces. One uploads the backup for storage, and the other downloads the backup when we want to restore data.

Then we will have two shell scripts: one for backup and one for restore.

For backup, the script connects to our MongoDB database and uses mongodump to extract the database data. Once the data is extracted and compressed, this script runs our Python script to upload the file.

For restore, the script first runs the Python script to download the backup. Then, once the backup has been retrieved, it uses mongorestore to restore the data.

The backup script will be launched automatically by Cron, while the restore script will be launched manually if needed.

Setting up the project

We will start from the Parse server configuration from the previous article.

Once the sources are ready, we will create a folder at the root of the project that contains all the scripts and the Dockerfile.

mkdir cron && cd cron

Creating the Python scripts

Now we will create our Python scripts, which will be used to upload and download backups.

Uploading the backup

Create a scripts folder and add the backup.py file inside it.

import boto3
import os
import datetime
 
session = boto3.session.Session()
client = session.client('s3',
                        region_name= os.environ['SPACE_REGION'],
                        endpoint_url=os.environ['SPACE_URL'] ,
                        aws_access_key_id= os.environ['SPACE_ACCESS_KEY'],
                        aws_secret_access_key= os.environ['SPACE_ACCESS_SECRET'])
 
client.upload_file('/home/backups/tmp_dump.gz',
                   os.environ['SPACE_NAME'],
                   datetime.datetime.today().strftime("%Y-%m-%d-%H:%M:%S")+".gz")
  1. We import the required libraries.
  2. We declare a client with the environment variables containing our Spaces information.
  3. We upload the file to Spaces.

The backup file is named tmp_dump.gz because it will be automatically deleted once the upload is complete. In our Space, however, the file will be named with the date and time when it was uploaded.

Downloading the backup

Still in the scripts folder, add the download.py file.

import boto3
import os
import datetime
import sys
 
session = boto3.session.Session()
client = session.client('s3',
                        region_name= os.environ['SPACE_REGION'],
                        endpoint_url=os.environ['SPACE_URL'] ,
                        aws_access_key_id= os.environ['SPACE_ACCESS_KEY'],
                        aws_secret_access_key= os.environ['SPACE_ACCESS_SECRET'])
 
client.download_file(os.environ['SPACE_NAME'],  sys.argv[1], '/home/backups/'+sys.argv[1])
  1. We import the required libraries.
  2. We declare a client with the environment variables containing our Spaces information.
  3. We download the file.

For the download, we use sys.argv[1]. This represents the name of the file we want to download, placed at the end of the command launched for the restore.

Example:

python download.py 2018-07-06-01:00:02.gz

Our Python scripts are now ready. However, as you noticed, they only handle uploading and downloading backups. To generate and restore the backup, we will need shell scripts that handle that for us.

Creating the shell scripts

Backup script

We start with the backup script. Still in the scripts folder, add the backup_mongo.sh file with the following code:

#!/bin/bash
 
set -e
 
DB=$DB_DATABASE
USER=$DB_USERNAME
PWD=$DB_PASSWORD
PORT=$DB_PORT
NAME=tmp_dump.gz
 
echo "=============================="
echo "Backing up the database to the DigitalOcean Space: $SPACE_NAME"
echo "=============================="
echo ""
echo "=============================="
echo "Creating the archive containing the MONGODB database data : $DB"
echo "=============================="
echo ""
 
mongodump  --db $DB --username $USER --password $PWD --host mongo:$PORT --archive=/home/backups/$NAME --gzip
 
echo "=============================="
echo "Copying the archive to the DigitalOcean Space : $SPACE_NAME"
echo "=============================="
echo ""
 
python /home/scripts/backup.py
 
echo "=============================="
echo "Deleting the local archive"
echo "=============================="
echo ""
 
rm /home/backups/$NAME
 
echo "=============================="
echo 'Backup complete!'
echo "=============================="
  1. We retrieve the environment variables.
  2. We use mongodump to generate an archive containing the database data.
  3. We run the Python script that copies the archive to DigitalOcean.
  4. Finally, we delete the temporary archive from our system.

Restore script

Now we will create the script that lets us restore data if there is a problem:

Create the restore_mongo.sh file.

#!/bin/bash
 
set -e
 
DB=$DB_DATABASE
USER=$DB_USERNAME
PWD=$DB_PASSWORD
PORT=$DB_PORT
NAME=$1
 
echo "=============================="
echo "Downloading the archive from the DigitalOcean Space : $SPACE_NAME"
echo "=============================="
echo ""
 
python /home/scripts/download.py $NAME
 
echo "=============================="
echo "Restoring data into the MongoDB database $DB "
echo "=============================="
echo ""
 
mongorestore --db $DB --username $USER --password $PWD --host mongo:$PORT --drop --gzip --archive=/home/backups/$NAME
 
 
echo "=============================="
echo 'Restore complete'
echo "=============================="
  1. We retrieve the environment variables.
  2. We run the archive download script.
  3. We use mongorestore to restore the data into our database.

As with the Python script and sys.argv[1], we retrieve the file name using $1.

So, to restore the data, you need to run the following command from the script folder in your container:

restore_mongo.sh 2018-07-06-01:00:02.gz

Creating the Cron job

Our scripts are ready. We now want to automate the backup so it runs every day at 01:00. To do this, we will create a crontab file at the root of the cron folder and add the following code:

 0 1 * * * root . /root/project_env.sh; /home/scripts/backup_mongo.sh >> /var/log/cron.log 2>&1
# Don't remove the empty line at the end of this file. It is required to run the cron job
  1. We define the job recurrence with 0 1 * * * (every day at 01:00). If you want to learn more about the different scheduling options, click here.
  2. We indicate that we want to run the script as the root user.
  3. We specify where the environment variables are located.
  4. We indicate which script to run.

For point 3, cron does not know which environment it is running in, so we specify which environment variables to use: /root/project_env.sh;

However, the whole point of environment variables is to specify them only when starting our container, without having to write them elsewhere.

We will see, when creating the Dockerfile, how to automatically generate the project_env.sh file with the right environment variables when the container starts.

Creating the Dockerfile

We are nearly at the end of the configuration. We still need to create the Dockerfile at the root of our cron folder:

FROM ubuntu:16.04
 
USER root
# Install Python, cron, and boto3
RUN \
  apt-get update && \
  apt-get install -y python python-dev python-pip python-virtualenv cron apt-transport-https  && \
  rm -rf /var/lib/apt/lists/* && pip install boto3
# Install mongodump and mongorestore
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4 && \
  echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-4.0.list && \
  apt-get update && apt-get install -y mongodb-org-tools
# Add the scripts and the cron job
COPY crontab /etc/cron.d/backup-cron
COPY scripts /home/scripts/
RUN mkdir /home/backups/
# Create the log file
RUN touch /var/log/cron.log
# Set execution permissions
RUN chmod 0644 /etc/cron.d/backup-cron
RUN ["chmod", "+x", "/home/scripts/backup_mongo.sh"]
RUN ["chmod", "+x", "/home/scripts/restore_mongo.sh"]
 
 
# Start the service
CMD printenv | sed 's/^\(.*\)$/export \1/g' > /root/project_env.sh && cron && tail -f /var/log/cron.log

As we saw earlier, we need to create the project_env.sh file so cron can access the environment variables. The following command is responsible for doing that:

printenv | sed 's/^\(.*\)$/export \1/g' > /root/project_env.sh

docker-compose configuration

Now that our backup/restore service is ready, we will add it to the docker-compose configuration of our Parse server.

In the Parse server docker-compose.yml file, add the following code after the previous services:

cron:
  build: ./cron/
  container_name: 'tutorial-cron-backup'
  image: tutorial/parse-server
  environment:
    SPACE_REGION: 'region'
    SPACE_URL: 'url_spaces'
    SPACE_ACCESS_KEY: 'access_key'
    SPACE_ACCESS_SECRET: 'access_secret'
    SPACE_NAME: 'space_name'
    DB_USERNAME: 'user'
    DB_PASSWORD: 'MONGODB_PASSWORD'
    DB_DATABASE: 'db_name'
    DB_PORT: 27017
  volumes:
    - ./cron/scripts:/home/scripts
    - ./cron/backup:/home/backups
  depends_on:
    - mongo

All that remains is to add your Spaces information and make sure the environment variables for access to your MongoDB database match those in the MongoDB service configuration.

mongo:
  image: 'bitnami/mongodb:latest'
  container_name: 'tutorial-mongo-db'
  restart: always
  ports:
    - '27017:27017'
  environment:
    MONGODB_ROOT_PASSWORD: 'MONGODB_ROOT_PASSWORD'
    MONGODB_USERNAME: 'user'
    MONGODB_PASSWORD: 'MONGODB_PASSWORD'
    MONGODB_DATABASE: 'db_name'
  volumes:
    - ./mongo:/bitnami

Configuration example:

Imagine that a Space named test-backup was created in the Amsterdam 3 region on DigitalOcean. The environment variable values for the Space would be:

SPACE_REGION: 'ams3'
SPACE_URL: 'https://ams3.digitaloceanspaces.com'
SPACE_ACCESS_KEY: 'GENERATED_ACCESS_KEY'
SPACE_ACCESS_SECRET: 'GENERATED_ACCESS_SECRET'
SPACE_NAME: 'test-backup'

Usage

Once you have started the services with the command:

docker-compose up

Backups will run automatically thanks to the cron job. However, if you want to manually test the upload to validate your configuration, or if you want to restore your data, here are the steps to follow:

1- Retrieve the cron container ID:

docker ps

2- Once you have copied the container ID, run the following command:

 docker exec -it [REPLACE WITH THE CONTAINER ID]  bin/bash

3- Once you are in the container:

To generate the backup:

home/scrips/backup_mongo.sh

To retrieve the backup:

home/scrips/restore_mongo.sh [BACKUP FILE NAME]

That's it. You can now easily set up the data backup system on your server.