Backing Up MySQL Database using Openshift/Kubernetes Cron Job

Periodic Backups

When running an application in production, the data and app should be backed up periodically. One way to do that is by using git repository for the source code and periodically dumping the database. This article shows how to do such periodic jobs using CronJob functionality of Openshift/Kubernetes.

Technique

When running a cronjob, the filesystem that being used to run the  pod will be ephemeral, that is, non-persistent. To work around this limit we should mount from external volume, which can be in the form of persistent volume claim or directly provided volume specification. 

The Job specs

apiVersion: batch/v1
kind: Job
metadata:
  name: job04-test 
spec:
  template:
    spec:
      containers:
      - name: job04c-test
        image: centos/mysql-57-centos7
        command: ["/bin/sh","-c",  "/opt/rh/rh-mysql57/root/usr/bin/mysqldump --single-transaction -u $MYSQL_USER $MYSQL_DB -h $MYSQL_HOST | gzip > /tmp/dump`date '+%Y%m%d'`.sql.gz"]
        volumeMounts:
        - mountPath: /tmp
          name: dbdumpsvol
          subPath: job03
        env :
        - name: MYSQL_USER
          value: "dbuser"
        - name: MYSQL_DB
          value: "dbname"
        - name: MYSQL_PWD
          value: "xxxx"
        - name: MYSQL_HOST
          value: "10.1.2.3"
      restartPolicy: Never
      volumes:
      - name: dbdumpsvol
        persistentVolumeClaim:
          claimName: dbdumps
  backoffLimit: 4

This job is using persistent volume claim, in which the persistent volume need to be prepared before. If you want to use direct specification, for instance referencing hostpath, the spec will be like this :

apiVersion: batch/v1
kind: Job
metadata:
  name: job05-hostpath
spec:
  template:
    spec:
      containers:
      - name: job05
        image: centos/mysql-57-centos7
        command: ["/bin/sh","-c",  "/opt/rh/rh-mysql57/root/usr/bin/mysqldump --single-transaction -u $MYSQL_USER $MYSQL_DB -h $MYSQL_HOST | gzip > /tmp/dump`date '+%Y%m%d'`.sql.gz"]
        volumeMounts:
        - mountPath: /tmp
          name: dbdumpsvol
          subPath: job05
        env :
        - name: MYSQL_USER
          value: "dbuser"
        - name: MYSQL_DB
          value: "dbname"
        - name: MYSQL_PWD
          value: "xxxx"
        - name: MYSQL_HOST
          value: "10.1.2.3"
      restartPolicy: Never
      volumes:
      - name: dbdumpsvol
        hostPath:
          path: /data/dbdumps
          type: Directory
  backoffLimit: 4

To run the job, save the job spec into a file (example: job05.yaml) first. If using kubernetes, this should do the job :
kubectl apply -f job05.yaml
or if using openshift command line interface :
oc create -f job05.yaml
or if using openshift web console:

  1. login to console, choose one existing project 
  2. click add to project
  3. click import YAML/JSON
  4. paste the job spec
  5. click create


Converting To CronJob


A cronjob is just a job with a cron schedule, for instance this cronjob is running once every hour at the 27th minute :

apiVersion: batch/v2alpha1
kind: CronJob
metadata:
  name: crondbdump1
spec:
  schedule: "27 * * * *"  
  successfulJobsHistoryLimit: 7
  jobTemplate:             
    spec:
      template:
        metadata:
          labels:          
            parent: "crondumpifrs"
        spec:
          containers:
          - name: crondbdump1
            image: centos/mysql-57-centos7
            command: ["/bin/sh","-c",  "/opt/rh/rh-mysql57/root/usr/bin/mysqldump --single-transaction --routines -u $MYSQL_USER $MYSQL_DB -h $MYSQL_HOST | gzip > /tmp/dump`date '+%Y%m%d_%H%M%S'`.sql.gz"]
            volumeMounts:
            - mountPath: /tmp
              name: dbdumpsvol
              subPath: job03
            env :
            - name: MYSQL_USER
              value: "dbuser"
            - name: MYSQL_DB
              value: "dbname"
            - name: MYSQL_PWD
              value: "xxxx"
            - name: MYSQL_HOST
              value: "10.1.2.3"
          restartPolicy: OnFailure 
          volumes:
          - name: dbdumpsvol
            hostPath:
              path: /data/dbdumps
              type: Directory

To run the cronjob, use the same procedure (oc create -f and kubectl apply -f ).  In Openshift, we are limited to command-line interface monitoring the cron jobs :


 oc get cronjobs
NAME                 SCHEDULE      SUSPEND   ACTIVE    LAST-SCHEDULE
crondumpifrs2        27 * * * *    True      0         Mon, 16 Apr 2018 16:27:00 +0700

oc edit crondumpifrs

References


Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more