使用 crontab 将 mongodb 备份到 GCP 存储

情况是要把原本写入local mongodb 的usage log dump 出csv 档案(并且从mongodb内删除)，并且把较旧的csv 档案移去GCP storage。

mongodump.sh

 #!/usr/bin/env bash
MY_DATE="'$(date +"%Y-%m-%d")'"
#echo $MY_DATE
MY_DATE_QUERY="{'recorddatetime': {\$lt: ${MY_DATE} } }"
#echo $MY_DATE_QUERY

#backup
echo "$(mongoexport -d transdb -c validtrans --query " ${MY_DATE_QUERY} " --fields=fieldA,fieldB,fieldC,fieldD,recorddatetime --quiet --out=/var/opt/mongodump/validtrans_$(date "+%Y%m%d").csv --type=csv)"

#delete
echo "$(mongo transdb --quiet --eval "db.validtrans.deleteMany( ${MY_DATE_QUERY} ) " )"


#move to bucket
echo "$(find /var/opt/mongodump/*.csv -type f -ctime +30 -exec gsutil mv {} gs://backup_bucket/path_to_destination/ \;)"

实际上执行的时候会有错误讯息

$ bash ./dump_mongo_keyusage.sh 
Copying file:///var/opt/mongodump/validtrans_20220131.csv [Content-Type=text/csv]...
Your "Oauth 2.0 User Account" credentials are invalid. Please run
  $ gcloud auth login
Failure: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590).

除了做gcloud auth login 之外，还要做gcloud auth activate-service-account。详细可以参考https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account 。同时要搭配key json file
，会长得类似这样…

 {
  "type": "service_account",
  "project_id": "my-services",
  "private_key_id": "PRIVATE_KEY_ID",
  "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY==\n-----END PRIVATE KEY-----\n",
  "client_email": "my-services@appspot.gserviceaccount.com",
  "client_id": "",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-services%40appspot.gserviceaccount.com"
}

执行gcloud auth activate-service-account

 $ gcloud auth activate-service-account my-services@appspot.gserviceaccount.com --key-file=/var/opt/key.json --project=my-services

然后执行mongodump.sh 就可以了

$ ./mongodump.sh
Uploading gs://keyusage_raw_backup/validtrans_20220131.csv: 106.39 MiB/106.39 MiB    
Removing file:///var/opt/mongodump/validtrans_20220131.csv...
Copying file:///var/opt/mongodump/validtrans_20220131.csv [Content-Type=text/csv]...

Original link: Phanix's Blog