Backup mongodb to GCP storage with crontab
情況是要把原本寫入 local mongodb 的 usage log dump 出 csv 檔案(並且從mongodb內刪除),並且把較舊的 csv 檔案移去 GCP storage。
mongodump.sh
#!/usr/bin/env bash MY_DATE="'$(date +"%Y-%m-%d")'" #echo $MY_DATE MY_DATE_QUERY="{'recorddatetime': {\$lt: ${MY_DATE} } }" #echo $MY_DATE_QUERY #backup echo "$(mongoexport -d transdb -c validtrans --query " ${MY_DATE_QUERY} " --fields=fieldA,fieldB,fieldC,fieldD,recorddatetime --quiet --out=/var/opt/mongodump/validtrans_$(date "+%Y%m%d").csv --type=csv)" #delete echo "$(mongo transdb --quiet --eval "db.validtrans.deleteMany( ${MY_DATE_QUERY} ) " )" #move to bucket echo "$(find /var/opt/mongodump/*.csv -type f -ctime +30 -exec gsutil mv {} gs://backup_bucket/path_to_destination/ \;)"
實際上執行的時候會有錯誤訊息
$ bash ./dump_mongo_keyusage.sh Copying file:///var/opt/mongodump/validtrans_20220131.csv [Content-Type=text/csv]... Your "Oauth 2.0 User Account" credentials are invalid. Please run $ gcloud auth login Failure: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590).
除了做 gcloud auth login 之外,還要做 gcloud auth activate-service-account。詳細可以參考 https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account。同時要搭配 key json file
,會長得類似這樣…
{ "type": "service_account", "project_id": "my-services", "private_key_id": "PRIVATE_KEY_ID", "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY==\n-----END PRIVATE KEY-----\n", "client_email": "my-services@appspot.gserviceaccount.com", "client_id": "", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-services%40appspot.gserviceaccount.com" }
執行 gcloud auth activate-service-account
$ gcloud auth activate-service-account my-services@appspot.gserviceaccount.com --key-file=/var/opt/key.json --project=my-services
然後執行 mongodump.sh 就可以了
$ ./mongodump.sh Uploading gs://keyusage_raw_backup/validtrans_20220131.csv: 106.39 MiB/106.39 MiB Removing file:///var/opt/mongodump/validtrans_20220131.csv... Copying file:///var/opt/mongodump/validtrans_20220131.csv [Content-Type=text/csv]...
Original link: Phanix's Blog