驗證人被 Tombstoned

在一星期前我們驗證人 BusinessAsMission 看到 節點 Server 的 storage 已被佔用了 90%, 加上最近執行liked-command 時不能成功運行，並出現下以下的 Error:

“ERROR: for likecoin-chain_liked-command_run Cannot create container for service liked-command: invalid mount config for type "bind": bind source path does not exist: xxxxxx”

嘗試過很多方法甚至重新安裝 Docker 也不成功，所以決定 Migrate 驗證人去新 node.

所以我們開始做 Migration 的工作，就在新的節點成功運行，並且 “catching_up = false” 的時候。作為一個傳統的 IT 人來說，通常是會想著新舊兩個系統同時運行，好讓節點不用 offline, 並在新節點完成 “recover” 的動作後，經過一輪快速的檢查及測試，就馬上把舊validator 關閉。 但現在知道這做法不適宜在遷移節點上運用...

當時 https://likecoin.bigdipper.live/ 及 https://stake.like.co/ 並沒有什麼異樣，但不久後就出現了 “Tombstoned”

我嘗試了多次 unjial 也失敗，並有以下的response:

{"height":"2502809","txhash":"3257318FBB87542B16538A09D641E65FF3B8A1641081EB6BBFC6710A8990BCDC","codespace":"slashing","code":4,"data":"","raw_log":"failed to execute message; message index: 0: validator still jailed; cannot be unjailed","logs":[],"info":"","gas_wanted":"200000","gas_used":"43291","tx":null,"timestamp":""}

我就立刻到 discord 求助，但可惜這段時間就是香港及台灣的清晨時間，相信大部分驗證人也在睡覺中。

我只好在 Google 上找 tombstoned 及 cosmos. SDK，有下以下的資料。

https://docs.cosmos.network/v0.44/modules/slashing/07_tombstone.html

Proposal: infinite jail

We propose setting the "jail time" for a validator who commits a consensus safety fault, to infinite (i.e. a tombstone state). This essentially kicks the validator out of the validator set and does not allow them to re-enter the validator set. All of their delegators (including the operator themselves) have to either unbond or redelegate away. The validator operator can create a new validator if they would like, with a new operator key and consensus key, but they have to "re-earn" their delegations back. Implementing the tombstone system and getting rid of the slashing period tracking will make the slashing module way simpler, especially because we can remove all of the hooks defined in the slashing module consumed by the staking module (the slashing module still consumes hooks defined in staking).

https://docs.like.co/user-guide/background#9e68

If, however, a validator does not perform its duty or act maliciously, there will be penalty: • Unavailability. Offline for more than 95% of times in the previous 10,000 blocks, i.e. the validator participated for fewer than 500 blocks in the previously 10,000 blocks. In this case 0.01% of the delegated LikeCoin will be slashed and the validator be “jailed”, until the validator unjail itself after at least 10 minutes. • Double signing. Validates 2 conflicting blocks at the same time, which is fatal because it allows an amount to be spent twice. In this case, the validator will be permanently removed, and 5% of the delegations will be slashed. In both cases, delegated LikeCoin from Likers will be slashed at the same ratio. It is important for Likers to delegate only to validators they can trust. In blockchain jargon, the above mechanism of decentralized consensus is called Byzantine Fault Tolerance (BFT), achieved through Delegated Proof of Stake (DPoS).



"Double signing" 之前雖然曾經看過，但並未完全了解怎樣會被觸發出來。 觸發Double signing 的就是兩個相同節點同一時間運行，那怕只是很短的時間。

今次的事件我們學到：

1.    兩個相同的節點不能相時間運行。

2.    就算是遷移節點，寧願 offline 節點，因大約需要 15 小時的 offline 才會坐牢，應有足夠時間為節點更新及修正。

3.    盡量不要在香港及台灣時間 00:00-09:00am 才緊急在 DISCORD 求救，因為該時間大部分驗證人也未必在線。

4.    驗證人要以委托人及likecoin community 的好處為大前提，每個改動也要深思而行。

今次的事件叫支持我們的人失望及在likecoin 上有損失我們十分抱歉。

我會把我們所經歷及學到的東西分享出來，讓社群及其他持分者不用再經歷損失。

也希望能集合各種節點操作的文章及FAQ，一起建立穏建的驗證人網絡來造福社群。

在這我們（@保羅．真 及@楊軍 YK ）要向每一位「BusinessAsMission」節點委託人致歉，令你有不必要的損失。希望你體諒。

**舊的「BusinessAsMission」節點，已刪除了名稱。你在Inactive中見到「BusinessAsMission」節點是新的，可以放心轉移委託(Restake)，謝謝。

https://stake.like.co/validators/cosmosvaloper1xnpa0k9ywwmk3lmysxred94zamffwkqmna5wur

再次為到每一位委託人帶來的損失抱歉。我們吸取了今次經驗，相信有助我們往後在這社群會有更好的貢獻。多謝您的信任，請繼續支持我們的工作。🙏 謝謝 🙇🏻‍♂️