DODO Pool Incident Postmortem: With a Little Help from Our Friends
Two days ago, we published an official statement regarding the March 9 security incident impacting DODO’s Crowdpooling contracts. Humbled by the experience, we have updated our security practices to the highest industry standards and are reinforcing our auditing processes. We anticipate Crowdpooling to resume next week.
Tracing the steps of how the funds were recovered in this incident has been a valuable lesson, both for us and for the community, on the intricate symbiosis of the state-of-the-art trading bots and DeFi protocols in the Dark Forest of Ethereum.
The story of exactly how the funds were recovered evokes a mystery novel with a villain, selfless heroes, plenty of plot twists, and ultimately, a happy ending.
Here is the account of what transpired over the last few days penned from the perspective of DODO CEO Radar Bear:
Ethereum is a Dark Forest
Last August, there was a widely circulated article in DeFi titled “Ethereum is a Dark Forest”, which introduced advanced predators in the ecosystem called “generalized front running bots”. These bots constantly monitor the mempool for transactions that have been broadcast but not yet landed on-chain. Once a pending original transaction is found to be profitable, these bots send the exact same transaction with their own addresses at a higher gas to get ahead of, or “frontrun”, the original transaction.
If you have seen the anime Naruto, these frontrunning bots are like Kakashi, who used his Sharingan to copy and preemptively initiate any attack techniques his opponent Zabuza intended to use.
With these ruthless bots running rampant, the Ethereum ecosystem is a “Dark Forest”, where every move of yours is being watched, and any intentions, good or evil, will be exposed to these robotic predators in the dark once they become pending transactions. Fortunately, our investigation unveils a story of a Dark Forest full of warmth and chivalry.
We Made a Mistake
8:00AM, March 9th UTC+8
I got an urgent call from our operations team saying that there was a vulnerability in the DODO smart contracts that was exploited. I immediately called a meeting with the dev team to investigate.
After a quick probe, we discovered the root cause: the init() function in the pool creation contract could be called multiple times, leading to repeated initializations. The attackers took advantage of this and borrowed out the tokens in the liquidity pools using flash loans. The pool contract was then initialized again and counterfeit tokens created by the attackers were returned in lieu of the original tokens, bypassing the flash loan return check logic.
It is important to note that the presence of this vulnerability had nothing to do with PeckShield, who was in charge of auditing our smart contracts. Rather, we had merged, after the audit, several code changes to simplify the logic before going live in February, and missed this critical permission management step. We made a huge mistake.
Fortunately, this issue only impacted part of our V2 pool functionality. The trading module was unaffected. In addition, only project teams that worked with us during pool creation and provided liquidity for their pools saw losses. DODO users’ assets were untouched.
We began our rescue and remediation efforts immediately. Within 15 minutes, our dev team identified all the pools that were still exposed to the vulnerability and rescued the remaining funds at risk (roughly $80,000). Then, we disabled the pool creation portal on the frontend side. Our operations team sent out an announcement to inform the DODO community of this incident and reached out to project teams.
Meanwhile, we monitored the on-chain movements of lost assets and estimated the total to be $3.8 million worth of USDT, ETH, and various project tokens.
Good News from Samczsun
8:30AM, March 9th UTC+8
As our investigation was underway, well-known white hat hacker Samczsun contacted me via DMs. He told me that an entity had “accidentally obtained” some of the stolen assets, and was willing to return them under the condition of anonymity.
Since we are an animal-loving bunch, I will refer to this entity as Leopard in our Dark Forest analogy. Who exactly is Leopard? How exactly did they manage to obtain some of the assets? Do they know the whereabouts of the remaining funds?
On-Chain Evidence Led to an Unexpected Turn of Events
10:00AM, March 9th UTC+8
Further analysis revealed that two addresses executed the attack. We will refer to these addresses as Hippo (0x368) (“Individual A” in our preliminary report) and Gazelle (0x355) (“Individual B” in our preliminary report), respectively.
Hippo executed two attacks. One of them, worth $200,000, went to a centralized exchange, which we were able to freeze immediately after contacting the exchange support staff. The other one, worth $1.89 million, was exactly the same amount as the amount Leopard was going to return to us via Samczsun. Therefore, we speculated that Hippo and Leopard were the same person and assumed their identity to be a white hat hacker.
On the other hand, Gazelle, executed their attacks with an arbitrage bot. It’s worth noting that they used up to 90,000 gwei of gasPrice to send their on-chain transactions, with one of the transactions costing an astonishing 8 ETH! It seemed likely that Gazelle’s bot automatically snapped up the attacker Mr. Hippo’s transaction, frontrunning it, and Gazelle probably didn’t even know about it!
This was another good news for us. If we could get in touch with Gazelle, then there was a possibility that the funds taken by them could be recovered as well.
A Puzzling Series of Events
9:00PM, March 9th UTC+8
We were glad and relieved to receive and recover $1.89 million worth of assets from Leopard, but what came as a surprise was that Leopard admitted it was not Hippo. So there were at least three entities involved in this incident! Moreover, we didn’t know how Leopard gained access to Hippo’s assets.
At the time, the only entity we could possibly reach was Leopard, who was well-versed in the law of the jungle that is the Dark Forest. Although Leopard continued to remain anonymous, we were able to directly communicate with Leopard through the help of Samczsun and other white hat friends.
It’s a Small World
1:30AM, March 10th UTC+8
I never thought in a million years that Leopard would be someone I knew. I met them back in 2018, when I was still a core developer at DDEX. We had lots of discussions on smart contract development and best practices together. I lost touch with Leopard shortly after I left DDEX and Leopard didn’t know that I became the co-founder of DODO either.
Leopard told me that Hippo was the original hacker and Hippo transferred the assets from the hack into a contract. However, this contract had a loophole that allowed anyone to withdraw assets from it. When Hippo sent a transaction to withdraw the assets, Leopard was able to frontrun it, thus “accidentally obtaining” the funds. What a plot twist.
While we were discussing internally how to contact Gazelle, they reached out to me first.
The Whole Story
3:00AM, March 10th UTC+8
Gazelle emailed me anonymously and offered to return the $1.2 million worth of assets they drained from our pools. My team and I breathed a collective sigh of relief, knowing that we would soon be able to recover the majority of the lost assets.
Gazelle gave us a detailed account of what happened and all the events they were monitoring, allowing us to finally see the full picture. We will omit the transaction hashes here, as Gazelle wanted to keep a low profile.
In summary,
- The only real attacker was Hippo.
- Hippo executed two attacks, but both were frontrun by Gazelle.
- Hippo got very frustrated and spent some time writing a contract to bypass Gazelle’s bot. They succeeded this time, and the funds went into Hippo’s contract.
- Hippo’s withdrawals from the contract were again frontran by bots. Gazelle and Leopard had a “gas battle”, and eventually Leopard won. By this point, Hippo had already executed 3 attacks, but got nothing to show for it, since all of them were frontrun by the bots sniping from the Dark Forest!
- Hippo was eventually able to execute two successful attacks, but both involved relatively small amounts, netting them a total of about $200,000. We are still exploring ways to recover these funds.
- Before Gazelle returned the drained vETH tokens, their bot was trapped by a honeypot contract designed and deployed specifically for it. This trap contact used 0.05 ETH as bait and stole 324 vETHs (worth about $500,000) from the bot. We still don’t know who set this trap — it could have been Hippo, or it could have been some other individual. We will refer to this trap-setter as Skunk.
- In the end, Gazelle generously decided to share the loss with us, for which we are very grateful for.
- Within 24 hours of the attacks, we were able to recover $3.1 million out of the $3.8 million stolen, with a further $200,000 frozen on an exchange at our request and $300,000 given as a special bounty.
♫ Oh, We Get by with a Little Help from Our friends
There are many predators in the Dark Forest, but they are not all as cold and ruthless as one might think. We were fortunate enough to be saved by the Good Samaritans of the bot ecosystem, who intercept stolen assets from malicious actors and return them to the victims.
To this day, there are still many people who believe that the crypto world is full of scammers and hackers. In our opinion, such an overgeneralization is unfair. We are genuinely blessed and humbled by everyone who lent us a helping hand during DODO’s most difficult time. We send our heartfelt gratitude and respect to the amazing DeFi and crypto community:
- PeckShield
- SlowMist
- Binance Smart Chain security team
- @samczsun
- @tzhen
- 1inch
- TokenLon
- Binance
- Huobi
- Etherscan
We received words of encouragement and consolations from many, including our competitors, who stood with us in this time of crisis. Thank you!
P.S. The Honeypot and Portals in the Dark Forest
Meanwhile, it is immensely encouraging to see that security researchers and mining pools are building “portals” designed to help users navigate the Dark Forest, such as MEV-Geth from MEV research organization Flashbots, which directly connects to multiple mining pools, and Taichi Network, which connects users to Sparkpool.
These “portals” connect the sender of transactions directly with privacy and expediency to the mining pool(s), which can prevent transactions from being front run.
Welcome to the DODO Zoo!
Official Website: https://dodoex.io/
FAQs: https://dodoex.github.io/docs/docs/
GitHub: https://github.com/DODOEX
Telegram: https://t.me/dodoex_official
Twitter: https://twitter.com/BreederDodo
Discord: https://discord.gg/tyKReUK
Forum: https://community.dodoex.io/
