Low Severity Recipient Cancellation Bug Post-Mortem
What Happened
On June 30, 2021, 8pm GMT, a white-hat member of the Sablier community notified our team of a low-severity bug discovered in a middleware contract used by the Sablier frontends.
The bug does not allow for any funds to be stolen by a third-party attacker. However, it allows recipients of streams to manipulate their stream in such a way that unstreamed tokens could become unrecoverable by the stream creator — effectively burning any unstreamed tokens. No loss of funds has occurred.
Root Causes
Sablier v1.0 is made up of three smart contracts, all non-upgradeable:
- Payroll.sol — Dapp middleware
- Sablier.sol — Money streaming engine
- CTokenManager.sol — Compound token manager
The frontends used to rely on the “Payroll.sol” middleware for creating the streams. That contract has a “createSalary” function that calls “createStream” on “Sablier.sol” under the hood. The rationale for why we decided to route streams through this contract instead of using “Sablier.sol” directly was threefold:
- Separate the payroll use-case, which our frontends were envisioned for, from the more generic money streaming protocol.
- Support the Ethereum Gas Station Network through the middleware but not the money streaming engine.
- Make the middleware upgradeable but not the money streaming engine (we ended making neither upgradeable).
After creating a stream, there are two actions that can be performed on it: withdraw and cancel. The “Sablier.sol” exposes these actions via two functions:
- withdrawFromStream
- cancelStream
While the “Payroll.sol” middleware exposes the same two actions via these other functions:
- withdrawFromSalary (calls “withdrawFromStream”)
- cancelSalary (calls “cancelStream”)
The bug occurs when a stream is created via “Payroll.sol” but is cancelled via “Sablier.sol” by the recipient. In all other cases, the protocol is behaving correctly. But if the recipient triggers the “cancelStream” function on a stream created via the middleware, any remaining unstreamed tokens would end up locked in the “Payroll.sol” contract, effectively burning them.
Resolution
Immediately after being notified, we paused all other development efforts and set out to patch the bug. Within 24 hours, we had a patched contract deployed internally, and we began working on updates for the front-end and subgraph to go live with an update as soon a possible. Specifically, we:
- Deployed a new Sablier.sol contract (v1.1), which supersedes v1.0
- Updated the subgraphs to index both the v1.0 and the v1.1 contracts
- Updated the official interfaces at pay.sablier.finance, app.sablier.finance, and the Gnosis app to use the v1.1 contract for creating streams while maintaining backwards-compatibility with v1.0 for cancelling and withdrawing from streams.
Next, we gave notice to stream creators who had the most significant exposure to the bug, giving them a chance to recreate their streams before going live with the public disclosure. Fortunately, we managed to migrate and rescue the vast majority of streams.
Impact
The bug affects all streams created via the “Payroll.sol” middleware contract, but certain users are not affected, or they bear a very low risk:
- If you used our interfaces after July 12, 2021, 05:33:43 PM, you are NOT affected.
- If you used the Sablier.sol contract to create the streams, instead of the Payroll.sol middleware, you are NOT affected.
- If your streams ended, you are NOT affected.
- If you’re streaming money to yourself, you have a VERY LOW risk of becoming affected.
- If you’re streaming money to trustworthy counterparties, you have a LOW risk of becoming affected.
Low risk means that the recipient has no economic incentive to exploit the bug. If they do, they won’t receive the unstreamed portion of the stream.
Also, the cancel stream function would have to be triggered in a programatic way, since both the official interface and the Gnosis integration do not allow users to interact in a way that could exploit the bug.
Action Items
If your streams are affected, you can either do nothing (if you think the risk is acceptable for your circumstance) or you can cancel and recreate the streams. For your conveniency, we built a frontend utility that makes it easy for you to migrate the streams:
Alternatively, you can manually recreate the streams via the user interfaces:
- Go to pay.sablier.finance or gnosis-safe.io.
- Cancel the stream; this will automatically distribute the streamed amount to the recipient and the unstreamed amount back to you.
- Calculate the difference between the older stream’s stop time and the current time.
- Recreate the stream with the unstreamed amount that you received back and the duration you calculated at step 2.
If you have questions, please join the the Sablier Discord server; our team, and members of the community, look forward to helping you.
Timeline
All times are in UTC.
Jun-30–2021 08:00:14 PM
The white-hat hacker reported the bug via a private communication channel.
Jul-01–2021 04:48:52 PM
Removed the buggy “Payroll.sol” contract from the code base. Started testing a patched version of the protocol on a local development network.
Jul-02–2021 10:35:20 AM
Deployed patched contract on Rinkeby, for testing purposes.
Jul-02–2021 04:42:52 PM
Deployed patched contract on Mainnet.
Jul-05–2021 11:49:09 AM
Finished reimplementing our subgraph so that it indexes both v1.0 and v1.1 streams. See commits 8cdfc3f and 06eca34. Created throwaway subgraphs on The Graph’s hosted service, for testing purposes.
Jul-06–2021 09:02:35 PM
Finished reimplementing our official interfaces so that they work with the new subgraph and the new contracts, while still remaining backwards compatible with v1.0 streams. Started testing on Rinkeby.
Jul-10–2021 08:06:00 PM
Finished reimplemented our Gnosis Safe app so that it works with the the new subgraph and the new contracts. See commit 1363fb2. Started testing on Rinkeby.
Jul-12–2021 03:09:54 PM
Deployed the updated subgraph implementation in production.
Jul-12–2021 05:33:43 PM
Deployed the updated official interfaces to pay.sablier.finance and app.sablier.finance.
Jul-14–2021 07:47:12 AM
The Gnosis team merged our PR to update our Gnosis Safe app on gnosis-safe.io.
Jul-14–2021 09:52:32 AM
Started notifying stream creators about about the bug, and recommended them to migrate the streams to the patched contract.
Jul-27–2021 06:45:04 PM
Published this port-mortem publicly.
Lessons Learned
First, this was a fresh reminder that high test coverage does only so much to ensure safety of use. Even if we added a test case that fended off the bug, test coverage wouldn’t have increased because all the logical branches that cause the bug to exist had already been covered. See GitHub and Coveralls (test coverage was around 98.19%).
Finally, the impacted contract has been within scope of audits internally, from a professional firm, and many other third-party audits from teams using Sablier for token vesting. This experience highlights one of the most significant values that comes from open-source development in continuously evaluating the security and soundness of your published work. Appropriately, you can expect that we remain committed to continue rewarding, based on severity, anyone who responsibly discloses bugs or vulnerabilities to our team.
Bounty Payout
While the scope of circumstance in which an attacker might leverage this bug is narrow, we still feel a great sense of responsibility in ensuring that the right incentives exist for responsible disclosure. Accordingly, we’ve paid out 5,000 DAI to Jack Aldridge for his discovery and cooperation in the matter.
We remain committed to following industry best practices for security and responsible bug disclosure. Thank you for your continued support and trust in Sablier.
