Engineering

Writing Accurate Time-Dependent Truffle Tests

Paul Razvan Berg

• 6 min read
Writing Accurate Time-Dependent Truffle Tests

When you’re building a dapp for continuous salaries, writing accurate time-dependent tests is a necessity. I recently learned how to do it the hard way, bashing my head against the wall, so I wrote this article to spare you the hassle.

I will further assume that:

  1. Your testing framework is Truffle, Ganache + Mocha.
  2. You want to your tests to be as accurate as possible (<5 seconds error margin).

Manipulating time in tests written for Ethereum smart contracts comes with some gotchas.

Gotcha #1: RPC

While it is totally possible to jump forwards and backwards in time in Ganache, there are multiple ways to achieve this. However, to get accurate results, you should use the evm_mine RPC method in the following way:

{
  "jsonrpc": "2.0",
  "method": "evm_mine",
  "params": ["NUMBER_OF_SECONDS"],
  "id": 1
}

As a promise in Javascript:

const advanceBlockAtTime = (time) => {
  return new Promise((resolve, reject) => {
    web3.currentProvider.send(
      {
        jsonrpc: "2.0",
        method: "evm_mine",
        params: [time],
        id: new Date().getTime(),
      },
      (err, _) => {
        if (err) {
          return reject(err);
        }
        const newBlockHash = web3.eth.getBlock("latest").hash;

        return resolve(newBlockHash);
      },
    );
  });
};

Notes:

  1. Without the NUMBER_OF_SECONDS parameter, the RPC call increases only the block height but doesn’t jump in time.
  2. The “id” parameter is optional but good to have. It doesn’t really matter what value you put in there while testing.

There is also evm_increaseTime, which increases the “internal clock” of Ganache so that whenever the next block is mined it has a timestamp offset. This adds overhead though:

// Not what you want
advanceTimeAndBlock = async (time) => {
  await advanceTime(time)
  await advanceBlock()
  return Promise.resolve(web3.eth.getBlock('latest'))
}

That’s right, you’d have to make two RPC calls, compared to only one with the first approach.

Props to

Jakub Wojciechowskifor coming up withPR #13for ganache-core. It would’ve been hard to write accurate tests without a deterministic and atomic way to jump in time in Ganache.

Gotcha #2: Run Time

Code itself takes time to execute. Specifically, javascript promises take a non-negligible amount of time to resolve.

This is obvious, right? When writing time-dependent tests for Ethereum smart contracts, things get delicate.

Consider the following:

  1. The time it takes for the test case to run
  2. The amount of seconds you want to jump in the future
  3. The unix timestamp for the moment when you submit yarn run test in your console

Contingent on these variables, your test block may get caught in between the passage of one or more seconds, hence your assertions may break.

For instance:

describe("when the stream did start but not end", function() {
  beforeEach(async function() {
    await advanceBlockAtTime(
      now
        .plus(STANDARD_TIME_OFFSET)
        .plus(5)
        .toNumber(),
    );
  });

  describe("when the withdrawal amount is within the available balance", function() {
    const amount = new BigNumber(5).multipliedBy(1e18).toString(10);

    it("makes the withdrawal", async function() {
      const balance = await this.token.balanceOf(recipient);
      await this.sablier.withdraw(streamId, amount, opts);
      const newBalance = await this.token.balanceOf(recipient);
      balance.should.be.bignumber.equal(newBalance.minus(amount));
    });
  });
});

What this does is that it calls the Sablier contract to withdraw a previously deposited wad of money. The rules that dictate how much the caller can withdraw are specified in ERC-1620.

I expected the test to pass consistently, but I was wrong. So wrong. It started to break ~1 in 8 times and the thing I feared most happened, that is, non-deterministic variance.

I logged the unix timestamp in mocha’s before block and I measured how long it takes for the test to run using node’s performance timing api:

t0 1565455128964
Call to sablier.withdraw took 115.92673601210117 milliseconds.
            1) makes the withdrawal

If you add 115 to 1565455128964, you end up with a number that ends with 9079, thus the number of seconds increases from 8 to 9. This is what broke the assertion, because I was expecting a balance of x, when I was actually getting x + 1 (more seconds passed = more money).

While it’s impossible to write a program P1 that can compute how long it takes for another program P2 to finish (see Turing’s Halting Problem), we could safely assume that no more than 1 second should pass between your beforeEach and it blocks. This is assuming the back and forth communication between your node instance and ganache is almost instantaneous, even when running coverage.

Here’s the fix:

balance.should.bignumber.satisfy(function(num) {
  return (
    num.isEqualTo(newBalance.minus(amount)) || num.isEqualTo(newBalance.minus(amount).plus(ONE_UNIT))
  );
});

Where ONE_UNIT is one monetary unit allocated per second, as per the Sablier model. It is imperfect, but way better than using a “greater than” or “less than” equality check.

Finally, as the OpenZeppelin team argues here, you might not need this level of precision. If your dapp doesn’t involve timestamps or block numbers directly, tolerating larger chronological offsets is perfectly fine. Nonetheless, it’s good to be aware of run time.

Gotcha #3: BeforeEach and AfterEach

Your mileage may vary, but you may want to jump forwards in “beforeEach” and jump backwards in “afterEach”. This is because your contract might have some variables defined in the scope of the “describe” block and you want to run a sequential set of “it” blocks that all assume the same state. Not reverting back in “afterEach” would only increase the timestamp forever.

Example:

describe("when the stream did start but not end", function() {
  const amount = new BigNumber(5).multipliedBy(1e18).toString(10);

  beforeEach(async function() {
    await web3.utils.advanceBlockAtTime(
      now
        .plus(STANDARD_TIME_OFFSET)
        .plus(5)
        .toNumber(),
    );
  });

  it("test1", function() {});
  
  it("test2", function() {});
  
  it("test3", function() {});

  afterEach(async function() {
    await web3.utils.advanceBlockAtTime(now.toNumber());
  });
});

As you can see in the snippet above, we have three tests in which we assume the amount withdrawn is 5. In the context of Sablier, advancing in time 15 seconds would yield a withdrawable amount of 15, thus we have to go back to the original state in the “afterEach” block.

Gotcha #4: Snapshots

While truffle provides its own clean-room environment, it’s not a bad idea to implement your own snapshotting mechanism. That is, going back to the original state after all tests are done. It may be helpful in CI or other external environments.

takeSnapshot = async () => {
  return new Promise((resolve, reject) => {
    web3.currentProvider.send(
      {
        jsonrpc: "2.0",
        method: "evm_snapshot",
        id: new Date().getTime(),
      },
      (err, snapshotId) => {
        if (err) {
          return reject(err);
        }
        return resolve(snapshotId);
      },
    );
  });
};

revertToSnapshot = async (id) => {
  return new Promise((resolve, reject) => {
    web3.currentProvider.send(
      {
        jsonrpc: "2.0",
        method: "evm_revert",
        params: [id],
        id: new Date().getTime(),
      },
      (err, result) => {
        if (err) {
          return reject(err);
        }
        return resolve(result);
      },
    );
  });
};

Define those functions in your codebase and then insert this in one of your root test files:

let snapshot;
let snapshotId;

before(async () => {
  snapshot = await takeSnapshot();
  snapshotId = snapshot.result;
});

after(async () => {
  await revertToSnapshot(snapshotId);
});

Voilà, now your blockchain will revert to its original timestamp after all magical time jumpings.

Wrap Up

Some of the bits and pieces used throughout this article are inspired or taken from other writings, such as Ethan Wessel’s amazing Standing the Time of Test with Truffle. ̶T̶h̶e̶ ̶o̶n̶l̶y̶ ̶c̶a̶v̶e̶a̶t̶ ̶w̶i̶t̶h̶ ̶t̶h̶a̶t̶ ̶a̶r̶t̶i̶c̶l̶e̶ ̶i̶s̶ ̶t̶h̶e̶ ̶u̶s̶a̶g̶e̶ ̶o̶f̶ ̶b̶o̶t̶h̶ ̶e̶v̶m̶_̶m̶i̶n̶e̶ ̶a̶n̶d̶ ̶e̶v̶m̶_̶i̶n̶c̶r̶e̶a̶s̶e̶T̶i̶m̶e̶,̶ ̶a̶n̶d̶ ̶w̶e̶ ̶e̶x̶p̶l̶a̶i̶n̶e̶d̶ ̶a̶b̶o̶v̶e̶ ̶w̶h̶y̶ ̶t̶h̶i̶s̶ ̶i̶s̶ ̶n̶o̶t̶ ̶i̶d̶e̶a̶l̶.̶

Update: Ethan was really cool and he updated his post and ganache-time-traveler package with a few methods that use evm_mine as indicated in this article.

Also, here’s a very good StackExchange thread on the inherent security of block.timestamp and some GitHub threads that shed some light on the history of deterministic time jumping in Ganache (1 and 2).

Thanks for reading! More on Sablier:

If you want to get in touch personally, I’m on Twitter and Keybase.

If you have any questions, ideas, or issues, ping us on Discord or Twitter — we’d love to hear from you.