Benchmarking

Benchmarking provides insights into Phink’s performance in real-world scenarios, in order to vizualise its efficiency and fuzzing ink! smart contracts. Below are the benchmark results for various smart contracts, detailing coverage, speed, corpus size, and the potential usage of generate-seed. Each contract were fuzzed for maximum a day. Statistics (especially average speed) are given for one core only. The coverage percent is calculated using the number of lines covered divided the number of reachable lines, as a percentage.

⚠️ The point of the benchmark is to demonstrate how much coverage is reachable within a day of fuzzing without doing proper seed creation. In a real fuzzing campaign, the developers would aim for 100% coverage, by creating seeds, adding GenesisConfig values, more (E2E) tests extracted with seed-generator, etc.

Benchmarks

• Github for abax_governor : AbaxFinance/dao-contracts/tree/main/src/contracts/abax_governor
• Github for multisig : use-ink/ink-examples/blob/main/multisig/lib.rs
• Github for erc1155 : use-ink/ink-examples/blob/main/erc1155/lib.rs

Dummy benchmark

The dummy benchmark involves a simple nested if-condition. It acts as a reference to ensure that the fuzzer is effectively coverage guided. The results for this benchmark are as follows:

• Average speed: 7,500 executions per second in average
• Number of cores used: 10
• Time until invariant triggered: 48 seconds
• Stability: 99.43%
• Fuzzing origin: false
• Final corpus size: 12 seeds

Dummy logic

The logic tested in the dummy benchmark can simply be represented that way:

if data.len() > 3 && data.len() < 7 {
    if data.chars().nth(0).unwrap() == 'f' {
        if data.chars().nth(1).unwrap() == 'u' {
            if data.chars().nth(2).unwrap() == 'z' {
                if data.chars().nth(3).unwrap() == 'z' {
                    self.forbidden_number = 42;
                }
            }
        }
    }
}

Contracts

ERC-1155

The ERC-1155 contract is a standard for creating multiple token types within a single contract. It allows for the creation of both fungible and non-fungible tokens and enables batch transfers, making it easy to transfer multiple tokens at once.

Multisig Wallet

The Multisig Wallet contract is a multi-owner wallet that requires a certain number of owners to agree on a transaction before it can be executed. Each owner can submit a transaction, and when enough owners confirm, it can be executed.

AbaxGovernor

The Abax Governor contract is a governance contract that allows for staking of PSP22 tokens in exchange for non-transferrable PSP22Vault shares (votes). It enables users to propose and vote on proposals, with the number of shares held by a user determining their voting power.

Explanation of terms

• Coverage: Represents the percentage of the code that have been executed during the fuzzing campaign. Higher coverage indicates more thorough testing (the higher the better).

• Average speed (for 1 core): The number of executions per second that the fuzzer can handle on a single CPU core. As a reminder, one execution contains multiple calls up to max_messages_per_exec.

• AFL++ corpus size: The size of the corpus generated by AFL++ during fuzzing. A larger corpus implies a diverse set of inputs to test the contract.

• generate-seed usage: Indicates whether generate-seed was used to seed the initial tests. This depends if the contract include tests or not.

Environment details

• CPU: AMD EPYC 7282 16-Cores
• Operating System: Linux 5.4.0-189-generic #209-Ubuntu x86_64
• Phink Version: 0.1.4

Contributing to the benchmarks

We encourage contributions to our benchmarks! If you have a contract you would like to see benchmarked, please submit a pull request to our repository.