Skip to content

An explanation of test IDs generated by fill

Test Case Organization

Test cases are implemented in Python "modules", which are organized by folder, named after the fork where the feature-under-test was introduced, for example:

  • tests/cancun/ contains tests for features added in the Cancun hardfork.
  • tests/prague/ contains tests for features added in the Prague hardfork.

Test ID generation

A test ID is a string, generated by the test framework, that uniquely identifies a test case. It's format is:

<python_test_module_path>::<python_test_function_name>[<parameter_1_id>-...-<parameter_N_id]

Let's use the following simple test case example to understand this format:

tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_Cancun-blockchain_test]

The components map as following:

  • <python_test_module_path> - the Python "module" (file) where the test is implemented tests/istanbul/eip1344_chainid/test_chainid.py,
  • <python_test_function_name> - the Python test function: test_chainid,
  • [<parameter_1_id>-...-<parameter_N_id] - the test parameters, this test case has the following parameters:
    • fork_Cancun, to read "fork = Cancun", the fork the test case is filled for,
    • blockchain_test, the test type.

Although test_chainid is not directly parametrized on the test function level, the framework always parametrizes tests by fork and, if possible, by test type. If we inspect its function signature, we see that it's native test format is of type StateTestFiller:

@pytest.mark.valid_from("Istanbul")
def test_chainid(state_test: StateTestFiller, pre: Alloc):
    """Test CHAINID opcode."""
    ...

Parametrization by Fork

Every test case is parametrized by the fork where the feature-under-test remains valid, for example:

  • Test cases in tests/cancun/ get filled for Cancun and Prague.
  • Test cases in tests/prague/, only get filled for Prague, at the time of writing, as Prague is the current fork in development.

All test IDs contain the fork_<name> parameter used to generate the test case in the test ID, e.g., test_Cancun.

Transition Forks

The test framework can also generate blockchain tests containing blocks that span a fork boundary, these test cases receive a special fork parameter name which contain both fork names and the timestamp at which the transition was made, e.g., fork_CancunToPragueAtTime15k.

Parametrization by Test Type

Each Python test case is also typically parametrized by test type, respectively fixture format. For example, if the test is implemented as a state_test, the test framework will additionally generate the following blockchain test fixtures (consisting of a single block with a single transaction):

  • a blockchain_test which can be tested via the Hive eest/consume-rlp simulator (or directly via a dedicated client interface).
  • a blockchain_engine_test (for post-merge forks) which can be tested via the Hive eest/consume-engine simulator.

Example: The Test IDs generated for test_chainid

To see all the test cases and their IDs that get generated from London until Shanghai for test_chainid, we can execute fill with the --collect-only -q flags:

fill tests/istanbul/eip1344_chainid/test_chainid.py --from London --until Shanghai --collect-only -q

which lists the following test IDs:

tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_London-blockchain_test]
tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_London-state_test]
tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_Paris-blockchain_test]
tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_Paris-blockchain_test_engine]
tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_Paris-state_test]
tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_Shanghai-blockchain_test]
tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_Shanghai-blockchain_test_engine]
tests/istanbul/eip1344_chainid/test_chainid.py::test_chainid[fork_Shanghai-state_test]

Additional Parametrization

The test function tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_type_tx_pre_fork is additionally parametrized:

tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_type_tx_pre_fork[fork_ShanghaiToCancunAtTime15k-blockchain_test-no_blob_tx]
tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_type_tx_pre_fork[fork_ShanghaiToCancunAtTime15k-blockchain_test-one_blob_tx]

The strings no_blob_tx and one_blob_tx are additional descriptive parameter IDs that indicate the focus of the test.