Skip to content

Changelog

Test fixtures for use by clients are available for each release on the Github releases page.

Key: โœจ = New, ๐Ÿž = Fixed, ๐Ÿ”€ = Changed, ๐Ÿ’ฅ = Breaking change.

๐Ÿ”œ Unreleased - 2024-XX-XX

๐Ÿงช Test Cases

  • โœจ EIP-4844 test tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py includes an EOF test case (#610).
  • โœจ Example test tests/frontier/opcodes/test_dup.py now includes EOF parametrization (#610).
  • โœจ Convert all opcodes validation test tests/frontier/opcodes/test_all_opcodes.py (#748).
  • โœจ Update EIP-7702 tests for Devnet-3 (#733)
  • ๐Ÿž Fix EIP-7702+EOF tests due to incorrect test expectations and faulty Conditional test generator in EOF mode (#821)

๐Ÿ› ๏ธ Framework

  • ๐Ÿž Fixed consume hive commands from spawning different hive test suites during the same test execution when using xdist (#712).
  • โœจ consume hive command is now available to run all types of hive tests (#712).
  • โœจ Generated fixtures now contain the test index index.json by default (#716).
  • โœจ A metadata folder .meta/ now stores all fixture metadata files by default (#721).
  • ๐Ÿž Fixed fill command index generation issue due to concurrency (#725).
  • โœจ Added with_all_evm_code_types, with_all_call_opcodes and with_all_create_opcodes markers, which allow automatic parametrization of tests to EOF (#610, #739).
  • โœจ Added with_all_system_contracts marker, which helps parametrize tests with all contracts that affect the chain on a system level (#739).
  • โœจ Code generators Conditional and Switch now support EOF by adding parameter evm_code_type (#610).
  • โœจ fill command now supports parameter --evm-code-type that can be (currently) set to legacy or eof_v1 to force all test smart contracts to deployed in normal or in EOF containers (#610).
  • ๐Ÿž Fixed fixture index generation on EOF tests (#728).
  • ๐Ÿž Fixes consume genesis mismatch exception for hive based simulators (#734).
  • โœจ Adds reproducible consume commands to hiveview (#717).
  • ๐Ÿ’ฅ Added multiple exceptions to the EOF fixture format (#759).
  • โœจ Added optional parameter to all with_all_* markers to specify a lambda function that filters the parametrized values (#739).
  • โœจ Added extend_with_defaults utility function, which helps extend test case parameter sets with default values. @pytest.mark.parametrize (#739).
  • โœจ Added Container.Init to ethereum_test_types.EOF.V1 package, which allows generation of an EOF init container more easily (#739).
  • โœจ Introduce method valid_opcodes() to the fork class (#748).
  • ๐Ÿž Fixed consume exit code return values, ensuring that pytest's return value is correctly propagated to the shell. This allows the shell to accurately reflect the test results (e.g., failures) based on the pytest exit code (#765).
  • โœจ Added a new flag --solc-version to the fill command, which allows the user to specify the version of the Solidity compiler to use when compiling Yul source code; this version will now be automatically downloaded by fill via solc-select (#772).
  • ๐Ÿž Fix usage of multiple @pytest.mark.with_all* markers which shared parameters, such as with_all_call_opcodes and with_all_create_opcodes which shared evm_code_type, and now only parametrize compatible values (#762).
  • โœจ Added selector and marks fields to all @pytest.mark.with_all* markers, which allows passing lambda functions to select or mark specific parametrized values (see documentation for more information) (#762).
  • โœจ Improves consume input flags for develop and stable fixture releases, fixes --help flag for consume (#745).
  • ๐Ÿž Fix erroneous fork message in pytest session header with development forks (#806).
  • ๐Ÿž Fix Conditional code generator in EOF mode (#821)
  • ๐Ÿ”€ ethereum_test_rpc library has been created with what was previously ethereum_test_tools.rpc (#822)
  • โœจ Add Wei type to ethereum_test_base_types which allows parsing wei amounts from strings like "1 ether", "1000 wei", "10**2 gwei", etc (#825)
  • ๐Ÿ”€ Replace ethereum.base_types with ethereum-types (#850)

๐Ÿ”ง EVM Tools

๐Ÿ“‹ Misc

  • โœจ Feature releases can now include multiple types of fixture tarball files from different releases that start with the same prefix (#736).
  • โœจ Releases for feature eip7692 now include both Cancun and Prague based tests in the same release, in files fixtures_eip7692.tar.gz and fixtures_eip7692-prague.tar.gz respectively (#743). โœจ Re-write the test case reference doc flow as a pytest plugin and add pages for test functions with a table providing an overview of their parametrized test cases (#801).
  • ๐Ÿ”€ Simplify Python project configuration and consolidate it into pyproject.toml (#764).
  • ๐Ÿ”€ Created pytest_plugins.concurrency plugin to sync multiple xdist processes without using a command flag to specify the temporary working folder (#824)
  • ๐Ÿ”€ Move pytest plugin pytest_plugins.filler.solc to pytest_plugins.solc.solc (#823).

๐Ÿ’ฅ Breaking Change

  • The EOF fixture format contained in eof_tests may now contain multiple exceptions in the "exception" field in the form of a pipe (|) separated string (#759).
  • Remove redundant tests within stable and develop fixture releases, moving them to a separate legacy release (#788).

v3.0.0 - 2024-07-22

๐Ÿงช Test Cases

๐Ÿ› ๏ธ Framework

  • ๐Ÿž Fix incorrect != operator for FixedSizeBytes (#477).
  • โœจ Add Macro enum that represents byte sequence of Op instructions (#457)
  • โœจ Number of parameters used to call opcodes (to generate bytecode) is now checked (#492).
  • โœจ Libraries have been refactored to use pydantic for type checking in most test types (#486, #501, #508).
  • โœจ Opcodes are now subscriptable and it's used to define the data portion of the opcode: Op.PUSH1(1) == Op.PUSH1[1] == b"\x60\x01" (#513)
  • โœจ Added EOF fixture format (#512).
  • โœจ Verify filled EOF fixtures using evmone-eofparse during fill execution (#519).
  • โœจ Added --traces support when running with Hyperledger Besu (#511).
  • โœจ Use pytest's "short" traceback style (--tb=short) for failure summaries in the test report for more compact terminal output (#542).
  • โœจ The fill command now generates HTML test reports with links to the JSON fixtures and debug information (#537).
  • โœจ Add an Ethereum RPC client class for use with consume commands (#556).
  • โœจ Add a "slow" pytest marker, in order to be able to limit the filled tests until release (#562).
  • โœจ Add a CLI tool that generates blockchain tests as Python from a transaction hash (#470, #576).
  • โœจ Add more Transaction and Block exceptions from existing ethereum/tests repo (#572).
  • โœจ Add "description" and "url" fields containing test case documentation and a source code permalink to fixtures during fill and use them in consume-generated Hive test reports (#579).
  • โœจ Add git workflow evmone coverage script for any new lines mentioned in converted_ethereum_tests.txt (#503).
  • โœจ Add a new covariant marker with_all_contract_creating_tx_types that allows automatic parametrization of a test with all contract-creating transaction types at the current executing fork (#602).
  • โœจ Tests are now encouraged to declare a pre: Alloc parameter to get the pre-allocation object for the test, and use pre.deploy_contract and pre.fund_eoa to deploy contracts and fund accounts respectively, instead of declaring the pre as a dictionary or modifying its contents directly (see the state test tutorial for an updated example) (#584).
  • โœจ Enable loading of ethereum/tests/BlockchainTests (#596).
  • ๐Ÿ”€ Refactor gentest to use ethereum_test_tools.rpc.rpc by adding to get_transaction_by_hash, debug_trace_call to EthRPC (#568).
  • โœจ Write a properties file to the output directory and enable direct generation of a fixture tarball from fill via --output=fixtures.tgz(#627).
  • ๐Ÿ”€ ethereum_test_tools library has been split into multiple libraries (#645).
  • โœจ Add the consume engine simulator and refactor the consume simulator suite. (#691).
  • ๐Ÿž Prevents forcing consume to use stdin as an input when running from hive. (#701).

๐Ÿ“‹ Misc

  • ๐Ÿž Fix CI by using Golang 1.21 in Github Actions to build geth (#484).
  • ๐Ÿ’ฅ "Merge" has been renamed to "Paris" in the "network" field of the Blockchain tests, and in the "post" field of the State tests (#480).
  • โœจ Port entry point scripts to use click and add tests (#483).
  • ๐Ÿ’ฅ As part of the pydantic conversion, the fixtures have the following (possibly breaking) changes (#486):
  • State test field transaction now uses the proper zero-padded hex number format for fields maxPriorityFeePerGas, maxFeePerGas, and maxFeePerBlobGas
  • Fixtures' hashes (in the _info field) are now calculated by removing the "_info" field entirely instead of it being set to an empty dict.
  • ๐Ÿž Relax minor and patch dependency requirements to avoid conflicting package dependencies (#510).
  • ๐Ÿ”€ Update all CI actions to use their respective Node.js 20 versions, ahead of their Node.js 16 version deprecations (#527).
  • โœจ Releases now contain a fixtures_eip7692.tar.gz which contains all EOF fixtures (#573).
  • โœจ Use solc-select for tox when running locally and within CI (#604).

๐Ÿ’ฅ Breaking Change

  • Cancun is now the latest deployed fork, and the development fork is now Prague (#489).
  • Stable fixtures artifact fixtures.tar.gz has been renamed to fixtures_stable.tar.gz (#573)
  • The "Blockchain Test Hive" fixture format has been renamed to "Blockchain Test Engine" and updated to more closely resemble the engine_newPayload format in the execution-apis specification (https://github.com/ethereum/execution-apis/blob/main/src/engine/prague.md#request) and now contains a single "params" field instead of multiple fields for each parameter (#687).
  • Output folder for fixtures has been renamed from "blockchain_tests_hive" to "blockchain_tests_engine" (#687).

v2.1.1 - 2024-03-09

๐Ÿงช Test Cases

  • ๐Ÿž Dynamic create2 collision from different transactions same block (#430).
  • ๐Ÿž Fix beacon root contract deployment tests so the account in the pre-alloc is not empty (#425).
  • ๐Ÿ”€ All beacon root contract tests are now contained in tests/cancun/eip4788_beacon_root/test_beacon_root_contract.py, and all state tests have been converted back to blockchain tests format (#449)

๐Ÿ› ๏ธ Framework

  • โœจ Adds two consume commands #339:

  • consume direct - Execute a test fixture directly against a client using a blocktest-like command (currently only geth supported).

  • consume rlp - Execute a test fixture in a hive simulator against a client that imports the test's genesis config and blocks as RLP upon startup. This is a re-write of the ethereum/consensus Golang simulator.

  • โœจ Add Prague to forks (#419).

  • โœจ Improve handling of the argument passed to solc --evm-version when compiling Yul code (#418).
  • ๐Ÿž Fix fill -m yul_test which failed to filter tests that are (dynamically) marked as a yul test (#418).
  • ๐Ÿ”€ Helper methods to_address, to_hash and to_hash_bytes have been deprecated in favor of Address and Hash, which are automatically detected as opcode parameters and pushed to the stack in the resulting bytecode (#422).
  • โœจ Opcodes enum now contains docstrings with each opcode description, including parameters and return values, which show up in many development environments (#424) @ThreeHrSleep.
  • ๐Ÿ”€ Locally calculate state root for the genesis blocks in the blockchain tests instead of calling t8n (#450).
  • ๐Ÿž Fix bug that causes an exception during test collection because the fork parameter contains None (#452).
  • โœจ The _info field in the test fixtures now contains a hash field, which is the hash of the test fixture, and a hasher script has been added which prints and performs calculations on top of the hashes of all fixtures (see hasher -h) (#454).
  • โœจ Adds an optional verify_sync field to hive blockchain tests (EngineAPI). When set to true a second client attempts to sync to the first client that executed the tests (#431).
  • ๐Ÿž Fix manually setting the gas limit in the genesis test env for post genesis blocks in blockchain tests (#472).

๐Ÿ“‹ Misc

  • ๐Ÿž Fix deprecation warnings due to outdated config in recommended VS Code project settings (#420).
  • ๐Ÿž Fix typo in the selfdestruct revert tests module (#421).

v2.1.0 - 2024-01-29: ๐Ÿ๐Ÿ–๏ธ Cancun

Release v2.1.0 primarily fixes a small bug introduced within the previous release where transition forks are used within the new StateTest format. This was highlighted by @chfast within #405 (#405), where the fork name ShanghaiToCancunAtTime15k was found within state tests.

๐Ÿงช Test Cases

  • โœจ EIP-4844: Adds test_blob_gas_subtraction_tx() verifying the blob gas fee is subtracted from the sender before executing the blob tx (#407).

๐Ÿ› ๏ธ Framework

  • ๐Ÿž State tests generated with transition forks no longer use the transition fork name in the fixture output, instead they use the actual enabled fork according to the state test's block number and timestamp (#406).

๐Ÿ“‹ Misc

  • โœจ Use run-parallel and shared wheel packages for tox (#408).

v2.0.0 - 2024-01-25: ๐Ÿ๐Ÿ–๏ธ Cancun

Release v2.0.0 contains many important framework changes, including introduction of the StateTest format, and some additional Cancun and other test coverage.

Due to changes in the framework, there is a breaking change in the directory structure in the release tarball, please see the dedicated "๐Ÿ’ฅ Breaking Changes" section below for more information.

๐Ÿงช Test Cases

  • โœจ EIP-4844: Add test_sufficient_balance_blob_tx() and test_sufficient_balance_blob_tx_pre_fund_tx() (#379).
  • โœจ EIP-6780: Add a reentrancy suicide revert test (#372).
  • โœจ EIP-1153: Add test_run_until_out_of_gas() for transient storage opcodes (#401).
  • โœจ EIP-198: Add tests for the MODEXP precompile (#364).
  • โœจ Tests for nested CALL and CALLCODE gas consumption with a positive value transfer (previously lacking coverage) (#371).
  • ๐Ÿž EIP-4844: Fixed test_invalid_tx_max_fee_per_blob_gas() to account for extra gas required in the case where the account is incorrectly deduced the balance as if it had the correct block blob gas fee (#370).
  • ๐Ÿž EIP-4844: Fixed test_insufficient_balance_blob_tx() to correctly calculate the minimum balance required for the accounts (#379).
  • ๐Ÿž EIP-4844: Fix and enable test_invalid_blob_tx_contract_creation (#379).
  • ๐Ÿ”€ Convert all eligible BlockchainTests to StateTests (and additionally generate corresponding BlockchainTests) (#368, #370).

๐Ÿ› ๏ธ Framework

  • โœจ Add StateTest fixture format generation; StateTests now generate a StateTest and a corresponding BlockchainTest test fixture, previously only BlockchainTest fixtures were generated (#368).
  • โœจ Add StateTestOnly fixture format is now available and its only difference with StateTest is that it does not produce a BlockchainTest (#368).
  • โœจ Add evm_bytes_to_python command-line utility which converts EVM bytecode to Python Opcodes (#357).
  • โœจ Fork objects used to write tests can now be compared using the >, >=, <, <= operators, to check for a fork being newer than, newer than or equal, older than, older than or equal, respectively when compared against other fork (#367).
  • โœจ Add solc 0.8.23 support (#373).
  • โœจ Add framework unit tests for post state exception verification (#350).
  • โœจ Add a helper class ethereum_test_tools.TestParameterGroup to define test parameters as dataclasses and auto-generate test IDs (#364).
  • โœจ Add a --single-fixture-per-file flag to generate one fixture JSON file per test case (#331).
  • ๐Ÿž Storage type iterator is now fixed (#369).
  • ๐Ÿž Fix type coercion in FixtureHeader.join() (#398).
  • ๐Ÿ”€ Locally calculate the transactions list's root instead of using the one returned by t8n when producing BlockchainTests (#353).
  • ๐Ÿ”€ Change custom exception classes to dataclasses to improve testability (#386).
  • ๐Ÿ”€ Update fork name from "Merge" to "Paris" used within the framework and tests (#363).
  • ๐Ÿ’ฅ Replace = with _ in pytest node ids and test fixture names (#342).
  • ๐Ÿ’ฅ The StateTest, spec format used to write tests, is now limited to a single transaction per test (#361).
  • ๐Ÿ’ฅ Tests must now use BlockException and TransactionException to define the expected exception of a given test, which can be used to test whether the client is hitting the proper exception when processing the block or transaction (#384).
  • ๐Ÿ’ฅ fill: Remove the --enable-hive flag; now all test types are generated by default (#358).
  • ๐Ÿ’ฅ Rename test fixtures names to match the corresponding pytest node ID as generated using fill (#342).

๐Ÿ“‹ Misc

  • โœจ Docs: Add a "Consuming Tests" section to the docs, where each test fixture format is described, along with the steps to consume them, and the description of the structures used in each format (#375).
  • ๐Ÿ”€ Docs: Update t8n tool branch to fill tests for development features in the readme (#338).
  • ๐Ÿ”€ Filling tool: Updated the default filling tool (t8n) to go-ethereum@master (#368).
  • ๐Ÿž Docs: Fix error banner in online docs due to mermaid syntax error (#398).
  • ๐Ÿž Docs: Fix incorrectly formatted nested lists in online doc (#403).

๐Ÿ’ฅ Breaking Changes

A concrete example of the test name renaming and change in directory structure is provided below.

  1. Fixture output, including release tarballs, now contain subdirectories for different test types:

  2. blockchain_tests: Contains BlockchainTest formatted tests

  3. blockchain_tests_hive: Contains BlockchainTest with Engine API call directives for use in hive

  4. state_tests: Contains StateTest formatted tests

  5. StateTest, spec format used to write tests, is now limited to a single transaction per test.

  6. In this release the pytest node ID is now used for fixture names (previously only the test parameters were used), this should not be breaking. However, = in both node IDs (and therefore fixture names) have been replaced with _, which may break tooling that depends on the = character.
  7. Produced blockchain_tests fixtures and their corresponding blockchain_tests_hive fixtures now contain the named exceptions BlockException and TransactionException as strings in the expectException and validationError fields, respectively. These exceptions can be used to test whether the client is hitting the proper exception when processing an invalid block.

Blockchain test:

"blocks": [
      {
         ...
         "expectException": "TransactionException.INSUFFICIENT_ACCOUNT_FUNDS",
         ...
      }
      ...
]

Blockchain hive test:

"engineNewPayloads": [
      {
         ...
         "validationError": "TransactionException.INSUFFICIENT_ACCOUNT_FUNDS",
         ...
      }
      ...
]

Renaming and Release Tarball Directory Structure Change Example

The fixture renaming provides a more consistent naming scheme between the pytest node ID and fixture name and allows the fixture name to be provided directly to pytest 5on the command line to execute individual tests in isolation, e.g. pytest tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier].

  1. Pytest node ID example:

  2. Previous node ID: tests/frontier/opcodes/test_dup.py::test_dup[fork=Frontier].

  3. New node ID: tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier].

  4. Fixture name example:

  5. Previous fixture name: 000-fork=Frontier

  6. New fixture name: tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier] (now the same as the pytest node ID).

  7. Fixture JSON file name example (within the release tarball):

  8. Previous fixture file name: fixtures/frontier/opcodes/dup/dup.json (BlockChainTest format).

  9. New fixture file names (all present within the release tarball):

    • fixtures/state_tests/frontier/opcodes/dup/dup.json (StateTest format).
    • fixtures/blockchain_tests/frontier/opcodes/dup/dup.json (BlockChainTest format).
    • fixtures/blockchain_tests_hive/frontier/opcodes/dup/dup.json (a blockchain test in HiveFixture format).

v1.0.6 - 2023-10-19: ๐Ÿ๐Ÿ–๏ธ Cancun Devnet 10

๐Ÿงช Test Cases

  • ๐Ÿ”€ EIP-4844: Update KZG point evaluation test vectors to use data from the official KZG setup and Mainnet Trusted Setup (#336).

๐Ÿ› ๏ธ Framework

  • ๐Ÿ”€ Fixtures: Add a non-RLP format field (rlp_decoded) to invalid blocks (#322).
  • ๐Ÿ”€ Spec: Refactor state and blockchain spec (#307).

๐Ÿ”ง EVM Tools

  • โœจ Run geth's evm blocktest command to verify JSON fixtures after test case execution (--verify-fixtures) (#325).
  • โœจ Enable tracing support for ethereum-spec-evm (#289).

๐Ÿ“‹ Misc

๐Ÿ’ฅ Breaking Change

v1.0.5 - 2023-09-26: ๐Ÿ๐Ÿ–๏ธ Cancun Devnet 9 Release 3

This release mainly serves to update the EIP-4788 beacon roots address to 0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02, as updated in ethereum/EIPs/pull/7672.

๐Ÿงช Test Cases

  • ๐Ÿž EIP-4844: Fix invalid blob txs pre-Cancun engine response (#306).
  • โœจ EIP-4788: Final update to the beacon root address (#312).

๐Ÿ“‹ Misc

  • โœจ Docs: Changelog added (#305).
  • โœจ CI/CD: Run development fork tests in Github Actions (#302).
  • โœจ CI/CD: Generate test JSON fixtures on push (#303).

๐Ÿ’ฅ Breaking Change

Please use development fixtures from now on when testing Cancun. These refer to changes that are currently under development within clients:

  • fixtures: All tests until the last stable fork (Shanghai)
  • fixtures_develop: All tests until the last development fork (Cancun)
  • fixtures_hive: All tests until the last stable fork (Shanghai) in hive format (Engine API directives instead of the usual BlockchainTest format)
  • fixtures_develop_hive: All tests until the last development fork (Cancun) in hive format

v1.0.4 - 2023-09-21: ๐Ÿ Cancun Devnet 9 Release 2

This release adds additional coverage to the current set of Cancun tests, up to the Devnet-9 Cancun specification.

Note: Additional EIP-4788 updates from ethereum/EIPs/pull/7672 will be included in the next release.

๐Ÿงช Test Cases

๐Ÿ› ๏ธ Framework

  • ๐Ÿ”€ EIP-4788: Beacon root contract is pre-deployed at 0xbEAC020008aFF7331c0A389CB2AAb67597567d7a (@spencer-tb in #297).
  • โœจ Deprecate empty accounts within framework (@spencer-tb in #300).
  • โœจ Fixture generation split based on hive specificity (@spencer-tb in #301).
  • ๐Ÿ’ฅ fill: --disable-hive flag removed; replaced by --enable-hive (@spencer-tb in #301).
  • โœจ Add engine API forkchoice updated information in fixtures (@spencer-tb in #256).

v1.0.3 - 2023-09-14: ๐Ÿ Cancun Devnet 9 Release

See v1.0.3.

v1.0.2 - 2023-08-11: ๐Ÿ Cancun Devnet 8 + 4788 v2 Pre-Release

See v1.0.2.

v1.0.1 - 2023-08-03: ๐Ÿ Cancun Devnet-8 Pre-Release

See v1.0.1.

v1.0.0 - 2023-06-27: ๐Ÿงช Welcome to the Pytest Era

See v1.0.0.

Older releases can be found on the releases page.