HIP-1056: Block Streams

Abstract

This HIP introduces a new output data format for consensus nodes, called Block Streams, that replaces the existing event streams, record streams, state files, signature files and side cars with one single stream. Each block within the block stream is a self-contained entity, including every event and all transactions that were part of the block, the state changes as a result of those transactions, and a network signature (using a BLS threshold signature scheme we call TSS) that proves the block was signed by a subset of nodes whose stake accounts for more than 1/3 of the network’s consensus stake weight.

By including state changes, downstream services can seamlessly maintain and verify state alongside consensus nodes. This enhancement fosters greater transparency and trust within the Hiero network. Any downstream service can independently rebuild and verify the state of consensus nodes at the time of the block, verified by the TSS network signature. Using this state, they can provide additional services such as state proofs, state snapshots, and more.

With the event information now within blocks, downstream services can reconstruct events and the network activity through the hashgraph algorithm can be replayed, permitting anyone to verify the correct execution of the consensus algorithm. In doing so Hiero users gain comprehensive visibility into network activities through an easily consumable format that can be delivered with low latency.

A key design criteria is for the block stream to be easily consumed by any programming language, and with minimal complexity or dependencies. For example, state data can be utilized for basic queries without having to reconstruct a merkle tree.

Block streams are an upgrade to the existing RecordStream V6. Block streams restructure and aggregate multiple record types in record streams, including EventStream, RecordStream, and Hiero state data to produce a single unified stream of items.

The key enhancements offered by block streams include:

• Unified Data Stream: Block stream consolidates event streams, record streams, sidecars, and signature files into a single cohesive data stream.
• State Change Data: Each block will include state change data for the given round of consensus.
• Verifiable Proof: Each block will be independently verifiable, containing a full proof of transactions, network consensus and state changes.
• Comprehensive Protobuf specification By defining blocks in protobuf, the inputs, outputs, and state changes of consensus nodes are clearly specified, paving the way for future implementations and greater client diversity.

With the adoption of block streams, data output by Hiero consensus nodes will be consolidated into a single, verifiable chronicle of the network, thereby strengthening the integrity and transparency of the Hiero ecosystem.

NOTE: This HIP also introduces the concepts of a Block Node and TSS Signatures.

Block Nodes are a new proposed node type designed to consume block streams, maintain state, store history, and provide additional block and state related APIs that benefit the Hiero community (e.g., state proofs). They are one type of “downstream service” that can consume block streams, Mirror Nodes will also consume block streams.

TSS signatures refer to a single, aggregated TSS-BLS signature that verifies a block has been signed by a threshold of Hiero consensus stake weight. Each block will include TSS signatures, offering verifiable assurance that the block was produced by a configured subset of nodes whose stake accounts for more than 1/3 of the network’s consensus stake weight.

Subsequent HIPs, to be introduced at a later date, will provide details on Block Nodes and TSS Signatures.

Context

Today, each consensus node independently gossips with other consensus nodes, agrees on the order of transactions, handles those transactions, stores saved states locally and emits signed records, events and signatures.

The publicly consumable output from the consensus node are 2 streams - the event stream (ordered collection of events post consensus) and the record stream (ordered collection of transaction body input pre-execution and transaction record results post-execution):

With block streams, we will produce only one stream. It will contain all the data from the previous two (events and records) without duplication and with state changes added.

Motivation

With the record stream design nodes had to independently upload record streams and their signatures files. These were stored in varied public cloud buckets and anyone in the world could download the files (e.g. Mirror Nodes do this today to support queries).

However, this approach had unintended consequences including it becoming expensive for products to access files from public cloud buckets.

Block streams resolve many issues with the previous Record Stream v6 specification from HIP 435 currently deployed on mainnet. Below are some key issues along with the potential resolutions that block streams provide:

• Mixed Serialization Formats: The record stream mixes protobuf with other serialization formats instead of using pure protobuf serialization. Block streams will use only protobuf definitions, making it much easier to consume and validate.
• Data Duplication: Transaction data was redundantly duplicated between event streams and record streams, resulting in wasted space and bandwidth. Block streams aim to reduce this duplication by being more explicit about the data being serialized, and streamlining the format.
• Event Stream Documentation: Currently, the Event stream is not in a publicly documented protobuf format. Block streams integrate Event Stream data in protobuf which increases the transparency of consensus logic. Event streams data can be used to validate how consensus was reached and that it was fair. They can also be used for event replay in crash recovery and node restart operations.
• Signature Files: Each node currently produces a v6 record stream with a corresponding signature file for verification. Mirror Nodes must download a strong minority (1/3 of consensus stake weight) worth of these signature files, along with the record file, to verify v6 records. This frequent access to cloud storage (GCS/S3) is costly. To reduce this expense and complexity, blocks include a TSS signature from a threshold (minimum of 1/3) stake weight of consensus node. With this a single Block can be used to verify a threshold of Hiero consensus stake weight agreed on the activity.
• File Naming and Retrieval: v6 record stream files are named after the consensus timestamp of the first transaction they contain, requiring Mirror Nodes to perform frequent and costly LIST operations on cloud storage (GCS/S3) to find new files. The new block streams use sequential block numbers, eliminating the need for these expensive LIST operations. Mirror Nodes will read the stream from a block node that manages its own storage at a lower cost and won’t need to deal with cloud storage buckets.
• Sidecar Data Integration: In v6 record streams, sidecar files are created to serialize extra data such as smart contract logs outside the main record stream. Block streams will incorporate sidecar data into the unified stream.
• Subset Support: Any block creating or parsing client will be able to decide individually which data in the block stream to stream or retain on disk respecitvely, without breaking the cryptographic integrity of the block stream. Different operators in different legal jurisdictions can make different decisions about what data to retain. Or, operators may subset the stream to retain minimal data and minimize storage costs.
• Errata Handling: Errata refers to software bugs where a node executed the transaction correctly but did not record the correct output in the stream. The block stream format will allow for signed errata transactions from the council to correct stream history, providing transparency, while maintaining the original proof and integrity of the chain.

Another key motivation for introducing block streams is their ability to enable downstream services to maintain and verify the merkle tree state of consensus nodes as each block reaches consensus. This capability allows critical but resource-intensive services to be offloaded from consensus nodes to other servers that can now maintain the merkle state alongside the consensus nodes. By offloading these services, consensus nodes can dedicate more resources to processing transactions, thereby increasing network scalability.

Specific examples of services that can be moved to a Block Node include:

• State Snapshots: Consensus nodes are currently required to periodically upload snapshots of the saved state for backup purposes. Since block streams contain all state changes, block nodes can now provide this service.
• Reconnect Services: Reconnect enables nodes that have fallen behind other nodes on the network to request data from nodes that are in sync. This process requires significant complexity and resources. We can enhance stability and simplify node software by allowing nodes to reconnect with a block node, instead of another consensus node. This is a significant and important enhancement to reconnect that is needed for permissionless nodes.
• Query Endpoints: As Block Nodes or other consumers of the Block Stream can maintain a live state in sync with the consensus nodes, they can answer any queries against state. They can also produce cryptographic state proofs to prove any data in state (a state proof) or any data was part of a block (a block proof). For example, they can provide a state proof that says a particular account has a specific balance at a block X, a block contents proof that a given HCS message was part of block X and a block proof that confirms the network came to agreement on the block X containing those transactions.

Rationale

As mentioned in the motivation section above, the design rationale of Block Streams is driven by the following goals:

Consolidation of Streams: Block streams should efficiently consolidate all information across event, state, and record streams, optimizing data size to reduce network storage costs over time.

Verifiability: Each block must be self-contained and independently verifiable. Block streams will provide the data to enable verification of hashgraph consensus as well as verification of transaction execution.

State Update Data: Block streams will include all state changes to enable downstream services to maintain merkle state alongside mainnet nodes. Prior to block streams the contents of consensus state was not directly visible from outside the consensus network.

Easy consumption: The Block stream is designed to be easy to consume by users. Where trade-offs exist between a small amount of extra work on consensus nodes or a large amount of work or complex understanding needed on consumers, we have chosen to perform extra work on consensus nodes to produce clean block streams.

Designed for Streaming or as Block files: The stream is designed to both be packaged up as self-contained block files and to be streamed as individual items with low latency.

Design for Consolidation

The block stream is designed to be a continuous stream of items. This allows for continuous streaming over gRPC. The order of items is significant, for example all items between and inclusive of a BlockHeader and BlockProof represent contents of one block. All items following an EventTransaction before the next EventTransaction are related to that first transaction. Block items for one block can also be packaged in a Block message so they can be stored in a protobuf file and served as a single entity if desired.

The Block Item design promotes granular stream processing ensuring all items in a block prior to the proof can be sent out immediately after processing. The result is reduced end-to-end latency for use cases prioritizing speed (e.g. high-frequency trading). Additionally, this means a following complete block will not be sent until a block proof is pushed out over the stream.

Design for substreams

The block stream items are split into several subtypes: events, inputs, outputs, state changes, and trace data, initially. These are hashed separately into different block stream merkle sub-trees. This is done so we can filter blocks based on data classification and maintain efficient block content proofs. A filtered block stream could then be used for only transaction replay or transaction execution verification, for example.

Consensus nodes have a state merkle tree, which is a merkle tree containing the current state of the consensus node. At each block boundary, the state merkle tree is a subtree of a conceptual block merkle tree. The block merkle tree is a “virtual” binary merkle tree that items are organized into for hashing purposes only, and whose eight subtrees at depth three also include the merkle tree of the previous block, as well as the state merkle tree. Of course Hash computation for the block N merkle tree reuses the hash computed for the block N-1 merkle tree.

Merkle Item Order vs Block Item Order: Items are split into multiple subtrees, and strict in-order traversal of the combined tree will not match the stream order. That is, the order in which items appear in the block merkle tree are not the same order in which they are streamed. The left-to-right order of merkle leaves however, must be the order in which those items are encountered in the stream. Some items are in one subtree and some are in another subtree, but within those trees the order will be the same as stream-order.

State Merkle Hash: A block contains the state hash that represents the state at the beginning of the block. The hash of the state at the end of the block is found in the next block. This is because it takes time to gather state changes of the block and hash them, and waiting for this process would result in the block delivery time being held up. Rapid block times is a goal of the block stream and thus a tradeoff is made to ensure minimum latency. The impact of this choice is that a state proof for verification of block N requires block N+1. However, the state proof can still be calculated by all the data contained in block N. A block is self contained!

Design for Verifiability

Existing record files contain the following cryptographic properties.

Record file contents are summarized in a chain of hashes, with each file’s contents containing the hash of the previous file so that the hash of the current file depends on the contents of the entire record stream. Additionally, each consensus node on Hiero signs a hash of the record files as it is produced, enabling mirror nodes and other downstream services to verify that the record file was produced and attested by a majority stake weight of the network nodes. A major drawback of the current design is that multiple signature files and the address book with network stake weight must be collected and verified before a record file can be verified. This design choice makes it difficult to support a “dynamic address book”, which is a hard requirement for permissionless nodes.

To make each block within a block stream independently verifiable, the use of an aggregated signature is leveraged. Aggregated signatures allow the verification of multiple node signatures to be processed using a single public key. Therefore, a user can verify that a block was signed by a sufficient stake weight within the network by verifying a single aggregated signature. TSS-BLS signatures were selected because of their ability to support partial share signatures assigned on a weighted number of shares per node with a threshold value required for verification. Additionally, adopting TSS-BLS allows the network to utilize a single semi-permanent public ledger id to verify the signature. This offers a valuable system optimization as no matter how much the network node membership changes, the same ledger id can be used to verify the aggregated signature. TSS-BLS signatures are also reasonably efficient when used within smart contracts. A state proof based on this signature can be verified by a smart contract on another ledger for little cost.

A subsequent HIP will be created to outline the changes and justification required for implementing Hiero TSS-BLS signatures.

Notably the block stream will enable 3 types of proof

1. Block Proof - This proof proves the blockchain and illustrates that given a starting block and its proof, a given set of transaction inputs and outputs as well as resulting state changes and a final merkle root hash was agreed upon by the majority stake weight of the network. In detail the Block Proof is a message in the Block Stream that contains a Ledger ID signature on the root hash of the Block Merkle Tree, as well as all of the information necessary to, with the Block Items for that Block, reconstruct the Block Merkle Tree, calculate the root hash, and verify the signature with the network Ledger ID.
2. Block Contents Proof - This proof proves that a given object was present in a Block. This is just as powerful as proving a value was in state. In detail, the Block Contents Proof is a Merkle Proof created for a single BlockItem within a Block. This proves that the network consensus agreed to the content of that specific item (input, output, or state change) in that specific block. This can prove, among other values, the content of an HCS message, the state of an Account changed in that block, or the content of a transaction submitted in that block. We can construct a Block Contents Proof from a Block Proof item and contents of the Block.
3. State Proof - This proof proves a given state value at a given block number was agreed upon by a majority stake weight of the network. In detail, the state proof is a Merkle Proof created for a value in network consensus state at a particular point in consensus time. This proves that the network consensus state contained a particular value at a specific consensus timestamp. This might prove, among other values, the balance of an Account, the state of a Smart Contract, or the properties of an HTS non-fungible token.

Each block will contain its own Block Proof, which provides proof of consensus agreement on the content of the block. The state hash contained in the proof for block n, however is the hash at the beginning of block n (i.e. the end of the previous block, n-1). The state hash representing the changes to state made in a given block n is present in the proof for the next block n+1. State proofs for block n, therefore, must be created using the Block Proof from the following block n+1 as well as a live, or snapshot, copy of state at the end of block n.

The presence of a transaction in block n can be proved with a block item proof using block n and the block proof for block n.

State proofs proving the value of any given value in the state merkle tree after the transactions in block n are applied require the state hash in the following block n+1

Proof scenarios for Block, BlockItem, and State

Design for State Changes

Including state changes within the block stream is a significant foundational step towards enhancing the decentralization of the network. This inclusion allows for the development of alternative node implementations and enables downstream state-related services such as state proofs, state snapshots, and reconnect services. It also provides a low latency feed of the contents of state as it changes to applications. This can open doors to exciting new latency critical application ideas and concepts that could not be built before.

To enable this, state in the consensus node will be restructured into simple named states. This allows for state changes to be expressed as simple CRUD operations on named states. This is so downstream consumers do not have to understand the full complexity of the merkle state tree.

Personas and User Stories

Personas

• Downstream Developer: A provider of value-added services utilizing block streams
• Verifier: Any person or service required to verify the integrity of a block or the information contained within a block.

User Stories

1. As a downstream developer, I want a streamlined feed that contains all event, transactional, and state details of the network so that I can offer value-added services to the ecosystem.
2. As a downstream developer, I want each block of data to be verifiable with a single, aggregated signature, so that I can authenticate transactional data without the cost and complexity of processing multiple signatures and the limitations of relying on an address book for verification.
3. As a verifier for a network, I want a single, self-contained stream of data with an aggregated signature signed by a threshold of stake weights, so that I can cost-effectively confirm that the stream represents the consensus output of the network.
4. As a downstream developer, I want a low latency feed that provides the state changes associated with transaction(s) within a block, so that I can interact with state changes with the smallest possible delay.

Specification

The Block Stream will be defined by the continuous transmission of Block information defined as follows.

• Each Block represents all inputs and outputs of the consensus network for zero or more whole Rounds. The maximum number of rounds per block will be configurable and start at 1 round per block. It could be increased later if necessary for performance reasons or if rounds become very frequent and small.
  • A block may be zero rounds because the last block prior to a network “freeze” must communicate the state hash (via a block proof) of the state immediately prior to network shutdown. This is only possible with an extra empty block because the state hash in each block proof is the state hash at the beginning of that block/round. This delay of state hash communication is required to simultaneously meet Hiero performance goals and also meet EVM expectations for immediate access to recent (the last 256) block hash values.
• A Round contains a sequence of zero or more Events that have reached consensus at the same time. Round length in real world time depends on internet latency and event creation rate. In Hiero today it is around 1 round per second. In the future this could be shorter to lower latency, but would never make sense to be less than ping time.
• An Event contains a sequence of zero or more EventTransactions submitted by a single node.
• Each EventTransaction is either a SignedTransaction or a system transaction.
• A SignedTransaction is transmitted as a byte array, contains the signed transaction bytes of a user-submitted transaction exactly as processed by the network, and is followed by transaction outputs and state changes resulting from that transaction.
• System transactions are represented by explicitly defined protobuf messages and used for consensus node to consensus node communication.

These concepts are core to the Hashgraph consensus mechanism and have direct impacts on the structure of data in the streams.

Important Note Regarding Map Fields

The network will never add a map to a protobuf message in the Block Stream, and will not use a map for any value in state. The protocol buffer implementations across different languages do not, and cannot, guarantee iteration order for map entries and some implementations deliberately randomize the iteration order to “aggressively” prevent any code from depending on map iteration order.

This unreliable iteration order has the consequence that block validation would fail non-deterministically when encountering a different iteration order because the serialized bytes would not be in the same sequence and would not generate the same hash. All implementations of block validation would be required to use custom protobuf libraries, which would be required to use the same deterministic iteration order when serializing bytes, so that the generated hash value would match every other implementation. The network values the freedom to implement a compliant Block Stream consumer using any valid protocol buffer library and any computing language of choice, and therefore chooses not to use the map type in protobuf messages.

Protobuf Definitions

The following protobuf specification defines a Block and it’s components. Notably, the protobufs in this HIP have partial comments for brevity and where they depend on existing message types defined in HAPI specification they are not repeated here. The full and detailed specification is present at the time of this HIP in Block Stream Protobuf and maintained in the public hedera-protobufs github repository.

Block

A collection of block items that represent a single block of transactions on the network.

/**
 * Block is used to serialize all the data for a block into record stream of
 * block files. This structure represents a block on Hiero.
 */
message Block {
    /**
     * The items contained in the block that make up the running hash.
     * This list is strictly and deterministically ordered
     */
    repeated BlockItem items = 1;
}

The order of block items in a block stream is important and in some cases it designates useful positions in the stream that a client may make inferences on. Particularly the start of a Block, Event and Transaction are useful considerations, in addition to the end of a Block or Transaction.

BlockItem

A block item is a logical separation of components that together make up a block. A Block item will fall into one of 3 categories - input, output or other. A Block item may be one of the 10 initial types noted below.

message BlockItem {
    // Reserved for future items that require separate handling for block hash purposes.
    reserved 12,13,14,15,16,17,18,19;

    oneof item {
        /**
         * An header for the block, marking the start of a new block.
         */
        com.hedera.hapi.block.stream.output.BlockHeader block_header = 1;

        /**
         * An header emitted at the start of a new network "event".
         */
        com.hedera.hapi.block.stream.input.EventHeader event_header = 2;

        /**
         * An header emitted at the start of a new consensus "round".
         * <p>
         * This item SHALL contain the properties relevant to a single
         * consensus round.
         */
        com.hedera.hapi.block.stream.input.RoundHeader round_header = 3;

        /**
         * A single transaction.
         */
        com.hedera.hapi.platform.event.EventTransaction event_transaction = 4;

        /**
         * The result of running a transaction.
         */
        com.hedera.hapi.block.stream.output.TransactionResult transaction_result = 5;

        /**
         * A transaction output.
         */
        com.hedera.hapi.block.stream.output.TransactionOutput transaction_output = 6;

        /**
         * A set of state changes.
         */
        com.hedera.hapi.block.stream.output.StateChanges state_changes = 7;

        /**
         * Verification data for items filtered from the stream.<br/>
         * This is a hash for a merkle tree node where the contents of that
         * part of the merkle tree have been removed from this stream.
         */
        FilteredItemHash filtered_item_hash = 8;

        /**
         * A signed block proof.<br/>
         * The signed merkle proof for this block. This will validate
         * a "virtual" merkle tree containing the previous block "virtual"
         * root, an "input" subtree, an "output" subtree, and
         * a "state changes" subtree.
         */
        BlockProof block_proof = 9;

        /**
         * A record file and associated data.
         */
        RecordFileItem record_file = 10;

        /**
         * Trace data.
         */
        TraceData trace_data = 11;
    }
}

All items in the block other than the final BlockProof, a RecordFileItem, or a FilteredItemHash are assigned to specific subtrees. This separation is important as they are hashed into different sub-merkle-trees in the final proof. We did not want to carry, in each block item, data representing which tree it was hashed in as that would waste a lot of bytes and also require excess parsing. Therefore, the subtree to use is defined by a convention for the field number assigned to the item in this message. The reason they are split into different sets is that this opens the door for using the block stream format in a wider set of uses and supporting privacy concerns for private Hiero networks that interoperate with public networks. The field number convention is detailed in the specification for Block Items, and repeated in Forward Compatibility below.

The division into subtrees also enables filtering of data based on data type or fine-grained content filters. For example a downstream services could filter for only transactional data if there is no requirement for event or state information. Another service could filter on only accounts they are interested in. Block items removed from a filtered stream are represented by one or more FilteredItemHash’s which provide the missing hashes required to validate the filtered stream with the same block proof as the full stream.

The parent-child relationship of some transactions is preserved in the Block Streams and will see the appropriate BlockItems for each level. For example, a CryptoTransfer transaction to a non existing EVM address results in the auto creation of a new Account with the said EVM address. In the Block Streams the CryptoTransfer transaction will be described by its EventTransaction, TransactionResult, TransactionOutput and StateChange BlockItems. The CryptoCreate transaction will also be described by its own EventTransaction, TransactionResult, TransactionOutput and StateChange BlockItem`s. In this way a transactions input, output and impacts on the network are clearly described.

Forward Compatibility

In order to maximize forward compatibility, and minimize the need to coordinate deployments of different systems creating and processing block streams in the future, the following rules SHALL be followed for field numbering in this message.

• The first 19 field numbers SHALL be assigned to the fields present in the first release. Unused fields in this range SHALL remain reserved until needed for additional options that do not fit into existing subtree categories.
• Fields numbered 20 and above MUST be numbered as follows.
  • Calculate the category number as N modulo 10, where N is the actual field number.
    • 0 - Consensus Headers
    • 1 - Inputs
    • 2 - Outputs
    • 3 - State Changes
    • 4 - Trace Data
    • 5 - Extension 0
    • 6 - Extension 1
    • 7 - Extension 2
    • 8 - Extension 3
    • 9 - Not hashed (not part of the block proof merkle tree)

Forward Compatibility Example

A future update adding three new items. A “BlockTrailer” item which is not part of the merkle tree, a new “ConsensusTransform” which is in Consensus Headers, and a new “BridgeTransform” which is in Trace Data.

• All three fields are at least 20, so they are additions.
• The “BlockTrailer” is field 29.
• The “ConsensusTransform” is field 20 (20 modulo 10 is 0, so it is a Consensus Header).
• The “BridgeTransform” field is 24 (24 modulo 10 is 4, so it is Trace Data).

BlockHeader

A BlockHeader is one type of output BlockItem and will always be the first element in a Block. It contains Block metadata which describes a block’s identifying information such as its number and HAPI specification version

message BlockHeader {
    /**
     * Version of the HAPI specification that was used to serialize the block.
     */
    proto.SemanticVersion hapi_proto_version = 1;

    /**
     * The software version used to execute transactions in this block.
     */
    proto.SemanticVersion software_version = 2;

    /**
     * The block number of this block.
     * <p>
     * This value MUST be exactly `1` more than the previous block.<br/>
     * Client systems SHOULD optimistically reject any block with a gap or
     * reverse in `number` sequence, and MAY assume the block stream has
     * encountered data loss, data corruption, or unauthorized modification.
     */
    uint64 number = 3;

    /**
     * The block hash of the previous block.
     * This is here so a client parsing the BlockHeader can optimistically reject
     * a block if this hash does not match the root hash of the previous block.
     */
    bytes previous_block_hash = 4;

    /**
     * The timestamp for this block.<br/>
     * The block timestamp is the consensus time stamp of the first round 
     * in the block.
     */
    proto.Timestamp block_timestamp = 5;

    /**
     * The hash algorithm used in this block.
     */
    proto.BlockHashAlgorithm hash_algorithm = 6;
}

/**
 * A specific hash algorithm used within a block.
 *
 * We did not reuse HashAlgorithm here because in all cases for now it will be SHA2_384 and if that's the default value
 * then we can save space by not serializing it, whereas HASH_ALGORITHM_UNKNOWN = 0; is the default for HashAlgorithm.
 */
enum BlockHashAlgorithm {
    SHA2_384 = 0;
}

A block’s timestamp is the timestamp of the first round within the block.

Timestamp on empty round: Currently, for rounds with events and transactions, the round timestamp is equal to the timestamp of the last transaction in the round. For empty rounds with no events, the round timestamp is equal to the timestamp of the last transaction in the previous round plus 1,000 nanoseconds, rounded up to the nearest 1,000 nanoseconds. If there is no previous round, then the round timestamp is the median of the judge created times. This logic derives the value of the BlockHeader item block_timestamp property.

EventHeader

The event header depicts the beginning of an event. All items after this and before the next EventHeader or BlockProof are part of this event or results of execution of this event’s contents. It is designed so that it is easy to reconstruct GossipEvents as needed for hashgraph reconstruction from the contents in the block stream. To make that easy EventHeader includes the the EventCore that is part of GossipEvent and all transactions are defined asEventTransaction which can be used unchanged to construct the original GossipEvent.

To save space in each block and in the block stream, the signature in a GossipEvent is replaced with a boolean modeling the middle bit of the signature in the EventHeader. Trust that the event is authentic will come from the block proof for the block containing the event. The EventHeader will contain all the data necessary to reconstruct the GossipEvent except for the original signature. The primary space-saving comes from replacing the EventDescriptors used to indicate the parents of each event. If an EventDescriptor of a parent points to an event outside the block, the EventHeader will continue to preserve the EventDescriptor that points to that parent event. If the parent is contained within this block, however, the EventDescriptor is replaced with a uint32 index that indicates which event from the start of the block is the parent event. This index is 0 based. The first few events from each node in each block will start out with EventDescriptors for parents outside the block. As the index increases, events will begin to have parents in the block. Each EventDescriptor is 72 bytes whereas each uint32 is 4 bytes. This space-saving is important for networks with low user transaction activity to minimize the space occupied by events not containing transactions.

To reconstruct the GossipEvent from an EventHeader, set the signature in the GossipEvent to the byte (signature_middle_bit ? 255 : 0). Then replace each parent reference index with the EventDescriptor of the event at that index within the block. This process is recursive. In order to construct the hash in an EventDescriptor, the EventCore data for the EventHeader at the given parent index, along with the EventDescriptors of the event’s parents and transactions in the event, must be serialized and hashed. If the EventHeader contains no event indices for parents, the base case is reached and the GossipEvent can be constructed without a recursive call. To reconstruct all EventCore data in a block, it is simplest to use dynamic programming techniques and build a lookup table from parent index to EventDescriptor starting at index 0.

Note: Platform state changes are made as a result of the rounds changes and not transaction level changes. The sum of all changes to the transaction receipt queue however, is a single state change from many transactions.

/**
 * Contains information about an event and its parents.
 */
message EventHeader {
    /**
     * An event core value.<br/>
     */
    com.hedera.hapi.platform.event.EventCore event_core = 1;

    /**
     * A list of references to parent events. <br/>
     */
    repeated ParentEventReference parents = 2;

    /**
     * The middle bit of the node's signature on the event.<br/>
     */
    bool signature_middle_bit = 3;
}

/*
 * A reference to a parent event.
 */
message ParentEventReference {

    oneof parent {
        /**
         * An EventDescriptor for the parent event outside of the containing block.
         */
        proto.EventDescriptor descriptor = 1;

        /**
         * An index of the parent event within the containing block.
         */
        uint32 index = 2;
    }
}

Round Header

The round header describes the beginning of a round. A block may contain zero or more rounds, although the initial release is expected to be 1 block containing 1 round. There is some fixed cost to each block, so batching multiple rounds into a single block increases efficiency. However, the more rounds that are batched, the longer the end-to-end latency for a transaction. So it is generally expected that the number of rounds per block will be kept low.

/**
 * A header for a single round.<br/>
 * This message delivers information about a consensus round.
 */
message RoundHeader {
    /**
    * A round number.<br/>
    * This is the number assigned to the round for consensus.
    */
    uint64 round_number = 1;
}

EventTransactions

A user transaction submitted to create a state change or query node state information. This message unifies the concepts of user and system transactions, and matches the structure of Events within the hashgraph algorithm. A state signature system transaction is a non-user transaction internally created to carry out a state change in conformance with network hashgraph logic. This transaction does not count towards network TPS and can only be issued internally via the consensus node software.

message EventTransaction {
    oneof transaction {
        /**
         * An application transaction.
         */
        bytes application_transaction = 1;

        /**
         * A single state signature.
         */
        StateSignatureSystemTransaction state_signature_transaction = 2;
    }
}

Reconstructing GossipEvents for Consensus Validation

The block stream supports complete reconstruction of the consensus event stream in a simple and secure manner. Any entity wishing to reconstruct the event stream may begin with the EventHeader and append each following EventTransaction in order as it appears in the Block Stream. The event ends when the next EventHeader or Block Proof is encountered in the stream.

Once the event data is gathered, the event may be rebuilt as a GossipEvent protocol buffer message. Gossiped events can then be fed into the Hashgraph consensus algorithm. The new resulting rounds and event order can be compared to the original order in the block stream to validate that consensus was executed correctly.

Event contents can also be validated by computing the SHA384 hash of the EventCore encoded bytes followed by the SHA384 hashes of each EventTransaction in order. The resulting hash can be checked against the signature originally present in the EventHeader.

Transaction Result

TransactionResult captures the impacts and network metadata of an individual transaction at the time the network processes it. It mainly represents items that are global to all transactions.

message TransactionResult {
    /**
    * The response code that indicates the current status of the transaction.
    */
    proto.ResponseCodeEnum status = 1;

    /**
    * The consensus timestamp of the transaction.
    */
    proto.Timestamp consensus_timestamp = 2;

    /**
    * In the record of an internal transaction, the consensus timestamp of the
    * user transaction that spawned it.
    */
    proto.Timestamp parent_consensus_timestamp = 3;

    /**
    * A schedule that executed this transaction, if this transaction
    * was scheduled.
    * <p>
    * This value SHALL NOT be set unless this transaction result represents
    * the result of a _scheduled_ child transaction.
    */
    proto.ScheduleID schedule_ref = 4;

    /**
    * The actual transaction fee charged, not the original transactionFee value
    * from TransactionBody.
    */
    uint64 transaction_fee_charged = 5;

    /**
    * All hbar transfers as a result of this transaction, such as fees, or
    * transfers performed by the transaction, or by a smart contract it calls, or
    * by the creation of threshold records that it triggers.
    */
    proto.TransferList transfer_list = 6;

    /**
    * All Token transfers as a result of this transaction.
    */
    repeated proto.TokenTransferList token_transfer_lists = 7;

    /**
    * All token associations implicitly created while handling this transaction.
    */
    repeated proto.TokenAssociation automatic_token_associations = 8;

    /**
    * List of accounts with the corresponding staking rewards paid as a result of
    * a transaction.
    */
    repeated proto.AccountAmount paid_staking_rewards = 9;

    /**
    * Congestion pricing multiplier at the time the transaction was executed.
    */
    uint64 congestion_pricing_multiplier = 10;
}

Transaction Output

Output data from a transaction data that is neither present in the transaction nor stored in state. Not all transactions have output, each one that does has a corresponding output message type with any data it needs to include.

For example a CryptoTransfer transaction input will result in both a TransactionResult and a CryptoTransferOutput to cover the outputs of the network.

message TransactionOutput {
    oneof transaction {
        /**
        * Output from a contract call transaction.
        */
        ContractCallOutput contract_call = 1;

        /**
        * Output from a contract create transaction.
        */
        ContractCreateOutput contract_create = 2;

        /**
        * Output from a crypto transfer transaction.
        */
        CryptoTransferOutput crypto_transfer = 3;

        /**
        * Output from an ethereum transaction.
        */
        EthereumOutput ethereum = 4;

        /**
        * Output from a schedule create transaction that executed
        * immediately on creation.
        */
        ScheduleCreateOutput schedule_create = 5;

        /**
        * Output from a schedule sign transaction that resulted in
        * executing the scheduled transaction.
        */
        ScheduleSignOutput schedule_sign = 6;

        /**
          * Output from a token airdrop transaction.
          */
        TokenAirdropOutput token_airdrop = 7;

        /**
        * Output from a UtilPrng transaction to request a
        * deterministic pseudo-random number.
        */
        UtilPrngOutput util_prng = 8;
    }
}

message ContractCallOutput {
    /**
     * Result details for an EVM transaction execution
     */
    EVMTransactionResult evm_transaction_result = 1;
}

message ContractCreateOutput {
    /**
     * Result details for an EVM transaction execution
     */
    EVMTransactionResult evm_transaction_result = 1;
}

message CreateAccountOutput {
    /**
     * A newly created account identifier.
     */
    proto.AccountID created_account_id = 1;
}

message EthereumOutput {
    /**
     * Result details for an EVM transaction execution
     */
    EVMTransactionResult evm_transaction_result = 1;
}

message ScheduleCreateOutput {
    /**
    * A scheduled transaction identifier.
    */
    proto.TransactionID scheduled_transaction_id = 2;
}

message ScheduleSignOutput {
    /**
    * A scheduled transaction identifier.
    */
    proto.TransactionID scheduled_transaction_id = 1;
}

message TokenAirdropOutput {
    /**
    * Custom fees assessed during a TokenAirdrop.
    */
    repeated proto.AssessedCustomFee assessed_custom_fees = 1;

    /**
     * A list of token associations.
     */
    repeated proto.TokenAssociation automatic_token_associations = 2;

    /**
     * A list of _non-HBAR_ token transfers, in single-entry form.
     */
    repeated proto.TokenTransferList token_transfer_lists = 3;
}

message UtilPrngOutput {
    oneof entropy {
        /**
        * A deterministic pseudo-random sequence of 48 bytes.
        */
        bytes prng_bytes = 1;

        /**
        * A deterministic pseudo-random number generated within a
        * specified range.
        */
        uint32 prng_number = 2;
    }
}

State Changes

The state changes to the merkle tree for a given transaction or outside a transaction as part of executing a round.

The aim of state changes in the block stream is two fold:

1. To give low latency visibility to state contents for apps and to allow external construction of a merkle tree state matching that in the consensus node.
2. To make it easy to use for apps the state is expressed at the high conceptual level of names states(like named database tables).

There are 3 types of named states supported in the first version of block streams: Maps, Queue’s and Singleton Objects. The algorithm for how states are translated into a merkle tree will be provided. The block proof will contain a signed root hash of the state merkle tree allowing those reconstructing a merkle tree outside the consensus node to have complete cryptographic trust of its contents. The granularity for state changes will support the ability to understand the changes made by an individual transaction.

message StateChanges {
    /**
     * The consensus timestamp of this set of changes.
     * See the full specification above for how this value is determined.
     * <p>
     * This value SHALL be deterministic.
     */
    proto.Timestamp consensus_timestamp = 1;

    /**
     * An ordered list of individual changes.
     * <p>
     * These changes MUST be applied in the order listed to produce
     * a correct modified state.
     */
    repeated StateChange state_changes = 2;
}

State changes are expressed in terms of changes to named states at the high level conceptual model of the state type like map key/values or queue appends. To determine which state each change affects we will include an integer number in place of a state name. This is done for performance and efficiency as there will be 10s of thousands of state changes in a block.

• If we have an extra 8-10 bytes per state change, then at 40-50K state changes per second that is an extra 3-4Mbits of bandwidth. Compression should help a lot but that is not guaranteed.
• When the state name is used as part of a complex key in the big state merkle map. The smaller the key is, in bytes, the more efficient the database is because more keys can fit in a single disk page.
• When parsing keys, parsing a UTF8 string to a Java String is a many times more expensive than parsing an integer from a varint.

For convenience, an enum will be included in the HAPI spec for each known state and the integer value will always match the enum ordinal. The reason we use an integer field instead of an enum field is to function better for forwards compatibility. This is where a block node with HAPI version 1 wants to process blocks from HAPI version 2, for example. If we use a protobuf enum, then when that is mapped to a language like Java or Rust it can cause errors because those languages do not support unknown enum values. ProtoC has a workaround for this but it is super ugly and creates more issues than it solves in this specific situation. We believe that the integer field with a separate enum useable to map that integer to a well-defined value in code is the best compromise.

/**
 * A change to any item in the merkle tree.
 */
message StateChange {
    /**
    * A state identifier.
    * The "mapping" for these values is available as an _informational_ enum
    * named `StateIdentifier`.
    */
    uint32 state_id = 1;

    // StateAdded and StateRemoved were requested. Can they not be inferred by the below? Do we need an add version for Singleton, Map and Queue?
    oneof change_operation {
        /**
        * Addition of a new state.
        */
        NewStateChange state_add = 2;

        /**
        * Removal of an existing state.
        */
        RemovedStateChange state_remove = 3;

        /**
        * An add or update to a `Singleton` state.
        */
        SingletonUpdateChange singleton_update = 4;

        /**
        * An add or update to a single item in a `VirtualMap`.
        */
        MapUpdateChange map_update = 5;

        /**
        * A removal of a single item from a `VirtualMap`.
        */
        MapDeleteChange map_delete = 6;

        /**
        * Addition of an item to a `Queue` state.
        */
        QueuePushChange queue_push = 7;

        /**
        * Removal of an item from a `Queue` state.
        */
        QueuePopChange queue_pop = 8;
    }
}

/**
 * An informational enumeration of all known states.
 * This enumeration is included here So that people know the mapping from
 * integer to state "name".
 *
 * Note: This enumeration is never transmitted directly in the block stream.
 * This enumeration is provided for clients to _interpret_ the value
 * of the `StateChange`.`state_id` field.
 */
enum StateIdentifier {
    /**
    * A state identifier for the Topics state.
    */
    STATE_ID_TOPICS = 0;

    // many other enumerated values omitted for brevity.
    // Full detail is available in the Hiero-protobuf github repo.

    /**
    * A state for the `159` upgrade file data
    */
    STATE_ID_UPGRADE_DATA_159 = 10010;
}

/**
 * An addition of a new named state.
 */
message NewStateChange {
    /**
    * The type (e.g. Singleton, Virtual Map, Queue) of state to add.
    */
    NewStateType state_type = 1;    
}

/**
 * An enumeration of the types of named states.
 */
enum NewStateType {
    SINGLETON = 0;
    VIRTUAL_MAP = 1;
    QUEUE = 2;
}

/**
 * A removal of a named state.
 */
message RemovedStateChange {
}

/**
 * An update to a `Singleton` state.
 */
message SingletonUpdateChange {
    oneof new_value {
        /**
        * A change to the block info singleton.
        */
        proto.BlockInfo block_info = 1;

        /**
        * A change to the congestion level starts singleton.
        */
        proto.CongestionLevelStarts congestion_level_starts = 2;

        /**
        * A change to the Entity Identifier singleton.
        */
        google.protobuf.UInt64Value entity_number = 3;

        /**
        * A change to the exchange rates singleton.
        */
        proto.ExchangeRateSet exchange_rate_set = 4;

        /**
        * A change to the network staking rewards singleton.
        */
        proto.NetworkStakingRewards network_staking_rewards = 5;

        /**
        * A change to a raw byte array singleton.
        */
        google.protobuf.BytesValue bytes = 6;

        /**
        * A change to a raw string singleton.
        */
        google.protobuf.StringValue string = 7;

        /**
        * A change to the running hashes singleton.
        */
        proto.RunningHashes running_hashes = 8;

        /**
        * A change to the throttle usage snapshots singleton.
        * <p>
        * Throttle usage snapshots SHALL be updated for _every transaction_
        * to reflect the amount used for each tps throttle and
        * for the gas throttle.
        */
        proto.ThrottleUsageSnapshots throttle_usage_snapshots = 9;

        /**
        * A change to a raw `Timestamp` singleton.<br/>
        * An example of a raw `Timestamp` singleton is the
        * "network freeze time" singleton state, which, if set, stores
        * the time for the next scheduled freeze.
        */
        proto.Timestamp timestamp = 10;

        /**
        * A change to the block stream status singleton.
        */
        com.hedera.hapi.node.state.blockstream.BlockStreamInfo block_stream_info = 11;

        /**
        * A change to the platform state singleton.
        */
        com.hedera.hapi.platform.state.PlatformState platform_state = 12;

        /**
        * A change to the roster state singleton. 
        */
        com.hedera.hapi.node.state.roster.RosterState roster_state = 13;
    }
}

/**
 * An update to a single item in a `VirtualMap`.<br/>
 * Each update consists of a "key" and a "value".
 * Keys are often identifiers or scalar values.
 * Values are generally full messages or byte arrays.
 */
message MapUpdateChange {
    /**
      * A boolean flag indicating if the written value was identical to the original state value.
      * <p>
      * If this flag is set to true, it indicates that the value being
      * written to the map is identical to the value that was already
      * present in the map for the given key. This means that no actual
      * change has occurred in the state of the map entry, and thus
      * the state change can be considered a no-op for the purposes of
      * state updates.
      * <p>
      * If this flag is set to false, it indicates that the value being
      * written to the map is different from the value that was already
      * present in the map for the given key. In this case, the state
      * change represent in value represents an actual update to the map entry
      * <p>
      * This field is OPTIONAL. If not set, it is assumed to be false.
      * <p>
      * Note: This field is useful for optimization purposes, allowing
      * clients to determine whether an update was necessary or if it
      * was a no-op. It can help reduce unnecessary state updates and
      * improve performance in scenarios where many updates are being
      * made to the same map entry without any actual change in value.
      * <p>
      */
    bool identical = 1;

    /**
    * A key in a virtual map.
    * <p>
    * This key MUST be mapped to the value added or updated.<br/>
    * This field is REQUIRED.
    */
    MapChangeKey key = 2;
        

    /**
      * A value in a virtual map.
      * <p>
      * This value MUST correctly represent the state of the map entry
      * _after_ the asserted update.<br/>
      * This value MAY be reduced to only transmit fields that differ
      * from the prior state.<br/>
      * This field is REQUIRED.
      */
    MapChangeValue value = 3;
}

/**
 * A removal of a single item from a `VirtualMap`.
 */
message MapDeleteChange {
    /**
    * A key in a virtual map.
    * <p>
    * This key SHALL be removed.<br/>
    * The mapped value SHALL also be removed.<br/>
    * This field is REQUIRED.
    */
    MapChangeKey key = 1;
}

/**
 * A key identifying a specific entry in a key-value "virtual map".
 */
message MapChangeKey {
    oneof key_choice {
        /**
        * A key for a change affecting a map keyed by an Account identifier.
        */
        proto.AccountID account_id = 1;

        /**
        * A change to the token relationships virtual map.<br/>
        * This map is keyed by the pair of account identifier and
        * token identifier.
        */
        proto.TokenAssociation token_relationship = 2;

        /**
        * A change to a map keyed by an EntityNumber (which is a whole number).
        */
        google.protobuf.UInt64Value entity_number = 3;

        /**
        * A change to a virtual map keyed by File identifier.
        */
        proto.FileID file_id = 4;

        /**
        * A change to a virtual map keyed by NFT identifier.
        */
        proto.NftID nft_id = 5;

        /**
        * A change to a virtual map keyed by a byte array.
        */
        google.protobuf.BytesValue proto_bytes = 6;

        /**
        * A change to a virtual map keyed by an int64 value.
        */
        google.protobuf.Int64Value proto_long = 7;

        /**
        * A change to a virtual map keyed by a string value.
        */
        google.protobuf.StringValue proto_string = 8;

        /**
        * A change to a virtual map keyed by a Schedule identifier.
        */
        proto.ScheduleID schedule_id = 9;

        /**
        * A change to the EVM storage "slot" virtual map.
        */
        proto.SlotKey slot_key = 10;

        /**
        * A change to a virtual map keyed by a Token identifier.
        */
        proto.TokenID token_id = 11;

        /**
        * A change to a virtual map keyed by a Topic identifier.
        */
        proto.TopicID topic_id = 12;

        /**
        * A change to a virtual map keyed by contract id identifier.
        */
        proto.ContractID contract_id = 13;

        /**
        * A change to a virtual map keyed by pending airdrop id identifier.
        */
        proto.PendingAirdropId pending_airdrop_id = 14;
    }
}

/**
 * A value updated in, or added to, a virtual map.
 */
message MapChangeValue {
    oneof value_choice {
        /**
        * An account value.
        */
        proto.Account account = 1;

        /**
        * An account identifier.<br/>
        * In some cases a map is used to connect a value or identifier
        * to another identifier.
        */
        proto.AccountID account_id = 2;

        /**
        * Compiled EVM bytecode.
        */
        proto.Bytecode bytecode = 3;

        /**
        * An Hiero "file" value.
        */
        proto.File file = 4;

        /**
        * A non-fungible/unique token value.
        */
        proto.Nft nft = 5;

        /**
        * A string value.
        */
        google.protobuf.StringValue proto_string = 6;

        /**
        * A scheduled transaction value.
        */
        proto.Schedule schedule = 7;

        /**
        * A list of scheduled transactions.<br/>
        * An example for this value is the map of consensus second to
        * scheduled transactions that expire at that consensus time.
        */
        proto.ScheduleList schedule_list = 8;

        /**
         * An EVM storage slot value.
         */
        proto.SlotValue slot = 9;

        /**
        * An updated set of staking information for all nodes in
        * the address book.
        */
        proto.StakingNodeInfo staking_node_info = 10;

        /**
        * An HTS token value.
        */
        proto.Token token = 11;

        /**
        * A token relationship value.<br/>
        * These values track which accounts are willing to transact
        * in specific HTS tokens.
        */
        proto.TokenRelation token_relation = 12;

        /**
        * An HCS topic value.
        */
        proto.Topic topic = 13;

        /**
        * An network node value.
        */
        com.hedera.hapi.node.state.addressbook.Node node = 14;

        /**
        * A pending airdrop value.
        */
        proto.AccountPendingAirdrop account_pending_airdrop = 15;

        /**
        * A roster value.
        */
        com.hedera.hapi.node.state.roster.Roster roster = 16;
    }
}

/**
 * Addition of an item to a `Queue` state.
 */
message QueuePushChange {
    oneof value {
        /**
        * A byte array added to the queue state.
        */
        google.protobuf.BytesValue proto_bytes_element = 1;

        /**
        * A string added to the queue state.
        */
        google.protobuf.StringValue proto_string_element = 2;

        /**
        * All transaction receipts for a round added to queue state.
        */
        proto.TransactionReceiptEntries transaction_receipt_entries_element = 3;
    }
}

/**
 * Removal of an item from a `Queue` state.<br/>
 */
message QueuePopChange {
}

Note: The block stream design ensures consistency of singleton and queue states at round boundaries so as to reduce the block stream size. That is singleton and queue state changes are written to the block stream at the end of a round. The platform computes the state hash at round boundaries, and the consensus node tracks these hashes. When a block ends the node uses the hash from the last non-empty round as the start_of_block_state_root_hash in the BlockProof. With that in mind, while intermediate states may temporarily diverge during block stream replay, the final state after applying all state changes in a round will match the live network’s state at that same round boundary.

FilteredBlockItem

The block stream will support the ability to omit BlockItems from the stream while retaining full verification of the block proof. This is possible because all items are double hashed. Their hash is hashed into the parent node not the raw contents of the item. The block proof is based on the signing of the root hash of a block merkle tree, therefore any item or subtree in the proof can be replaced by a FilteredItemHash. Notably, the replacement of an item with a hash will not impact the ability for downstream services to validate block proofs.

message FilteredItemHash {
    /**
     * A hash of items filtered from the stream.
    */
    bytes hash = 1;
    
    /**
    * A binary tree path to a merkle subtree.
    */
    uint64 filtered_path = 2;

    /**
     * The log2 value of the number of filtered items.<br/>
     */
    uint64 log2_item_count = 3;
}

💡 Sidebar: Filtering and Errata for Block Stream. Block stream supports filtering the block stream through the use of a FilteredItemHash to replace any item or subtree that has been removed. The removed subtree is replaced by its hash and the path that was filtered which preserves the ability to verify the integrity of the block stream and block proof. Errata are handled similarly for removed items. Adding items after the fact or replacing an erroneous item requires potentially removing a prior item, and adding the correction (errata) as a new item in the block stream at the point of the “errata” transaction, which is then part of the block proof for the block where the “errata” was processed.

Block Proof

The block proof carries the trust of the consensus network to the contents of the block. A block can conceptually be represented as a binary merkle tree with 8 sub-trees at depth 3 which create the block proof and tree when hashed to obtain a merkle root.

The 8 leaves (with associated merkle binary path numbers) represent

1. (000) Previous block proof hash - the root of previous blocks merkle tree, this forms a blockchain
2. (001) Merkle root of state tree - root hash of consensus node state merkle tree at the beginning before the state changes of the block are applied
3. (010) Merkle root of consensus header tree - root hash of combined round and event header items merkle tree
4. (011) Merkle root of transaction inputs tree - root hash of input item merkle tree
5. (100) Merkle root of transaction outputs tree - root hash of output item merkle tree
6. (101) Merkle root of state changes outputs tree - root hash of state changes output merkle tree
7. (110) Merkle root of trace data outputs tree - root hash of trace data item merkle tree
8. (111) Merkle root of extensions subtree - root of a depth 2 subtree for up to 4 additional “future” subtrees. These subtrees are pre-defined to support efficient forward-compatibility. Until these extensions are defined in a future HIP, the leaves of this tree are all NULL.
   1. (1100) Merkle root of extension 0
   2. (1101) Merkle root of extension 1
   3. (1110) Merkle root of extension 2
   4. (1111) Merkle root of extension 3

The root hash of the block merkle tree is signed by the aggregate signature of the consensus network nodes. Thus proving the super majority of consensus nodes agree on the contents of this block, the state and all previous blocks. Consensus nodes will sign the root hash at the end of executing a block of transactions then gossip its signature fragments. It will then listen on gossip till it receives enough signature fragments from other nodes to produce an aggregate signature. The block proof item will not be produced and sent until that aggregate signature is created. This also means a consensus node will not begin to emit block items for future blocks until the current block can be proved.

💡Sidebar: Future extension
The future extensions subtree allows for new subtrees to be defined in the future without any requirement to update Block Stream processors! With careful definition and use of field number rules for new BlockItem types, we can pre-define which items belong in which extension subtree and software processing the block stream can properly assign those items to the correct subtree without knowing the content of the items. The full specification for block items describes how this works in detail.

Q: Why is the state hash taken at the beginning of the block and not the end?

This is a very important decision, it was made to solve the performance/timing issue. This is because smart contracts have an EVM opcode for getting the hash of the last 256 blocks including the immediately preceding block. That opcode could be the first instruction of the first transaction in a block. This means we need to be able to compute the block hash of previous block very quickly at the start of a block. The problem is because of the high TPS and large state size of Hiero the amount of work to compute the updated state hash can take a whole round/block. By having the hash of state at beginning of block stored in the BlockProof item at the end of the block it gives us one block’s worth of time to compute that state hash.

💡 The amazing thing from this is we can have one “Block Hash” and use the same one everywhere, in Block Nodes, Mirror Nodes, JSON-RPC relays, EVM etc. That “Block Hash” is the root hash of the block merkle tree.

There is one downside of this choice which is when a stream consumer is applying state changes to a local merkle tree it cannot verify that they were applied correctly until the start of the next block. Block Nodes can still trust the changes, however, as the changes are part of the block hash and signed. So it is only a small added latency for the extra reliability.

There is also one edge case, which is when shutting down a consensus node after a freeze for the final state hash to be signed and saved one extra empty block is need to be created containing just a BlockHeader and BlockProof items.

Q: What happens if signature fragments are slow to arrive?

If there was an issue causing a consensus node to not see enough signature fragments for block 1 before it has finished executing block 2 and received all the signature fragments for block 2. It can use the signature for block 2’s root hash to create a block proof for block 1 based on its root hash being present in block 2’s merkle tree. This allows the stream to continue even if the extremely unlikely case happens where it can not gather enough signature fragments to produce an aggregate signature for a block. The sibling_hashes field will be zero length in all cases other than this rare case where it must be used to create a merkle proof for the root hash of the block.

🗒️ Side note: Today the root hash of state is signed and gossiped by consensus nodes. After we move to block streams the state root hash will not be signed directly but included in the block and then the block root hash is signed. This is much more powerful as a single signature proves all of state(prior to the current block) as well as the contents of the block and all previous blocks. Details:

• We will collect signatures pre-consensus.
• This means each consensus node may collect a different majority of fragments and therefore have a different but equally valid signature bytes on a block.
• We could end up with out of order arrival of block signatures. Block 2 signature could arrive before block 1. This is fine we will construct a proof item for block 1 based on its hash being in block 2 then send out both blocks and their proofs in order to the block node. This should be very rare as long as rounds are many times longer than ping times.
• We accept that blocks items will be identical and deterministic but the proofs could differ from one block node to another. But all proofs are equally valid. It is possible one block node’s proof will be bigger than another block node’s. That is ok.
• In the future block nodes could be clever and collect the signature fragments themselves from the system transactions in events. They could use those to go back and create and store more concise proofs for earlier blocks.
• There are cases were we could be missing a majority of signature fragments for a few blocks in a row. This can happen around address book changes. They can be buffered until we get a good block with a valid signature then we can create proofs for the blocks missing signatures and send all those blocks to a block node.
• Consensus nodes need to handle back pressure from block nodes. If the consensus node cannot send its blocks to a block node it can buffer them for a period of time. After which it has to stop handling transactions. This is because a consensus node can not safely determine if other consensus nodes are successfully storing blocks on their block nodes. So it has to assume the worst, ie. it is the last man standing. So it must stop producing blocks when it can no longer store them. As it is bad for the network to carry on executing transactions if the record of execution has nowhere to go.
• A block node will confirm each block it successfully receives, verifies, and stores back to the consensus nodes it is connected to. This allows the consensus node to delete its pre-consensus event stream up to that point. The confirmation will be by a GRPC message on the back channel of the bidirectional streaming connection between consensus node and block node.
• Block nodes will be able to receive blocks from more than one consensus node.
  • For direct streaming consumers they will pick one and just use that to send items directly.
  • For storing blocks they will collect all the items from all of them, though they may deduplicate them to reduce memory cost. Each node will pick one stream it is hashing as it goes. As soon as it gets a valid proof on any of the received streams it can use that to verify the hash it computed and store the verified block, with its proof, on disk.
  • They could hash more than one stream to lower latency in the case where the chosen source node is bad but that will waste a lot of CPU for a rare case and is probably unwise.

In the block streams output a block proof is represented as such

message BlockProof {
    /**
     * The block this proof secures.<br/>
     * We provide this because a proof for a future block can be used to prove
     * the state of the ledger at that block and the blocks before it.
     */
    uint64 block = 1;

    /**
     * A merkle root hash of the previous block.
     */
    bytes previous_block_root_hash = 2;

    /**
     * The hash of the merkle tree root for the network state at the start of
     * this block.
     */
    bytes start_of_block_state_root_hash = 3;

    /**
     * A network signature of the block root hash.
     */
    bytes block_signature = 4;

    /**
     * MerkleSiblingHash's starting from the root hash of the signed block down to and including the sibling of this
     * blocks root hash. Provided when for some reason a block signature for this block could not be produced.
     */
    repeated MerkleSiblingHash sibling_hashes = 5;
    
    oneof verification_reference {
        /**
         * The id of the hinTS scheme this signature verifies under.
         */
        uint64 scheme_id = 6;

        /**
         * The explicit hinTS key this signature verifies under
         */
        bytes verification_key = 7;
    }
}

message MerkleSiblingHash {
    /**
    * True if this MerkleSiblingHash is the first hash in a pair of siblings, false if it is the second hash.
    */
    bool is_first = 1;

    /**
    * BlockHashType(SHA_384) hash for the sibling at this level of the merkle tree.
    */
    bytes sibling_hash = 2;
}

The following diagram illustrates what the 8 sub trees contain and how blocks are chained.

The input and output item merkle trees are balanced binary trees. The leaves across the bottom are the items in the order they occur in the block. If the total number of input or output items contained in the block is less than a power of 2 then the empty leaves are assumed to be null (zero length bytes). The nature of the merkle tree creation logic allows for the consensus node or block validator to calculate the input and output tree merkle hashes as items are received in a streaming way.

In summary a consensus node can calculate the merkle hashes of the transaction inputs and outputs in real time and need only calculate the state merkle root at the end of a block.

Example Pseudo Code for computing streaming merkle tree root hash

elements = [ ... ] // array of the leaves to be streamed and hashed. Items may be transaction inputs, transaction outputs or state
hashList = [] // a new empty list of hashes

// loop over each element received and create a list of all the root hashes of the leaves of the tree 
for (int i=0, i<elements.size(), i++) {
    write elements[i] // output elemen to stream
    e = hash(elements[i]) // SHA 384 item data
    hashList.add(e) // add hash to end of list
    
    // for every odd located element remove and hash the last 2 hashes (representing the hash leaves) and shift right per loop to pick up the hash root
    for (int n=i, (n & 1 == 1), n>>=1) {
        y = hashList.getAndRemove(hashList.size() – 1); // remove last element of hashList
        x = hashList.getAndRemove(hashList.size() – 1); // remove last element of hashList
        hashList.add(hash(x, y)); // add the new hash root of the two hashes
    }    
}

merkleRootHash = hashList.get(hashList.size() – 1)

for (int i=hashList.size() – 2, i>=0, i--) {
    merkleRootHash = hash(hashList.get(i), merkleRootHash)
}

write merkleRootHash; // output

RecordFileItem

A RecordFileItem block stream item is added to act as a wrapper around the contents of a legacy block. This allows block nodes and other handlers of the block stream to have single standardized APIs and handling code for blocks before and after the switch to block streams. It is not possible to change the format because it is signed by the consensus nodes in the original format. As a result of this a reader will still have to understand record file formats version 2, 3, 5 and 6 that existed prior to block streams. Each block of record style will have just one item: a RecordFileItem.

/**
 * A Block Item for record files.
 */
message RecordFileItem {
    /**
    * The consensus time the record file was produced for.
    */
    proto.Timestamp creation_time = 1;

    /**
    * The contents of a record file.
    */
    bytes record_file_contents = 2;

    /**
    * The contents of sidecar files for this block.
    */
    repeated SidecarFile sidecar_file = 3;

    /**
    * A collection of RSA signatures from consensus nodes.
    * The first 4 bytes are a 32bit little endian version number and multiple version exist including v2, v5 and v6
    */
    repeated bytes record_file_hash_signatures = 4;
}

TraceData

A TraceData block item is added to contain all information that could be cosnidered trace or debugging in nature. The consensus node will export all necessary and important data for a transactions execution in the block streams, however every byte of information is stored fore er and needed in the proof of the network data. As such it’s important to ensure that data is required but also laid out in a way that makes it easier for filtering by downstream clients. To this point the TraceData would initially contain smart contracts EVMTraceData that support traceability information (e.g. contract actions, read values etc). Future trace like data can be added to TraceData and Block Nodes, Mirror Nodes and other block stream parsign clients can decided to store or filter it out based on their needs.

message ContractSlotReads {
     
     message SlotRead {
          oneof identifier {
               /**
               * The contract storage slot counter in this block 
               * This is populated in place of the 256 bit word when the given slot is written to in state changes
               */
               int32 index = 1;

               /**
               * The key of this contratc storage slot, may be left-padded with zeros to form a 256-bit word.
               * This is populated when the slot was not written and only read
               */
               bytes key = 2;
          }

          /**
          * The storage value in this slot, may be left-padded with zeros to form a 256-bit word.
          */
          bytes read_value = 3;
     }
     
     /**
      * The contract associated with the storage slots this slot belongs to.
      */
     ContractID contract_id = 1;

     /**
      * The storage slots that were read in this EVM exectuion. They may or may not have assocaited slot writes
      */
     repeated SlotRead slot_reads = 2;    
}

/**
 * EVM transaction execution log storage details
 * Details maps to the log object in eth_getTransactionReceipt response without repeating info already available in the input transaction
 * Log bloom logic is removed as it may be calculated by block parser for both transactions and block level
 */
message EVMTransactionLog {
     /**
      * The contract emitting the log. ContractID vs 20 bye address is used to preserve space
      */
     ContractID contract_id = 1;

     /**
      * The Log data
      */
     bytes data = 2;

     /**
      * The logc topics left padding of 0's by EVM is stripped to save space. 
      * Indexers should left-pad with zeros to form a 256-bit word.
      */
     repeated bytes topics = 3;
}

/**
 * The init bytecode components for child contracts created as part of the contract execution.
 * init_bytecode = deploy_bytecode + runtime_byteode + metadata_bytecode
 */
message ContractInitByteCode {
    /**
     * The bytecode that predeces and deploys the runtime bytecode in a contracts deployment
     */
    google.protobuf.BytesValue deploy_bytecode = 1;

    /**
     * The bytecode that follows the runtime bytecode (found in contract state) in a contracts deployment.
     */
    google.protobuf.BytesValue metadata_bytecode = 2;

    /**
     * The bytecode that defines a deployed contracts logic. Only present if not in state.
     */
    google.protobuf.BytesValue runtime_bytecode = 3;
}

/**
 * EVM tranaction execution trace details
 * Details maps to the variable needed for debugging executions
 */
message EVMTraceData { 
    /**
     * The init bytecode component for this child contract created as part of the contract execution.
     */
    ContractInitByteCode init_bytecode = 1;

    /**
     * The inter contract interaction details. This represents the internal EVM message frames.
     */
    repeated ContractAction contract_actions = 2;    
 
    /**
     * Contract slot values that were read during the execution of the EVM transaction. Associated written values will be in state changes
     */
    repeated ContractSlotReads contract_slot_reads = 3;

    /**
     * Any error message produced by the contract call.
     */
    string full_error_message = 4;

    /**
     * Any Log events produced by this contract call.
     */
    repeated EVMTransactionLog logs = 5;
}

Backwards Compatibility

Mirror Node Operators

The Block Streams format is a new format and is not an iteration on previous record, event, balance or state stream format. It is designed to be the continuation of the existing record stream.

This is achieved in two ways

1. The previous block hash of the first block stream block will be the hash of the previous record stream block. This continues the blockchain unbroken from genesis.
2. The block stream can contain wrapped record stream blocks. This allows for a single standard container and streaming format for blocks in all formats since genesis.

All current users of the record streams like the Mirror Node product will need to be updated to support the new block stream format when the block stream is produced otherwise they may experience a gap in data, and will, eventually, cease to receive record stream data.

We plan to conduct a preview release which will produce Block Stream alongside the Record Stream for a short time. The Block Stream will be packaged into files stored in buckets, similar to the Record Stream files. This preview period will permit applications to process both data sources side-by-side; ensuring that the Block Stream continues to provide all of the needed information and that applications are able to process Block Stream data effectively.

Developers <mark add section on mirror node endpoints/node GRPC not changing> maybe add section for node operators

Security Implications

The Blocks Stream provides many improvements in the ability for machines outside the consensus network to validate what was done in the consensus network. This opens the door for increased trust. Also much more information is made available like state, which will increase transparency of the networks operation.

One notable consideration is the increased size of a Block in comparison to a Record. To support state proof a Block contains more data than a record, however, it also removes duplication previously found in the streams between events, records, balances and side cars. Collectively a net reduction in combined stream size is expected.

As always the costs of the contents of a transaction and its effect on state are captured in transaction costs and in the future state rent.

How to Teach This

To effectively educate and inform users about block streams, comprehensive technical documentation, blogs, and webinars will be essential. Technical documentation will provide detailed and in-depth explanations of block stream format, usage, and best practices, ensuring that developers and mirror node operators can fully understand and transition to block streams.

Blogs will offer more accessible and engaging content, highlighting use cases, real-world applications, and the benefits of block streams, catering to a broader audience of Hiero stakeholders. Webinars will serve as interactive platforms for live demonstrations, Q&A sessions, and expert insights, enabling participants to gain a deeper understanding through direct engagement with subject matter experts.

Additionally, a block stream producer simulator will be provided to the community to allow viewing and testing as clients consuming a block stream.

Rejected Ideas

Re-writing pre block stream transaction body protobufs

The previous record streams protobuf contained deprecated fields and messages that could be better grouped for efficiency and consistency. With a new data stream there was a consideration to rewrite the previous format. Unfortunately, the TransactionBody message itself cannot be re-written due to signing verification implications on backwards compatibility. This is because mirror node and other products need to be able to parse historical transaction bytes utilizing older formats and verify them. However, the transaction wrappers in the old formats could still be optimized. This may be explored in the future.

Ensuring all events across upgrade boundaries are captured

The block streams design ensures that all events are captured and that blocks are signed. However, how does one denote the last block before upgrade if the act of signing a block n produces new events that must be recorded in block n+1? One consideration was to create a marker after which a future block may be ignored in terms of signing requirements. Though this could work it would result in Blocks that are unsigned even if they are ignored. This also makes stream consideration by clients a bit more complex as they would have to be aware of and handle this marker case. The final solution was to capture events created by signing Block n in the Pre Consensus Event Stream (PCES) across the upgrade boundary for insertion into a future Block n+1 after upgrade.

Impact of large entities when externalizing individual entity state.

The Block Stream plans to ensure visibility of all modified entities by externalizing the full entity state in the stream. This is to solve the issue of partial data in record streams where a client may miss the externalization of state data and would therefore possess a partial state for entities with few easy options to retrieve the missing data. By externalizing the full entity state in the block stream a client can get the complete entity details on any update and won’t have to employ “upsert” logic. An emergent feature when externalizing the full entity state of modified entities is that when entity properties are modified multiple times those entities may be externalized in the stream multiple times. These accounts may have multiple keys and other potentially large list properties (e.g. approvals) that may result in a large block size. To manage this an idea inspired from video iframe approach (when the image changes the whole entity is sent, if not only diffs are sent) was considered. In this case every entity could have a last updated counter that is updated on entity changes, if the value is over a certain threshold then the whole entity is externalized, if not only deltas would be externalized. This would optimize how far back a client would have to go to retrieve state info. However, this approach would require the counter to be stored in state for every entity which has undesired impacts on state size. Another inspired approach was that on the first observation of an entity in a block the CN would externalize the full entity state, after that only the delta changes would be externalized for the remainder of the block. In this way any consumer of a block would have the complete state by applying the diffs onto the initial state. This shares the downside of requiring additional temporary storage, and also requires the creation of a “difference” form for every entity potentially stored in state, and could, paradoxically, increase the total size of blocks with very many changes to small entities. Upon early POC considerations of compression the state size gains of the block stream size when compared to the record v6 externalized stream mitigated the concerns and thus the explored solutions were not needed.

Including exchange_rate in the TransactionResult

In the past the exchange_rate details were in the transaction record. However, this information doesn’t change often and is based on the value of a system file 0.0.112. Changes to this file are externalized in the block stream, thus the current exchange rate value is always available via a MN (and future BN). Thus it is no longer necessary to store the exchange_rate in the stream and we can remove unnecessary data from the block stream.

Changing EthereumOutput to EVMOutput

EthereumOutput seemed more appropriate if called EVMOutput. However, this is a legacy item referring ot the EthereumTransaction input whose name can’t be changed for backwards compatibility. Since outputs in the Block Stream must match the transaction input it would nopt be easy to change it. Additionally, there’s a ContractCall, ContractCreate and EthereumTransaction, EVMOutput would actually apply to all 3 and is thus too general.

String error message

With record stream versions HAPI developers and users in the explorer often are presented with error messages that are confusing and not always actionable. The same error code can apply to multiple issues so it’s not clear what went wrong. We pondered if we could include a string error message in the TransactionResult to give an improved DexExp when troubleshooting transaction failures. This would allow HAPI logic to say specifically the cause of the issue. An alternative suggestion was made to offer a detail error number, defined on a per-result-code basis, and published as a set of enumerations. This numeric option was noted to greatly reduce extra data in the block stream and avoid unclear or “bit-rotted” error strings. Eventually this was rejected as there was a concern it would introduce a lot of extra bytes and that could not be localized. Additionally, it was noted that it is challenging API wise as every letter in that string becomes API.

FileID refs for child contract creations

In some contract deployments on Hiero the contract bytecode of A may be found in a FileID entity, that bytecode may include logic to deloy contracts B and C. Since contracts B and C’s bytecode may be found in A’s bytecode it was explored if a reference could be made to the FileID noting which part of its contents defined B and C. This was not feasible as the File entity itself may be modified or deleted outside of the lifecycle of the contract. This would make a unique Hiero contract feature more challenging and potentially confusing to developers as to when a files contents described the contracts B and C. Additionally, it would require further optimization to provide pointers to where exactly in the FileID B and Cs init or runtime bytecode may be found. The topic may still be revisited in the future to explore if a simpler and developer friendly solution would work to avoid data replication whiles making it easy to discover the init byte code components of child contracts. For instance could it be possible to note that at the time of create the bytecodes for B and C started and ended at specific locations in the file bytearray. Indexers would require access to historical state in many cases to access this information.

Open Issues

Open Issues captured here and in https://github.com/hiero-ledger/hiero-block-node/issues/37

• Q: Are there additional protocol buffer changes needed to reduce the stream overhead for Events, Transactions, or other frequent entities.
  • Transaction rearrangement - utilize new wrapper and remove old references
  • Potentially move previous block has from block header to block proof
  • Update smart contract output as its missing details
  • Optimizations to reduce number of message objects
  • Ensure we are not double nesting in Transaction result vs Transaction output
  • How to handle File contents for large files e.g upgrade files
• Q: Should TransferList be included in the TransactionResult object or not. It contains redundant data, but makes mirror node processing much simpler.
  • Could a MN that parses the block stream reconstruct the transfer list from state changes? There are considerations for emergent transfer logic that isn’t in the transaction body which may prevent this.
  • There may be some transfers only exposed in the transfer_list
• Q: What’s the algorithm for how states are translated into a merkle tree? Should it be documented here or referenced from here. This has a few parts:
  • Document that all state changes result into key/values in a single huge binary merkle tree.
  • Document the VirtualMap algorithm for insertion and deletion of leaves
  • Document how internal hashes are computed
  • Note: Megamap implementation is expected to change a few things noted here. Thus we’ll wait for that to update the HIP description
• Q: Should transaction hash be added to the stream? In record stream the HAPI transaction hash is present and for EVM equivalence the hash is needed as EVM tools utilize that as the unique identifier. There’s a state concern as it’s an additional 32 bytes which could be computed from the transaction. However, this would be a behavioural change as non Hiero DApps and client that may expect it would be responsible for obtaining the initial transaction and doing the hash. if not included maybe this is a feature the MN or BN adds
• TransactionResult should be update to remove fields that are not common to all transaction types. Those fields should be moved to all the applicable transaction types

Copyright/license

This document is licensed under the Apache License, Version 2.0 – see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

Citation

Please cite this document as: