HIP-1127: Unified transaction record format

Abstract

The Transaction protobuf message in the record stream supports two ways to represent a transaction.

1. The Transaction.signedTransactionBytes field.
2. Deprecated The Transaction.bodyBytes and Transaction.sigMap fields combination.

The definition of the transaction hash differs between the two representations. For (1), the transaction hash is the SHA-384 hash of the raw bytes in the signedTransactionBytes field. But for (2), the transaction hash is the SHA-384 hash of the protobuf serialization of the entire Transaction message; where this serialization must be done using ascending protobuf field order.

This HIP defines a single transaction representation for the block stream and preserves existing hash-based look-ups, so no client code changes are required.

This HIP does not propose any changes to the Hiero API (HAPI) or to the existing record stream. The consensus node will do all needed conversions from and to the supported forms of Transaction messages for the foreseeable future.

Motivation

Unnecessary nesting in the Transaction message bloats the block stream and increases the serialization work done by consumers. The SignedTransaction message is a better, cleaner candidate for representing transactions in the block stream. This message is,

message SignedTransaction {
    /**
     * A byte array containing a serialized `TransactionBody`.
     * <p>
     * This content is what the signatures in `sigMap` MUST sign.
     */
    bytes bodyBytes = 1;

    /**
     * A set of cryptographic signatures.
     * <p>
     * This set MUST contain all signatures required to authenticate
     * and authorize the transaction.<br/>
     * This set MAY contain additional signatures.
     */
    SignatureMap sigMap = 2;
}

Rationale

We could specify the block stream’s representation with a TransactionBody body field instead of the serialized bodyBytes. But we feel it is important to make the block stream definition explicitly encode the signed bytes, as this is such a fundamental part of the protocol.

We could also extend HAPI to let clients submit SignedTransactions instead of Transactions. This would be nice, but does not offer a compelling return on investment at the current stage of the project.

User stories

1. As a transaction sender, I want to be able to send transactions using both Transaction representations so that I am not forced to rewrite the calling code (which may be impossible).
2. As a record stream/block stream consumer, I want a simple way to parse transactions and extract data.
3. As a transaction signer, I want to be sure that my signatures can always be validated, even if transactions are modified to follow the normalized format.
4. As a user of APIs that take transaction hashes as input, I want block stream consumers to preserve the current transaction hash definitions for both Transaction representations; no matter what SDK is used to submit the transaction.

Specification

The protobuf definition of the Transaction object will not change. The SignedTransaction message will get one new field, a boolean that is true iff the transaction was originally submitted using the deprecated bodyBytes and sigMap combination,

message SignedTransaction {
    ...
    /**
     * If false then the hash of this transaction is the SHA-384 hash of the
     * serialization of this SignedTransaction message as it arrived on the wire. 
     * <p>
     * If true then the hash of this transaction is the SHA-384 hash of the
     * ascending field order serialization of the Transaction whose `bodyBytes`
     * and sigMap fields are deserialized from the contents of this message. 
     */
    boolean use_serialized_tx_message_hash_algorithm = 3;
}

The use_serialized_tx_message_hash_algorithm field can only be set by the network. If a HAPI client tries to set this field, its transaction will be rejected with status INVALID_SERIALIZED_TX_HASH_ALGORITHM at ingest.

Modification of incoming transactions

When a node receives a Transaction using the signedTransactionBytes field, it will submit those signedTransactionBytes contents directly into consensus gossip.

When a node receives a Transaction using the bodyBytes and sigMap combination, it will repackage those two fields inside a SignedTransaction along with use_serialized_tx_message_hash_algorithm=true, and submit the serialization of the repackaged message to consensus gossip.

Changes in record stream and block stream

We do not believe any Hedera users that will see a functional difference in the record stream. However, in the unlikely case that a network client is,

1. Submitting transactions using the deprecated bodyBytes and sigMap combination; and,
2. Serializing the Transaction wrapper using a non-ascending protobuf field order; and,
3. Scanning the record stream to find a Transaction with serialized bytes identical to their submission; then,

This client will need to switch to submitting transactions with the non-deprecated Transaction#signedTransactionBytes field, since a side effect of this HIP will be to re-serialize their submitted Transaction in ascending protobuf field order when it uses the deprecated Transaction fields.

In the block stream, the oneof item in the block stream’s BlockItem message will replace its EventTransaction choice with a bytes that is the serialization of a SignedTransaction.

message BlockItem {
    ...
    oneof item {
        ...
        /**
         * A single logical transaction in the block stream; the
         * serialization of a SignedTransaction message.
         * <p>
         * If this item is not followed by a `state_changes` item
         * before the appearance of the next `signed_transaction`,
         * it MUST be taken as purely informational; that is, as a
         * "synthetic" transaction whose only purpose is to increase
         * the legibility of the block stream semantics to consumers
         * that do not themselves simulate consensus node logic.
         */
        bytes signed_transaction = 4;
        ...
    }
}

Backwards Compatibility

Before this HIP, a user could always find the original serialized bytes of their submitted Transaction in the record stream, even if they used the deprecated bodyBytes and sigMap combination. This HIP will not preserve this property. Clients must use the long-standard Transaction#signedTransactionBytes field to have the exact bytes of their submission reproduced in the output streams.

Important: Even so, this HIP will not prevent a user from matching their deprecated submissions using non-ascending protobuf serialization order by TransactionID or hash.

Security Implications

This change has no known security implications.

How to Teach This

The block stream standardizes to a single transaction representation, the SignedTransaction. Users that look up transactions by hash can continue to do this as they do today.

Rejected Ideas

• Keep EventTransaction in the block stream. (The parsing and wrapper overhead was deemed excessive.)
• Use any of the other fields as the standard format.
• Introduce a new transaction type.

Copyright/license

This document is licensed under the Apache License, Version 2.0 – see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

Citation

Please cite this document as: