Interchain Privacy is Here

Penumbra has achieved another major milestone in our journey to mainnet. The Penumbra testnet now integrates IBC and ICS-20 token transfers, bringing privacy to the Interchain: decentralized, permissionless token transfers from any IBC-compatible chain directly into and out of Penumbra's multi-asset shielded pool.

We believe that for privacy to succed, it must be useful. That's why we built an advanced DEX from the future that allows users to trade on-chain without leaking their alpha, protecting their trading strategy and history. But where do those assets come from? That's where Penumbra's IBC implementation comes in: it connects the shielded pool and DEX to any IBC-compatible chain.

This functionality is live today, with an IBC channel established between the Penumbra and Osmosis testnets. For instance, this transaction shields OSMO from the Osmosis testnet by moving it into Penumbra, and this transaction unlinkably transfers it back to a different Osmosis account. Last summer, we were the first non-Cosmos chain to open an IBC connection to the Cosmos Hub. This milestone is another first: the first Cosmos assets to be moved into a shielded pool over IBC.

In this post, we'll dig into how Penumbra's IBC functionality works, top-to-bottom: starting with a high-level overview of the user experience, and then working through a deep dive of the IBC stack and the challenges we overcame along the way.

Interchain Privacy

Let's walk through the steps required for a user of an IBC chain to shield their funds on Penumbra.

1. Generate a Penumbra address using a Penumbra-compatible wallet.
2. Send funds to that address, using a standard ICS-20 cross-chain transfer.

That's it. There are no extra steps. The user's funds are now shielded, and will appear in their Penumbra wallet.

Unlike Zcash or Namada, where users have to think about whether their assets are recorded privately, Penumbra delivers conceptual simplicity. Transferring tokens into Penumbra shields them. Transferring tokens out of Penumbra unshields them. The entire Interchain is our transparent pool.

This user experience is made possible by deep protocol-level integration work that ensures Penumbra integrates seamlessly with IBC. Let's look under the hood to see how it works.

Shielded ICS-20 Transfers

To initiate a cross-chain transfer into Penumbra, an IBC-connected counterparty chain will write an ICS-20 packet into its state. The ICS-20 packet describes information about the transfer: the sender address, the destination address, the amount, and the token denomination. This behavior is all standard, and requires no special Penumbra support.

An IBC relayer observes this packet, and forwards it to the Penumbra chain, along with a proof that it was created by the counterparty chain. When the Penumbra chain validates and processes the packet, it mints a new note controlled by the destination address and inserts it directly into the shielded pool. Penumbra's shielded pool is a next-generation iteration on the Zcash design, with support for arbitrary IBC assets, efficiency improvements, and more.

The new note is then detected by the user's Penumbra wallet as it privately scans the chain, the same way it would detect any other shielded transfer to their account. And no special handling on the client side is required for IBC assets, because Penumbra's shielded pool identifies assets by asset ID, designed to be computed directly from any ICS-20 denomination string, so IBC assets are handled just like any other asset.

The user can now transact privately on Penumbra, without publicly linking their future activity to the original deposit. To transfer back out of Penumbra, they create a transaction that burns shielded assets on Penumbra and instructs the Penumbra chain to create an outgoing transfer packet addressed to the counterparty chain.

Handling outbound transfers would seem to pose a problem. ICS-20 specifies that transfer packets include the sender address. This is crucial from a safety perspective, because it ensures that funds can be returned if the transfer times out without being acknowledged by the other chain. But putting the same address in inbound and outbound transfer packets would link them, destroying privacy. To handle this and similar use cases, Penumbra accounts are designed to support randomized ephemeral addresses, which are publicly unlinkable but privately point to the same Penumbra account. Penumbra clients automatically generate a new ephemeral address when creating a withdrawal transaction, ensuring that users can recover funds from failed transfers without compromising privacy.

Crucially, however, users don't need to consider or even know about any of these details. Because Penumbra's entire protocol and cryptography stack was holistically designed around this use case, we can provide a seamless user experience without compromising on privacy or security.

Privacy Beyond Transfers

Our strategy has been to start with one concrete use case, interchain private trading, and use it to deeply inform our protocol design. But this doesn't mean our ambition is limited -- it just means we think it's better to move from the specific to the general, iterating on designs as they interact with concrete technical requirements.

So while Penumbra will only support ordinary token transfers at launch, we haven't stopped thinking about what's next. After building private ICS-20 for token transfers, private Interchain Accounts (ICA) seem like an interesting stepping stone towards Penumbra's endgame as a fully general interchain privacy layer.

Like our ICS-20 implementation, which doesn't require any special handling on the counterparty chain, these interchain accounts would appear to the counterparty chain like any other. However, they'd be controlled by a bearer NFT recorded in the Penumbra shielded pool, and funded via shielded transfers. This would allow Penumbra users to privately interact with public state on other chains, before withdrawing back to the shielded pool.

If you think this could enable interesting use cases, let us know in Discord!

Penumbra's IBC stack

As one of the first non-Cosmos-SDK chains to build in the Cosmos ecosystem with Tendermint/CometBFT, much of the required tooling simply didn't exist, and so we've had to build a lot of the stack from the ground up, extracting reusable code along the way, and our IBC stack is no exception.

An IBC implementation interfaces between the messages defined by interchain standards (ICSs) and the host chain's state. It processes messages from counterparty chains, executes state changes, and writes new messages into the chain state. For this reason, while the abstract IBC protocol is generic, a specific IBC implementation is fundamentally intertwined with the host chain's state and execution model.

Our IBC implementation has two parts:

• ibc-types, a minimal, general-purpose Rust crate defining the foundational data types defined by the IBC protocol;
• penumbra-ibc / ibc-async, an asynchronous IBC implementation built on top of ibc-types.

The ibc-types library is intended to provide a common language for working with IBC data, independent of the choices made by any specific implementation, similar to the way that tendermint-rs provides a common language for working with Tendermint/CometBFT data across the Rust IBC ecosystem.

The penumbra-ibc crate is our IBC implementation, designed around unique features of Penumbra's state and execution model.

Penumbra's state model (defined by our penumbra-storage crate) is built around asynchronous reads, to allow more efficient disk I/O. A part of the Penumbra code that needs multiple pieces of data from the chain state can read them concurrently, rather than waiting for the first access to complete before starting the second. Writes are performed synchronously, in lightweight copy-on-write snapshots, so all execution is simulated in a state fork and only applied later.

Penumbra's execution model (defined by our penumbra-component crate, akin to SDK modules) splits processing into multiple phases: stateless checks (like proof verification) that can be parallelized across transactions, stateful checks that can be parallelized within a transaction, and execution, which operates serially.

Together, these design choices let us push most work out to the end-user device, parallelize as much verification as possible, and cut down dependency chains in the critical path of the remaining on-chain execution work. Our IBC implementation can take advantage of these properties and integrate cleanly with the rest of our code.

To our great disappointment, however, we weren't able to reuse or share code with other Rust IBC efforts, such as Informal Systems' ibc-rs. We originally built the Penumbra IBC implementation using data types defined by ibc-rs. Later, the ibc-rs team started filling out a complete implementation of IBC, initially intended for Anoma. We weren't able to reuse it, because Penumbra's state is accessed asynchronously, and it went through significant churn on its way to discovering the right abstractions for its initial use-case, just as Penumbra's execution model churned on our way to discovering the right abstractions for our use case.

In our discussions with the ibc-rs team last year, we highlighted our view that the right move was to collaborate on a common set of domain types for modeling IBC messages, and work towards deeper integration as codebases stabilize. Historically, this worked fine, because IBC messages themselves are specified as part of the IBC protocol, and so are relatively insulated from implementation churn.

However, last February, the ibc-rs team let us know that they were planning to remove the IBC message types entirely, and shift ibc-rs from a general-purpose library to only provide an opaque IBC implementation. When we let them know this would break our implementation, they suggested forking the code, as they'd done with Hermes, Informal's IBC relayer, when its needs also conflicted with the ibc-rs direction. (Hermes now maintains its own entirely distinct library for modeling IBC messages, called ibc-relayer-types). As a result, we were forced to fork when those changes landed and we no longer had an upstream.

Our hope is that going forward, we can reduce fragmentation and duplication of effort in the Rust IBC ecosystem, as we pull out minimally-scoped components bottom-up that can be reused by other projects. This strategy has worked in the past, such as our domain types modeling the ABCI protocol used to communicate with Tendermint (upstreamed into tendermint-rs), our ABCI server implementation (tower-abci, reused by Namada), our Jellyfish merkle tree implementation (the jmt crate, now co-developed with Sovereign Labs), and our async storage and state management library (penumbra-storage, reused by Astria).

The first step is ibc-types, a minimally-opinionated set of crates for modeling IBC data. We'd be interested in having this code reused by either ibc-rs or Hermes, were either of those teams so inclined. Longer-term, we plan to continue to extract our IBC implementation into a standalone, Penumbra-independent crate called ibc-async, as described in our IBC Rescue Plan tracking issue.

Coordinating ICS-23 Upgrades

One remaining speedbump relates to our merkle tree structure. Penumbra uses the Jellyfish Merkle Tree, a sparse merkle tree originally designed for Libra/Diem. To implement IBC, we needed to define an ICS-23 proof specification, to allow counterparty chains to understand our merkle proofs. ICS-23 is the Interchain Standard defining a generic merkle proof format, so that different chains can use different merkle trees without each chain having to specially implement verification for every kind of of merkle tree used by any counterparty chain.

ICS-23 specifies both existence proofs (proving that some key has some value) and _non_existence proofs (proving that some key does not exist in the tree). Nonexistence proofs are critical to supporting packet timeouts, which enable users to recover funds from failed transfers.

However, the original version of ICS-23 accidentally baked assumptions from the IAVL tree used by the Cosmos SDK into non-existence proof specification. In ICS-23, a non-inclusion proof for key k is a pair of inclusion proofs for keys k1, k2, a check that k1 and k2 are adjacent in the tree, and a check that k1 < k < k2. The problem is that this mechanism only works on ordered trees like the IAVL tree where the key ordering corresponds to the tree structure. On a sparse merkle tree like our JMT, this isn't the case -- sparsity is achieved by using the hash H(k) of the key, so keys are randomly ordered.

To fix this, we upstreamed a prehash_key_before_comparison field into the ICS23 spec. This provides a backwards-compatible extension supporting trees like the JMT: if the field is absent, it's false, and noninclusion proofs are verified as before; if it's present, it's true, and the noninclusion proof verifier checks that H(k1) and H(k2) are adjacent and that H(k1) < H(k) < H(k2)`.

This change was included in ICS-23 v0.10.0. Unfortunately, however, it isn't forwards compatible, because the Cosmos SDK rejects proof specifications with unknown fields.

To ensure support for connections to Penumbra (and other chains with sparse merkle trees), we're working to coordinate deployment of ICS-23 v0.10.0 across the ecosystem. Any ibc-go version greater than v7.1.0 includes it already, and we've been working with the Interchain team to make backports available. Because only ics23 needs updating, it may be possible to just use a replace directive in go.mod, without any other code changes.

In the meantime, we temporarily patched our IBC implementation to allow counterparty chains to be created with a proof specification with an incorrect prehash_key_before_comparison field. This means timeouts won't work, but allows us to exercise all other IBC functionality against existing Cosmos-SDK testnets.

Getting Started

Shielded IBC is live now! Check out the user guide to participate in the testnet and test out interchain transfers using pcli.

Stay tuned for more updates on our pathway to mainnet. Slowly, and then all at once.