<
Back to arcticles
Interchain Privacy is Here
Penumbra has achieved another major milestone in our journey to mainnet. ThePenumbra testnet now integrates IBC and ICS-20 token transfers, bringing privacyto the Interchain: decentralized, permissionless token transfers from anyIBC-compatible chain directly into and out of Penumbra's multi-asset shieldedpool.
We believe that for privacy to succed, it must be useful. That's why we builtan advanced DEX from the future that allows users to tradeon-chain without leaking their alpha, protecting their trading strategy andhistory. But where do those assets come from? That's where Penumbra's IBCimplementation comes in: it connects the shielded pool and DEX to any IBC-compatible chain.
This functionality is live today, with an IBC channel established between thePenumbra and Osmosis testnets. For instance, thistransaction shields OSMO from the Osmosis testnet by movingit into Penumbra, and this transaction unlinkably transfersit back to a different Osmosis account. Last summer, we were the firstnon-Cosmos chain to open an IBC connection to the Cosmos Hub.This milestone is another first: the first Cosmos assets to be moved into ashielded pool over IBC.
In this post, we'll dig into how Penumbra's IBC functionality works, top-to-bottom: starting with a high-level overview of the user experience, and then working through a deep dive of the IBC stack and the challenges we overcame along the way.
Let's walk through the steps required for a user of an IBC chain to shield their funds on Penumbra.
That's it. There are no extra steps. The user's funds are now shielded, and willappear in their Penumbra wallet.
Unlike Zcash or Namada, where users have to think about whether their assets arerecorded privately, Penumbra delivers conceptual simplicity. Transferringtokens into Penumbra shields them. Transferring tokens out of Penumbra unshieldsthem. The entire Interchain is our transparent pool.
This user experience is made possible by deep protocol-level integration workthat ensures Penumbra integrates seamlessly with IBC. Let's look under the hoodto see how it works.
To initiate a cross-chain transfer into Penumbra, an IBC-connected counterpartychain will write an ICS-20 packet into its state. The ICS-20 packet describesinformation about the transfer: the sender address, the destination address, theamount, and the token denomination. This behavior is all standard, and requiresno special Penumbra support.
An IBC relayer observes this packet, and forwards it to the Penumbra chain,along with a proof that it was created by the counterparty chain. When thePenumbra chain validates and processes the packet, it mints a new notecontrolled by the destination address and inserts it directly into the shieldedpool. Penumbra's shielded pool is a next-generation iteration on the Zcashdesign, with support for arbitrary IBC assets, efficiency improvements, andmore.
The new note is then detected by the user's Penumbra wallet as it privatelyscans the chain, the same way it would detect any other shielded transfer totheir account. And no special handling on the client side is required for IBCassets, because Penumbra's shielded pool identifies assets by asset ID,designed to be computed directly from any ICS-20 denomination string, so IBCassets are handled just like any other asset.
The user can now transact privately on Penumbra, without publicly linking theirfuture activity to the original deposit. To transfer back out of Penumbra, theycreate a transaction that burns shielded assets on Penumbra and instructs thePenumbra chain to create an outgoing transfer packet addressed to thecounterparty chain.
Handling outbound transfers would seem to pose a problem. ICS-20 specifies thattransfer packets include the sender address. This is crucial from a safetyperspective, because it ensures that funds can be returned if the transfer timesout without being acknowledged by the other chain. But putting the same addressin inbound and outbound transfer packets would link them, destroying privacy.To handle this and similar use cases, Penumbra accounts are designed to supportrandomized ephemeral addresses, which are publicly unlinkable but privatelypoint to the same Penumbra account. Penumbra clients automatically generate anew ephemeral address when creating a withdrawal transaction, ensuring thatusers can recover funds from failed transfers without compromising privacy.
Crucially, however, users don't need to consider or even know about any of thesedetails. Because Penumbra's entire protocol and cryptography stack washolistically designed around this use case, we can provide a seamless userexperience without compromising on privacy or security.
Our strategy has been to start with one concrete use case, interchain privatetrading, and use it to deeply inform our protocol design. But this doesn't meanour ambition is limited -- it just means we think it's better to move from thespecific to the general, iterating on designs as they interact with concretetechnical requirements.
So while Penumbra will only support ordinary token transfers at launch, wehaven't stopped thinking about what's next. After building private ICS-20 fortoken transfers, private Interchain Accounts (ICA) seem like an interestingstepping stone towards Penumbra's endgame as a fully general interchain privacylayer.
Like our ICS-20 implementation, which doesn't require any special handling onthe counterparty chain, these interchain accounts would appear to thecounterparty chain like any other. However, they'd be controlled by a bearerNFT recorded in the Penumbra shielded pool, and funded via shielded transfers.This would allow Penumbra users to privately interact with public state on otherchains, before withdrawing back to the shielded pool.
If you think this could enable interesting use cases, let us know in Discord!
As one of the first non-Cosmos-SDK chains to build in the Cosmos ecosystem withTendermint/CometBFT, much of the required tooling simply didn't exist, and sowe've had to build a lot of the stack from the ground up, extracting reusablecode along the way, and our IBC stack is no exception.
An IBC implementation interfaces between the messages defined by interchainstandards (ICSs) and the host chain's state. It processes messages fromcounterparty chains, executes state changes, and writes new messages into thechain state. For this reason, while the abstract IBC protocol is generic, aspecific IBC implementation is fundamentally intertwined with the host chain'sstate and execution model.
Our IBC implementation has two parts:
ibc-types, a minimal, general-purpose Rust crate defining the foundational data types defined by the IBC protocol;penumbra-ibc / ibc-async, an asynchronous IBC implementation built on top of ibc-types.The ibc-types library is intended to provide a common language for workingwith IBC data, independent of the choices made by any specific implementation,similar to the way that tendermint-rs provides a common language for workingwith Tendermint/CometBFT data across the Rust IBC ecosystem.
The penumbra-ibc crate is our IBC implementation, designed around uniquefeatures of Penumbra's state and execution model.
Penumbra's state model (defined by our penumbra-storage crate) is built aroundasynchronous reads, to allow more efficient disk I/O. A part of the Penumbracode that needs multiple pieces of data from the chain state can read themconcurrently, rather than waiting for the first access to complete beforestarting the second. Writes are performed synchronously, in lightweightcopy-on-write snapshots, so all execution is simulated in a state fork and onlyapplied later.
Penumbra's execution model (defined by our penumbra-component crate, akin toSDK modules) splits processing into multiple phases: stateless checks (likeproof verification) that can be parallelized across transactions, statefulchecks that can be parallelized within a transaction, and execution, whichoperates serially.
Together, these design choices let us push most work out to the end-user device,parallelize as much verification as possible, and cut down dependency chains inthe critical path of the remaining on-chain execution work. Our IBCimplementation can take advantage of these properties and integrate cleanly withthe rest of our code.
To our great disappointment, however, we weren't able to reuse or share codewith other Rust IBC efforts, such as Informal Systems' ibc-rs. We originallybuilt the Penumbra IBC implementation using data types defined by ibc-rs.Later, the ibc-rs team started filling out a complete implementation of IBC,initially intended for Anoma. We weren't able to reuse it, because Penumbra'sstate is accessed asynchronously, and it went through significant churn on itsway to discovering the right abstractions for its initial use-case, just asPenumbra's execution model churned on our way to discovering the rightabstractions for our use case.
In our discussions with the ibc-rs team last year, we highlighted our viewthat the right move was to collaborate on a common set of domain types formodeling IBC messages, and work towards deeper integration as codebasesstabilize. Historically, this worked fine, because IBC messages themselves arespecified as part of the IBC protocol, and so are relatively insulated fromimplementation churn.
However, last February, the ibc-rs team let us know that they were planning toremove the IBC message types entirely, and shift ibc-rs from a general-purposelibrary to only provide an opaque IBC implementation. When we let them know thiswould break our implementation, they suggested forking the code, as they'd donewith Hermes, Informal's IBC relayer, when its needs also conflicted with theibc-rs direction. (Hermes now maintains its own entirely distinct library formodeling IBC messages, called ibc-relayer-types). As a result, we were forcedto fork when those changes landed and we no longer had an upstream.
Our hope is that going forward, we can reduce fragmentation and duplication ofeffort in the Rust IBC ecosystem, as we pull out minimally-scoped componentsbottom-up that can be reused by other projects. This strategy has worked in thepast, such as our domain types modeling the ABCI protocol used to communicatewith Tendermint (upstreamed into tendermint-rs), our ABCI serverimplementation (tower-abci, reused by Namada), our Jellyfish merkle treeimplementation (the jmt crate, now co-developed with Sovereign Labs), and ourasync storage and state management library (penumbra-storage, reused byAstria).
The first step is ibc-types, a minimally-opinionated set ofcrates for modeling IBC data. We'd be interested in having this code reused byeither ibc-rs or Hermes, were either of those teams so inclined. Longer-term,we plan to continue to extract our IBC implementation into a standalone,Penumbra-independent crate called ibc-async, as described in our IBC RescuePlan tracking issue.
One remaining speedbump relates to our merkle tree structure. Penumbra uses theJellyfish Merkle Tree, a sparse merkle tree originally designed forLibra/Diem. To implement IBC, we needed to define an ICS-23 proofspecification, to allow counterparty chains to understand our merkle proofs.ICS-23 is the Interchain Standard defining a generic merkle proof format, sothat different chains can use different merkle trees without each chain havingto specially implement verification for every kind of of merkle tree used by anycounterparty chain.
ICS-23 specifies both existence proofs (proving that some key has some value) and_non_existence proofs (proving that some key does not exist in the tree).Nonexistence proofs are critical to supporting packet timeouts, which enableusers to recover funds from failed transfers.
However, the original version of ICS-23 accidentally baked assumptions from theIAVL tree used by the Cosmos SDK into non-existence proof specification. InICS-23, a non-inclusion proof for key k is a pair of inclusion proofs for keysk1, k2, a check that k1 and k2 are adjacent in the tree, and a checkthat k1 < k < k2. The problem is that this mechanism only works on orderedtrees like the IAVL tree where the key ordering corresponds to the treestructure. On a sparse merkle tree like our JMT, this isn't the case -- sparsityis achieved by using the hash H(k) of the key, so keys are randomly ordered.
To fix this, we upstreamed a prehash_key_before_comparison field into theICS23 spec. This provides a backwards-compatible extension supporting trees likethe JMT: if the field is absent, it's false, and noninclusion proofs areverified as before; if it's present, it's true, and the noninclusion proofverifier checks that H(k1) and H(k2) are adjacent and that H(k1) < H(k) <H(k2)`.
This change was included in ICS-23 v0.10.0. Unfortunately, however, it isn'tforwards compatible, because the Cosmos SDK rejects proof specifications withunknown fields.
To ensure support for connections to Penumbra (and other chains with sparsemerkle trees), we're working to coordinate deployment of ICS-23 v0.10.0 acrossthe ecosystem. Any ibc-go version greater than v7.1.0 includes it already,and we've been working with the Interchain team to make backports available.Because only ics23 needs updating, it may be possible to just use a replacedirective in go.mod, without any other code changes.
In the meantime, we temporarily patched our IBC implementation to allowcounterparty chains to be created with a proof specification with an incorrectprehash_key_before_comparison field. This means timeouts won't work, butallows us to exercise all other IBC functionality against existing Cosmos-SDKtestnets.
Shielded IBC is live now! Check out the user guide to participate in thetestnet and test out interchain transfers using pcli.
Stay tuned for more updates on our pathway to mainnet. Slowly, and then all at once.
