Penumbra Winter 2022 Update
Penumbra is building the first shielded DEX, allowing on-chain trading and marketmaking with private strategies. Since our summer update, we've continued to realize this vision, building and breaking in public, shipping weekly testnets, and growing our community. As 2022 draws to a close, here's a comprehensive update on what we've built since the summer, and what we're building next.
What we've built
Since our last comprehensive update this summer, we've made huge progress on a number of fronts:
We've run through two design iterations on Penumbra's shielded swap mechanism, which allows Penumbra users to privately swap tokens without leaving the shielded pool. On Penumbra, all swaps in each block are executed in a single batch, with a common clearing price, eliminating frontrunning by eliminating the entire concept of intra-block ordering.
Shielded swaps reveal an important technical challenge: the challenge of providing private interaction with public shared state. We want individual users' trades and account balances to stay private, while retaining a public view of the aggregate market state, like clearing prices, available liquidity, and trading volume. We think this is the key problem to solve to make blockchain privacy mainstream, and that focusing on solving one concrete use-case first will point the way to better solutions for blockchain privacy generally. For more on this aspect, check out Henry's talks at DevCon Bogota or at ZKSummit 8.
To solve it, we designed a new, asynchronous ZK execution model that separates
the private, per-user state from the public, aggregate state, and explicitly
models communication between the two using message-passing. Our insight was
that we could model
Promises) on-chain, pausing execution at
.await point by creating a SNARK-friendly commitment to all of the
intermediate execution state, and then resuming execution in a later transaction
by opening the commitment to the intermediate execution state.
For instance, in a swap, the initial transaction's
Swap action can't know the
clearing price of the batch, as it hasn't happened yet, so instead, it commits
to the intermediate execution state – the user's input amounts, trading pair,
and claim address – and once the batch prices are published, a followup
SwapClaim action can open that commitment, and privately mint
the user's pro-rata share of the batch, proving consistency between the public
prices and the private input data. Importantly, the followup transaction
doesn't require any additional signing, so it can be submitted automatically by
In September, we shipped the first version of ZSwap, our mechanism for sealed-bid batch swaps; our announcement post has details on our initial design. That initial design had some rough edges, however, and while thinking through some other aspects of the system design in November, we landed on a second design iteration which simplifies the design and fixes a security flaw. A roundup of the design simplification and bugfix are described in our announcement post for Testnet 38, which we shipped in December.
Transaction Plans, Perspectives, and Views
It should be as easy to interact with the shielded chain state on Penumbra as it is to interact with transparent chain state on a transparent chain. But solving on-chain privacy creates much more complex problems about legibility: now that transaction data is private by default, how do users understand it? On a transparent chain, everyone has complete visibility into every transaction, and this assumption permeates nearly every aspect of existing blockchain tooling. For instance, now that transactions are opaque, how does a user get the information required to decide whether to sign them? How does a user's wallet understand and show the effects of their transactions? How does it understand and show the effects of transactions where it has incomplete visibility?
We realized that to make it as easy to build interfaces and tooling for Penumbra as it is for transparent chains, we needed to address legibility of shielded transactions in a reusable, first-class way, solving the problem once, rather than forcing every interface or tool to build its own solution just to get started. So, we now specify data formats modeling the entire lifecycle of a shielded transaction, both before and after its creation.
Before a transaction is created, it's specified in a
contains a complete, plaintext description of all data that will be used to
build the transaction. This allows a custodian to review the exact effects of a
transaction before authorizing it, so Penumbra users can have confidence that
the transaction will do what they intended it to. And because the
TransactionPlan is specified as a Protobuf, it can be
serialized, sent across the network, and parsed by tooling in any language.
TransactionPlan is assembled into a
Transaction, the cleartext data
in the plan is turned into encrypted payload data, or omitted entirely and
replaced by opaque commitments. Different parties may have different levels of
visibility into the transaction's encrypted payloads. For instance, in a
payment, the sender can view the entire transaction contents, while the receiver
can only view the encrypted memo and the outputs sent to them, but not the
sender's spends and change outputs.
We capture the notion that there are different vantage points on the same
transaction data with a
generated by viewing a
Transaction from some particular
TransactionView models a decrypted, interpreted view of a transaction, and
is intended to be consumed by ecosystem tools and interfaces. For instance,
Spend action of a
Transaction only has opaque commitment data and
ZK proofs, the corresponding
SpendView in a
TransactionView is an enum,
SpendView::Visible with the plaintext note that
was spent. This means that rather than requiring every ecosystem tool to
implement its own logic to decrypt and understand the effects of shielded
transactions, tools can consume
TransactionViews, without having to support
Penumbra-specific cryptography. And, like the
TransactionView is also specified as a Protobuf, so it can be
serialized and parsed by tooling in any language.
TransactionPerspective is a bundle of per-transaction key
material and commitment openings, generated by applying an account's long-term
full viewing key to the transaction data. The transaction perspective can be
thought of as a per-transaction viewing key, but we decided to avoid using that
term, because referring to a "transaction viewing key" could give the impression
that each transaction has a unique viewing key, rather than multiple possible
Beyond developer tooling and ergonomics, this design also improves Penumbra's security and auditability properties. Because the transaction perspective is a self-contained, serializable object, we now have the ability to selectively disclose information about specific transactions, rather than just disclosing an account's long-term full viewing keys, which give access to all past and future activity. This gives users the ability to disclose fine-grained selections from their transaction graph for accounting, auditing, or compliance purposes. It also means that we can design wallets or frontend interfaces that don't require access to long-term key material and allow users to revoke access.
Improved State and Execution Model
The chain state is the backbone of the Penumbra protocol: it's what the network comes to consensus on, it's how the network records data, it's how nodes implement application logic and thus intertwined with the chain's execution model, and so on. So it's critical to get right. As one of the first projects building a Tendermint chain in pure Rust, we didn't have an existing state layer and application framework to use, which has been both a curse and a blessing: a curse, because designing and building one is a lot of work, and a blessing, because the one we've ended up building is extremely powerful.
Our first iteration was Postgres-based, and only useful as an MVP. Our second iteration moved all of the chain state into a Jellyfish Merkle Tree forked from the Diem project, and used it to build a component-based node framework. This worked well, but further iteration revealed some major limitations. This fall, we addressed those limitations by landing the third iteration of Penumbra's state model, which we think is close to its final form, and will be generally useful for Rust-based blockchains other than Penumbra.
Penumbra's new storage system provides a versioned,
verifiable key-value store with lightweight, copy-on-write forks. A single
Storage instance records a sequence of versioned
States, each a lightweight
snapshot of a particular version of the chain state. Each
State instance can
also be used as a copy-on-write fork to build up changes to the chain state
before committing them to persistent storage, and those changes to the
can be built up in transactionally-applied groups.
State consists of two data stores:
- A verifiable key-value store, with UTF-8 keys and byte values, backed by the Jellyfish Merkle Tree. The JMT is a sparse merkle tree that records hashed keys, so we also record an index of the keys themselves to allow range queries on keys rather than key hashes. This index, however, is not part of the verifiable consensus state.
- A secondary, non-verifiable key-value store with byte keys and byte values, backed directly by RocksDB. This is intended for use building application-specific indexes of the verifiable consensus state – and in particular, indexes of liquidity positions used to accelerate our DEX engine.
penumbra-storage crate provides a generic, byte-oriented key-value store.
To use it, we layer on behavior with extension traits in our
crate that transform it into an object store with Protobuf-encoded data. This
means that every piece of data in the Penumbra chain state has a documented and
standardized schema, because the chain state is a kind of public API, and
should be treated as such. As a nice side effect, this design means that code
inspecting the Penumbra chain state can be written in any language with a
State snapshot is independent of all others, there's no need for
a global state mutex, as in the Cosmos SDK. For instance,
pd creates a new
State snapshot at the beginning of processing each RPC request, allowing that
request to be processed against a consistent version of the chain state without
requiring synchronization with the consensus worker. And, because those
snapshots can be mutated without committing them to the chain state, it's easy
to implement extremely powerful simulation capability. For instance, a fullnode
could run simulations of different batch trades against the current on-chain
liquidity, processing each member of the ensemble in parallel.
Our execution model also takes advantage of these capabilities. On Penumbra,
transaction verification and execution is modeled with the
trait and splits into three phases:
check_stateless, which performs all stateless checks, like signature or proof verification;
check_stateful, which performs stateful checks against the chain state prior to execution, like consistency checks between the public inputs to proofs and the chain state;
execute, which performs checks while writing changes to the chain state.
All three phases must succeed for a transaction's changes to be applied to the
state, but segmenting them allows the implementation to maximize parallelism and
check_stateless calls within a block can be run concurrently,
check_stateful calls within a transaction can be run concurrently,
leaving only the writes in
execute on the critical path. This means that the
bulk of the compute-intensive processing (in
check_stateless) and the bulk of
the state reads (in
check_stateful) can be run in parallel, using Tokio to
schedule the task graph over the available execution resources.
This architecture is still relatively Penumbra-specific, since our efforts are
focused on getting Penumbra to mainnet, but we expect that by the time Penumbra
is ready, we'll be able to extract it into a general-purpose application
framework for building blockchains in Rust. Already, the
crate is free of Penumbra-specific dependencies and is intended for general use.
Client Sync Improvements
Penumbra clients need to synchronize with the network to learn about their state and create transactions -- unlike a transparent chain, they can't just ask a fullnode for their account balance, because a fullnode doesn't have that data, and therefore can't leak it. To handle this, Penumbra includes a client protocol that allows clients to synchronize their local state with the chain. Making this protocol simple and efficient translates directly into faster and better client experiences.
In our summer update, we described optimization work we did to accelerate the sync process, and dove into more detail in our post on our Tiered Commitment Tree. However, while we'd made synchronization fast, some accumulated mistakes from previous design iterations meant it was more complex than it needed to be, making it more difficult to write clients for Penumbra. Meanwhile, further design iterations on other parts of the system gave us new insight on how we could simplify it. We combined these ideas into improvements to the unbonding mechanism and to the swap claim mechanism, and shipped them in December; full details are written up in the announcement post for Testnet 38.
Protocol Governance MVP
In August, we shipped an MVP of Penumbra's governance component, with support for validator voting. Penumbra's governance component is similar to the Cosmos SDK's governance module, with the biggest difference being that because Penumbra provides delegation privacy, on Penumbra, individual delegators' votes are private, while validator votes are transparent. This provides privacy for delegators and accountability for validators, who cast default votes on behalf of their delegation pool (as in the Cosmos SDK).
We also make a few other changes, based on lessons learned from the Cosmos ecosystem. For instance, validator votes are signed not with the validator's long-term identity key, but with a governance-specific subkey. This allows validators to participate in governance without having to constantly pull their long-term key material out of cold storage, or even to delegate their participation to other entities, opening up interesting possibilities (e.g., a validator that specializes in operating infrastructure could delegate their governance voting to a DAO).
ZK Proof Implementation
In order to iterate on Penumbra's cryptography and system design, we've built our current testnets using mock "transparent proofs". Everywhere we intend to have a ZK proof, we instead encode the witness data transparently as a mock proof, and check the proof statements directly in software. This gives us an opaque blob of data with the exact same verification interface as a real ZK proof, but allowed us to iterate much more quickly than we could if we were doing circuit programming.
This fall, we started the process of replacing our mock proofs with ZK proofs,
and wrote circuit implementations of our
Output proofs, which we
expect to integrate into testnets in the new year.
Tendermint 0.35/0.34 Migration
Last year, we started building Penumbra against Tendermint 0.35, which at the time was newly released. However, our testnets, as well as those of Celestia and Vega, revealed that the 0.35 release of Tendermint had serious stability problems when deployed in real network environments. At one point during the summer, for instance, we were unable to keep Penumbra testnets running for more than a day or two, not because of any problem with the Penumbra application, but because validators would mysteriously be unable to communicate with each other, causing chain halts unless we ensured that a single validator had a supermajority of voting power (allowing it to continue producing blocks even when unable to communicate with any other validators).
As a result of these stability issues, the Tendermint 0.35 release was discontinued by the Tendermint team, along with the unreleased work on Tendermint 0.36. Instead, Tendermint 0.37 will be based on the stable 0.34 code currently used in production.
Migration from 0.35 to 0.34 was a somewhat painful process. We interface to
Tendermint via ABCI, and in particular through the ABCI domain types we
contributed upstream to the
tendermint-rs crate. However, because we had
written that code against the 0.35 branch of
tendermint-rs, which had to be
rolled back, we had to backport the interface definitions, and then propagate
those changes through our dependency tree, including
ibc-rs. However, in
collaboration with the Informal Systems team who maintain the
ibc-rs crates, we were able to get through it, and have had no network
stability issues since migrating to Tendermint 0.34.
Unfortunately, this change means that ABCI++, the upgraded interface to Tendermint that allows application logic to hook into consensus and implement advanced features like threshold cryptography, will not ship in its full form until Tendermint 0.38 at the earliest. In addition, based on our experiences so far, we're hesitant to rely on features that affect the network layer (like vote extensions, which allow validators to gossip threshold shares) without testing them in real-world conditions.
These factors led us to decide to defer flow encryption to a future network upgrade, rather than including it in the initial mainnet launch scope. This means that Penumbra won't conceal users' contributions to batch totals at launch. While this is unfortunate, the protocol is still designed to enable encrypting them in the future, and we feel that even without flow encryption, Penumbra's privacy guarantees are still a huge advance over the status quo.
Better Tooling, Deployments and Integration Testing
Finally, we also took the time to invest in improved tooling. We're now using
Buf to host our Protobuf definitions, and auto-generate Go and
Typescript packages that can work with any Penumbra data types or communicate
pd's RPC. We added native
grpc-web support to
pd, so every fullnode
can serve GRPC to web clients without requiring a proxy, and started working on
integrating automatic HTTPS support so every fullnode can serve RPC requests
securely without any extra configuration beyond a domain name. Finally, we
added a GRPC proxy to
pd that provides access to the Tendermint RPC, so that
Penumbra clients only need to speak to one RPC endpoint.
In collaboration with the Strangelove team, we also built Kubernetes-based testnet deployment infrastructure, which we'll be using for testnet deployments going forward, and started work on integrating Penumbra into their IBC test framework to do automatic conformance testing between Penumbra and Cosmos SDK testnets.
What we're building next
We're aiming for a mainnet launch next year, and closed out 2022 by planning the major components left to build before then. We'll be diving into more detail on each of these projects in future posts, but for now, here are brief summaries:
Penumbra DEX Engine
The last major component of the Penumbra ledger is the DEX engine, which executes batch swaps against the active liquidity. Penumbra's batched execution model has interesting implications for the DEX engine: because the DEX only runs executes once per block, not once per transaction, we can afford to make the DEX engine considerably more sophisticated, as the execution cost is amortized over all transactions in the block.
Penumbra uses a hybrid, order-book-like AMM with automatic routing. Liquidity on Penumbra is recorded as many individual concentrated liquidity positions, akin to an order book. Each liquidity position is its own AMM, with its own fee tier, and that AMM has the simplest possible form, a constant-sum (fixed-price) market maker. These component AMMs are synthesized into a global AMM by the DEX engine, which optimally routes trades across the entire liquidity graph.
Penumbra has no intra-block trade ordering, so DEX execution operates at the end of the block in four phases:
- All newly opened liquidity positions are added to the active set.
- Trades are batched by liquidity pair and executed against the active liquidity.
- The chain arbitrages all active positions against each other and burns the arbitrage profits.
- All newly closed positions are removed from the active set.
In this model, intra-chain arbitrage (making prices consistent within the chain) is performed automatically by the chain, while inter-chain arbitrage (making prices on Penumbra consistent with other markets) is performed by arbitrageurs. This arbitrage game has interesting pro-rata dynamics, which you can read about in this research paper we collaborated on with the Bain Capital Crypto team.
It has other useful properties: market-makers can create fill-or-kill positions
with prices valid for exactly one block without having to compete for ordering
within that block, by opening and then closing a liquidity position in the same
transaction. Or, traders can execute on the market-maker side, by creating a
fill-or-kill position whose price crosses the spread, and waiting for the chain
to fill it through arbitrage. Because trades are optimally routed, the chain
can compose cheap stableswaps between different IBC paths on either end of a
trade, making available liquidity independent of the specific bridge path of an
asset. Finally, passive market-makers can replicate the payoff curves of other
AMM trading functions, like UniV2's
xy=k, though they may be outcompeted by
Our new state model is key to implementing this design efficiently, as it allows us to maintain application-specific indexes (like the liquidity positions on each trading pair) with transactional, copy-on-write semantics, so we can speculatively mutate state during routing and choose whether or not to apply the results.
We've been laying the groundwork for web interfaces to Penumbra, with our
modular client design, the
TransactionView design, and the Protobuf tooling that
allows accessing Penumbra data structures from Typescript. We've identified a
few key goals for interfaces to Penumbra:
We want to be able to support a wide variety of interfaces to Penumbra.
While it’s important that there be at least one first-class wallet experience at launch, Penumbra’s capabilities are multifaceted, and users will probably be best served by specialized interfaces: e.g., one for basic transfers or swaps, one for governance, one for power users to manage liquidity or examine the liquidity graph, etc.
Supporting a wide variety of interfaces is also important for decentralization: no one entity should “own” the Penumbra userbase via control of a single frontend, or be at risk of becoming a chokepoint for control of those users. This also means that it should be possible to build frontend interfaces to Penumbra that do not require custom backend infrastructure beyond an ordinary
pd full node.
We want those interfaces to be as easy to build as interfaces to transparent chains.
Historically, interfaces to shielded chains have been more difficult to build than interfaces to transparent chains, because they require the application developer to manage synchronization of users’ private state, unlike a transparent chain, where user state is accessible via RPC. Penumbra is about privacy without compromise, so we want to make it as easy to build those interfaces for Penumbra as it is for a shielded chain.
We want our users to have security when interacting with those interfaces.
In order for users to benefit from the availability of a wide variety of third-party interfaces, users need to be confident they can use them without risk of losing funds or being hacked. We need to ensure that only the user can authorize a transaction, and be able to understand exactly what actions they’re authorizing when they do so.
To accomplish these goals, we're building implementations of our view and custody services, and bundling them into a browser extension that can communicate with web content. All long-term key material is kept inside of the browser extension, so web content only has access to per-transaction data, and all transactions must be approved through the browser extension, which provides a secure display path for the user to review a proposed transaction. This allows a user to selectively grant viewing or spending permissions to web interfaces to relatively untrusted web interfaces, and to revoke those permissions later. The resulting architecture is shown below:
┌───────────┐ │ Extension │ ╭ │ ┌───────┐ │ custody spending │ │ │custody│ │ protocol capability │ │ │service│◀┼─────────────────┐ ╰ │ └───────┘ │ │ │ │ │ │ │ │ │ │ │ ╭ │ ┌───────┐ │ view │ full viewing │ │ │view │ │ protocol │ capability │ │ │service│◀┼──────┐ │ ╰ │ └───────┘ │ │ │ │ ▲ │ │ ┌───┼────────────┐ └───┼───────┘ │ │ │ Web Content│ │ │ │ ▼ │ ╭ │ │ │ ┌───────┐ │ transaction │ │ └──────┼▶│wallet │ │ perspectives │ │ │ │ logic │ │ ╰ │ │ └───────┘ │ │ │ ▲ │ │ │ specific/oblivious └───┼─┼──────────┘ │ client protocols │ │ ├─────────────────────────┘ │ tx │┌──────────────────────────┘ broadcast .───. ││ ,' `. ╭ ┌───┼┼─────────────────────────────────────┐ .───; : public │ │ ││ Penumbra Fullnode│ ; │ chain │ │ ││ grpc/grpc-web │ .─┤ ├──. data │ │ ▼▼ │ ,' `. │ │ ┌────┐ tm rpc proxy ┌──────────┐ │ ; Penumbra : │ │ │ │◀────────────────────▶│ │ │ : ┌───────────▶ Network ; │ │ │ pd │◀────────────────────▶│tendermint│◀┼─────────┘ ╱ │ │ └────┘ abci app └──────────┘ │ `. `. `. ,' ╰ └──────────────────────────────────────────┘ `───' `───' `───'
We're working with the Zpoken team to build this system and feed back improvements to Penumbra's RPCs along the way, and we're excited to be able to show the results when they're ready.
Narsil: Sharded Custody and Personal Rollup
Because Penumbra uses custom cryptography to achieve privacy, we can't easily make use of existing custody tooling like HSMs or hardware wallets without extending them to add support for our cryptography. However, secure custody is a critical operational requirement. Our plan to address it is to build a threshold signing implementation using FROST, providing off-chain multisignatures indistinguishable from any other transactions.
This tool will be called Narsil, and it will act as a custodian and audit log for a single Penumbra account, sharing control of that account's spending authority across multiple Narsil shards.
While FROST solves the cryptographic part of threshold signatures, it doesn't solve the engineering problem of building a way for the participants to reliably communicate with each other and agree on what messages they should be signing. That problem is also subtle and difficult to solve, but we realized we already have a tool that allows a set of participants to come to consensus on what messages have occurred: Tendermint consensus.
Using Tendermint internally gives us fault-tolerant state replication with an off-the-shelf tool, but it also has another important advantage. Because the Narsil cluster has fault-tolerant replication of the audit log of every transaction the account has authorized, by implication, it has fault-tolerant replication of the entire account state. This means that users can choose to maintain their account state off-chain, using their Narsil cluster as a personal rollup, and only post state commitments, nullifiers, and proofs to the L1.
This seems particularly useful for active market makers, who can cheaply perform frequent updates to their own state (e.g., updating their liquidity positions potentially every block), while saving on gas and saving other users from scanning irrelevant state updates.
We're excited about these possibilities and we'll have more details to share on the Narsil design in the new year. For now, here's a rough idea of how Narsil will be structured, and how it integrates with other Penumbra software:
┌────────────────────────┐ │ pcli (or other client) │ └────────────────────────┘ ▲ ▲ │ custody │ view │ protocol │ protocol ▼ ▼ ┌──────┬────────┬──────┬────────┬───────────────────────┐ ┌────────┐ │ │Custody │ │ View │ client protocol │ │Penumbra│ │ │Service │ │Service │◀──────────────────────┼─────│Fullnode│ │ └────────┘ └────────┘ │ └────────┘ │ │ ▲ ▲ │ │ │ │ View │ │ │ Auth │ │ Auth Advice │ absent for │ │ Request │ │ Data │ replicas │ │ │ │ ┌────────┐ ┌ ─ ─ ─ ─ ─ │ │ │ │ │ Narsil │ FROST Narsil │ │ │ │ └──────────│ Ledger │─────────▶│ Shard │ │ │ └────────┘ ─ ─ ─ ─ ─ ┘ │ │ │ ▲ FROST │ │ │ └─────────────┐ │ ┌─────────────────┘ │ │ │ ABCI │ │ │narsild │ │ │ │ └───────────────────────┼───┼──┼────────────────────────┘ │ │ │ ▼ │ ▼ ┌──────────┐ │Tendermint│ └──────────┘
Our current weekly testnets start from a new genesis state, and don't preserve any history, which has allowed us to iterate quickly. But before we go to mainnet, we need to build and exercise a real chain upgrade path. Upgrades to a shielded chain like Penumbra are tricky, because unlike a transparent chain, where all state is on-chain and can be snapshotted and migrated, state on Penumbra splits into two categories: public state, recorded on-chain, and private state, recorded off-chain.
While we can perform migrations on the public state, changes to the private state must be accretive, preserving all existing private state and allowing users to roll over their data on their own schedule. We're working on the design of this upgrade mechanism, and plan to exercise it on our testnets prior to mainnet.
We've implemented ZK circuits for two proof statements used in Penumbra. In the new year, we'll be integrating those ZK proofs into the rest of the system, replacing the other mock proofs with ZK implementations, now that the system design has matured.
Refinement, Testing, and Assurance
Finally, after finishing the remaining on-chain implementation tasks, we'll be changing gears from an expansive, iterative process of building out the system, exploring the design space, and moving as quickly as possible, to a process of careful refinement of the system we've built, cleaning up our code to make it easier to audit and understand, focusing on testing and assurance, and carefully checking every detail before we're ready to go to mainnet.
To stay up to date with the latest progress along the way, follow us on Twitter, join our Discord and subscribe to the
#announcements channel, and check out our weekly testnets!