<
Back to arcticles
Penumbra Winter 2022 Update
Penumbra is building the first shielded DEX, allowing on-chain trading andmarketmaking with private strategies. Since our summer update,we've continued to realize this vision, building and breaking in public,shipping weekly testnets, and growing our community. As 2022 drawsto a close, here's a comprehensive update on what we've built since the summer,and what we're building next.
Since our last comprehensive update this summer, we've madehuge progress on a number of fronts:
We've run through two design iterations on Penumbra's shielded swap mechanism,which allows Penumbra users to privately swap tokens without leaving theshielded pool. On Penumbra, all swaps in each block are executed in a singlebatch, with a common clearing price, eliminating frontrunning by eliminating theentire concept of intra-block ordering.
Shielded swaps reveal an important technical challenge: the challenge ofproviding private interaction with public shared state. We want individualusers' trades and account balances to stay private, while retaining a publicview of the aggregate market state, like clearing prices, available liquidity,and trading volume. We think this is the key problem to solve to makeblockchain privacy mainstream, and that focusing on solving one concreteuse-case first will point the way to better solutions for blockchain privacygenerally. For more on this aspect, check out Henry's talks at DevConBogota or at ZKSummit 8.
To solve it, we designed a new, asynchronous ZK execution model that separatesthe private, per-user state from the public, aggregate state, and explicitlymodels communication between the two using message-passing. Our insight wasthat we could model Futures (or Promises) on-chain, pausing execution ateach .await point by creating a SNARK-friendly commitment to all of theintermediate execution state, and then resuming execution in a later transactionby opening the commitment to the intermediate execution state.
For instance, in a swap, the initial transaction's Swap action can't know theclearing price of the batch, as it hasn't happened yet, so instead, it commitsto the intermediate execution state – the user's input amounts, trading pair,and claim address – and once the batch prices are published, a followuptransaction's SwapClaim action can open that commitment, and privately mintthe user's pro-rata share of the batch, proving consistency between the publicprices and the private input data. Importantly, the followup transactiondoesn't require any additional signing, so it can be submitted automatically bya wallet.
In September, we shipped the first version of ZSwap, our mechanism forsealed-bid batch swaps; our announcement post has details onour initial design. That initial design had some rough edges, however, andwhile thinking through some other aspects of the system design in November, welanded on a second design iteration which simplifies the design and fixes asecurity flaw. A roundup of the design simplification and bugfix are describedin our announcement post for Testnet 38, which we shipped inDecember.
It should be as easy to interact with the shielded chain state on Penumbra as itis to interact with transparent chain state on a transparent chain. But solvingon-chain privacy creates much more complex problems about legibility: now thattransaction data is private by default, how do users understand it? On atransparent chain, everyone has complete visibility into every transaction, andthis assumption permeates nearly every aspect of existing blockchain tooling.For instance, now that transactions are opaque, how does a user get theinformation required to decide whether to sign them? How does a user's walletunderstand and show the effects of their transactions? How does it understandand show the effects of transactions where it has incomplete visibility?
We realized that to make it as easy to build interfaces and tooling for Penumbraas it is for transparent chains, we needed to address legibility of shieldedtransactions in a reusable, first-class way, solving the problem once, ratherthan forcing every interface or tool to build its own solution just to getstarted. So, we now specify data formats modeling the entire lifecycle of ashielded transaction, both before and after its creation.
Before a transaction is created, it's specified in a TransactionPlan, whichcontains a complete, plaintext description of all data that will be used tobuild the transaction. This allows a custodian to review the exact effects of atransaction before authorizing it, so Penumbra users can have confidence thatthe transaction will do what they intended it to. And because theTransactionPlan is specified as a Protobuf, it can beserialized, sent across the network, and parsed by tooling in any language.
Once the TransactionPlan is assembled into a Transaction, the cleartext datain the plan is turned into encrypted payload data, or omitted entirely andreplaced by opaque commitments. Different parties may have different levels ofvisibility into the transaction's encrypted payloads. For instance, in apayment, the sender can view the entire transaction contents, while the receivercan only view the encrypted memo and the outputs sent to them, but not thesender's spends and change outputs.
We capture the notion that there are different vantage points on the sametransaction data with a TransactionPerspective: each TransactionView isgenerated by viewing a Transaction from some particularTransactionPerspective.
The TransactionView models a decrypted, interpreted view of a transaction, andis intended to be consumed by ecosystem tools and interfaces. For instance,while the Spend action of a Transaction only has opaque commitment data andZK proofs, the corresponding SpendView in a TransactionView is an enum,either SpendView::Opaque, or SpendView::Visible with the plaintext note thatwas spent. This means that rather than requiring every ecosystem tool toimplement its own logic to decrypt and understand the effects of shieldedtransactions, tools can consume TransactionViews, without having to supportPenumbra-specific cryptography. And, like the TransactionPlan, theTransactionView is also specified as a Protobuf, so it can beserialized and parsed by tooling in any language.
Internally, a TransactionPerspective is a bundle of per-transaction keymaterial and commitment openings, generated by applying an account's long-termfull viewing key to the transaction data. The transaction perspective can bethought of as a per-transaction viewing key, but we decided to avoid using thatterm, because referring to a "transaction viewing key" could give the impressionthat each transaction has a unique viewing key, rather than multiple possibleperspectives.
Beyond developer tooling and ergonomics, this design also improves Penumbra'ssecurity and auditability properties. Because the transaction perspective is aself-contained, serializable object, we now have the ability to selectivelydisclose information about specific transactions, rather than just disclosing anaccount's long-term full viewing keys, which give access to all past and futureactivity. This gives users the ability to disclose fine-grained selections fromtheir transaction graph for accounting, auditing, or compliance purposes. Italso means that we can design wallets or frontend interfaces that don't requireaccess to long-term key material and allow users to revoke access.
The chain state is the backbone of the Penumbra protocol: it's what the networkcomes to consensus on, it's how the network records data, it's how nodesimplement application logic and thus intertwined with the chain's executionmodel, and so on. So it's critical to get right. As one of the first projectsbuilding a Tendermint chain in pure Rust, we didn't have an existing state layerand application framework to use, which has been both a curse and a blessing: acurse, because designing and building one is a lot of work, and a blessing,because the one we've ended up building is extremely powerful.
Our first iteration was Postgres-based, and only useful as an MVP. Our seconditeration moved all of the chain state into a Jellyfish MerkleTree forked from the Diem project, and used it to build acomponent-based node framework. This worked well,but further iteration revealed some major limitations. This fall, weaddressed those limitations by landing the third iteration of Penumbra's statemodel, which we think is close to its final form, and will be generally usefulfor Rust-based blockchains other than Penumbra.
Penumbra's new storage system provides a versioned,verifiable key-value store with lightweight, copy-on-write forks. A singleStorage instance records a sequence of versioned States, each a lightweightsnapshot of a particular version of the chain state. Each State instance canalso be used as a copy-on-write fork to build up changes to the chain statebefore committing them to persistent storage, and those changes to the Statecan be built up in transactionally-applied groups.
Each State consists of two data stores:
The penumbra-storage crate provides a generic, byte-oriented key-value store.To use it, we layer on behavior with extension traits in our penumbra-protocrate that transform it into an object store with Protobuf-encoded data. Thismeans that every piece of data in the Penumbra chain state has a documented andstandardized schema, because the chain state is a kind of public API, andshould be treated as such. As a nice side effect, this design means that codeinspecting the Penumbra chain state can be written in any language with aProtobuf compiler.
Because each State snapshot is independent of all others, there's no need fora global state mutex, as in the Cosmos SDK. For instance, pd creates a newState snapshot at the beginning of processing each RPC request, allowing thatrequest to be processed against a consistent version of the chain state withoutrequiring synchronization with the consensus worker. And, because thosesnapshots can be mutated without committing them to the chain state, it's easyto implement extremely powerful simulation capability. For instance, a fullnodecould run simulations of different batch trades against the current on-chainliquidity, processing each member of the ensemble in parallel.
Our execution model also takes advantage of these capabilities. On Penumbra,transaction verification and execution is modeled with the ActionHandlertrait and splits into three phases:
check_stateless, which performs all stateless checks, like signature or proof verification;check_stateful, which performs stateful checks against the chain state prior to execution, like consistency checks between the public inputs to proofs and the chain state;execute, which performs checks while writing changes to the chain state.All three phases must succeed for a transaction's changes to be applied to thestate, but segmenting them allows the implementation to maximize parallelism andconcurrency: all check_stateless calls within a block can be run concurrently,and all check_stateful calls within a transaction can be run concurrently,leaving only the writes in execute on the critical path. This means that thebulk of the compute-intensive processing (in check_stateless) and the bulk ofthe state reads (in check_stateful) can be run in parallel, using Tokio toschedule the task graph over the available execution resources.
This architecture is still relatively Penumbra-specific, since our efforts arefocused on getting Penumbra to mainnet, but we expect that by the time Penumbrais ready, we'll be able to extract it into a general-purpose applicationframework for building blockchains in Rust. Already, the penumbra-storagecrate is free of Penumbra-specific dependencies and is intended for general use.
Penumbra clients need to synchronize with the network to learn about their stateand create transactions -- unlike a transparent chain, they can't just ask afullnode for their account balance, because a fullnode doesn't have that data,and therefore can't leak it. To handle this, Penumbra includes a clientprotocol that allows clients to synchronize their local state with the chain.Making this protocol simple and efficient translates directly into faster andbetter client experiences.
In our summer update, we described optimization work wedid to accelerate the sync process, and dove into more detail in our post on ourTiered Commitment Tree. However, while we'd made synchronizationfast, some accumulated mistakes from previous design iterations meant it wasmore complex than it needed to be, making it more difficult to write clients forPenumbra. Meanwhile, further design iterations on other parts of the systemgave us new insight on how we could simplify it. We combined these ideas intoimprovements to the unbonding mechanism and to the swap claim mechanism, andshipped them in December; full details are written up in theannouncement post for Testnet 38.
In August, we shipped an MVP of Penumbra's governance component, with supportfor validator voting. Penumbra's governance component is similar to the CosmosSDK's governance module, with the biggest difference being that because Penumbraprovides delegation privacy, on Penumbra, individual delegators' votes areprivate, while validator votes are transparent. This provides privacy fordelegators and accountability for validators, who cast default votes on behalfof their delegation pool (as in the Cosmos SDK).
We also make a few other changes, based on lessons learned from the Cosmosecosystem. For instance, validator votes are signed not with the validator'slong-term identity key, but with a governance-specific subkey. This allowsvalidators to participate in governance without having to constantly pull theirlong-term key material out of cold storage, or even to delegate theirparticipation to other entities, opening up interesting possibilities (e.g., avalidator that specializes in operating infrastructure could delegate theirgovernance voting to a DAO).
In order to iterate on Penumbra's cryptography and system design, we've builtour current testnets using mock "transparent proofs". Everywhere we intend tohave a ZK proof, we instead encode the witness data transparently as a mockproof, and check the proof statements directly in software. This gives us anopaque blob of data with the exact same verification interface as a real ZKproof, but allowed us to iterate much more quickly than we could if we weredoing circuit programming.
This fall, we started the process of replacing our mock proofs with ZK proofs,and wrote circuit implementations of our Spend and Output proofs, which weexpect to integrate into testnets in the new year.
Last year, we started building Penumbra against Tendermint 0.35, which at thetime was newly released. However, our testnets, as well as those of Celestiaand Vega, revealed that the 0.35 release of Tendermint had serious stabilityproblems when deployed in real network environments. At one point during thesummer, for instance, we were unable to keep Penumbra testnets running for morethan a day or two, not because of any problem with the Penumbra application, butbecause validators would mysteriously be unable to communicate with each other,causing chain halts unless we ensured that a single validator had asupermajority of voting power (allowing it to continue producing blocks evenwhen unable to communicate with any other validators).
As a result of these stability issues, the Tendermint 0.35 release wasdiscontinued by the Tendermint team, along with the unreleased work onTendermint 0.36. Instead, Tendermint 0.37 will be based on the stable 0.34 codecurrently used in production.
Migration from 0.35 to 0.34 was a somewhat painful process. We interface toTendermint via ABCI, and in particular through the ABCI domain types wecontributed upstream to the tendermint-rs crate. However, because we hadwritten that code against the 0.35 branch of tendermint-rs, which had to berolled back, we had to backport the interface definitions, and then propagatethose changes through our dependency tree, including ibc-rs. However, incollaboration with the Informal Systems team who maintain the tendermint-rsand ibc-rs crates, we were able to get through it, and have had no networkstability issues since migrating to Tendermint 0.34.
Unfortunately, this change means that ABCI++, the upgraded interface toTendermint that allows application logic to hook into consensus and implementadvanced features like threshold cryptography, will not ship in its full formuntil Tendermint 0.38 at the earliest. In addition, based on our experiences sofar, we're hesitant to rely on features that affect the network layer (like voteextensions, which allow validators to gossip threshold shares) without testingthem in real-world conditions.
These factors led us to decide to defer flow encryption to a futurenetwork upgrade, rather than including it in the initial mainnet launch scope.This means that Penumbra won't conceal users' contributions to batch totals atlaunch. While this is unfortunate, the protocol is still designed to enableencrypting them in the future, and we feel that even without flow encryption,Penumbra's privacy guarantees are still a huge advance over the status quo.
Finally, we also took the time to invest in improved tooling. We're now usingBuf to host our Protobuf definitions, and auto-generate Go andTypescript packages that can work with any Penumbra data types or communicatewith pd's RPC. We added native grpc-web support to pd, so every fullnodecan serve GRPC to web clients without requiring a proxy, and started working onintegrating automatic HTTPS support so every fullnode can serve RPC requestssecurely without any extra configuration beyond a domain name. Finally, weadded a GRPC proxy to pd that provides access to the Tendermint RPC, so thatPenumbra clients only need to speak to one RPC endpoint.
In collaboration with the Strangelove team, we also built Kubernetes-basedtestnet deployment infrastructure, which we'll be using for testnet deploymentsgoing forward, and started work on integrating Penumbra into their IBC testframework to do automatic conformance testing between Penumbra and Cosmos SDKtestnets.
We're aiming for a mainnet launch next year, and closed out 2022 by planning themajor components left to build before then. We'll be diving into more detail on each of these projects in future posts, but for now, here are brief summaries:
The last major component of the Penumbra ledger is the DEX engine, whichexecutes batch swaps against the active liquidity. Penumbra's batched executionmodel has interesting implications for the DEX engine: because the DEX only runsexecutes once per block, not once per transaction, we can afford to make the DEXengine considerably more sophisticated, as the execution cost is amortized overall transactions in the block.
Penumbra uses a hybrid, order-book-like AMM with automatic routing. Liquidity onPenumbra is recorded as many individual concentrated liquidity positions, akinto an order book. Each liquidity position is its own AMM, with its own fee tier,and that AMM has the simplest possible form, a constant-sum (fixed-price) marketmaker. These component AMMs are synthesized into a global AMM by the DEX engine,which optimally routes trades across the entire liquidity graph.
Penumbra has no intra-block trade ordering, so DEX execution operates at the endof the block in four phases:
In this model, intra-chain arbitrage (making prices consistent within thechain) is performed automatically by the chain, while inter-chain arbitrage(making prices on Penumbra consistent with other markets) is performed byarbitrageurs. This arbitrage game has interesting pro-rata dynamics, which youcan read about in this research paper we collaborated on with theBain Capital Crypto team.
It has other useful properties: market-makers can create fill-or-kill positionswith prices valid for exactly one block without having to compete for orderingwithin that block, by opening and then closing a liquidity position in the sametransaction. Or, traders can execute on the market-maker side, by creating afill-or-kill position whose price crosses the spread, and waiting for the chainto fill it through arbitrage. Because trades are optimally routed, the chaincan compose cheap stableswaps between different IBC paths on either end of atrade, making available liquidity independent of the specific bridge path of anasset. Finally, passive market-makers can replicate the payoff curves of otherAMM trading functions, like UniV2's xy=k, though they may be outcompeted byactive LPs.
Our new state model is key to implementing this design efficiently, as it allowsus to maintain application-specific indexes (like the liquidity positions oneach trading pair) with transactional, copy-on-write semantics, so we canspeculatively mutate state during routing and choose whether or not to apply theresults.
We've been laying the groundwork for web interfaces to Penumbra, with ourmodular client design, theTransactionPerspective/TransactionView design, and the Protobuf tooling thatallows accessing Penumbra data structures from Typescript. We've identified afew key goals for interfaces to Penumbra:
We want to be able to support a wide variety of interfaces to Penumbra.
While it’s important that there be at least one first-class wallet experience at launch, Penumbra’s capabilities are multifaceted, and users will probably be best served by specialized interfaces: e.g., one for basic transfers or swaps, one for governance, one for power users to manage liquidity or examine the liquidity graph, etc.
Supporting a wide variety of interfaces is also important for decentralization: no one entity should “own” the Penumbra userbase via control of a single frontend, or be at risk of becoming a chokepoint for control of those users. This also means that it should be possible to build frontend interfaces to Penumbra that do not require custom backend infrastructure beyond an ordinary pd full node.
We want those interfaces to be as easy to build as interfaces to transparent chains.
Historically, interfaces to shielded chains have been more difficult to build than interfaces to transparent chains, because they require the application developer to manage synchronization of users’ private state, unlike a transparent chain, where user state is accessible via RPC. Penumbra is about privacy without compromise, so we want to make it as easy to build those interfaces for Penumbra as it is for a shielded chain.
We want our users to have security when interacting with those interfaces.
In order for users to benefit from the availability of a wide variety of third-party interfaces, users need to be confident they can use them without risk of losing funds or being hacked. We need to ensure that only the user can authorize a transaction, and be able to understand exactly what actions they’re authorizing when they do so.
Web Architecture
To accomplish these goals, we're building implementations of our view andcustody services, and bundling them into a browser extension that cancommunicate with web content. All long-term key material is kept inside of thebrowser extension, so web content only has access to per-transaction data, andall transactions must be approved through the browser extension, which providesa secure display path for the user to review a proposed transaction. Thisallows a user to selectively grant viewing or spending permissions to webinterfaces to relatively untrusted web interfaces, and to revoke thosepermissions later. The resulting architecture is shown below:
┌───────────┐
│ Extension │
╭ │ ┌───────┐ │ custody
spending │ │ │custody│ │ protocol
capability │ │ │service│◀┼─────────────────┐
╰ │ └───────┘ │ │
│ │ │
│ │ │
│ │ │
╭ │ ┌───────┐ │ view │
full viewing │ │ │view │ │ protocol │
capability │ │ │service│◀┼──────┐ │
╰ │ └───────┘ │ │ │
│ ▲ │ │ ┌───┼────────────┐
└───┼───────┘ │ │ │ Web Content│
│ │ │ ▼ │
╭ │ │ │ ┌───────┐ │
transaction │ │ └──────┼▶│wallet │ │
perspectives │ │ │ │ logic │ │
╰ │ │ └───────┘ │
│ │ ▲ │ │
│ specific/oblivious └───┼─┼──────────┘
│ client protocols │ │
├─────────────────────────┘ │ tx
│┌──────────────────────────┘ broadcast .───.
││ ,' `.
╭ ┌───┼┼─────────────────────────────────────┐ .───; :
public │ │ ││ Penumbra Fullnode│ ; │
chain │ │ ││ grpc/grpc-web │ .─┤ ├──.
data │ │ ▼▼ │ ,' `.
│ │ ┌────┐ tm rpc proxy ┌──────────┐ │ ; Penumbra :
│ │ │ │◀────────────────────▶│ │ │ : ┌───────────▶ Network ;
│ │ │ pd │◀────────────────────▶│tendermint│◀┼─────────┘ ╱
│ │ └────┘ abci app └──────────┘ │ `. `. `. ,'
╰ └──────────────────────────────────────────┘ `───' `───' `───'
We're working with the Zpoken team to build this system and feed back improvements to Penumbra's RPCs along the way, and we're excited to be able to show the results when they're ready.
Because Penumbra uses custom cryptography to achieve privacy, we can't easilymake use of existing custody tooling like HSMs or hardware wallets withoutextending them to add support for our cryptography. However, secure custody isa critical operational requirement. Our plan to address it is to build athreshold signing implementation using FROST, providing off-chainmultisignatures indistinguishable from any other transactions.
This tool will be called Narsil, and it will act as a custodian and audit logfor a single Penumbra account, sharing control of that account's spendingauthority across multiple Narsil shards.
While FROST solves the cryptographic part of threshold signatures, it doesn'tsolve the engineering problem of building a way for the participants to reliablycommunicate with each other and agree on what messages they should be signing.That problem is also subtle and difficult to solve, but we realized we alreadyhave a tool that allows a set of participants to come to consensus on whatmessages have occurred: Tendermint consensus.
Using Tendermint internally gives us fault-tolerant state replication with anoff-the-shelf tool, but it also has another important advantage. Because theNarsil cluster has fault-tolerant replication of the audit log of everytransaction the account has authorized, by implication, it has fault-tolerantreplication of the entire account state. This means that users can choose tomaintain their account state off-chain, using their Narsil cluster as a personalrollup, and only post state commitments, nullifiers, and proofs to the L1.
This seems particularly useful for active market makers, who can cheaply performfrequent updates to their own state (e.g., updating their liquidity positionspotentially every block), while saving on gas and saving other users fromscanning irrelevant state updates.
We're excited about these possibilities and we'll have more details to share onthe Narsil design in the new year. For now, here's a rough idea of how Narsilwill be structured, and how it integrates with other Penumbra software:
┌────────────────────────┐
│ pcli (or other client) │
└────────────────────────┘
▲ ▲
│ custody │ view
│ protocol │ protocol
▼ ▼
┌──────┬────────┬──────┬────────┬───────────────────────┐ ┌────────┐
│ │Custody │ │ View │ client protocol │ │Penumbra│
│ │Service │ │Service │◀──────────────────────┼─────│Fullnode│
│ └────────┘ └────────┘ │ └────────┘
│ │ ▲ ▲ │
│ │ │ View │ │
│ Auth │ │ Auth Advice │ absent for │
│ Request │ │ Data │ replicas │
│ │ │ ┌────────┐ ┌ ─ ─ ─ ─ ─ │
│ │ │ │ Narsil │ FROST Narsil │ │
│ │ └──────────│ Ledger │─────────▶│ Shard │
│ │ └────────┘ ─ ─ ─ ─ ─ ┘ │
│ │ ▲ FROST │ │
│ └─────────────┐ │ ┌─────────────────┘ │
│ │ ABCI │ │
│narsild │ │ │ │
└───────────────────────┼───┼──┼────────────────────────┘
│ │ │
▼ │ ▼
┌──────────┐
│Tendermint│
└──────────┘
Our current weekly testnets start from a new genesis state, and don't preserveany history, which has allowed us to iterate quickly. But before we go tomainnet, we need to build and exercise a real chain upgrade path. Upgrades to ashielded chain like Penumbra are tricky, because unlike a transparent chain,where all state is on-chain and can be snapshotted and migrated, state onPenumbra splits into two categories: public state, recorded on-chain, andprivate state, recorded off-chain.
While we can perform migrations on the public state, changes to the privatestate must be accretive, preserving all existing private state and allowingusers to roll over their data on their own schedule. We're working on thedesign of this upgrade mechanism, and plan to exercise it on our testnets priorto mainnet.
We've implemented ZK circuits for two proof statements used in Penumbra. In thenew year, we'll be integrating those ZK proofs into the rest of the system,replacing the other mock proofs with ZK implementations, now that the systemdesign has matured.
Finally, after finishing the remaining on-chain implementation tasks, we'll bechanging gears from an expansive, iterative process of building out the system,exploring the design space, and moving as quickly as possible, to a process ofcareful refinement of the system we've built, cleaning up our code to make iteasier to audit and understand, focusing on testing and assurance, and carefullychecking every detail before we're ready to go to mainnet.
To stay up to date with the latest progress along the way, follow us on Twitter, join our Discord and subscribe to the #announcements channel, and check out our weekly testnets!
