<
Back to arcticles
Testnet 38: Kalyke
We've been building Penumbra in public, with weekly testnet releases named afterthe moons of Jupiter. All of our engineering discussion happens in Discord,where we've been summarizing the changes each week, and now that the protocol ismaturing, we'll also be writing up progress updates here on the blog.
Today, we released our 38th testnet, codenamed Kalyke, which contains majorimprovements that make it much easier and simpler to write clients for Penumbra,and lay the groundwork for exciting future work. To test it out, check out theguide documentation on how to use the command-line client, pcli(pronounced "pickle-y").
As described in one of our first posts on shielded staking,Penumbra records delegations using delegation tokens, which represent a shareof a particular validator's delegation pool. Delegation to a validator is aprotocol-native exchange of staking tokens for delegation tokens, andundelegation exchanges delegation tokens back to staking tokens. Rather thandistributing staking rewards, the chain prices them into each delegation token'sexchange rate. This means that delegations can be recorded in Penumbra'smulti-asset shielded pool like any other asset, and allows us to maintaintransparency and accountability for validators, while having privacy fordelegators.
However, handling unbonding is a challenge. While we can neatly handleslashing, by marking down the exchange rate to price in a slashing penalty, weneed to maintain an unbonding period, so that a malicious validator cannotmisbehave, then immediately withdraw stake before they're slashed. This meanswe need a way to freeze the results of an undelegation until the end of theunbonding period, and apply any slashing penalties that happen in the meantime.
Our initial approach to unbonding involved "quarantining". Because alltransactions are private by default, and all value is recorded in the sameshielded pool, we built a system that would place the outputs from a transactionwith an undelegation in a special quarantine state, and only add them to theshielded pool after unbonding if no slashings had occurred, or otherwise rollback the effects of the transaction.
While this sounds simple, it turned out to be extremely complex: to start,preserving the ability to unapply particular state transitions is quite complexand error-prone, but even worse, that complexity multiplies across every otherpart of the system. For instance, a key thing to realize about shieldedblockchains is that, fundamentally, the way they achieve privacy is by movingexecution off-chain, out to the client device at the "edge" of the network --the role of the ZK proofs is to certify that the client's execution was donecorrectly -- and so execution happens on the client. But that meant that notonly did the fullnode have to pay the complexity of separately maintainingquarantined transactions and the ability to roll them back, so did every singleclient. Moreover, because outputs are shielded, we had to shield everyoutput of a transaction, meaning that unless an undelegation was carefullyconstructed with exact change, it could accidentally lock a user's other funds.
This was definitely suboptimal. To build privacy without compromise, we need tomake it as easy to build clients for Penumbra as it is to build them for atransparent chain. That's a hard enough problem to start, but this designdecision made it much worse.
Instead, we followed a design principle that's emerged in other contexts aswe've built Penumbra: any time a token can be in a different state, it shouldbe a different stateful token. In this case, our problems arise from having tomaintain a different state for staking tokens that are still unbonding, ratherthan having a distinct "unbonding token" that represents that state.
In our new design, that's exactly what we do. Rather thanexchanging delegation tokens for staking tokens, undelegation is now a two-stepprocess. First, the Undelegate action exchanges delegation tokens(parameterized by validator) for unbonding tokens (parameterized by validatorand unbonding period), with an exchange rate that prices in staking rewards.Then, the UndelegateClaim action exchanges unbonding tokens for stakingtokens, with a penalty rate that prices in slashings over the unbonding period.The chain validates that the unbonding period is over, and that the penalty rateis correct. In the happy path, the penalty is 0, and unbonding tokens convert tostaking tokens 1:1.
This change was a massive simplification for client implementations, which nolonger need to unapply transactions, and only have to update forwards throughtime.
Another feature we shipped this week is minimal support for personal rollups,so that transactions can just post opaque state commitments and ZK proofs to thechain, and omit the other transaction contents.
One way to understand the basic design of Zcash-like systems such as Penumbra isthat every transaction contains both a micro-rollup, with opaque commitments tothe notes consumed and produced by the transaction and ZK proofs those noteswere well-formed, and also the rolled-up data itself, encrypted to the senderand/or receiver, who can scan the blockchain to learn about their transactionsand decrypt the note contents.
Rather than simply recording notes, Penumbra's commitment tree nowrecords commitments to arbitrary state fragments, with shielded notes just onespecial case. This means that we maintain forward compatibility with expansionsto the kinds of state fragments we record, or changes to the format of existingstate fragments. And, rather than requiring that the encrypted payload contentsare always present, we now support bare state commitments without payloads,representing state fragments rolled-up off-chain. This also provides scalabilitybenefits, because every other client saves the effort that would be required toscan those state commitments' encrypted payloads.
Currently, we only use this to implement swaps, as described in the nextsection, but future work building on this foundation will allow users likemarketmakers to maintain their state off-chain, unlocking significantperformance and scalability gains. Operationalizing this poses some interestingchallenges, which we've been thinking about and are excited to share our ideason soon.
Earlier this fall, we described our initial implementation of shieldedswaps, which allow Penumbra users to swap assets from one typeto another without leaving the shielded pool. To do this, we need a way forclients to execute state updates asynchronously, because the clearing pricefor the batch only becomes known after the transaction is submitted. Ourinsight was that we could model Futures (or Promises) on-chain, pausingexecution at an .await point by creating a SNARK-friendly Merklization of allof the intermediate execution state, and then resuming execution in a latertransaction by opening the commitment to the intermediate execution state. Forinstance, in a swap, the initial transaction's Swap action commits to theuser's input amounts, trading pair, and claim address, and once the batch pricesare published, the SwapClaim action privately mints their pro-rata share ofthe batch, proving consistency between the public prices and the private inputdata.
In our initial implementation, we recorded each user's execution state bycreating a "swap NFT", 1 unit of an asset whose asset ID was a commitment totheir swap inputs, and recording it in a shielded note like any other asset.This was a convenient way to get the system working, but we realized it had afew ill-fitting parts:
SwapClaim in the second phase to be possible to submit automatically, and not require separate signing, since that would be bad UX, and because the SwapClaim doesn't represent any new intent, it's just finishing a computation already started by the Swap, which specifies the claim address up front.SwapClaim shouldn't require authorization, we ended up building a special-case proof statement that allowed ignoring the spend capability when spending a note, and instead proving that the note recorded a swap NFT, and that the output notes were minted to the correct claim address. This felt wrong: if the capability is going to be ignored, it shouldn't be recorded in the first place!SwapClaim used the correct output prices, we proved that the prices used as the public input to the SwapClaim proof matched the height at which the note recording the swap NFT was created. But this is definitely wrong, because the height at which the note recording the NFT was created is not necessarily the same as the height the swap NFT was created. Since the swap NFT is just another asset in the shielded pool, a user can send their swap NFT to themselves, effectively getting a free option on any later clearing price!SwapClaim creates output notes, the claimer needs to be able to learn about them, in order to have effective control over the output funds. But while the note commitment is checked in the circuit, the encrypted note payload isn't, so a malicious client could correctly claim a user's output funds, but encrypt garbage data in the payload, effectively burning the money.However, after the design changes described in the last section, all theingredients for a much simpler solution that avoids these issues were in place.
Once we generalized our commitment tree to arbitrary state fragments, we couldcommit directly to users' swap inputs: we recast the note commitment tree as astate commitment tree, define both note commitments and swap commitments asdifferent kinds of state commitments, and insert the swap commitments directlyinto the state commitment tree rather than creating "swap NFTs". On the clientside, we expand the set of possible state payloads to add encrypted swaps aswell as encrypted notes, allowing clients to detect their own swaps whilescanning.
Because the swap commitment is inserted into the tree by the chain, and can't berelocated after the fact, we can use its height in the SwapClaim proof, whichjust has to prove knowledge of an inclusion path for the swap commitment, andthat the inclusion path passes through the claimed height. To prevent the DoSattack, we make the output notes deterministically derivable from the originalswap plaintext and the public clearing prices -- but at that point, why includethe output notes at all? Now that we have support for rolled-up statecommitments, we can skip including the output notes on-chain, saving scanningwork for every other client.
In addition to fixing the security problems in the initial proof of concept,these changes streamline the client-side implementation of synchronization andscanning, by removing special cases and features-jammed-into-other features infavor of a cleaner, more extensible system for client state.
We've also been pushing forward on other fronts, with the following work inprogress but not yet ready to land in a user-visible way:
Spend proof; in an upcoming release, we'll begin replacing our mock transparent proofs with ZK versions as we finalize Penumbra's functionality.#deployments channel in Discord.Stay tuned for more news on what we're building soon!
