Add encryption chapter

2026-04-13 21:21:52 -07:00
parent 5364854881
commit 2f334fa3be
8 changed files with 1259 additions and 27 deletions
@@ -0,0 +1,325 @@
+# Research: 03-encryption
+
+## Topic Summary
+
+Chapter covers nostr encryption: ECDH shared key derivation over secp256k1, NIP-04
+(legacy AES-256-CBC), and NIP-44 v2 (ChaCha20 + HKDF + HMAC-SHA256). Likely extends
+`PrivateKey` with encrypt/decrypt methods, but reference implementations also offer
+useful abstractions like a reusable `ConversationKey` type. Should explain why NIP-44
+exists, the message format, and the per-message key derivation pipeline.
+
+## Philosophy
+
+From `ref/building-nostr`:
+
+- Nostr is **publicity technology**, not privacy technology. Encryption is situational,
+  not foundational. Frame it as an opt-in privacy mechanism layered on top of an
+  inherently public event broadcast system.
+- The NIP-04 → NIP-17 evolution is a case study in protocol churn: NIP-04 leaks
+  significant metadata (sender, recipient, timing); NIP-17 leaks much less but broke
+  backwards compatibility. Two major clients resisted NIP-17 to preserve UX/delivery.
+  Lesson: privacy and ergonomics are in tension; libraries must support both.
+- Encryption is a **content privacy mechanism**, not an access control mechanism.
+  Access control belongs in relays.
+- User agency over identity is paramount: never send keys to custodians; encryption
+  primitives must be usable by hardware/remote signers without exposing the key.
+- Metadata is the harder problem than message content. Even encrypted DMs leak
+  unless carefully designed (gift wrap pattern).
+
+Implications for the chapter: introduce encryption as a tool with trade-offs, be clear
+that NIP-04 is deprecated but still required for compatibility, and design the API so
+ECDH operations can be delegated to a signer abstraction later.
+
+## Reference Implementation Analysis
+
+### applesauce
+
+Encryption is owned by signers, not a standalone module. Core types in
+`packages/core/src/helpers/encrypted-content.ts`:
+
+- `EncryptedContentSigner` interface defines `nip04` and `nip44` methods on signers
+- `EncryptionMethods` pairs `encrypt(pubkey, plaintext)` and `decrypt(pubkey, ciphertext)`
+- Event-kind → encryption-method mapping (kind 4 → nip04, kind 13/1059 → nip44)
+- Plaintext cached on the event object via `Symbol.for("encrypted-content")` (Reflect API)
+
+`PrivateKeySigner` (`packages/signers/src/signers/private-key-signer.ts:25-30`) holds a
+Uint8Array key and exposes `.nip04` and `.nip44` properties with async encrypt/decrypt.
+NIP-44 calls `nip44.getConversationKey(secret, pubkey)` per message; no conversation key
+caching at the applesauce level.
+
+Aggressive plaintext caching via RxJS observables persists decrypted content to
+IndexedDB for offline-read UX. Gift wrap (NIP-59) is a clean three-layer cascade
+(rumor → seal → gift wrap) using symbols to track refs without polluting JSON.
+
+Dependencies: `nostr-tools ~2.19`, `@noble/secp256k1 ^1.7.1`, `@noble/hashes`.
+
+Worth borrowing: signer-as-HSM pattern, kind→method mapping, capability probing.
+
+### ndk
+
+`NDKEncryptionScheme = "nip04" | "nip44"` union (core/src/types.ts).
+
+Signer interface:
+```ts
+encrypt(recipient: NDKUser, value: string, scheme?: NDKEncryptionScheme): Promise<string>
+decrypt(sender: NDKUser, value: string, scheme?: NDKEncryptionScheme): Promise<string>
+encryptionEnabled(scheme?): Promise<NDKEncryptionScheme[]>
+```
+
+Event-level encrypt/decrypt in `core/src/events/encryption.ts` delegates to signer.
+Auto-detects NIP-04 vs NIP-44 by checking for `?iv=` pattern in the ciphertext.
+Decrypted events cached by event ID via pluggable cache adapter (no conversation key
+cache).
+
+NIP-46 backend separates encrypt/decrypt strategies (`backend/nip44-encrypt.ts`,
+`backend/nip44-decrypt.ts`).
+
+### nostr-gadgets
+
+No direct encryption implementation. Delegates entirely to `@nostr/tools` (nip04, nip44
+exports). Not useful as a reference for our implementation.
+
+### nostr-tools
+
+The canonical low-level reference. Pure-sync API, minimal deps, all from the @noble
+ecosystem.
+
+**Dependencies:** `@noble/curves` (secp256k1 ECDH), `@noble/hashes` (SHA-256, HKDF,
+HMAC), `@noble/ciphers` (AES-CBC, ChaCha20), `@scure/base` (base64).
+
+**NIP-04** (`nip04.ts`, 41 lines):
+1. ECDH: `secp256k1.getSharedSecret(privkey, '02' + pubkey)` → 65-byte uncompressed
+2. Take x-coordinate: `key.slice(1, 33)` → 32-byte AES key
+3. AES-256-CBC with random 16-byte IV
+4. Format: `{base64_ciphertext}?iv={base64_iv}`
+
+**NIP-44 v2** (`nip44.ts`, 128 lines):
+
+`getConversationKey(privkey, pubkey)`:
+```
+sharedX = ecdh(privkey, '02' + pubkey)[1:33]   // 32 bytes
+return hkdf_extract(sha256, ikm=sharedX, salt='nip44-v2')
+```
+
+Per-message key derivation (HKDF-Expand with nonce as info, output 76 bytes):
+- `[0:32]` ChaCha20 key
+- `[32:44]` ChaCha20 nonce (12 bytes)
+- `[44:76]` HMAC-SHA256 key
+
+Padding: messages 1–65535 bytes; ≤32 → 32; else round up to next power-of-2 chunk.
+Length prefix is 2-byte big-endian, prepended before padding.
+
+Encryption pipeline:
+1. Generate 32-byte random nonce
+2. Pad plaintext (length prefix + zero-padding to chunk size)
+3. Derive message keys from conversation key + nonce
+4. ChaCha20 encrypt
+5. HMAC-SHA256(nonce || ciphertext) with derived MAC key
+6. Payload: `[version=2][nonce(32)][ciphertext][mac(32)]`, base64-encoded
+
+Decryption verifies version byte, recomputes HMAC (constant-time compare), decrypts,
+and unpads with length-prefix validation.
+
+Test vectors live in `nip44.vectors.json` (official NIP-44 vectors from spec).
+
+### rust-nostr
+
+The most directly relevant reference — Rust idioms we may want to mirror or diverge
+from. All in the `nostr` crate.
+
+**Module layout:**
+- `src/util/mod.rs:72-83` — `generate_shared_key()`
+- `src/util/hkdf.rs` — HKDF extract/expand
+- `src/nips/nip04.rs` — NIP-04
+- `src/nips/nip44/mod.rs` — NIP-44 versioning wrapper
+- `src/nips/nip44/v2.rs` — v2 implementation
+
+**Dependencies:** `secp256k1` (ECDH), `chacha20`, `aes`, `cbc`, `bitcoin_hashes`
+(HMAC-SHA256, SHA-256), `base64`. Behind `nip04` and `nip44` feature flags.
+
+**ECDH** (`util/mod.rs:72-83`):
+```rust
+pub fn generate_shared_key(sk: &SecretKey, pk: &PublicKey) -> Result<[u8; 32], _> {
+    let pk = pk.xonly()?;
+    let normalized = NormalizedPublicKey::from_x_only_public_key(pk, Parity::Even);
+    let ssp: [u8; 64] = ecdh::shared_secret_point(&normalized, sk);
+    let mut shared = [0u8; 32];
+    shared.copy_from_slice(&ssp[..32]);
+    Ok(shared)
+}
+```
+Uses x-only (BIP340) pubkey with `Parity::Even` normalization; takes first 32 bytes
+of the shared secret point.
+
+**API:** Free functions, not methods on `Keys`:
+- `nip04::encrypt(sk, pk, content)`, `decrypt(sk, pk, payload)`
+- `nip44::encrypt(sk, pk, content, version)`, `decrypt(sk, pk, payload)`
+- `ConversationKey::derive(sk, pk)` — first-class type for v2
+
+**ConversationKey** (`v2.rs:108-153`):
+- Newtype wrapping HMAC state (via Deref) to avoid re-hashing
+- Result of `hkdf_extract(salt=b"nip44-v2", ikm=shared_key)`
+- Reusable across messages in the same conversation
+
+**Message keys derivation (`v2.rs:252-258`):**
+```rust
+fn get_message_keys(ck: &ConversationKey, nonce: &[u8]) -> Result<MessageKeys, _> {
+    let expanded = hkdf::expand(ck.as_bytes(), nonce, 76);
+    MessageKeys::from_slice(&expanded)
+}
+```
+
+**Padding (`v2.rs:260-287`):**
+- 1–65408 byte limit
+- ≤32 → 32; else round to next power-of-2, with chunk = nextpow2/8 for large messages
+
+**Errors:** Layered enum types (`Error`, `ErrorV2`) covering key errors, base64,
+UTF-8, HKDF length, HMAC mismatch, padding, message size.
+
+**Tests:** Official NIP-44 test vectors in `nip44/nip44.vectors.json`, exercised by
+`test_valid_get_conversation_key`, `test_valid_calc_padded_len`,
+`test_valid_encrypt_decrypt`, plus invalid-input tests.
+
+### welshman
+
+Encryption lives in the signer package
+(`packages/signer/src/util.ts`), wrapping `nostr-tools` directly:
+
+```ts
+interface EncryptionImplementation {
+  encrypt(pubkey, message): Promise<string>
+  decrypt(pubkey, message): Promise<string>
+}
+interface ISigner {
+  nip04: EncryptionImplementation
+  nip44: EncryptionImplementation
+}
+```
+
+**Notable: aggressive ECDH conversation-key caching** via an LRU cache (maxSize 10,000)
+keyed by `${secret}:${pubkey}`. Wraps `nip44.v2.utils.getConversationKey()`. None of
+the other references cache shared secrets.
+
+NIP-46 remote signer defaults to NIP-44 with NIP-04 fallback. Standalone `decrypt()`
+helper auto-detects the format.
+
+## Common Patterns
+
+1. **ECDH = secp256k1 shared point, take x-coordinate.** Every implementation does
+   this. The NIP-04 path uses the raw x-coordinate as the AES key. The NIP-44 path
+   feeds it into HKDF-Extract with salt `"nip44-v2"` to produce a conversation key.
+
+2. **Message format for NIP-44 v2 is fixed:** `[version=2][nonce(32)][ct][hmac(32)]`,
+   base64-encoded. HMAC covers `nonce || ciphertext` (the nonce is AAD, not encrypted).
+
+3. **Padding scheme is uniform:** 2-byte big-endian length prefix, then zero-pad to
+   the next power-of-2 chunk (min 32 bytes, max ~65KB). Decryption must verify the
+   length prefix and that padding bytes are zero.
+
+4. **Per-message keys derived via HKDF-Expand** with the conversation key as PRK and
+   the random nonce as info, producing 76 bytes split into ChaCha20 key (32),
+   ChaCha20 nonce (12), HMAC key (32).
+
+5. **NIP-04 is universally deprecated** but still implemented for compatibility. Its
+   format `{ct}?iv={iv}` is easy to detect, which enables auto-routing in
+   higher-level decrypt helpers.
+
+6. **Architectural divergence** on where encryption lives:
+   - **Free functions** (rust-nostr, nostr-tools): standalone modules, no key class
+     coupling. Most flexible.
+   - **Methods on a signer** (applesauce, ndk, welshman): encryption belongs to the
+     thing that holds the key. Enables hardware/remote signers.
+   - We can do **both**: free functions in this chapter, signer abstraction added
+     in a later chapter that calls them.
+
+7. **Conversation key as a first-class type** (rust-nostr's `ConversationKey`,
+   welshman's LRU cache) is worth borrowing. Without it, every message re-runs ECDH
+   and HKDF-Extract, which is wasteful for chat-like workloads. With it, callers can
+   derive once and encrypt many.
+
+8. **Plaintext caching is an application concern**, not a library concern. Several
+   libraries (applesauce, ndk) cache decrypted event content, but always at a layer
+   above the raw crypto.
+
+## Considerations for Our Implementation
+
+**Crate placement:** Encryption belongs in `coracle-lib` next to `PrivateKey` and
+`PublicKey` (`coracle-lib/src/keys.rs`). Keep it stateless and signer-agnostic; a
+later signer chapter can wrap it.
+
+**Dependencies to add:**
+- `secp256k1` is already pulled in by the keys chapter — use its `ecdh` module.
+  Alternative: `k256` (RustCrypto). secp256k1 is simpler and matches BIP340 semantics.
+- `aes` + `cbc` for NIP-04
+- `chacha20` for NIP-44
+- `hmac` + `sha2` (RustCrypto) or `bitcoin_hashes` for HMAC-SHA256 + HKDF. RustCrypto
+  is more idiomatic and we can use the `hkdf` crate directly.
+- `base64` (or `base64ct`) for payload encoding
+- `rand` / `rand_core` for IV and nonce generation (already a transitive dep)
+
+Keep dependencies minimal — prefer one ecosystem (RustCrypto: `aes`, `cbc`, `chacha20`,
+`hmac`, `sha2`, `hkdf`) for consistency.
+
+**API shape proposal:**
+
+Free functions in a new `coracle-lib/src/encryption.rs` module:
+```rust
+pub fn shared_secret(sk: &PrivateKey, pk: &PublicKey) -> [u8; 32];
+
+pub mod nip04 {
+    pub fn encrypt(sk: &PrivateKey, pk: &PublicKey, plaintext: &str) -> String;
+    pub fn decrypt(sk: &PrivateKey, pk: &PublicKey, payload: &str) -> Result<String, Error>;
+}
+
+pub mod nip44 {
+    pub struct ConversationKey([u8; 32]);
+    impl ConversationKey {
+        pub fn derive(sk: &PrivateKey, pk: &PublicKey) -> Self;
+    }
+    pub fn encrypt(ck: &ConversationKey, plaintext: &str) -> Result<String, Error>;
+    pub fn decrypt(ck: &ConversationKey, payload: &str) -> Result<String, Error>;
+}
+```
+
+Plus convenience methods on `PrivateKey` that wrap these (per the user's hint):
+```rust
+impl PrivateKey {
+    pub fn nip04_encrypt(&self, pk: &PublicKey, msg: &str) -> String;
+    pub fn nip04_decrypt(&self, pk: &PublicKey, payload: &str) -> Result<String, Error>;
+    pub fn nip44_encrypt(&self, pk: &PublicKey, msg: &str) -> Result<String, Error>;
+    pub fn nip44_decrypt(&self, pk: &PublicKey, payload: &str) -> Result<String, Error>;
+}
+```
+
+The `ConversationKey` type is the load-bearing abstraction — methods on `PrivateKey`
+are the friendly wrapper that derives one ad-hoc and discards it.
+
+**Error type:** A single `EncryptionError` enum covering: invalid payload format,
+base64 decode, version mismatch, HMAC mismatch, padding error, message too long,
+message empty, UTF-8 error. Don't split into NIP-04 and NIP-44 enums — too much
+ceremony for a teaching resource.
+
+**Padding & format constants:** Define them as named constants with comments
+referencing the NIP, since the magic numbers (32, 65535, "nip44-v2", 76) are
+otherwise opaque.
+
+**Test vectors:** Pull a small subset of the official NIP-44 vectors into
+`coracle-lib/tests/nip44_vectors.rs` for confidence. NIP-04 has no official vectors
+but a round-trip test plus interop with rust-nostr's known ciphertext is sufficient.
+
+**Out of scope for this chapter:**
+- Gift wrap (NIP-59) — separate chapter
+- Encrypted event helpers (kind→scheme dispatch) — belongs to events/signer chapter
+- Conversation key caching — application concern; mention in passing
+- NIP-49 password-encrypted keys — separate concern, possibly a signer chapter
+
+**Narrative arc:**
+1. Why encryption is situational, not foundational. NIP-04 → NIP-44 motivation.
+2. ECDH shared secret derivation: the common foundation.
+3. NIP-04: the simple, deprecated path. Show it first because it's smaller.
+4. NIP-44 v2: the message format, padding, conversation key, per-message keys,
+   encrypt/decrypt pipeline. Show why it's an improvement.
+5. Wiring it onto `PrivateKey` for ergonomics.
+6. Tests against round-trips and known vectors.
+
+Research complete. You can proceed with `/plan-chapter 03-encryption`.