coracle-rust/book/research/03-encryption.md

# Research: 03-encryption

## Topic Summary

Chapter covers nostr encryption: ECDH shared key derivation over secp256k1, NIP-04
(legacy AES-256-CBC), and NIP-44 v2 (ChaCha20 + HKDF + HMAC-SHA256). Likely extends
`PrivateKey` with encrypt/decrypt methods, but reference implementations also offer
useful abstractions like a reusable `ConversationKey` type. Should explain why NIP-44
exists, the message format, and the per-message key derivation pipeline.

## Philosophy

From `ref/building-nostr`:

- Nostr is **publicity technology**, not privacy technology. Encryption is situational,
  not foundational. Frame it as an opt-in privacy mechanism layered on top of an
  inherently public event broadcast system.
- The NIP-04 → NIP-17 evolution is a case study in protocol churn: NIP-04 leaks
  significant metadata (sender, recipient, timing); NIP-17 leaks much less but broke
  backwards compatibility. Two major clients resisted NIP-17 to preserve UX/delivery.
  Lesson: privacy and ergonomics are in tension; libraries must support both.
- Encryption is a **content privacy mechanism**, not an access control mechanism.
  Access control belongs in relays.
- User agency over identity is paramount: never send keys to custodians; encryption
  primitives must be usable by hardware/remote signers without exposing the key.
- Metadata is the harder problem than message content. Even encrypted DMs leak
  unless carefully designed (gift wrap pattern).

Implications for the chapter: introduce encryption as a tool with trade-offs, be clear
that NIP-04 is deprecated but still required for compatibility, and design the API so
ECDH operations can be delegated to a signer abstraction later.

## Reference Implementation Analysis

### applesauce

Encryption is owned by signers, not a standalone module. Core types in
`packages/core/src/helpers/encrypted-content.ts`:

- `EncryptedContentSigner` interface defines `nip04` and `nip44` methods on signers
- `EncryptionMethods` pairs `encrypt(pubkey, plaintext)` and `decrypt(pubkey, ciphertext)`
- Event-kind → encryption-method mapping (kind 4 → nip04, kind 13/1059 → nip44)
- Plaintext cached on the event object via `Symbol.for("encrypted-content")` (Reflect API)

`PrivateKeySigner` (`packages/signers/src/signers/private-key-signer.ts:25-30`) holds a
Uint8Array key and exposes `.nip04` and `.nip44` properties with async encrypt/decrypt.
NIP-44 calls `nip44.getConversationKey(secret, pubkey)` per message; no conversation key
caching at the applesauce level.

Aggressive plaintext caching via RxJS observables persists decrypted content to
IndexedDB for offline-read UX. Gift wrap (NIP-59) is a clean three-layer cascade
(rumor → seal → gift wrap) using symbols to track refs without polluting JSON.

Dependencies: `nostr-tools ~2.19`, `@noble/secp256k1 ^1.7.1`, `@noble/hashes`.

Worth borrowing: signer-as-HSM pattern, kind→method mapping, capability probing.

### ndk

`NDKEncryptionScheme = "nip04" | "nip44"` union (core/src/types.ts).

Signer interface:
```ts
encrypt(recipient: NDKUser, value: string, scheme?: NDKEncryptionScheme): Promise<string>
decrypt(sender: NDKUser, value: string, scheme?: NDKEncryptionScheme): Promise<string>
encryptionEnabled(scheme?): Promise<NDKEncryptionScheme[]>
```

Event-level encrypt/decrypt in `core/src/events/encryption.ts` delegates to signer.
Auto-detects NIP-04 vs NIP-44 by checking for `?iv=` pattern in the ciphertext.
Decrypted events cached by event ID via pluggable cache adapter (no conversation key
cache).

NIP-46 backend separates encrypt/decrypt strategies (`backend/nip44-encrypt.ts`,
`backend/nip44-decrypt.ts`).

### nostr-gadgets

No direct encryption implementation. Delegates entirely to `@nostr/tools` (nip04, nip44
exports). Not useful as a reference for our implementation.

### nostr-tools

The canonical low-level reference. Pure-sync API, minimal deps, all from the @noble
ecosystem.

**Dependencies:** `@noble/curves` (secp256k1 ECDH), `@noble/hashes` (SHA-256, HKDF,
HMAC), `@noble/ciphers` (AES-CBC, ChaCha20), `@scure/base` (base64).

**NIP-04** (`nip04.ts`, 41 lines):
1. ECDH: `secp256k1.getSharedSecret(privkey, '02' + pubkey)` → 65-byte uncompressed
2. Take x-coordinate: `key.slice(1, 33)` → 32-byte AES key
3. AES-256-CBC with random 16-byte IV
4. Format: `{base64_ciphertext}?iv={base64_iv}`

**NIP-44 v2** (`nip44.ts`, 128 lines):

`getConversationKey(privkey, pubkey)`:
```
sharedX = ecdh(privkey, '02' + pubkey)[1:33]   // 32 bytes
return hkdf_extract(sha256, ikm=sharedX, salt='nip44-v2')
```

Per-message key derivation (HKDF-Expand with nonce as info, output 76 bytes):
- `[0:32]` ChaCha20 key
- `[32:44]` ChaCha20 nonce (12 bytes)
- `[44:76]` HMAC-SHA256 key

Padding: messages 1–65535 bytes; ≤32 → 32; else round up to next power-of-2 chunk.
Length prefix is 2-byte big-endian, prepended before padding.

Encryption pipeline:
1. Generate 32-byte random nonce
2. Pad plaintext (length prefix + zero-padding to chunk size)
3. Derive message keys from conversation key + nonce
4. ChaCha20 encrypt
5. HMAC-SHA256(nonce || ciphertext) with derived MAC key
6. Payload: `[version=2][nonce(32)][ciphertext][mac(32)]`, base64-encoded

Decryption verifies version byte, recomputes HMAC (constant-time compare), decrypts,
and unpads with length-prefix validation.

Test vectors live in `nip44.vectors.json` (official NIP-44 vectors from spec).

### rust-nostr

The most directly relevant reference — Rust idioms we may want to mirror or diverge
from. All in the `nostr` crate.

**Module layout:**
- `src/util/mod.rs:72-83` — `generate_shared_key()`
- `src/util/hkdf.rs` — HKDF extract/expand
- `src/nips/nip04.rs` — NIP-04
- `src/nips/nip44/mod.rs` — NIP-44 versioning wrapper
- `src/nips/nip44/v2.rs` — v2 implementation

**Dependencies:** `secp256k1` (ECDH), `chacha20`, `aes`, `cbc`, `bitcoin_hashes`
(HMAC-SHA256, SHA-256), `base64`. Behind `nip04` and `nip44` feature flags.

**ECDH** (`util/mod.rs:72-83`):
```rust
pub fn generate_shared_key(sk: &SecretKey, pk: &PublicKey) -> Result<[u8; 32], _> {
    let pk = pk.xonly()?;
    let normalized = NormalizedPublicKey::from_x_only_public_key(pk, Parity::Even);
    let ssp: [u8; 64] = ecdh::shared_secret_point(&normalized, sk);
    let mut shared = [0u8; 32];
    shared.copy_from_slice(&ssp[..32]);
    Ok(shared)
}
```
Uses x-only (BIP340) pubkey with `Parity::Even` normalization; takes first 32 bytes
of the shared secret point.

**API:** Free functions, not methods on `Keys`:
- `nip04::encrypt(sk, pk, content)`, `decrypt(sk, pk, payload)`
- `nip44::encrypt(sk, pk, content, version)`, `decrypt(sk, pk, payload)`
- `ConversationKey::derive(sk, pk)` — first-class type for v2

**ConversationKey** (`v2.rs:108-153`):
- Newtype wrapping HMAC state (via Deref) to avoid re-hashing
- Result of `hkdf_extract(salt=b"nip44-v2", ikm=shared_key)`
- Reusable across messages in the same conversation

**Message keys derivation (`v2.rs:252-258`):**
```rust
fn get_message_keys(ck: &ConversationKey, nonce: &[u8]) -> Result<MessageKeys, _> {
    let expanded = hkdf::expand(ck.as_bytes(), nonce, 76);
    MessageKeys::from_slice(&expanded)
}
```

**Padding (`v2.rs:260-287`):**
- 1–65408 byte limit
- ≤32 → 32; else round to next power-of-2, with chunk = nextpow2/8 for large messages

**Errors:** Layered enum types (`Error`, `ErrorV2`) covering key errors, base64,
UTF-8, HKDF length, HMAC mismatch, padding, message size.

**Tests:** Official NIP-44 test vectors in `nip44/nip44.vectors.json`, exercised by
`test_valid_get_conversation_key`, `test_valid_calc_padded_len`,
`test_valid_encrypt_decrypt`, plus invalid-input tests.

### welshman

Encryption lives in the signer package
(`packages/signer/src/util.ts`), wrapping `nostr-tools` directly:

```ts
interface EncryptionImplementation {
  encrypt(pubkey, message): Promise<string>
  decrypt(pubkey, message): Promise<string>
}
interface ISigner {
  nip04: EncryptionImplementation
  nip44: EncryptionImplementation
}
```

**Notable: aggressive ECDH conversation-key caching** via an LRU cache (maxSize 10,000)
keyed by `${secret}:${pubkey}`. Wraps `nip44.v2.utils.getConversationKey()`. None of
the other references cache shared secrets.

NIP-46 remote signer defaults to NIP-44 with NIP-04 fallback. Standalone `decrypt()`
helper auto-detects the format.

## Common Patterns

1. **ECDH = secp256k1 shared point, take x-coordinate.** Every implementation does
   this. The NIP-04 path uses the raw x-coordinate as the AES key. The NIP-44 path
   feeds it into HKDF-Extract with salt `"nip44-v2"` to produce a conversation key.

2. **Message format for NIP-44 v2 is fixed:** `[version=2][nonce(32)][ct][hmac(32)]`,
   base64-encoded. HMAC covers `nonce || ciphertext` (the nonce is AAD, not encrypted).

3. **Padding scheme is uniform:** 2-byte big-endian length prefix, then zero-pad to
   the next power-of-2 chunk (min 32 bytes, max ~65KB). Decryption must verify the
   length prefix and that padding bytes are zero.

4. **Per-message keys derived via HKDF-Expand** with the conversation key as PRK and
   the random nonce as info, producing 76 bytes split into ChaCha20 key (32),
   ChaCha20 nonce (12), HMAC key (32).

5. **NIP-04 is universally deprecated** but still implemented for compatibility. Its
   format `{ct}?iv={iv}` is easy to detect, which enables auto-routing in
   higher-level decrypt helpers.

6. **Architectural divergence** on where encryption lives:
   - **Free functions** (rust-nostr, nostr-tools): standalone modules, no key class
     coupling. Most flexible.
   - **Methods on a signer** (applesauce, ndk, welshman): encryption belongs to the
     thing that holds the key. Enables hardware/remote signers.
   - We can do **both**: free functions in this chapter, signer abstraction added
     in a later chapter that calls them.

7. **Conversation key as a first-class type** (rust-nostr's `ConversationKey`,
   welshman's LRU cache) is worth borrowing. Without it, every message re-runs ECDH
   and HKDF-Extract, which is wasteful for chat-like workloads. With it, callers can
   derive once and encrypt many.

8. **Plaintext caching is an application concern**, not a library concern. Several
   libraries (applesauce, ndk) cache decrypted event content, but always at a layer
   above the raw crypto.

## Considerations for Our Implementation

**Crate placement:** Encryption belongs in `coracle-lib` next to `PrivateKey` and
`PublicKey` (`coracle-lib/src/keys.rs`). Keep it stateless and signer-agnostic; a
later signer chapter can wrap it.

**Dependencies to add:**
- `secp256k1` is already pulled in by the keys chapter — use its `ecdh` module.
  Alternative: `k256` (RustCrypto). secp256k1 is simpler and matches BIP340 semantics.
- `aes` + `cbc` for NIP-04
- `chacha20` for NIP-44
- `hmac` + `sha2` (RustCrypto) or `bitcoin_hashes` for HMAC-SHA256 + HKDF. RustCrypto
  is more idiomatic and we can use the `hkdf` crate directly.
- `base64` (or `base64ct`) for payload encoding
- `rand` / `rand_core` for IV and nonce generation (already a transitive dep)

Keep dependencies minimal — prefer one ecosystem (RustCrypto: `aes`, `cbc`, `chacha20`,
`hmac`, `sha2`, `hkdf`) for consistency.

**API shape proposal:**

Free functions in a new `coracle-lib/src/encryption.rs` module:
```rust
pub fn shared_secret(sk: &PrivateKey, pk: &PublicKey) -> [u8; 32];

pub mod nip04 {
    pub fn encrypt(sk: &PrivateKey, pk: &PublicKey, plaintext: &str) -> String;
    pub fn decrypt(sk: &PrivateKey, pk: &PublicKey, payload: &str) -> Result<String, Error>;
}

pub mod nip44 {
    pub struct ConversationKey([u8; 32]);
    impl ConversationKey {
        pub fn derive(sk: &PrivateKey, pk: &PublicKey) -> Self;
    }
    pub fn encrypt(ck: &ConversationKey, plaintext: &str) -> Result<String, Error>;
    pub fn decrypt(ck: &ConversationKey, payload: &str) -> Result<String, Error>;
}
```

Plus convenience methods on `PrivateKey` that wrap these (per the user's hint):
```rust
impl PrivateKey {
    pub fn nip04_encrypt(&self, pk: &PublicKey, msg: &str) -> String;
    pub fn nip04_decrypt(&self, pk: &PublicKey, payload: &str) -> Result<String, Error>;
    pub fn nip44_encrypt(&self, pk: &PublicKey, msg: &str) -> Result<String, Error>;
    pub fn nip44_decrypt(&self, pk: &PublicKey, payload: &str) -> Result<String, Error>;
}
```

The `ConversationKey` type is the load-bearing abstraction — methods on `PrivateKey`
are the friendly wrapper that derives one ad-hoc and discards it.

**Error type:** A single `EncryptionError` enum covering: invalid payload format,
base64 decode, version mismatch, HMAC mismatch, padding error, message too long,
message empty, UTF-8 error. Don't split into NIP-04 and NIP-44 enums — too much
ceremony for a teaching resource.

**Padding & format constants:** Define them as named constants with comments
referencing the NIP, since the magic numbers (32, 65535, "nip44-v2", 76) are
otherwise opaque.

**Test vectors:** Pull a small subset of the official NIP-44 vectors into
`coracle-lib/tests/nip44_vectors.rs` for confidence. NIP-04 has no official vectors
but a round-trip test plus interop with rust-nostr's known ciphertext is sufficient.

**Out of scope for this chapter:**
- Gift wrap (NIP-59) — separate chapter
- Encrypted event helpers (kind→scheme dispatch) — belongs to events/signer chapter
- Conversation key caching — application concern; mention in passing
- NIP-49 password-encrypted keys — separate concern, possibly a signer chapter

**Narrative arc:**
1. Why encryption is situational, not foundational. NIP-04 → NIP-44 motivation.
2. ECDH shared secret derivation: the common foundation.
3. NIP-04: the simple, deprecated path. Show it first because it's smaller.
4. NIP-44 v2: the message format, padding, conversation key, per-message keys,
   encrypt/decrypt pipeline. Show why it's an improvement.
5. Wiring it onto `PrivateKey` for ergonomics.
6. Tests against round-trips and known vectors.

Research complete. You can proceed with `/plan-chapter 03-encryption`.