coracle-rust/book/research/03-encryption.md

Research: 03-encryption

Topic Summary

Chapter covers nostr encryption: ECDH shared key derivation over secp256k1, NIP-04 (legacy AES-256-CBC), and NIP-44 v2 (ChaCha20 + HKDF + HMAC-SHA256). Likely extends PrivateKey with encrypt/decrypt methods, but reference implementations also offer useful abstractions like a reusable ConversationKey type. Should explain why NIP-44 exists, the message format, and the per-message key derivation pipeline.

Philosophy

From ref/building-nostr:

• Nostr is publicity technology, not privacy technology. Encryption is situational, not foundational. Frame it as an opt-in privacy mechanism layered on top of an inherently public event broadcast system.
• The NIP-04 → NIP-17 evolution is a case study in protocol churn: NIP-04 leaks significant metadata (sender, recipient, timing); NIP-17 leaks much less but broke backwards compatibility. Two major clients resisted NIP-17 to preserve UX/delivery. Lesson: privacy and ergonomics are in tension; libraries must support both.
• Encryption is a content privacy mechanism, not an access control mechanism. Access control belongs in relays.
• User agency over identity is paramount: never send keys to custodians; encryption primitives must be usable by hardware/remote signers without exposing the key.
• Metadata is the harder problem than message content. Even encrypted DMs leak unless carefully designed (gift wrap pattern).

Implications for the chapter: introduce encryption as a tool with trade-offs, be clear that NIP-04 is deprecated but still required for compatibility, and design the API so ECDH operations can be delegated to a signer abstraction later.

Reference Implementation Analysis

applesauce

Encryption is owned by signers, not a standalone module. Core types in packages/core/src/helpers/encrypted-content.ts:

• EncryptedContentSigner interface defines nip04 and nip44 methods on signers
• EncryptionMethods pairs encrypt(pubkey, plaintext) and decrypt(pubkey, ciphertext)
• Event-kind → encryption-method mapping (kind 4 → nip04, kind 13/1059 → nip44)
• Plaintext cached on the event object via Symbol.for("encrypted-content") (Reflect API)

PrivateKeySigner (packages/signers/src/signers/private-key-signer.ts:25-30) holds a Uint8Array key and exposes .nip04 and .nip44 properties with async encrypt/decrypt. NIP-44 calls nip44.getConversationKey(secret, pubkey) per message; no conversation key caching at the applesauce level.

Aggressive plaintext caching via RxJS observables persists decrypted content to IndexedDB for offline-read UX. Gift wrap (NIP-59) is a clean three-layer cascade (rumor → seal → gift wrap) using symbols to track refs without polluting JSON.

Dependencies: nostr-tools ~2.19, @noble/secp256k1 ^1.7.1, @noble/hashes.

Worth borrowing: signer-as-HSM pattern, kind→method mapping, capability probing.

ndk

NDKEncryptionScheme = "nip04" | "nip44" union (core/src/types.ts).

Signer interface:

encrypt(recipient: NDKUser, value: string, scheme?: NDKEncryptionScheme): Promise<string>
decrypt(sender: NDKUser, value: string, scheme?: NDKEncryptionScheme): Promise<string>
encryptionEnabled(scheme?): Promise<NDKEncryptionScheme[]>

Event-level encrypt/decrypt in core/src/events/encryption.ts delegates to signer. Auto-detects NIP-04 vs NIP-44 by checking for ?iv= pattern in the ciphertext. Decrypted events cached by event ID via pluggable cache adapter (no conversation key cache).

NIP-46 backend separates encrypt/decrypt strategies (backend/nip44-encrypt.ts, backend/nip44-decrypt.ts).

nostr-gadgets

No direct encryption implementation. Delegates entirely to @nostr/tools (nip04, nip44 exports). Not useful as a reference for our implementation.

nostr-tools

The canonical low-level reference. Pure-sync API, minimal deps, all from the @noble ecosystem.

Dependencies: @noble/curves (secp256k1 ECDH), @noble/hashes (SHA-256, HKDF, HMAC), @noble/ciphers (AES-CBC, ChaCha20), @scure/base (base64).

NIP-04 (nip04.ts, 41 lines):

1. ECDH: secp256k1.getSharedSecret(privkey, '02' + pubkey) → 65-byte uncompressed
2. Take x-coordinate: key.slice(1, 33) → 32-byte AES key
3. AES-256-CBC with random 16-byte IV
4. Format: {base64_ciphertext}?iv={base64_iv}

NIP-44 v2 (nip44.ts, 128 lines):

getConversationKey(privkey, pubkey):

sharedX = ecdh(privkey, '02' + pubkey)[1:33]   // 32 bytes
return hkdf_extract(sha256, ikm=sharedX, salt='nip44-v2')

Per-message key derivation (HKDF-Expand with nonce as info, output 76 bytes):

• [0:32] ChaCha20 key
• [32:44] ChaCha20 nonce (12 bytes)
• [44:76] HMAC-SHA256 key

Padding: messages 1–65535 bytes; ≤32 → 32; else round up to next power-of-2 chunk. Length prefix is 2-byte big-endian, prepended before padding.

Encryption pipeline:

1. Generate 32-byte random nonce
2. Pad plaintext (length prefix + zero-padding to chunk size)
3. Derive message keys from conversation key + nonce
4. ChaCha20 encrypt
5. HMAC-SHA256(nonce || ciphertext) with derived MAC key
6. Payload: [version=2][nonce(32)][ciphertext][mac(32)], base64-encoded

Decryption verifies version byte, recomputes HMAC (constant-time compare), decrypts, and unpads with length-prefix validation.

Test vectors live in nip44.vectors.json (official NIP-44 vectors from spec).

rust-nostr

The most directly relevant reference — Rust idioms we may want to mirror or diverge from. All in the nostr crate.

Module layout:

• src/util/mod.rs:72-83 — generate_shared_key()
• src/util/hkdf.rs — HKDF extract/expand
• src/nips/nip04.rs — NIP-04
• src/nips/nip44/mod.rs — NIP-44 versioning wrapper
• src/nips/nip44/v2.rs — v2 implementation

Dependencies: secp256k1 (ECDH), chacha20, aes, cbc, bitcoin_hashes (HMAC-SHA256, SHA-256), base64. Behind nip04 and nip44 feature flags.

ECDH (util/mod.rs:72-83):

pub fn generate_shared_key(sk: &SecretKey, pk: &PublicKey) -> Result<[u8; 32], _> {
    let pk = pk.xonly()?;
    let normalized = NormalizedPublicKey::from_x_only_public_key(pk, Parity::Even);
    let ssp: [u8; 64] = ecdh::shared_secret_point(&normalized, sk);
    let mut shared = [0u8; 32];
    shared.copy_from_slice(&ssp[..32]);
    Ok(shared)
}

Uses x-only (BIP340) pubkey with Parity::Even normalization; takes first 32 bytes of the shared secret point.

API: Free functions, not methods on Keys:

• nip04::encrypt(sk, pk, content), decrypt(sk, pk, payload)
• nip44::encrypt(sk, pk, content, version), decrypt(sk, pk, payload)
• ConversationKey::derive(sk, pk) — first-class type for v2

ConversationKey (v2.rs:108-153):

• Newtype wrapping HMAC state (via Deref) to avoid re-hashing
• Result of hkdf_extract(salt=b"nip44-v2", ikm=shared_key)
• Reusable across messages in the same conversation

Message keys derivation (v2.rs:252-258):

fn get_message_keys(ck: &ConversationKey, nonce: &[u8]) -> Result<MessageKeys, _> {
    let expanded = hkdf::expand(ck.as_bytes(), nonce, 76);
    MessageKeys::from_slice(&expanded)
}

Padding (v2.rs:260-287):

• 1–65408 byte limit
• ≤32 → 32; else round to next power-of-2, with chunk = nextpow2/8 for large messages

Errors: Layered enum types (Error, ErrorV2) covering key errors, base64, UTF-8, HKDF length, HMAC mismatch, padding, message size.

Tests: Official NIP-44 test vectors in nip44/nip44.vectors.json, exercised by test_valid_get_conversation_key, test_valid_calc_padded_len, test_valid_encrypt_decrypt, plus invalid-input tests.

welshman

Encryption lives in the signer package (packages/signer/src/util.ts), wrapping nostr-tools directly:

interface EncryptionImplementation {
  encrypt(pubkey, message): Promise<string>
  decrypt(pubkey, message): Promise<string>
}
interface ISigner {
  nip04: EncryptionImplementation
  nip44: EncryptionImplementation
}

Notable: aggressive ECDH conversation-key caching via an LRU cache (maxSize 10,000) keyed by ${secret}:${pubkey}. Wraps nip44.v2.utils.getConversationKey(). None of the other references cache shared secrets.

NIP-46 remote signer defaults to NIP-44 with NIP-04 fallback. Standalone decrypt() helper auto-detects the format.

Common Patterns

1. ECDH = secp256k1 shared point, take x-coordinate. Every implementation does this. The NIP-04 path uses the raw x-coordinate as the AES key. The NIP-44 path feeds it into HKDF-Extract with salt "nip44-v2" to produce a conversation key.

2. Message format for NIP-44 v2 is fixed: [version=2][nonce(32)][ct][hmac(32)], base64-encoded. HMAC covers nonce || ciphertext (the nonce is AAD, not encrypted).

3. Padding scheme is uniform: 2-byte big-endian length prefix, then zero-pad to the next power-of-2 chunk (min 32 bytes, max ~65KB). Decryption must verify the length prefix and that padding bytes are zero.

4. Per-message keys derived via HKDF-Expand with the conversation key as PRK and the random nonce as info, producing 76 bytes split into ChaCha20 key (32), ChaCha20 nonce (12), HMAC key (32).

5. NIP-04 is universally deprecated but still implemented for compatibility. Its format {ct}?iv={iv} is easy to detect, which enables auto-routing in higher-level decrypt helpers.

6. Architectural divergence on where encryption lives:

   • Free functions (rust-nostr, nostr-tools): standalone modules, no key class coupling. Most flexible.
   • Methods on a signer (applesauce, ndk, welshman): encryption belongs to the thing that holds the key. Enables hardware/remote signers.
   • We can do both: free functions in this chapter, signer abstraction added in a later chapter that calls them.

7. Conversation key as a first-class type (rust-nostr's ConversationKey, welshman's LRU cache) is worth borrowing. Without it, every message re-runs ECDH and HKDF-Extract, which is wasteful for chat-like workloads. With it, callers can derive once and encrypt many.

8. Plaintext caching is an application concern, not a library concern. Several libraries (applesauce, ndk) cache decrypted event content, but always at a layer above the raw crypto.

Considerations for Our Implementation

Crate placement: Encryption belongs in coracle-lib next to PrivateKey and PublicKey (coracle-lib/src/keys.rs). Keep it stateless and signer-agnostic; a later signer chapter can wrap it.

Dependencies to add:

• secp256k1 is already pulled in by the keys chapter — use its ecdh module. Alternative: k256 (RustCrypto). secp256k1 is simpler and matches BIP340 semantics.
• aes + cbc for NIP-04
• chacha20 for NIP-44
• hmac + sha2 (RustCrypto) or bitcoin_hashes for HMAC-SHA256 + HKDF. RustCrypto is more idiomatic and we can use the hkdf crate directly.
• base64 (or base64ct) for payload encoding
• rand / rand_core for IV and nonce generation (already a transitive dep)

Keep dependencies minimal — prefer one ecosystem (RustCrypto: aes, cbc, chacha20, hmac, sha2, hkdf) for consistency.

API shape proposal:

Free functions in a new coracle-lib/src/encryption.rs module:

pub fn shared_secret(sk: &PrivateKey, pk: &PublicKey) -> [u8; 32];

pub mod nip04 {
    pub fn encrypt(sk: &PrivateKey, pk: &PublicKey, plaintext: &str) -> String;
    pub fn decrypt(sk: &PrivateKey, pk: &PublicKey, payload: &str) -> Result<String, Error>;
}

pub mod nip44 {
    pub struct ConversationKey([u8; 32]);
    impl ConversationKey {
        pub fn derive(sk: &PrivateKey, pk: &PublicKey) -> Self;
    }
    pub fn encrypt(ck: &ConversationKey, plaintext: &str) -> Result<String, Error>;
    pub fn decrypt(ck: &ConversationKey, payload: &str) -> Result<String, Error>;
}

Plus convenience methods on PrivateKey that wrap these (per the user's hint):

impl PrivateKey {
    pub fn nip04_encrypt(&self, pk: &PublicKey, msg: &str) -> String;
    pub fn nip04_decrypt(&self, pk: &PublicKey, payload: &str) -> Result<String, Error>;
    pub fn nip44_encrypt(&self, pk: &PublicKey, msg: &str) -> Result<String, Error>;
    pub fn nip44_decrypt(&self, pk: &PublicKey, payload: &str) -> Result<String, Error>;
}

The ConversationKey type is the load-bearing abstraction — methods on PrivateKey are the friendly wrapper that derives one ad-hoc and discards it.

Error type: A single EncryptionError enum covering: invalid payload format, base64 decode, version mismatch, HMAC mismatch, padding error, message too long, message empty, UTF-8 error. Don't split into NIP-04 and NIP-44 enums — too much ceremony for a teaching resource.

Padding & format constants: Define them as named constants with comments referencing the NIP, since the magic numbers (32, 65535, "nip44-v2", 76) are otherwise opaque.

Test vectors: Pull a small subset of the official NIP-44 vectors into coracle-lib/tests/nip44_vectors.rs for confidence. NIP-04 has no official vectors but a round-trip test plus interop with rust-nostr's known ciphertext is sufficient.

Out of scope for this chapter:

• Gift wrap (NIP-59) — separate chapter
• Encrypted event helpers (kind→scheme dispatch) — belongs to events/signer chapter
• Conversation key caching — application concern; mention in passing
• NIP-49 password-encrypted keys — separate concern, possibly a signer chapter

Narrative arc:

1. Why encryption is situational, not foundational. NIP-04 → NIP-44 motivation.
2. ECDH shared secret derivation: the common foundation.
3. NIP-04: the simple, deprecated path. Show it first because it's smaller.
4. NIP-44 v2: the message format, padding, conversation key, per-message keys, encrypt/decrypt pipeline. Show why it's an improvement.
5. Wiring it onto PrivateKey for ergonomics.
6. Tests against round-trips and known vectors.

Research complete. You can proceed with /plan-chapter 03-encryption.