Add vitepress docs

This commit is contained in:
Ticruz
2025-02-04 14:43:40 +01:00
committed by Jon Staab
parent 43255bcb74
commit 94375a56ec
84 changed files with 10821 additions and 139 deletions
+101
View File
@@ -0,0 +1,101 @@
# Feed Compiler
The `FeedCompiler` class is responsible for transforming feed definitions into executable relay requests. It handles the complex task of converting various feed types into optimized filters and relay selections.
## Overview
```typescript
class FeedCompiler {
constructor(readonly options: FeedOptions)
canCompile(feed: Feed): boolean
compile(feed: Feed): Promise<RequestItem[]>
}
```
## Feed Compilation Process
The compiler transforms feed definitions into `RequestItem[]`, where each item contains:
```typescript
type RequestItem = {
relays?: string[] // Specific relays to query
filters?: Filter[] // Nostr filters to apply
}
```
## Examples
### Basic Feed Compilation
```typescript
const compiler = new FeedCompiler(options)
// Simple author feed
const feed = [FeedType.Author, "pubkey1", "pubkey2"]
const requests = await compiler.compile(feed)
// => [{ filters: [{ authors: ["pubkey1", "pubkey2"] }] }]
```
### Complex Feed Compilation
```typescript
// Complex feed with multiple operations
const feed = [
FeedType.Intersection,
[FeedType.Kind, 1],
[
FeedType.Union,
[FeedType.Scope, Scope.Follows],
[FeedType.List, { addresses: ["trending"] }]
]
]
const requests = await compiler.compile(feed)
// Compiles to optimized filters for relay queries
```
### DVM Integration
```typescript
const feed = [
FeedType.DVM,
{
kind: 5300,
mappings: [
["p", [FeedType.Author]],
["t", [FeedType.Tag, "#t"]]
]
}
]
const requests = await compiler.compile(feed)
// Queries DVM and compiles resulting tags into feeds
```
## Implementation Notes
### Optimization Strategies
1. **Filter Merging**: Similar filters are combined when possible
```typescript
// Before: [{ authors: ["a"] }, { authors: ["b"] }]
// After: [{ authors: ["a", "b"] }]
```
2. **Relay Grouping**: Requests are grouped by relay where possible
```typescript
// Filters are organized by relay to minimize connections
filtersByRelay: Map<string, Filter[]>
```
3. **Deduplication**: Duplicate values are removed using `uniq`
```typescript
uniq(scopes.flatMap(this.options.getPubkeysForScope))
```
### Error Handling
The compiler includes various safety checks:
```typescript
canCompile(feed: Feed): boolean {
// Checks if feed type is supported
// Recursively checks sub-feeds
}
```
+132
View File
@@ -0,0 +1,132 @@
# Feed Controller
The `FeedController` class is responsible for managing and executing feed queries in a performant and organized manner. It handles the compilation of feed definitions into executable queries and manages the loading of events based on those queries.
## Usage
```typescript
import { FeedController } from '@welshman/feeds'
const controller = new FeedController({
feed: yourFeedDefinition,
request: async ({ filters, relays, onEvent }) => {
// Your implementation for fetching events
},
requestDVM: async ({ kind, tags, relays, onEvent }) => {
// Your implementation for DVM requests
},
getPubkeysForScope: (scope) => {
// Return pubkeys for given scope
return ['pubkey1', 'pubkey2']
},
getPubkeysForWOTRange: (min, max) => {
// Return pubkeys within WOT range
return ['pubkey1', 'pubkey2']
},
onEvent: (event) => {
// Handle received events
},
onExhausted: () => {
// Called when no more events are available
},
useWindowing: true, // Optional: enable time-window based loading
})
```
## API Reference
### Constructor
```typescript
constructor(options: FeedOptions)
```
Creates a new feed controller with the given options:
- `feed`: The feed definition to execute
- `request`: Function to fetch events from relays
- `requestDVM`: Function to fetch events from DVMs
- `getPubkeysForScope`: Function to get pubkeys for a scope
- `getPubkeysForWOTRange`: Function to get pubkeys within a WOT range
- `onEvent`: Optional callback for received events
- `onExhausted`: Optional callback when feed is exhausted
- `useWindowing`: Optional flag to enable time-window based loading
### Methods
#### `load(limit: number): Promise<void>`
```typescript
const controller = new FeedController(options)
await controller.load(10) // Load 10 events
```
Loads events from the feed up to the specified limit.
#### `getLoader(): Promise<(limit: number) => Promise<void>>`
Gets the loader function for this feed. Usually called internally by `load()`.
#### `getRequestItems(): Promise<RequestItem[] | undefined>`
Gets the compiled request items for this feed. Usually called internally.
## Advanced Features
### Time Windowing
When `useWindowing` is enabled, the controller uses a time-based window approach to load events:
```typescript
const controller = new FeedController({
...options,
useWindowing: true
})
```
This is useful for:
- Loading recent events first
- Handling large datasets efficiently
- Progressive loading of historical data
## Examples
### Basic Loading
```typescript
const controller = new FeedController(options)
await controller.load(20) // Load 20 events
```
### Custom Loading Strategy
```typescript
const controller = new FeedController({
...options,
useWindowing: true,
onEvent: (event) => {
console.log('Received event:', event.id)
},
onExhausted: () => {
console.log('No more events available')
}
})
// Load events in batches
async function loadAllEvents() {
while (!exhausted) {
await controller.load(10)
await new Promise(resolve => setTimeout(resolve, 1000))
}
}
```
### Error Handling
```typescript
try {
await controller.load(10)
} catch (error) {
if (error.message.includes('relay')) {
// Handle relay errors
} else {
// Handle other errors
}
}
```
+341
View File
@@ -0,0 +1,341 @@
# Feed Types and Core Definitions
This module defines the core types and structures used to build Nostr feeds.
It provides a type-safe way to define complex feed compositions using various filtering mechanisms and set operations.
## Feed Types
```typescript
enum FeedType {
Address = "address", // Filter by event addresses
Author = "author", // Filter by author pubkeys
CreatedAt = "created_at", // Filter by timestamp
DVM = "dvm", // Data Vending Machine based feed
Difference = "difference", // Set difference operation
ID = "id", // Filter by event IDs
Intersection = "intersection", // Set intersection operation
Global = "global", // Global feed (no filters)
Kind = "kind", // Filter by event kinds
List = "list", // List-based feed
Label = "label", // Label-based feed
WOT = "wot", // Web of Trust based feed
Relay = "relay", // Relay-specific feed
Scope = "scope", // Scoped feed (followers, network)
Search = "search", // Search-based feed
Tag = "tag", // Filter by specific tags
Union = "union" // Set union operation
}
```
## Scope Types
```typescript
enum Scope {
Followers = "followers", // People who follow the user
Follows = "follows", // People the user follows
Network = "network", // Extended network
Self = "self" // The signed in user
}
```
## Feed Definitions
Each feed type has its own structure:
### Basic Filter Feeds
```typescript
type AddressFeed = [type: FeedType.Address, ...addresses: string[]]
type AuthorFeed = [type: FeedType.Author, ...pubkeys: string[]]
type IDFeed = [type: FeedType.ID, ...ids: string[]]
type KindFeed = [type: FeedType.Kind, ...kinds: number[]]
type TagFeed = [type: FeedType.Tag, key: string, ...values: string[]]
```
### Time-based Feeds
```typescript
type CreatedAtItem = {
since?: number
until?: number
relative?: string[] // For relative time references
}
type CreatedAtFeed = [type: FeedType.CreatedAt, ...items: CreatedAtItem[]]
```
### Advanced Filter Feeds
```typescript
// DVM-based feed
type DVMItem = {
kind: number
tags?: string[][]
relays?: string[]
mappings?: TagFeedMapping[]
}
type DVMFeed = [type: FeedType.DVM, ...items: DVMItem[]]
// List-based feed
type ListItem = {
addresses: string[]
mappings?: TagFeedMapping[]
}
type ListFeed = [type: FeedType.List, ...items: ListItem[]]
// Label-based feed
type LabelItem = {
relays?: string[]
authors?: string[]
[key: `#${string}`]: string[]
mappings?: TagFeedMapping[]
}
type LabelFeed = [type: FeedType.Label, ...items: LabelItem[]]
// Web of Trust feed
type WOTItem = {
min?: number
max?: number
}
type WOTFeed = [type: FeedType.WOT, ...items: WOTItem[]]
```
## Tag Feed Mapping
`TagFeedMapping` is a mechanism to convert event tags into feed definitions. It's particularly useful when working with DVMs, Lists, and Labels where you want to interpret tags in a specific way.
```typescript
type TagFeedMapping = [string, Feed]
```
### Usage
```typescript
// Example mappings
const mappings: TagFeedMapping[] = [
// Convert 'p' tags into author feeds
["p", [FeedType.Author]],
// Convert 't' tags into hashtag filters
["t", [FeedType.Tag, "#t"]],
// Convert 'e' tags into event ID feeds
["e", [FeedType.ID]],
// Convert 'r' tags into relay feeds
["r", [FeedType.Relay]]
]
// Using mappings in a DVM feed
const dvmFeed: Feed = [
FeedType.DVM,
{
kind: 5300,
mappings: mappings
}
]
// Using mappings in a List feed
const listFeed: Feed = [
FeedType.List,
{
addresses: ["list_id"],
mappings: mappings
}
]
```
### Default Mappings
The system comes with default mappings for common tags:
```typescript
const defaultTagFeedMappings: TagFeedMapping[] = [
["a", [FeedType.Address]], // Address references
["e", [FeedType.ID]], // Event references
["p", [FeedType.Author]], // Person/Pubkey references
["r", [FeedType.Relay]], // Relay references
["t", [FeedType.Tag, "#t"]], // Hashtags
]
```
## Set Operation Feeds
### Union Feed
A Union feed combines multiple feeds with an OR operation. Events matching any of the constituent feeds will be included.
```typescript
type UnionFeed = [type: FeedType.Union, ...feeds: Feed[]]
// Example: Events from either Alice OR Bob
const unionFeed: UnionFeed = [
FeedType.Union,
[FeedType.Author, "alice_pubkey"],
[FeedType.Author, "bob_pubkey"]
]
// Example: Events from a list OR matching a search term
const complexUnion: UnionFeed = [
FeedType.Union,
[FeedType.List, { addresses: ["trending_list"] }],
[FeedType.Search, "bitcoin"]
]
```
### Intersection Feed
An Intersection feed combines multiple feeds with an AND operation. Only events that match all constituent feeds will be included.
```typescript
type IntersectionFeed = [type: FeedType.Intersection, ...feeds: Feed[]]
// Example: Text notes (kind 1) from trusted authors
const intersectionFeed: IntersectionFeed = [
FeedType.Intersection,
[FeedType.Kind, 1],
[FeedType.WOT, { min: 0.5 }]
]
// Example: Recent posts from followed users
const timeAndScope: IntersectionFeed = [
FeedType.Intersection,
[FeedType.CreatedAt, { since: Date.now() - 86400000 }], // Last 24h
[FeedType.Scope, Scope.Follows]
]
```
### Difference Feed
A Difference feed excludes events from the second feed from the first feed (NOT operation).
```typescript
type DifferenceFeed = [type: FeedType.Difference, ...feeds: Feed[]]
// Example: Posts from everyone except blocked users
const differenceFeed: DifferenceFeed = [
FeedType.Difference,
[FeedType.Global], // All events
[FeedType.List, { addresses: ["blocked_users"] }] // Except from blocked users
]
// Example: Posts from follows except reposts
const noReposts: DifferenceFeed = [
FeedType.Difference,
[FeedType.Scope, Scope.Follows],
[FeedType.Kind, 6] // Kind 6 is repost
]
```
### Complex Combinations
You can nest set operations to create sophisticated feed definitions:
```typescript
// Posts that are either:
// - from trusted authors AND about bitcoin
// - OR from a curated list
const complexFeed: Feed = [
FeedType.Union,
[
FeedType.Intersection,
[FeedType.WOT, { min: 0.7 }],
[FeedType.Search, "bitcoin"]
],
[FeedType.List, { addresses: ["curated_content"] }]
]
// Posts that are:
// - from follows
// - AND (from the last 24h OR highly rated by DVMs)
// - AND NOT marked as sensitive content
const advancedFeed: Feed = [
FeedType.Difference,
[
FeedType.Intersection,
[FeedType.Scope, Scope.Follows],
[
FeedType.Union,
[FeedType.CreatedAt, { since: Date.now() - 86400000 }],
[FeedType.DVM, { kind: 5300, pubkey: "rating_dvm" }]
]
],
[FeedType.Label, { authors: ["content_warning_dvm"] }]
]
```
## Feed Controller Options
The `FeedOptions` interface defines the configuration required to execute a feed:
```typescript
interface FeedOptions {
// The feed definition to execute
feed: Feed
// Function to request events from relays
request: (opts: RequestOpts) => Promise<void>
// Function to request events from DVMs
requestDVM: (opts: DVMOpts) => Promise<void>
// Function to get pubkeys for a given scope
getPubkeysForScope: (scope: Scope) => string[]
// Function to get pubkeys within a WOT range
getPubkeysForWOTRange: (minWOT: number, maxWOT: number) => string[]
// Event handler
onEvent?: (event: TrustedEvent) => void
// Called when feed is exhausted
onExhausted?: () => void
// Enable time-window based loading
useWindowing?: boolean
// Optional abort controller
abortController?: AbortController
}
```
## Examples
### Simple Author Feed
```typescript
const authorFeed: Feed = [FeedType.Author, "pubkey1", "pubkey2"]
```
### Time-filtered Feed
```typescript
const recentFeed: Feed = [
FeedType.CreatedAt,
{
since: Date.now() - 24 * 60 * 60 * 1000, // Last 24 hours
relative: ["since"]
}
]
```
### Complex Feed Composition
```typescript
const complexFeed: Feed = [
FeedType.Intersection,
[FeedType.Kind, 1], // Text notes
[FeedType.WOT, { min: 0.5 }], // Trusted authors
[
FeedType.Union,
[FeedType.Scope, Scope.Follows], // From follows
[FeedType.List, { addresses: ["list_id"] }] // Or from list
]
]
```
### DVM Feed with Mappings
```typescript
const dvmFeed: Feed = [
FeedType.DVM,
{
kind: 5300,
mappings: [
["p", [FeedType.Author]], // Map 'p' tags to authors
["t", [FeedType.Tag, "#t"]] // Map 't' tags to hashtags
]
}
]
```
This core module provides the foundation for building complex, type-safe feed definitions that can be executed by the [feed controller](/feeds/controller).
+19
View File
@@ -0,0 +1,19 @@
# @welshman/feeds
A powerful package for building and executing dynamic Nostr feeds.
It provides a declarative way to define complex feed compositions using set operations (union, intersection, difference) and various filtering mechanisms.
## What's Included
- **Feed Core** - Declarative feed definition with composable operations
- **Feed Compiler** - Transforms feed definitions into optimized relay requests
- **Feed Controller** - Manages feed execution and event loading
- **Feed Utils** - Helper functions for creating and manipulating feeds
- **Feed Types** - Supports authors, kinds, tags, DVMs, lists, WOT, and more
## Installation
```bash
npm install @welshman/feeds
```
+175
View File
@@ -0,0 +1,175 @@
# Feed Utilities
The utils module provides helper functions for creating, type-checking, and manipulating feed definitions. It includes factory functions, type guards, feed transformation utilities, and feed traversal tools.
## Feed Factory Functions
Create strongly-typed feed definitions:
```typescript
// Basic Feeds
const authors = makeAuthorFeed("pubkey1", "pubkey2")
const kinds = makeKindFeed(1, 6)
const search = makeSearchFeed("bitcoin", "nostr")
const global = makeGlobalFeed()
// Time-based Feeds
const recent = makeCreatedAtFeed({
since: Date.now() - 86400000,
relative: ["since"]
})
// Advanced Feeds
const dvm = makeDVMFeed({
kind: 5300,
mappings: [["p", [FeedType.Author]]]
})
const list = makeListFeed({
addresses: ["list_id"],
mappings: [["t", [FeedType.Tag, "#t"]]]
})
// Set Operations
const union = makeUnionFeed(authors, kinds)
const intersection = makeIntersectionFeed(authors, recent)
const difference = makeDifferenceFeed(global, authors)
```
## Type Guards
Check feed types safely:
```typescript
const feed: Feed = makeDVMFeed({ kind: 5300 })
if (isDVMFeed(feed)) {
// feed is now typed as DVMFeed
const [kind] = feed.slice(1)
}
if (hasSubFeeds(feed)) {
// feed is now typed as UnionFeed | IntersectionFeed | DifferenceFeed
const subFeeds = getFeedArgs(feed)
}
```
## Feed Transformations
### Tag to Feed Conversion
```typescript
// Default tag mappings
const defaultTagFeedMappings: TagFeedMapping[] = [
["a", [FeedType.Address]], // address tags
["e", [FeedType.ID]], // event references
["p", [FeedType.Author]], // people/pubkeys
["r", [FeedType.Relay]], // relay URLs
["t", [FeedType.Tag, "#t"]], // hashtags
]
// Convert event tags to feeds
const tags = [["p", "pubkey1"], ["t", "bitcoin"]]
const feeds = feedsFromTags(tags)
// => [[FeedType.Author, "pubkey1"], [FeedType.Tag, "#t", "bitcoin"]]
// Convert tags to a single intersection feed
const feed = feedFromTags(tags)
// => [FeedType.Intersection, [FeedType.Author, "pubkey1"], [FeedType.Tag, "#t", "bitcoin"]]
```
### Filter to Feed Conversion
```typescript
// Convert a single filter to feeds
const filter = {
kinds: [1],
authors: ["pubkey1"],
"#t": ["bitcoin"],
since: 1234567890
}
const feeds = feedsFromFilter(filter)
// => [
// [FeedType.CreatedAt, { since: 1234567890 }],
// [FeedType.Kind, 1],
// [FeedType.Author, "pubkey1"],
// [FeedType.Tag, "#t", "bitcoin"]
// ]
// Convert a filter to an intersection feed
const feed = feedFromFilter(filter)
// Convert multiple filters to a union feed
const feeds = feedFromFilters([filter1, filter2])
```
## Feed Traversal
Walk through a feed tree and visit each node:
```typescript
const feed = makeIntersectionFeed(
makeAuthorFeed("pubkey1"),
makeUnionFeed(
makeKindFeed(1),
makeTagFeed("#t", "bitcoin")
)
)
walkFeed(feed, (node) => {
console.log(`Visiting feed of type: ${node[0]}`)
})
```
## Type Extraction
Get typed arguments from feeds:
```typescript
function getFeedArgs(feed: IntersectionFeed): Feed[]
function getFeedArgs(feed: AuthorFeed): string[]
function getFeedArgs(feed: CreatedAtFeed): CreatedAtItem[]
function getFeedArgs(feed: WOTFeed): WOTItem[]
// ... and so on for each feed type
const feed = makeAuthorFeed("pubkey1", "pubkey2")
const pubkeys = getFeedArgs(feed) // => ["pubkey1", "pubkey2"]
```
## Best Practices
1. Use factory functions instead of raw arrays:
```typescript
// Good
const feed = makeAuthorFeed("pubkey1")
// Avoid
const feed = [FeedType.Author, "pubkey1"]
```
2. Use type guards for safe type narrowing:
```typescript
if (isAuthorFeed(feed)) {
const pubkeys = getFeedArgs(feed) // Properly typed
}
```
3. Use feed transformations for dynamic feed creation:
```typescript
// Convert event tags to feeds
const feeds = feedsFromTags(event.tags)
// Convert filters to feeds
const feed = feedFromFilter(filter)
```
4. Use feed traversal for analysis or transformation:
```typescript
const kinds = new Set<number>()
walkFeed(feed, (node) => {
if (isKindFeed(node)) {
getFeedArgs(node).forEach(k => kinds.add(k))
}
})
```