Architecture
Write Path
How content flows into Magneteco memory
Write Path
The write path handles all content ingestion—conversations, events, and webhooks—transforming them into structured memory.
Overview
Input (conversation/event)
↓
Store raw resource (S3)
↓
Create resource record (Postgres)
↓
Queue for extraction (SQS)
↓
Extract atomic facts (LLM)
↓
Classify by category
↓
Generate embeddings
↓
Write items to Postgres
↓
Evolve category summaries (LLM)
↓
Extract entities → Neo4j nodes
Extract relationships → Neo4j edgesComponents
Resource Writer
Handles the synchronous part of the write path:
- Stores raw content to S3 (immutable, timestamped)
- Creates reference record in PostgreSQL
- Queues for async extraction
// POST /memorize
async function memorize(request: MemorizeRequest) {
// 1. Store to S3
const s3Key = `${request.appId}/${request.userId}/${uuid()}.json`;
await s3.putObject(bucket, s3Key, request.content);
// 2. Create resource record
const resource = await db.insert(memoryResources).values({
appId: request.appId,
userId: request.userId,
s3Key,
contentType: request.contentType,
metadata: request.metadata,
});
// 3. Queue for extraction
await sqs.sendMessage(extractionQueue, {
resourceId: resource.id,
appId: request.appId,
});
return { status: 'queued', resourceId: resource.id };
}Fact Extractor (Lambda)
Processes queued extraction jobs:
- Receives SQS message
- Loads domain config for app
- Uses LLM to extract atomic facts
- Classifies facts into categories
- Identifies entities and relationships
- Queues results for writing
// Extraction prompt uses domain config
const extractionPrompt = `
You are extracting memories for ${config.appId}: ${config.description}
## Categories
${config.categories.map(c => `- ${c.name}: ${c.description}`).join('\n')}
## Entity Types
${config.entityTypes.map(e => `- ${e.name}: ${e.description}`).join('\n')}
## Content
${content}
Extract atomic facts, entities, and relationships as JSON.
`;Fact Writer (Lambda)
Persists extracted data:
- Generates embeddings for each fact
- Writes items to PostgreSQL
- Evolves category summaries
- Creates/updates Neo4j nodes and relationships
Category Summary Evolution
Summaries are evolved, not replaced:
async function evolveSummary(
existingSummary: string | null,
newFacts: string[],
config: CategoryDefinition
): Promise<string> {
const prompt = `
You are updating a memory summary for category "${config.name}".
${config.description}
Existing summary:
${existingSummary || '(No existing summary)'}
New facts to incorporate:
${newFacts.map((f, i) => `${i + 1}. ${f}`).join('\n')}
Rules:
- If new facts contradict old information, use the new information
- Keep the summary concise but comprehensive
- Preserve important historical context
- Maximum 500 words
Updated summary:`;
return await llm.complete(prompt);
}Conflict Resolution
When relationships conflict:
async function handleRelationship(rel: ExtractedRelationship) {
const relDef = config.relationshipTypes.find(r => r.name === rel.type);
if (relDef?.exclusive) {
// Archive any existing relationship of this type
await neo4j.run(`
MATCH (s:Entity {app_id: $appId, name: $subject})
-[old:RELATES_TO {type: $type}]->(:Entity)
WHERE old.archived = false
SET old.archived = true, old.archived_at = datetime()
`, { appId, subject: rel.subject, type: rel.type });
}
// Create new relationship
await neo4j.run(`
MATCH (s:Entity {app_id: $appId, name: $subject})
MATCH (o:Entity {app_id: $appId, name: $object})
CREATE (s)-[:RELATES_TO {
type: $type,
created_at: datetime(),
archived: false,
properties: $properties
}]->(o)
`, { appId, subject: rel.subject, object: rel.object, type: rel.type, properties: rel.properties });
}Performance
| Stage | Latency | Notes |
|---|---|---|
| S3 write | ~50ms | Parallel with DB |
| Postgres insert | ~20ms | Resource record |
| SQS queue | ~30ms | Async handoff |
| Total sync | ~100ms | Returns to client |
| LLM extraction | ~3-5s | Async |
| Embedding generation | ~500ms | Batch |
| Postgres items | ~100ms | Bulk insert |
| Neo4j updates | ~200ms | Batch |
| Total async | ~5-10s | Background |