Magneteco
Architecture

Write Path

How content flows into Magneteco memory

Write Path

The write path handles all content ingestion—conversations, events, and webhooks—transforming them into structured memory.

Overview

Input (conversation/event)

Store raw resource (S3)

Create resource record (Postgres)

Queue for extraction (SQS)

Extract atomic facts (LLM)

Classify by category

Generate embeddings

Write items to Postgres

Evolve category summaries (LLM)

Extract entities → Neo4j nodes
Extract relationships → Neo4j edges

Components

Resource Writer

Handles the synchronous part of the write path:

  1. Stores raw content to S3 (immutable, timestamped)
  2. Creates reference record in PostgreSQL
  3. Queues for async extraction
// POST /memorize
async function memorize(request: MemorizeRequest) {
  // 1. Store to S3
  const s3Key = `${request.appId}/${request.userId}/${uuid()}.json`;
  await s3.putObject(bucket, s3Key, request.content);

  // 2. Create resource record
  const resource = await db.insert(memoryResources).values({
    appId: request.appId,
    userId: request.userId,
    s3Key,
    contentType: request.contentType,
    metadata: request.metadata,
  });

  // 3. Queue for extraction
  await sqs.sendMessage(extractionQueue, {
    resourceId: resource.id,
    appId: request.appId,
  });

  return { status: 'queued', resourceId: resource.id };
}

Fact Extractor (Lambda)

Processes queued extraction jobs:

  1. Receives SQS message
  2. Loads domain config for app
  3. Uses LLM to extract atomic facts
  4. Classifies facts into categories
  5. Identifies entities and relationships
  6. Queues results for writing
// Extraction prompt uses domain config
const extractionPrompt = `
You are extracting memories for ${config.appId}: ${config.description}

## Categories
${config.categories.map(c => `- ${c.name}: ${c.description}`).join('\n')}

## Entity Types
${config.entityTypes.map(e => `- ${e.name}: ${e.description}`).join('\n')}

## Content
${content}

Extract atomic facts, entities, and relationships as JSON.
`;

Fact Writer (Lambda)

Persists extracted data:

  1. Generates embeddings for each fact
  2. Writes items to PostgreSQL
  3. Evolves category summaries
  4. Creates/updates Neo4j nodes and relationships

Category Summary Evolution

Summaries are evolved, not replaced:

async function evolveSummary(
  existingSummary: string | null,
  newFacts: string[],
  config: CategoryDefinition
): Promise<string> {
  const prompt = `
You are updating a memory summary for category "${config.name}".
${config.description}

Existing summary:
${existingSummary || '(No existing summary)'}

New facts to incorporate:
${newFacts.map((f, i) => `${i + 1}. ${f}`).join('\n')}

Rules:
- If new facts contradict old information, use the new information
- Keep the summary concise but comprehensive
- Preserve important historical context
- Maximum 500 words

Updated summary:`;

  return await llm.complete(prompt);
}

Conflict Resolution

When relationships conflict:

async function handleRelationship(rel: ExtractedRelationship) {
  const relDef = config.relationshipTypes.find(r => r.name === rel.type);

  if (relDef?.exclusive) {
    // Archive any existing relationship of this type
    await neo4j.run(`
      MATCH (s:Entity {app_id: $appId, name: $subject})
            -[old:RELATES_TO {type: $type}]->(:Entity)
      WHERE old.archived = false
      SET old.archived = true, old.archived_at = datetime()
    `, { appId, subject: rel.subject, type: rel.type });
  }

  // Create new relationship
  await neo4j.run(`
    MATCH (s:Entity {app_id: $appId, name: $subject})
    MATCH (o:Entity {app_id: $appId, name: $object})
    CREATE (s)-[:RELATES_TO {
      type: $type,
      created_at: datetime(),
      archived: false,
      properties: $properties
    }]->(o)
  `, { appId, subject: rel.subject, object: rel.object, type: rel.type, properties: rel.properties });
}

Performance

StageLatencyNotes
S3 write~50msParallel with DB
Postgres insert~20msResource record
SQS queue~30msAsync handoff
Total sync~100msReturns to client
LLM extraction~3-5sAsync
Embedding generation~500msBatch
Postgres items~100msBulk insert
Neo4j updates~200msBatch
Total async~5-10sBackground

On this page