Table of Contents
- Architecture: Journal System
Architecture: Journal System
Layer 2 - Episodic Memory, Importance Scoring, and Reflection
Overview
The journal system provides episodic memory with importance-weighted retrieval:
- Journal entries - Narrative observations with source metadata
- Importance scoring - Two-tier system (heuristic + LLM)
- Reflection - Cumulative threshold triggers meta-learning
- Lifecycle - Creation → Sleep upgrade → Consolidation → Pruning
Research foundation: Generative Agents (Park et al. 2023)
1. Journal Entry Structure
Stored on character.db.journal:
db.journal = {
"entries": [
{
"id": 42,
"timestamp": "2025-12-06T14:30:00Z",
"content": "Player Alice prefers formal address and dislikes jokes",
# Source metadata (O-MEM inspired)
"source_type": "direct", # direct | observation | inference | environmental
"source_trust": 0.9, # 0.0-1.0
"source_entity": "Alice", # Who provided this info
# Importance scoring
"importance": 7, # 1-10 scale
"importance_method": "llm", # heuristic | llm | manual
# Organization
"tags": ["player_preference", "communication"],
"related_projects": ["alice_quest"],
},
],
"entry_count": 42,
"max_entries": 100,
"last_reflection": "2025-12-06T10:00:00Z",
"consolidated_entry_ids": [1, 2, 5, 12], # Moved to Mem0
}
Source Types and Trust
| Source Type | Default Trust | Meaning |
|---|---|---|
direct |
0.9 | Told directly by entity |
observation |
0.8 | Witnessed firsthand |
inference |
0.6 | Deduced from other facts |
environmental |
0.3 | Overheard, ambient |
Trust scores influence memory retrieval filtering.
2. Importance Scoring
Two-tier system balances immediate feedback with accuracy.
Tier 1: Heuristic Scoring (Immediate)
Called synchronously during add_journal_entry:
def score_importance_heuristic(content, source_type, tags):
score = 5 # Base score (neutral midpoint)
# Source type modifiers
source_modifiers = {
"direct": +2,
"observation": +1,
"inference": 0,
"environmental": -1,
}
score += source_modifiers.get(source_type, 0)
# High-importance keywords (+2 each, max +4)
high_keywords = [
"player", "conflict", "discovery", "secret", "revealed",
"attack", "danger", "important", "urgent", "critical",
"death", "birth", "marriage", "betrayal", "alliance",
"war", "peace", "treasure", "quest",
]
keyword_bonus = sum(2 for kw in high_keywords if kw in content.lower())
score += min(keyword_bonus, 4)
# Low-importance keywords (-1 each)
low_keywords = ["routine", "walked", "moved", "entered", "ordinary"]
score -= sum(1 for kw in low_keywords if kw in content.lower())
# Content bonuses
if len(content) > 200:
score += 1
if "!" in content or "?" in content:
score += 1
return max(1, min(10, score)) # Clamp to 1-10
Tier 2: LLM Scoring (Accurate)
Runs during sleep phase via score_pending_entries():
@inlineCallbacks
def score_pending_entries(script, character, max_per_tick=3):
# Find entries with heuristic scores
entries = [e for e in journal["entries"]
if e.get("importance_method") == "heuristic"]
for entry in entries[:max_per_tick]:
# LLM evaluates significance
prompt = f"""Rate the significance of this event (1-10):
{entry["content"]}
1 = mundane routine
5 = moderately notable
10 = extremely significant (major revelation, emotional event)
Return only the integer score."""
llm_score = yield llm_client.complete(prompt)
# Update entry
entry["importance"] = int(llm_score)
entry["importance_method"] = "llm"
3. Reflection System
Generative Agents-inspired periodic self-reflection.
Reflection State
Tracked in script.db.reflection_state:
db.reflection_state = {
"cumulative_importance": 0.0, # Running total
"threshold": 150, # Trigger when reached
"last_reflection_time": None,
"reflection_count": 0,
"entries_since_reflection": [41, 42], # Entry IDs for reflection window
}
Threshold Trigger
Each journal entry accumulates importance:
def update_reflection_state(script, importance, entry_id):
state = script.db.reflection_state
state["cumulative_importance"] += importance
state["entries_since_reflection"].append(entry_id)
# Check: cumulative_importance >= 150 triggers reflection
Three-Step Reflection Process
1. QUESTION GENERATION
Input: Recent entries from entries_since_reflection
Output: 3 questions about world state and patterns
Example questions:
- "What patterns do I observe in player behavior?"
- "How have relationships in the game world changed?"
- "What recurring challenges have emerged?"
2. EVIDENCE RETRIEVAL
For each question:
- Search memory client for supporting evidence
- Up to 10 memories per question
- Normalize to {id, content, score}
3. INSIGHT GENERATION
LLM synthesizes insights with evidence citations
→ Persona Protection Filter removes self-referential insights
→ Store as [SYNTHESIS] journal entry with importance=8
Persona Protection in Insights
# Filtered patterns (regex):
self_reference_patterns = [
r"I should",
r"my behavior",
r"the assistant",
r"I need to",
r"I will",
]
# Example:
# REJECTED: "I should be more patient with players"
# ACCEPTED: "Players respond better to patient guidance"
State Reset After Reflection
def reset_reflection_state(script):
script.db.reflection_state = {
"cumulative_importance": 0.0, # Reset counter
"threshold": 150, # Preserve threshold
"last_reflection_time": now(), # Record time
"reflection_count": state["reflection_count"] + 1,
"entries_since_reflection": [], # Clear window
}
4. Journal Tools
add_journal_entry
class AddJournalEntryTool(Tool):
name = "add_journal_entry"
category = ToolCategory.SAFE_CHAIN
parameters = {
"content": str, # Required: narrative text
"tags": list[str], # Optional: categorization
"related_projects": list[str],
"source_type": "direct|observation|inference|environmental",
"source_trust": float, # 0.0-1.0, auto-calculated if omitted
"source_entity": str, # Who provided info
"importance": int, # 1-10, auto-scored if omitted
}
Features:
- Automatic importance scoring if not provided
- Trust defaults from source type
- Updates reflection state
- Syncs to RAG if enabled
- Auto-prunes oldest when exceeding max_entries
search_journal
class SearchJournalTool(Tool):
name = "search_journal"
category = ToolCategory.SAFE_CHAIN
cacheable = True
cache_ttl = 300.0 # 5 minutes
parameters = {
"query": str, # Text or semantic search
"tags": list[str], # Filter by tags (AND logic)
"days_back": int, # Time window
"related_to_project": str, # Project filter
"limit": int, # Max results (default 10)
"semantic": bool, # Use RAG if available
}
Search modes:
- Semantic (default if RAG enabled): Vector similarity search
- Text fallback: Case-insensitive substring matching
review_journal
class ReviewJournalTool(Tool):
name = "review_journal"
category = ToolCategory.SAFE_CHAIN
parameters = {
"synthesis": str, # Required: your insights after review
"days_back": int, # Days to review (default 7)
"tags": list[str], # Focus on specific tags
"save_as_entry": bool, # Save synthesis as journal entry (default True)
}
Creates [SYNTHESIS] entries tagged with ["synthesis", "meta_learning"].
5. Episodic Search Scoring
Hybrid scoring for retrieval (Generative Agents formula):
def score_episodic_memory(entry, query, now, alpha_recency=1.0,
alpha_importance=1.0, alpha_relevance=1.0):
# Recency: exponential decay
age_hours = (now - entry["timestamp"]).total_seconds() / 3600
recency = math.exp(-0.99 * age_hours)
# Importance: normalized 1-10 → 0.1-1.0
importance = entry["importance"] / 10.0
# Relevance: keyword overlap
query_words = set(query.lower().split())
content_words = set(entry["content"].lower().split())
overlap = len(query_words & content_words)
relevance = overlap / max(len(query_words), 1)
# Weighted combination
total_alpha = alpha_recency + alpha_importance + alpha_relevance
score = (alpha_recency * recency +
alpha_importance * importance +
alpha_relevance * relevance) / total_alpha
return score
6. Journal Lifecycle
┌──────────────────────────────────────────────────────────────────┐
│ CREATION (Awake) │
├──────────────────────────────────────────────────────────────────┤
│ add_journal_entry() │
│ → Heuristic importance scoring │
│ → Update reflection state (cumulative += importance) │
│ → Sync to RAG if enabled │
│ → Auto-prune if > max_entries │
└──────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────┐
│ SLEEP PHASE: Compacting │
├──────────────────────────────────────────────────────────────────┤
│ score_pending_entries() │
│ → Find entries with importance_method="heuristic" │
│ → LLM upgrade to accurate scores (3 per tick) │
│ → Update importance_method="llm" │
│ │
│ run_sleep_consolidation() │
│ → Move unconsolidated entries to Mem0 │
│ → Mark entries as consolidated │
└──────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────┐
│ SLEEP PHASE: Dreaming │
├──────────────────────────────────────────────────────────────────┤
│ check_reflection_trigger() │
│ → If cumulative_importance >= 150: │
│ run_reflection() │
│ → Generate questions │
│ → Retrieve evidence │
│ → Synthesize insights │
│ → Filter self-references │
│ → Store [SYNTHESIS] entry │
│ reset_reflection_state() │
│ │
│ prune_low_importance_entries() │
│ → Remove entries with importance <= 3 AND age > 30 days │
│ → Up to 10 entries per call │
└──────────────────────────────────────────────────────────────────┘
Key Files
| File | Lines | Purpose |
|---|---|---|
tools/journal.py |
23-266 | AddJournalEntryTool |
tools/journal.py |
269-464 | SearchJournalTool |
tools/journal.py |
467-594 | ReviewJournalTool |
importance_scoring.py |
116-176 | score_importance_heuristic() |
importance_scoring.py |
184-237 | score_importance_llm() |
importance_scoring.py |
272-367 | score_pending_entries() |
importance_scoring.py |
375-449 | Reflection state management |
generative_reflection.py |
145-199 | generate_reflection_questions() |
generative_reflection.py |
275-345 | generate_insights() |
generative_reflection.py |
385-477 | run_reflection() |
helpers.py |
2244-2350 | score_episodic_memory() |
helpers.py |
2437-2559 | prune_low_importance_entries() |
See also: Architecture-Self-Management | Architecture-Memory-and-Sleep | Data-Flow-03-Memory-Consolidation