docs(user-guide): document SOUL/MEMORY/USER.md persona files (#604) (#610)

* docs(user-guide): document SOUL/MEMORY/USER.md persona files (#604)

The persistent-memory showcase links to the User Guide: Agents page for how
SOUL.md / MEMORY.md / USER.md are loaded at conversation start, but that page
never covered them (site search for the filenames returns nothing).

Add a "Persistent Persona" section to user-guide/agents.md: the three files and
what each holds, where they live (config dir + [memory_files]), how they load
(after the agent template, cached per conversation, per-section truncation),
named personas (--persona / personas/<name>/), and editing by hand or via the
memory_manage / user_profile_manage tools. Cross-links the distinct retrieval
memory backend to resolve the reporter's confusion.

Closes #604

* docs(user-guide): clarify memory_manage/user_profile_manage target the default MEMORY.md/USER.md

docs/user-guide/agents.md

@@ -19,6 +19,71 @@ Agents are the agentic logic layer of OpenJarvis. They determine how a query is
 
 ---
 
+## Persistent Persona: SOUL.md, MEMORY.md, USER.md
+
+Every agent's system prompt is assembled at conversation start by the `SystemPromptBuilder`, which injects up to three optional Markdown files -- the **persistent persona**. They are plain text you own and edit, loaded at the start of each conversation. There is no vector database or embedding cache behind them.
+
+| File | What it holds | Example line |
+|------|---------------|--------------|
+| `SOUL.md` | How the agent should behave -- tone, length, what to push back on | `Be concise. Challenge weak assumptions.` |
+| `MEMORY.md` | Facts about you, your projects, your preferences | `I deploy to Postgres, never MySQL.` |
+| `USER.md` | Who you are -- role, team, context | `Backend engineer at Acme, on the payments team.` |
+
+This persona is distinct from the retrieval [memory backend](memory.md): the persona is always-on Markdown context loaded into the prompt, while the memory backend is searchable long-term storage the agent queries on demand.
+
+### Where they live
+
+By default the files are read from the config directory:
+
+```
+~/.openjarvis/SOUL.md
+~/.openjarvis/MEMORY.md
+~/.openjarvis/USER.md
+```
+
+(The config directory honors `$OPENJARVIS_HOME` / `$XDG_DATA_HOME` when set.) The paths are configurable under `[memory_files]`:
+
+```toml
+[memory_files]
+soul_path    = "~/.openjarvis/SOUL.md"
+memory_path  = "~/.openjarvis/MEMORY.md"
+user_path    = "~/.openjarvis/USER.md"
+persona_name = ""    # optional named persona -- see below
+```
+
+### How they're loaded
+
+At the start of each conversation, `SystemPromptBuilder` reads each file as UTF-8 and adds its contents as a section of the system prompt, after the agent template and before the skill catalog:
+
+- **All three are optional.** A missing or empty file is skipped, so any subset works and an install with no persona files behaves exactly as before.
+- **Edits apply to the next conversation.** The files are read once when a conversation's prompt is built, so there is no restart or re-indexing -- edit or delete a line and it takes effect the next time you start a conversation.
+- **Each section is length-capped.** Files are truncated to a per-section character budget so a large `MEMORY.md` cannot crowd out the rest of the prompt.
+
+### Named personas
+
+A single install can answer as different personas without changing global config. A named persona lives in its own directory:
+
+```
+~/.openjarvis/personas/<name>/SOUL.md
+~/.openjarvis/personas/<name>/MEMORY.md
+~/.openjarvis/personas/<name>/USER.md
+```
+
+Select one per invocation, or opt out entirely:
+
+```bash
+jarvis ask --persona work  "summarize my open PRs"
+jarvis ask --persona none  "what is 2 + 2?"     # inject no persona
+```
+
+Set `persona_name` under `[memory_files]` to make a named persona the default. `persona_name = "none"` (equivalently `--persona none`) disables persona injection for that run.
+
+### Editing them
+
+`SOUL.md`, `MEMORY.md`, and `USER.md` are plain Markdown -- open them in any editor. `MEMORY.md` and `USER.md` can also be updated by the agent itself through the `memory_manage` and `user_profile_manage` tools when those are enabled, so the agent can record a new fact mid-conversation. These tools always target the default `MEMORY.md` and `USER.md` (under `~/.openjarvis/`), never a named persona's copies -- edit those by hand.
+
+---
+
 ## BaseAgent ABC
 
 All agents extend the abstract `BaseAgent` class.

src/openjarvis/agents/native_openhands.py

@@ -19,6 +19,7 @@ from openjarvis.agents.prompt_loader import (
 from openjarvis.core.events import EventBus
 from openjarvis.core.registry import AgentRegistry
 from openjarvis.core.types import Message, Role, ToolCall, ToolResult
+from openjarvis.engine._base import estimate_prompt_tokens
 from openjarvis.engine._stubs import InferenceEngine
 from openjarvis.tools._stubs import BaseTool, build_tool_descriptions
 
@@ -116,8 +117,7 @@ class NativeOpenHandsAgent(ToolUsingAgent):
         max_prompt_tokens: int = 3000,
     ) -> list[Message]:
         """Truncate messages if estimated token count exceeds limit."""
-        total_chars = sum(len(m.content) for m in messages)
-        estimated_tokens = total_chars // 4
+        estimated_tokens = estimate_prompt_tokens(messages)
         if estimated_tokens <= max_prompt_tokens:
             return messages
         # Find the last user message and truncate its content
@@ -125,7 +125,7 @@ class NativeOpenHandsAgent(ToolUsingAgent):
             if messages[i].role == Role.USER:
                 excess_tokens = estimated_tokens - max_prompt_tokens
                 excess_chars = excess_tokens * 4
-                original = messages[i].content
+                original = messages[i].content or ""
                 if len(original) > excess_chars + 200:
                     truncated = original[: len(original) - excess_chars]
                     messages[i] = Message(
@@ -258,7 +258,7 @@ class NativeOpenHandsAgent(ToolUsingAgent):
                 # still emitted before re-raising.
                 self._emit_turn_end(turns=1, error=True)
                 raise
-            content = self._strip_think_tags(result.get("content", ""))
+            content = self._strip_think_tags(result.get("content") or "")
             usage = result.get("usage", {})
             self._emit_turn_end(turns=1)
             return AgentResult(
@@ -315,7 +315,7 @@ class NativeOpenHandsAgent(ToolUsingAgent):
             for k in total_usage:
                 total_usage[k] += usage.get(k, 0)
 
-            content = result.get("content", "")
+            content = result.get("content") or ""
             # Strip think tags so they don't interfere with parsing
             content = self._strip_think_tags(content)
             last_content = content

src/openjarvis/core/types.py

@@ -63,7 +63,7 @@ class Message:
     """A single chat message (OpenAI-compatible structure)."""
 
     role: Role
-    content: str = ""
+    content: str | None = ""
     name: Optional[str] = None
     tool_calls: Optional[List[ToolCall]] = None
     tool_call_id: Optional[str] = None
@@ -73,6 +73,11 @@ class Message:
     # empty for text-only messages (the common case).
     images: Optional[List[str]] = None
 
+    @property
+    def text(self) -> str:
+        """Return message content as text, treating ``None`` as empty."""
+        return self.content or ""
+
 
 @dataclass(slots=True)
 class Conversation:

src/openjarvis/engine/_base.py

@@ -13,6 +13,22 @@ class EngineConnectionError(Exception):
     """Raised when an engine is unreachable."""
 
 
+_REASONING_METADATA_KEYS = ("reasoning_content", "thinking")
+
+
+def _message_estimated_chars(message: Message) -> int:
+    parts = [message.text]
+    for key in _REASONING_METADATA_KEYS:
+        value = message.metadata.get(key)
+        if isinstance(value, str):
+            parts.append(value)
+    for tc in message.tool_calls or []:
+        parts.extend((tc.id, tc.name, tc.arguments))
+    if message.tool_call_id:
+        parts.append(message.tool_call_id)
+    return sum(len(part) for part in parts)
+
+
 def messages_to_dicts(messages: Sequence[Message]) -> List[Dict[str, Any]]:
     """Convert ``Message`` objects to OpenAI-format dicts."""
     out: List[Dict[str, Any]] = []
@@ -53,9 +69,11 @@ def estimate_prompt_tokens(messages: Sequence[Message]) -> int:
     provider would charge.
 
     Uses ~4 characters per token (standard BPE average for English) plus
-    a small per-message overhead for role markers and separators.
+    a small per-message overhead for role markers and separators. Counts
+    content, reasoning metadata, tool-call payloads, and tool result IDs
+    because all are replayed into later prompt turns when present.
     """
-    total_chars = sum(len(m.content) for m in messages)
+    total_chars = sum(_message_estimated_chars(m) for m in messages)
     # ~4 tokens overhead per message for role markers / separators
     overhead = len(messages) * 4
     return max(1, total_chars // 4 + overhead)

tests/agents/test_native_openhands.py

@@ -8,7 +8,7 @@ from openjarvis.agents._stubs import AgentContext
 from openjarvis.agents.native_openhands import NativeOpenHandsAgent
 from openjarvis.core.events import EventBus, EventType
 from openjarvis.core.registry import AgentRegistry
-from openjarvis.core.types import Conversation, Message, Role, ToolResult
+from openjarvis.core.types import Conversation, Message, Role, ToolCall, ToolResult
 from openjarvis.tools._stubs import BaseTool, ToolSpec
 
 # ---------------------------------------------------------------------------
@@ -118,6 +118,51 @@ class TestNativeOpenHandsRegistration:
 
 
 class TestNativeOpenHandsAgent:
+    def test_truncate_handles_none_content_tool_call_turn(self):
+        """Tool-call assistant turns may carry content=None."""
+        engine = MagicMock()
+        engine.engine_id = "mock"
+        agent = NativeOpenHandsAgent(engine, "test-model")
+        messages = [
+            Message(role=Role.USER, content="hi"),
+            Message(
+                role=Role.ASSISTANT,
+                content=None,  # type: ignore[arg-type]
+                tool_calls=[ToolCall(id="call_1", name="calculator", arguments="{}")],
+            ),
+        ]
+
+        assert agent._truncate_if_needed(messages) == messages
+
+    def test_native_tool_call_with_none_content_does_not_crash(self):
+        """Native tool-call responses may omit assistant text content."""
+        engine = MagicMock()
+        engine.engine_id = "mock"
+        engine.generate.side_effect = [
+            _engine_response(
+                None,
+                tool_calls=[
+                    {
+                        "id": "call_1",
+                        "name": "calculator",
+                        "arguments": '{"expression": "2+2"}',
+                    }
+                ],
+            ),
+            _engine_response("The result is 4."),
+        ]
+        agent = NativeOpenHandsAgent(
+            engine,
+            "test-model",
+            tools=[_CalculatorStub()],
+        )
+
+        result = agent.run("What is 2+2?")
+
+        assert result.content == "The result is 4."
+        assert result.turns == 2
+        assert [tr.content for tr in result.tool_results] == ["4"]
+
     def test_simple_response(self):
         """No code -> direct answer."""
         engine = MagicMock()

tests/core/test_types.py

@@ -35,9 +35,15 @@ class TestMessage:
         msg = Message(role=Role.USER, content="hello")
         assert msg.role == Role.USER
         assert msg.content == "hello"
+        assert msg.text == "hello"
         assert msg.tool_calls is None
         assert msg.metadata == {}
 
+    def test_none_content_text_helper(self) -> None:
+        msg = Message(role=Role.ASSISTANT, content=None)
+        assert msg.content is None
+        assert msg.text == ""
+
     def test_tool_calls(self) -> None:
         tc = ToolCall(id="1", name="calc", arguments='{"x": 1}')
         msg = Message(role=Role.ASSISTANT, content="", tool_calls=[tc])

tests/engine/test_base.py

@@ -0,0 +1,60 @@
+from __future__ import annotations
+
+from openjarvis.core.types import Message, Role, ToolCall
+from openjarvis.engine._base import estimate_prompt_tokens
+
+
+def test_estimate_prompt_tokens_handles_none_content_tool_call_turn() -> None:
+    messages = [
+        Message(role=Role.USER, content="hi"),
+        Message(
+            role=Role.ASSISTANT,
+            content=None,
+            tool_calls=[ToolCall(id="call_1", name="lookup", arguments="{}")],
+        ),
+    ]
+
+    assert estimate_prompt_tokens(messages) == 12
+
+
+def test_estimate_prompt_tokens_counts_tool_call_arguments() -> None:
+    base = [
+        Message(role=Role.USER, content="hi"),
+        Message(role=Role.ASSISTANT, content=None),
+    ]
+    with_tool_call = [
+        Message(role=Role.USER, content="hi"),
+        Message(
+            role=Role.ASSISTANT,
+            content=None,
+            tool_calls=[ToolCall(id="", name="", arguments="abcdefgh")],
+        ),
+    ]
+
+    assert estimate_prompt_tokens(with_tool_call) - estimate_prompt_tokens(base) == 2
+
+
+def test_estimate_prompt_tokens_counts_reasoning_metadata() -> None:
+    base = [
+        Message(role=Role.USER, content="hi"),
+        Message(role=Role.ASSISTANT, content=None),
+    ]
+    with_reasoning = [
+        Message(role=Role.USER, content="hi"),
+        Message(
+            role=Role.ASSISTANT,
+            content=None,
+            metadata={"reasoning_content": "abcdefgh"},
+        ),
+    ]
+
+    assert estimate_prompt_tokens(with_reasoning) - estimate_prompt_tokens(base) == 2
+
+
+def test_estimate_prompt_tokens_counts_tool_result_ids() -> None:
+    messages = [
+        Message(role=Role.USER, content="hi"),
+        Message(role=Role.TOOL, content="ok", tool_call_id="abcdefgh"),
+    ]
+
+    assert estimate_prompt_tokens(messages) == 11