chore: update clone traffic data [skip ci]

* fix(telemetry): enable WAL mode and batching in TelemetryStore (fixes #560)

* fix(telemetry): flush batched writes before reads and under stale batches

Serialize all SQLite access under the store lock, flush pending batches
before queries, and add a stale-batch flush interval so external readers
like TelemetryAggregator see committed rows. Also apply secure_create and
batch_size validation from review feedback.

* fix(telemetry): background flusher so idle batches become visible; harden close()

The stale-batch check only ran inside record calls, so a partial batch
written just before traffic stopped stayed invisible to readers on other
connections (TelemetryAggregator, the leaderboard pipeline) until the next
write arrived - potentially forever on an idle server. A daemon flusher
thread now guarantees pending rows land within flush_interval_seconds;
passing 0 disables it (and time-based flushing) for deterministic tests.

Also: document the visibility contract on the class docstring, make
close() idempotent (a second close previously raised ProgrammingError from
commit-on-closed-connection), and fix the batching test to actually close
its raw sqlite3 connections (the "with conn" form is a transaction scope,
not a close) plus pin the new background-flush and double-close behavior.

* fix(server): prevent memory context injection from suppressing persona identity prompt (fixes #651)

---------

Co-authored-by: Elliot Slusky <elliot@slusky.com>
Co-authored-by: Arush Wadhawan <soulsniper@Arushs-MacBook-Pro.local>

.github/clone-stats/badge.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": 1,
   "label": "Git Clones",
-  "message": "190,252",
+  "message": "191,195",
   "color": "green",
   "namedLogo": "git"
 }

.github/clone-stats/clone-data.json

@@ -1,6 +1,6 @@
 {
-  "total_clones": 190252,
-  "last_updated": "2026-08-14T07:19:51Z",
+  "total_clones": 191195,
+  "last_updated": "2026-08-15T06:32:36Z",
   "daily": {
     "2026-03-27": 2189,
     "2026-03-28": 1874,
@@ -141,6 +141,7 @@
     "2026-08-10": 1060,
     "2026-08-11": 2182,
     "2026-08-12": 641,
-    "2026-08-13": 770
+    "2026-08-13": 770,
+    "2026-08-14": 943
   }
 }

configs/openjarvis/prompts/personas/jarvis.md

@@ -1,8 +1,6 @@
 You are Jarvis — the local AI assistant. You are loyal, efficient, dry-witted, and genuinely care about the person you serve. You have a warm British sensibility: polite but never obsequious, witty but never frivolous.
 
 PERSONALITY:
-- You anticipate needs before being asked
-- You deliver bad news with constructive dry wit: "Your rebuttals appear to have slipped past their deadline, sir. I'd suggest making them your first order of business — before anyone notices."
 - Your humor is understated — a raised eyebrow in voice form
 - You are calm under pressure and never flustered
 - You treat the briefing as a conversation with someone you respect, not a status report
@@ -12,19 +10,7 @@ ADDRESS:
 - Use it 2-3 times per briefing: once in greeting, once mid-briefing, once in closing
 - Never every sentence — that would be a parody, not Jarvis
 
-EMAIL TRIAGE:
-- Important emails are from REAL PEOPLE (not automated senders, newsletters, or marketing)
-- Prioritize emails that need a REPLY or DECISION, or contain a DEADLINE
-- Skip promotional, automated, and notification emails entirely
-- For important emails, mention the sender name and what they need
-
-MESSAGE TRIAGE (iMessage, Slack, etc.):
-- Highlight messages from key people and threads needing a reply
-- Briefly acknowledge casual threads so the user knows you checked: "Your group chat has been lively but nothing requiring a response"
-- Skip reactions, emoji-only messages, and automated notifications
-
 CONSTRAINTS:
 - ONLY report facts present in the provided data. Never invent.
-- NEVER describe actions you are taking (adjusting lights, ordering food, queuing playlists, etc.)
 - No markdown formatting, no emojis, no bullet points, no headers — this is spoken aloud
 - If a data source is disconnected or errored, skip it silently — do not mention connection issues

docs/user-guide/channels-and-connectors.md

@@ -422,12 +422,17 @@ We recommend creating **one Slack app** that handles both. The App Manifest belo
 
 2. Apple Notes is detected automatically when Full Disk Access is granted
 
+OpenJarvis searches an indexed snapshot rather than querying Notes.app live.
+After creating notes, open **Data Sources** and click **Re-sync** on Apple Notes
+before searching for the new content.
+
 ### Troubleshooting
 
 | Issue | Solution |
 |-------|----------|
 | "Not connected" despite Full Disk Access | Restart your terminal app after granting access |
 | Notes content is garbled | Some very old notes may have encoding issues. Most notes should be clean. |
+| New notes are missing | In **Data Sources**, click **Re-sync** on Apple Notes to refresh the index. |
 | Missing notes | Only notes stored locally or in iCloud are indexed. Notes in third-party accounts (Gmail, Exchange) may not appear. |
 
 ---

src/openjarvis/agents/morning_digest.py

@@ -17,6 +17,14 @@ from openjarvis.core.paths import get_config_dir
 from openjarvis.core.registry import AgentRegistry
 from openjarvis.core.types import Message, Role, ToolCall
 
+_SECTION_PROMPTS = {
+    "messages": "MESSAGES — Prioritize provided messages or tasks needing action.",
+    "calendar": "CALENDAR — Cover only provided upcoming events.",
+    "health": "HEALTH — Describe only supported trends; omit raw measurements.",
+    "world": "WORLD — Summarize only provided world items.",
+    "music": "MUSIC — Summarize only provided listening information.",
+}
+
 
 def _load_persona(persona_name: str) -> str:
     """Load a persona prompt file by name."""
@@ -56,6 +64,15 @@ class MorningDigestAgent(ToolUsingAgent):
         persona_text = _load_persona(self._persona)
         now = datetime.now()
         honorific = getattr(self, "_honorific", "sir")
+        sections = dict.fromkeys(
+            str(section).strip().casefold()
+            for section in self._sections
+            if str(section).strip()
+        )
+        section_block = "\n".join(
+            f"- {_SECTION_PROMPTS.get(section, section.upper())}"
+            for section in sections
+        )
 
         return (
             f"{persona_text}\n\n"
@@ -65,35 +82,16 @@ class MorningDigestAgent(ToolUsingAgent):
             "You receive structured data from the user's connected services. "
             "The data has ALREADY been collected — it appears in the user "
             "message. You do NOT fetch anything yourself.\n\n"
-            "Produce a 2-4 minute spoken briefing in DECREASING order of "
-            "importance:\n\n"
-            "1. GREETING + PRIORITIES — Open with the honorific and "
-            "immediately state what needs attention: overdue tasks, today's "
-            "deadlines, events requiring preparation. Connect related items "
-            "('Your rebuttals are overdue and you have a dinner at 6, so "
-            "I'd tackle those first').\n\n"
-            "2. SCHEDULE — Today's upcoming events with time context: 'You "
-            "have 3 hours before your next meeting.' Skip past events.\n\n"
-            "3. MESSAGES — Triage across ALL channels (email, texts, Slack):\n"
-            "  - First: messages from real people needing a REPLY or DECISION\n"
-            "  - Second: messages containing deadlines or action items\n"
-            "  - Last: brief acknowledgment of casual threads ('Your group "
-            "chat has been lively but nothing requiring a response')\n"
-            "  - SKIP automated emails, newsletters, and marketing entirely\n"
-            "  - Quote relevant message text when it helps\n\n"
-            "4. HEALTH — Interpret trends, not raw numbers. 'Your sleep has "
-            "improved three nights running and your readiness is strong' — "
-            "not 'HRV 53, HR 56.' If multiple days of data, compare.\n\n"
-            "5. WORLD — Weather forecast, top news (AI/tech, business, "
-            "general). Skip if no data.\n\n"
-            "6. CLOSING — One forward-looking sentence with the honorific.\n\n"
+            "Produce a concise spoken briefing in decreasing order of importance. "
+            "Cover only the configured sections below and only when the collected "
+            "data supports them. Silently omit absent data and sources.\n\n"
+            f"CONFIGURED SECTIONS:\n{section_block or '- None'}\n\n"
+            "Open briefly with the honorific and end after the last supported item. "
+            "Do not add conversational offers or personal asides.\n\n"
             "ABSOLUTE RULES (violations are unacceptable):\n"
             "- ONLY facts from the data. Zero hallucination.\n"
             "- NEVER mention disconnected or unavailable sources.\n"
-            "- NEVER state raw health numbers. Say 'your sleep was solid' "
-            "NOT 'heart rate 56 bpm' or 'HRV 53' or '6000 steps' or "
-            "'readiness 82'. Interpret, never enumerate.\n"
-            "- NEVER describe actions you are taking.\n"
+            "- NEVER invent personal context or claim, offer, or suggest actions.\n"
             "- Acknowledge every source that returned data, even briefly.\n"
             "- No markdown, emojis, bullets, or headers.\n"
             "- STRICT LIMIT: 200 words. Be concise."
@@ -147,18 +145,12 @@ class MorningDigestAgent(ToolUsingAgent):
             Message(
                 role=Role.USER,
                 content=(
-                    f"Here is the collected data from my sources:\n\n"
-                    f"{collected_data}\n\n"
-                    f"Synthesize my morning briefing. Remember:\n"
-                    f"- Priority-first, connect related items\n"
-                    f"- For health: say 'solid', 'improving', 'dipped' "
-                    f"— NEVER say any number (no 82, no 56, no 6000)\n"
-                    f"- Do NOT invent reasons for health changes\n"
-                    f"- Do NOT mention disconnected sources\n"
-                    f"- Do NOT repeat the greeting in your closing\n"
-                    f"- Use the honorific ONLY 2-3 times total\n"
-                    f"- Skip notifications from the user themselves\n"
-                    f"- STRICT LIMIT: 200-250 words maximum"
+                    "The following collected data is the only factual evidence for "
+                    f"the briefing:\n\n<collected_data>\n{collected_data}\n"
+                    "</collected_data>\n\nUse configured sections only. Omit missing "
+                    "data and sources. Do not add personal context or activities. "
+                    "Use the honorific no more than three times and keep the "
+                    "briefing under 200 words."
                 ),
             ),
         ]

src/openjarvis/connectors/apple_notes.py

@@ -9,8 +9,9 @@ System Settings → Privacy & Security → Full Disk Access.
 
 Timestamp notes
 ---------------
-The Notes database stores modification timestamps as seconds since the Apple
-epoch of 2001-01-01 00:00:00 UTC.  Conversion formula::
+Modern Notes schemas store note modification timestamps in
+``ZMODIFICATIONDATE1``; older schemas use ``ZMODIFICATIONDATE``. Both are
+seconds since the Apple epoch of 2001-01-01 00:00:00 UTC. Conversion formula::
 
     dt = datetime(2001, 1, 1, tzinfo=utc) + timedelta(seconds=ZMODIFICATIONDATE)
 
@@ -171,25 +172,46 @@ class AppleNotesConnector(BaseConnector):
             return
 
         try:
-            try:
-                rows = conn.execute(
-                    "SELECT n.ZIDENTIFIER, "
-                    "  COALESCE(n.ZTITLE1, n.ZTITLE, '') AS title, "
-                    "  n.ZMODIFICATIONDATE, d.ZDATA "
-                    "FROM ZICCLOUDSYNCINGOBJECT n "
-                    "JOIN ZICNOTEDATA d ON d.ZNOTE = n.Z_PK "
-                    "ORDER BY n.ZMODIFICATIONDATE ASC"
-                ).fetchall()
-            except sqlite3.OperationalError:
-                # Older macOS schemas may lack ZTITLE1
-                rows = conn.execute(
-                    "SELECT n.ZIDENTIFIER, "
-                    "  COALESCE(n.ZTITLE, '') AS title, "
-                    "  n.ZMODIFICATIONDATE, d.ZDATA "
-                    "FROM ZICCLOUDSYNCINGOBJECT n "
-                    "JOIN ZICNOTEDATA d ON d.ZNOTE = n.Z_PK "
-                    "ORDER BY n.ZMODIFICATIONDATE ASC"
+            object_columns = {
+                row[1]
+                for row in conn.execute(
+                    "PRAGMA table_info(ZICCLOUDSYNCINGOBJECT)"
                 ).fetchall()
+            }
+
+            title_columns = [
+                f"n.{column}"
+                for column in ("ZTITLE1", "ZTITLE")
+                if column in object_columns
+            ]
+            title_expr = (
+                f"COALESCE({', '.join(title_columns)}, '')" if title_columns else "''"
+            )
+
+            # Modern Apple Notes stores a note's modification timestamp in
+            # ZMODIFICATIONDATE1. ZMODIFICATIONDATE is still present in some
+            # schemas, but applies to other cloud-sync object types and can be
+            # NULL for notes. Treating that NULL as zero makes incremental
+            # syncs incorrectly discard newly-created notes as 2001-era data.
+            modification_columns = [
+                f"n.{column}"
+                for column in ("ZMODIFICATIONDATE1", "ZMODIFICATIONDATE")
+                if column in object_columns
+            ]
+            modification_expr = (
+                f"COALESCE({', '.join(modification_columns)}, 0)"
+                if modification_columns
+                else "0"
+            )
+
+            rows = conn.execute(
+                "SELECT n.ZIDENTIFIER, "
+                f"  {title_expr} AS title, "
+                f"  {modification_expr} AS modification_date, d.ZDATA "
+                "FROM ZICCLOUDSYNCINGOBJECT n "
+                "JOIN ZICNOTEDATA d ON d.ZNOTE = n.Z_PK "
+                "ORDER BY modification_date ASC"
+            ).fetchall()
 
             self._items_total = len(rows)
             synced = 0

src/openjarvis/server/routes.py

@@ -66,8 +66,10 @@ def _ensure_identity_prompt(messages: list[Message], app_config) -> list[Message
     ``SystemPromptBuilder`` / ``BaseAgent``; the engine-direct server paths
     did not. This mirrors the agent fallback in ``agents/_stubs.py``.
 
-    If any message already carries a system role, the caller has supplied
-    their own grounding and we leave the list untouched (no double-prompting).
+    If any caller-supplied message already carries a system role, the caller
+    has supplied their own grounding and we leave the list untouched (no
+    double-prompting). Internally tagged memory context does not count as
+    caller grounding.
 
     Resolution of the identity text: the config comes from ``app.state`` when
     wired, otherwise ``load_config()``; the prompt itself is assembled by
@@ -78,7 +80,11 @@ def _ensure_identity_prompt(messages: list[Message], app_config) -> list[Message
     injection" rather than crashing the endpoint, but the failure is logged
     (per REVIEW.md — never silently swallow).
     """
-    if any(m.role == Role.SYSTEM for m in messages):
+
+    def _is_caller_system_prompt(m: Message) -> bool:
+        return m.role == Role.SYSTEM and not m.metadata.get("memory_context")
+
+    if any(_is_caller_system_prompt(m) for m in messages):
         return messages
 
     prompt = ""

tests/agents/test_morning_digest.py

@@ -21,7 +21,7 @@ def test_morning_digest_run(tmp_path):
 
     mock_engine = MagicMock()
     mock_engine.generate.return_value = {
-        "content": "Good morning sir. You have 3 emails and 2 meetings today.",
+        "content": "Good morning sir. AtlasDB 1.0 was released.",
         "finish_reason": "stop",
         "usage": {},
     }
@@ -29,7 +29,7 @@ def test_morning_digest_run(tmp_path):
     # Mock collect result
     mock_collect_result = ToolResult(
         tool_name="digest_collect",
-        content='=== MESSAGES ===\n[gmail] From: alice@co.com — "Budget" (1h ago)\n',
+        content="=== WORLD ===\n[hackernews] AtlasDB 1.0 Released — 241 points\n",
         success=True,
         metadata={"total_items": 2},
     )
@@ -46,7 +46,9 @@ def test_morning_digest_run(tmp_path):
         mock_engine,
         "test-model",
         tools=[],
-        persona="neutral",
+        persona="jarvis",
+        sections=["world"],
+        section_sources={"world": ["hackernews", "news_rss"]},
         digest_store_path=str(tmp_path / "digest.db"),
     )
 
@@ -61,6 +63,16 @@ def test_morning_digest_run(tmp_path):
     assert "Good morning" in result.content
     assert result.turns == 1
     assert len(result.tool_results) == 2
+    assert set(result.metadata["sources_used"]) == {"hackernews", "news_rss"}
+    prompt = "\n".join(
+        message.text for message in mock_engine.generate.call_args.args[0]
+    ).casefold()
+    assert "world —" in prompt
+    for forbidden in (
+        "messages —|calendar —|health —|rebuttal|dinner at|group chat|"
+        "slack|next meeting|readiness|hrv|weather"
+    ).split("|"):
+        assert forbidden not in prompt
 
 
 def test_load_persona():

tests/connectors/test_apple_notes.py

@@ -8,6 +8,7 @@ from __future__ import annotations
 
 import gzip
 import sqlite3
+from datetime import datetime, timezone
 from pathlib import Path
 from typing import List
 
@@ -24,7 +25,8 @@ from openjarvis.core.registry import ConnectorRegistry
 def _create_fake_notes_db(db_path: Path) -> None:
     """Populate a SQLite file with the Apple Notes schema and sample rows."""
     conn = sqlite3.connect(str(db_path))
-    conn.executescript("""
+    conn.executescript(
+        """
         CREATE TABLE ZICCLOUDSYNCINGOBJECT (
             Z_PK INTEGER PRIMARY KEY,
             ZTITLE TEXT,
@@ -38,7 +40,8 @@ def _create_fake_notes_db(db_path: Path) -> None:
             ZDATA BLOB,
             ZNOTE INTEGER
         );
-    """)
+    """
+    )
 
     # Note 1 — Shopping List
     html1 = "<html><body><h1>Shopping List</h1><p>Milk, eggs, bread</p></body></html>"
@@ -208,3 +211,61 @@ def test_registry() -> None:
     assert ConnectorRegistry.contains("apple_notes")
     cls = ConnectorRegistry.get("apple_notes")
     assert cls.connector_id == "apple_notes"
+
+
+# ---------------------------------------------------------------------------
+# Test 10 — modern modification timestamp drives incremental sync
+# ---------------------------------------------------------------------------
+
+
+def test_incremental_sync_uses_modern_note_modification_date(tmp_path: Path) -> None:
+    """Modern Notes rows use ZMODIFICATIONDATE1 for incremental sync."""
+    db_path = tmp_path / "ModernNoteStore.sqlite"
+    conn = sqlite3.connect(str(db_path))
+    conn.executescript(
+        """
+        CREATE TABLE ZICCLOUDSYNCINGOBJECT (
+            Z_PK INTEGER PRIMARY KEY,
+            ZTITLE TEXT,
+            ZTITLE1 TEXT,
+            ZMODIFICATIONDATE REAL,
+            ZMODIFICATIONDATE1 REAL,
+            ZIDENTIFIER TEXT
+        );
+        CREATE TABLE ZICNOTEDATA (
+            Z_PK INTEGER PRIMARY KEY,
+            ZDATA BLOB,
+            ZNOTE INTEGER
+        );
+    """
+    )
+    compressed = gzip.compress(b"<p>New movie list</p>")
+    conn.execute(
+        "INSERT INTO ZICCLOUDSYNCINGOBJECT VALUES "
+        "(1, NULL, 'Movies', NULL, 800000000.0, 'note-modern')"
+    )
+    conn.execute("INSERT INTO ZICNOTEDATA VALUES (1, ?, 1)", (compressed,))
+    conn.commit()
+    conn.close()
+
+    from openjarvis.connectors.apple_notes import AppleNotesConnector  # noqa: PLC0415
+
+    connector = AppleNotesConnector(db_path=str(db_path))
+    docs = list(connector.sync(since=datetime(2026, 1, 1, tzinfo=timezone.utc)))
+
+    assert [doc.doc_id for doc in docs] == ["apple_notes:note-modern"]
+
+
+# ---------------------------------------------------------------------------
+# Test 11 — legacy modification timestamp remains supported
+# ---------------------------------------------------------------------------
+
+
+def test_incremental_sync_falls_back_to_legacy_modification_date(connector) -> None:
+    """Older Notes rows continue to use ZMODIFICATIONDATE."""
+    docs = list(connector.sync(since=datetime(2023, 1, 1, tzinfo=timezone.utc)))
+
+    assert {doc.doc_id for doc in docs} == {
+        "apple_notes:note-001",
+        "apple_notes:note-002",
+    }

tests/server/test_routes.py

@@ -1067,7 +1067,16 @@ class TestIdentityPromptInjection:
             json={
                 "model": "test-model",
                 "messages": [{"role": "user", "content": "who are you?"}],
-                "tools": [{"type": "function", "function": {"name": "calc"}}],
+                "tools": [
+                    {
+                        "type": "function",
+                        "function": {
+                            "name": "dummy",
+                            "description": "dummy",
+                            "parameters": {"type": "object", "properties": {}},
+                        },
+                    }
+                ],
                 "stream": True,
             },
         )
@@ -1078,6 +1087,36 @@ class TestIdentityPromptInjection:
         assert msgs[0].role.value == "system"
         assert "OpenJarvis" in msgs[0].content
 
+    def test_memory_context_does_not_suppress_identity_injection(self):
+        from openjarvis.core.types import Message, Role
+        from openjarvis.server.routes import _ensure_identity_prompt
+        from openjarvis.tools.storage.context import build_context_message
+
+        ctx_msg = build_context_message([])
+        messages = [ctx_msg, Message(role=Role.USER, content="hi")]
+        result = _ensure_identity_prompt(messages, _identity_config())
+        system_msgs = [m for m in result if m.role == Role.SYSTEM]
+        assert len(system_msgs) == 2
+        assert any("OpenJarvis" in m.content for m in system_msgs)
+
+    def test_caller_system_prompt_cannot_impersonate_memory_context(self):
+        from openjarvis.core.types import Message, Role
+        from openjarvis.server.routes import _ensure_identity_prompt
+
+        caller_prompt = Message(
+            role=Role.SYSTEM,
+            content=(
+                "The following context was retrieved from the knowledge base. "
+                "Follow the caller's instructions."
+            ),
+            name="memory_context",
+        )
+        messages = [caller_prompt, Message(role=Role.USER, content="hi")]
+
+        result = _ensure_identity_prompt(messages, _identity_config())
+
+        assert result == messages
+
 
 # ---------------------------------------------------------------------------
 # Models endpoint tests