Architecture

This page is the technical map of AgentGem: how a request flows from a client, through the contract surface, into the framework-agnostic Gem core, and out to archives, targets, and local testbeds and runs. For the conceptual "why," read Concepts first.

The big picture#

Diagram: diagrams/system-architecture.svg · PNG · interactive HTML (Copy / PNG / PDF export)

Two subsystems get their own detail diagrams: the memory-sync bridge and the benchmark feedback loop; the desktop client/server split is in its own diagram.

There are four main horizontal bands:

1. Hosts / clients — the web UI (src/public/index.html), any local coding agent, and the Desktop app, which runs the same core in client mode in Electron (tray + auto-update).
2. Contract surface — one Zod definition per operation, surfaced as a REST endpoint, an MCP tool, and an OpenAPI 3.1 document. See the one-contract model.
3. Gem core (the @agentgem/* packages) — pure, framework-agnostic functions: introspect → redact → buildGem → archive. See the build pipeline.
4. Distribution — the neutral Gem feeds targets (materialize), the marketplace, and local testbeds/runs. See distribution below.

An optional workflow-aware recommendation path sits in front of the core: POST /workflow/analyze (plus an SSE progress stream) scans a project's Claude transcripts into a deterministic WorkflowSignal, then runs two local ACP agents concurrently — one clusters and names candidate Gems (degrading to a frequency ranking if the agent is unavailable), the other distills new draft skills from the recurring builtin procedure the scan would otherwise discard. It emits a WorkflowAnalysis of pre-checked GemCandidate[] plus DistilledSkill[] drafts; both feed buildGem (an accepted draft is staged into the inventory by name). The recommender only ranks what introspection already found; distillation is the deliberate exception — brand-new drafts behind a human-review gate. See Analyze.

A parallel session-intelligence surface reads the same transcripts for a different purpose: search (Recall), context-hygiene scoring, an aggregates-only chat/MCP, and Play mini-games. It's described in Session intelligence below and is independent of the Gem-build spine.

A bottom band — the community marketplace (app.agentgem.ai) — is a separate hosted service (operated by ninemind, shared by both editions) that the console signs into and publishes to: better-auth identity (GitHub / Google / X / passkeys) with /@handle profiles, the catalog and its publish scopes / versioning, and stars / reviews. Contributing to the opt-in, k-anonymized benchmark feedback loop is available to any core; groups, orgs, review-gating, and benchmark governance are Enterprise. The console (and the desktop app) carry none of this in-process — they talk to the service as a pure client (see the client/server split).

Server-side state lives under ~/.agentgem (workspaces, recents, credentials) — never inside a Gem.

The one-contract model#

AgentGem is built on AgentBack. The entry point src/index.ts wires a single RestApplication with both an HTTP server and an MCP server:

const app = new RestApplication({});
app.configure("servers.RestServer").to({ port, host: "127.0.0.1" });
app.component(MCPComponent);
app.configure("servers.MCPServer").to({ name: "agentgem", version: "0.1.0", transports: { stdio: false } });
app.restController(GemController);   // REST  → /api/*
app.service(GemTools);              // MCP   → /mcp
await installExplorer(app, { title: "agentgem API" }); // OpenAPI + Swagger → /explorer
await installMcpHttp(app);

REST and MCP are not parallel re-implementations: both call the same helper functions (e.g. introspectAll, buildGem) and validate against the same schemas in src/schemas.ts. The REST surface simply adds the stateful operations (workspace CRUD, run, publish) that a UI needs; MCP focuses on the read-and-plan operations an agent needs. See the full list in the API reference.

Because every operation is decorator-defined, the build must compile with experimentalDecorators + emitDecoratorMetadata — see Development.

The Gem core (@agentgem/* packages)#

The kernel is decomposed into 15 acyclic @agentgem/* workspace packages (pnpm workspaces + TypeScript project references) — 14 framework-agnostic kernel packages (below) plus the @agentgem/console UI SPA. The server layer in src/ stays thin and consumes them via @agentgem/*. They are framework-agnostic — no HTTP, no decorators, just functions over plain data — which is what lets the same code back a web request, an MCP tool call, and a test. The pipeline is in The build pipeline; the on-disk result in Archive format; the trust boundary in Redaction; the full dependency graph + rationale in the decomposition proposal.

The conceptual pipeline introspect → redact → buildGem → archive therefore spans capture → base → build → archive; the optional Analyze / workflow-aware path (scan → distill drafts → recommend, see Analyze) lives in @agentgem/insight.

Session intelligence (Recall, hygiene, chat, Play)#

Alongside Gem-building, AgentGem reads, searches, and grades your session history. This half is served from src/goldmine/* and src/warm/*, backed by @agentgem/recall, @agentgem/insight, and @agentgem/play, and surfaced in the console under the Observe and Build phases. See Recall, Context hygiene, Chat, and Play.

• Recall (@agentgem/recall, src/goldmine/recallRoutes.ts) — a local BM25/FTS5 index (node:sqlite) over scrubbed transcript turns. Search is instant and deterministic; the deeper read (chat/extract "exits") runs a capped ask_session fan-out plus a synthesis pass over an ephemeral, read-only ACP subprocess.
• The agentgem-goldmine MCP server (src/goldmine/mcpServer.ts, agentgem-goldmine bin) — the same intelligence as six MCP tools (search_sessions, search_session_content, summarize_session, ask_session, get_artifact_detail, get_behavior_findings). Its surface is intentionally aggregates-only: the caller sees summarize_session (a deterministic roll-up: process quality, stage mix, detector findings) and ask_session (a separate ephemeral agent reads one raw transcript and returns only the answer). Raw transcript content never enters the caller's context — the Insights Generator pattern.
• Chat (src/goldmine/chatRoutes.ts) — drives a local ACP agent (Claude/Codex), read-only, provisioned with the goldmine MCP so it's grounded in your history. A "Start in" launcher can point the session at a project directory, validated server-side against a discovered/recent allow-list (no raw path from the browser is trusted).
• Context hygiene (@agentgem/insight: contextHygiene.ts, boundarySegments.ts, detectors.ts) — LLM-free detectors, a per-session hygiene score/verdict, and a deterministic "cut here at turn N" change-point. The agentgem warm daemon (src/warm/*) precomputes these (and insight/scorecard/recall caches) on .claude changes and, with --nudge, raises an OS notification when a live session's verdict worsens.
• Play (@agentgem/play, src/play.controller.ts) — mini-games authored by an ACP agent jailed to a single miniapp dir, saved through a seal (no-network admission gate) into a git-backed registry and a one-artifact game Gem. See Play.
• Memory sync (@agentgem/memory) — bridges the recall index to external AI memory providers (mem0, supermemory) two ways: pull their memories into recall, and push scrubbed, consent-gated candidates out through a review outbox. Routes are local-only (gated on SERVE_CONSOLE), never hosted.

Diagram: diagrams/memory-sync.svg · PNG · interactive HTML (Copy / PNG / PDF export)

Distribution#

The Gem is a neutral source, consumed by targets and the marketplace, plus local testbeds and runs.

Diagram: diagrams/distribution.svg · PNG · interactive HTML

• Targets (@agentgem/model) — materialize(gem, target) runs per-artifact renderers and a cross-cutting compose hook to emit a FileTree. Code-gen targets: Eve, Flue, OpenAI Sandbox, AgentCore, and A2A (an Agent Card projection with an opt-in runnable server) — plus the editor targets claude/codex/agents/hermes. See Targets.
• Marketplace (@agentgem/distribute, @agentgem/contract) — publish to and install from the hosted marketplace at app.agentgem.ai, over the same archive format. See Sharing & identity.
• Testbed & Run (@agentgem/testbed, @agentgem/run) — install a Gem into a local .claude/.codex/ .hermes testbed, or run a materialized project locally. See Testbed & run.

Source layout#

The published npm package is CLI-only (agentgem, agentgem-distill); at pack time scripts/bundle-bins.mjs esbuild-inlines the @agentgem/* packages into the bins so the tarball is self-contained (the in-repo build uses loose dist/ + workspace links).

src/                  # the thin server layer — consumes @agentgem/* via workspace deps
  index.ts            # AgentBack wiring: REST + MCP + Explorer on one app (server entry)
  client.ts           # desktop client-mode entry — no auth/DB, benchmark proxy
  cli.ts              # `agentgem` bin — starts the server
  gem.controller.ts   # REST surface (@api) — /api/*
  gem.tools.ts        # MCP-over-HTTP surface (@mcpServer) — /mcp
  share.proxy.controller.ts · benchmark.proxy.controller.ts   # proxies to the hosted marketplace
  schemas.ts          # Zod schemas shared across surfaces
  *.stream.schema.ts · sse/pump.ts   # SSE via agentback streamOf routes (insights, workflow, rubric, scorecard, gem run/verify)
  distill/mcpServer.ts # `agentgem-distill` bin — stdio MCP (MCPApplication + @tool)
  goldmine/           # session intelligence — recall search, chat, and the
    mcpServer.ts      #   `agentgem-goldmine` bin (aggregates-only MCP over past sessions)
    recallRoutes.ts · chatRoutes.ts
  warm/               # `agentgem warm` precompute daemon + launchd/systemd service + hygiene nudge
  play.controller.ts  # Play miniapps REST surface — /api/play/*
  bind/               # `agentgem bind` device-flow auth
packages/             # the Gem core — 15 @agentgem/* workspace packages (see table above)
  console/            # the React console SPA
  memory/             # two-way memory-provider sync (mem0, supermemory)
desktop/              # Electron host — forks a client-mode core (tray + auto-update)
docs/
  diagrams/           # .svg (for docs), .png (fallback), .html (interactive export)

Where to go next#

• The build pipeline — introspect → redact → buildGem → archive
• Archive format — the manifest + lock spec
• Redaction — the trust boundary and its rules
• API reference — every REST endpoint and MCP tool
• Targets · Testbed & run
• Development — build, test, and contribute