TL;DR: Current AI systems hit a wall when tasks require more context than fits in a single prompt. Shad solves this by treating your knowledge vault as an explorable environment — not a fixed input — enabling AI to tackle complex tasks that would be impossible otherwise.

The Problem

If you've used AI for anything beyond simple questions, you've hit the ceiling.

You want Claude or GPT to build you a complete application. You paste in your requirements, maybe some example code, your API docs. The model does a decent job on the first file. Then the second file contradicts the first. By the third file, it's forgotten the type definitions it wrote ten minutes ago. Import paths don't resolve. Interfaces don't match. The model is smart — it just can't see enough at once.

This isn't a model intelligence problem. It's an inference problem.

The best AI models in the world can reason brilliantly within their context window. But the moment your task requires more knowledge than fits in that window — a documentation vault, a codebase, architectural patterns, past decisions — everything falls apart. You end up babysitting the model, manually feeding it the right context at the right time, fixing inconsistencies between outputs, and wondering why you didn't just write the code yourself.

RAG (Retrieval-Augmented Generation) helps, but only for simple lookups. "Find me the relevant docs for this question" works fine. "Build a production application using patterns from these 200 files across 5 repositories" does not.

Long-context models help, but 200K tokens still isn't enough for serious work. And cramming everything in degrades quality — the model's attention spreads thin across a massive prompt, and the signal drowns in noise.

The real solution isn't bigger context windows. It's smarter context retrieval.

What Shad Does

Shad (Shannon's Daemon) enables AI to utilize virtually unlimited context.

Load your knowledge into a vault — documentation, code examples, architecture patterns, best practices, past decisions — then ask Shad to accomplish complex tasks. Shad recursively breaks the task into subtasks, retrieves precisely the context each subtask needs, generates outputs with cross-file consistency, verifies everything, and assembles a coherent result.

shad run "Build a task management app with auth, offline sync, and push notifications" \
  --vault ~/MobileDevVault \
  --strategy software \
  --write-files --output ./TaskApp

That single command:

  1. Decomposes the task into a dependency-aware graph (requirements → data model → type contracts → implementation → verification → synthesis)
  2. Retrieves targeted context for each subtask by generating custom search scripts against your vault
  3. Generates code with contracts-first type consistency — types are defined once and imported everywhere
  4. Verifies all imports resolve, syntax is valid, types check, and tests pass
  5. Assembles a complete, working codebase in your output directory

The vault contains the how — your curated patterns and knowledge. Shad provides the engine — decomposition, retrieval, generation, verification, assembly. Together: complex tasks in one shot.

How It Actually Works

Recursive Decomposition

Shad doesn't try to solve your entire problem at once. It decomposes tasks using strategy skeletons — templates that encode domain-specific best practices.

For software tasks, the skeleton enforces contracts-first development: define your types and interfaces before writing implementation code. This single constraint eliminates the most common failure mode in AI-generated code — files that don't agree on what types exist.

The LLM fills in task-specific details within the skeleton's guardrails. It can add modules, split implementations, define dependencies between subtasks. But it can't skip required stages like type contracts or verification.

Targeted Retrieval (Code Mode)

This is the key innovation. For each subtask, Shad doesn't just run a keyword search. It generates a Python script that programmatically queries your vault:

# LLM-generated retrieval for "Implement OAuth"
results = obsidian.search("OAuth refresh token", limit=10)
patterns = obsidian.read_note("Patterns/Auth/OAuth.md")

relevant = [r["content"][:2000] for r in results if "JWT" in r["content"]]
__result__ = {"context": "\n".join(relevant), "citations": [...], "confidence": 0.72}

The model decides what to search for and how to filter results based on what it's trying to accomplish. If retrieval confidence is low, Shad retries with broader queries or falls back to direct search. High-impact nodes with poor context get escalated for human review.

This approach means a 100MB vault can meaningfully contribute to output — not by loading it all at once, but by intelligently retrieving what's needed for each specific subtask.

Cross-Subtask Context

Subtasks don't operate in isolation. When a node completes, it produces a context packet — a compact summary of its outputs, key decisions, and artifacts. These packets flow to downstream nodes that might benefit from them.

The type contracts node produces the canonical type definitions. Every implementation node receives those definitions as ground truth. No guessing, no drift, no "I'll just define User here too."

Verification & Repair

After generation, Shad verifies outputs:

  • Import resolution — every import points to a real file and symbol
  • Syntax validation — code actually parses
  • Type checking — TypeScript types align (via tsc --noEmit)
  • Tests — generated tests pass

When something fails, Shad classifies the error and attempts repair. Syntax errors get local fixes. Type errors get repaired with contract context. Integration failures escalate to the parent coordinator. Error signature hashing prevents infinite repair loops.

Vault Curation

Your vault is only as good as what's in it. Shad includes tools to keep it healthy:

  • shad ingest github <url> — Clone and process repositories into your vault with configurable presets (docs, mirror, deep semantic indexing)
  • shad sources add — Schedule automatic syncs from GitHub repos, URLs, RSS feeds, and local folders
  • shad vault analyze — Score your vault for gaps using query history, coverage patterns, and optional LLM audit

Over time, your vault becomes a compound asset — every project you work on contributes knowledge that makes future projects easier.

What Makes This Different

From RAG: Shad doesn't just retrieve and append. It decomposes, retrieves per-subtask, generates with cross-file consistency, and verifies. RAG answers questions. Shad builds things.

From long-context models: Instead of cramming everything into one prompt and hoping the model attends to the right parts, Shad retrieves precisely what each subtask needs. A 100MB vault works fine. A 200K context window doesn't have to.

From AI coding tools: Copilot and Cursor work great for single-file suggestions. Shad works at the project level — multi-file codebases with consistent types, resolved imports, and verified outputs.

From agents: Most AI agents are loops that retry until something works. Shad is a structured execution engine with strategy enforcement, budget management, verification layers, and human-in-the-loop checkpoints. It converges instead of wandering.

The Bigger Picture

I built Shad because I believe the bottleneck in AI isn't model intelligence — it's context management. The models are already smart enough to build amazing things. They just can't see enough of the world at once to do it consistently.

Shad is the beginning of solving that. Give AI access to your accumulated knowledge — structured, searchable, version-controlled — and let it do work that actually requires expertise.

Your vault is your unfair advantage. Shad is the engine that makes it productive.

Shad is open source. github.com/jonesj38/shad

# Install
curl -fsSL https://raw.githubusercontent.com/jonesj38/shad/main/install.sh | bash

# Try it
shad run "Summarize the key patterns in this codebase" --vault ~/your-project