Grok files and cleanup
Grok files and cleanup
Grok reads attached files via the files[] parameter. Each entry has EXACTLY ONE of:
path- a local file. Delivery is controlled bymode(default"upload"- bridge uploads to the xAI Files API;"inline"embeds asinput_text;"auto"picks per heuristic - see "Inline vs upload delivery" below).file_id- an already-uploaded xAI file id (passed through, no upload).file_url- a public URL (passed through).dir- a local directory expanded recursively. Samemoderules; the walker applies the chosen mode to every selected file (see below).
A path entry may also set headBytes (positive integer) to send only the start of
the file: the bridge reads at most that many bytes (never the whole file), cuts at the
last line break (or a UTF-8 character boundary), and appends
[truncated: first N of M bytes]. A headBytes entry is always inline and is rejected
with mode: "upload"; content plus marker is clamped to GROK_INLINE_MAX_BYTES, and
a FIFO or other non-regular file is refused. headBytes is not allowed on dir,
file_id, or file_url; a dir entry keeps maxBytes as its walk cap. The
OpenRouter bridge accepts the same field on path entries.
A path or dir resolves against the top-level roots[] array (absolute directories,
first-root-wins for relative entries) or, when roots is omitted, against cwd. A
path that resolves outside every declared root is refused (no exfiltration); symlinks
that escape via realpath are also refused. An oversize file (>48 MB) returns
file-too-large.
Cross-repo example
mcp__deliberation-grok__grok({
prompt: "Compare the auth strategy in these two services.",
cwd: "/Users/me/work/service-a",
roots: ["/Users/me/work/service-a", "/Users/me/work/service-b"],
files: [
{ path: "src/auth.ts" }, // resolves under service-a (first root)
{ path: "/Users/me/work/service-b/src/auth.ts" }, // absolute, must lie under one root
{ dir: "docs", include: ["**/*.md"], maxFiles: 20 }, // expands service-a/docs
],
})Directory expansion ({dir})
The bridge bundles a zero-dep glob walker (server/grok/glob.js) so you do not have
to enumerate every file by hand:
include(default["**/*"]).excludeis appended to the bridge's safe defaults (it does NOT replace them). Defaults cover: VCS (.git); JS/Node (node_modules,dist,build,out,.next,.svelte-kit,.nuxt,.turbo,.cache,.parcel-cache,.pnpm-store); Yarn Berry (.yarn/cache,.yarn/unplugged); lockfiles (**/*.lock); Python (.venv,venv,__pycache__,.tox,.pytest_cache,.mypy_cache,.ruff_cache,.ipynb_checkpoints,.eggs,htmlcov); coverage (coverage,.nyc_output); Rust/Java/Gradle (target,.gradle); Go/PHP (vendor); Terraform (.terraform,.terragrunt-cache); plus security:**/*.tfstate*, granular.envvariants (keeps.env.examplereadable),.ssh/**, SSH keypairs (id_rsa,id_ed25519,id_ecdsa,id_dsaand.pub), and**/*.pem/**/*.key.- To replace defaults entirely instead of appending, set
excludeReset: trueon the same{dir}entry.excludeResetis validated as a strict boolean byvalidateFiles; non-boolean values are rejected. Use only when reviewing files defaults would block (e.g., Terraform state in a security audit, or legitimate.pempublic certs). Tradeoff is explicit: the bridge prefers a false positive (blocking a legitimate.pem) over a false negative (leaking a private key). maxFiles(default 50),maxBytes(default 128 MB). Exceeding either throws a hard error with counts - no silent truncation.- Walker is symlink-safe: dirs are pruned before descent; symlinks to dirs are
not followed (cycle safety); symlinks to files are followed only when
realpathstays inside the resolved root. - Patterns are POSIX (
/separator). Backslash escape sequences are rejected at validation; literalpath/dirvalues may contain backslashes (Windows OK).
Inline vs upload delivery (mode)
xAI's input_file references are searchable attachments; for large source files
the model may enumerate them rather than read line-by-line. To force a full
line-by-line read, deliver the content as input_text instead:
files: [
{ path: "app/apps/api/routes.py", mode: "inline" }, // forced inline
{ path: "modules/web.tm.hcl", mode: "auto" }, // text + small → inline
{ path: "design.pdf", mode: "auto" }, // binary or big → upload
{ dir: "src", include: ["**/*.ts"], mode: "auto" }, // each walked file decides
]"upload"(default) - always uses the xAI Files API. Matches the v2.0 behavior."inline"- embeds the file content directly as a separateinput_textpart with a=== {filename} ===header. No/filescall, no cache row, nouploadedFileIdsentry. Best for source code review."auto"- inlines when the file is probably text (no NUL byte; <5% non-printable bytes in the first 4 KB) AND its size is at or belowGROK_INLINE_MAX_BYTES(default 262144 = 256 KB). Otherwise uploads.
For {dir} entries the mode is inherited by every walked file. mode must
NOT be set on file_id / file_url entries (those bypass the upload path
entirely; setting mode on them returns -32602 from validateFiles).
Override the inline ceiling with GROK_INLINE_MAX_BYTES=<bytes> in the bridge
environment.
Content-hash cache
Uploads are deduplicated by SHA-256 content hash. A reuse hit requires the SAME content
plus the same API key, the same normalised apiBase, and the same effective filename
(see cache-key below); identical bytes uploaded under a different filename or a different
key produce separate cache rows:
- Cache file:
~/.cache/deliberation/grok-files.json(canonical XDG path; Windows%LOCALAPPDATA%\deliberation\grok-files.json). Override withDELIBERATION_CACHE. - Cache key:
sha256(bytes)@sha256(XAI_API_KEY)[:16]@normalize(apiBase)@effectiveFilename- Key rotation auto-invalidates entries (different
keyFp). - Different
apiBase(including port/protocol differences) → separate rows. - Different effective filename (basename or
filenameoverride) → separate rows.
- Key rotation auto-invalidates entries (different
- Reuse check: hit +
expiresAt > now + 60s+apiBase+keyFpall match. - In-process Promise dedup (
withInflight): concurrent uploads of the same content collapse into a single network call. - Cross-process safety: mkdir-based lock (
server/grok/lock.js) with token-specific owner markers + stale reclaim via atomic rename. (lock.heartbeat()is provided for long-running holders; cache writes complete sub-second so the 5s stale window is not at risk and the bridge does not call it.) - Stale xAI file id mid-
/v1/responses: when the responses call returns a 4xx whose body names afile_*/file-*id from the current refs (and the ref has asourcePath), the bridge evicts the cached row, re-uploads from the original disk path, and retries the responses call once. Errors that don't name a known file id are surfaced unchanged. XAI_DISABLE_FILE_CACHE=1(env) skips the cache layer entirely (debugging).
Stored upload filenames are deliberation-{sha256[:16]}-{basename}. Uploads also
carry expires_after set by GROK_FILE_TTL_SECONDS (default 604800 = 7 days,
clamped 1h..30d).
Cleanup (/grok-files)
The bundled server/grok/files-admin.js supports three subcommands:
list- shows total xAI file count and everydeliberation-*upload.prune --older-than <30m|24h|7d|seconds> [--yes]- dry run by default; deletes remote bridge-owned files matched by filename prefix + age. Works without the local cache; safe for environments where the cache was lost or never existed.gc [--all-keys] [--force-local-prune]- syncs the local cache with the remote file list via one paginatedGET /v1/files. Prunes local rows whosefileIdis no longer on xAI. Default scope is the currentXAI_API_KEY+XAI_API_BASErows only.--all-keyswidens to foreign rows but leaves them in place when remote absence is ambiguous (the current key can't see foreign files).--force-local-prunedrops ambiguous foreign rows anyway.
prune and gc are complementary: prune is the remote-side cleaner; gc keeps
the local cache aligned with remote state. The deliberation- filename prefix
is a hard safety invariant on both paths - your own xAI files are never touched.