Deliberationv4.6.1

Grok files and cleanup

Grok files and cleanup

Grok reads attached files via the files[] parameter. Each entry has EXACTLY ONE of:

  • path - a local file. Delivery is controlled by mode (default "upload" - bridge uploads to the xAI Files API; "inline" embeds as input_text; "auto" picks per heuristic - see "Inline vs upload delivery" below).
  • file_id - an already-uploaded xAI file id (passed through, no upload).
  • file_url - a public URL (passed through).
  • dir - a local directory expanded recursively. Same mode rules; the walker applies the chosen mode to every selected file (see below).

A path entry may also set headBytes (positive integer) to send only the start of the file: the bridge reads at most that many bytes (never the whole file), cuts at the last line break (or a UTF-8 character boundary), and appends [truncated: first N of M bytes]. A headBytes entry is always inline and is rejected with mode: "upload"; content plus marker is clamped to GROK_INLINE_MAX_BYTES, and a FIFO or other non-regular file is refused. headBytes is not allowed on dir, file_id, or file_url; a dir entry keeps maxBytes as its walk cap. The OpenRouter bridge accepts the same field on path entries.

A path or dir resolves against the top-level roots[] array (absolute directories, first-root-wins for relative entries) or, when roots is omitted, against cwd. A path that resolves outside every declared root is refused (no exfiltration); symlinks that escape via realpath are also refused. An oversize file (>48 MB) returns file-too-large.

Cross-repo example

js
mcp__deliberation-grok__grok({
  prompt: "Compare the auth strategy in these two services.",
  cwd: "/Users/me/work/service-a",
  roots: ["/Users/me/work/service-a", "/Users/me/work/service-b"],
  files: [
    { path: "src/auth.ts" },                          // resolves under service-a (first root)
    { path: "/Users/me/work/service-b/src/auth.ts" }, // absolute, must lie under one root
    { dir: "docs", include: ["**/*.md"], maxFiles: 20 }, // expands service-a/docs
  ],
})

Directory expansion ({dir})

The bridge bundles a zero-dep glob walker (server/grok/glob.js) so you do not have to enumerate every file by hand:

  • include (default ["**/*"]).
  • exclude is appended to the bridge's safe defaults (it does NOT replace them). Defaults cover: VCS (.git); JS/Node (node_modules, dist, build, out, .next, .svelte-kit, .nuxt, .turbo, .cache, .parcel-cache, .pnpm-store); Yarn Berry (.yarn/cache, .yarn/unplugged); lockfiles (**/*.lock); Python (.venv, venv, __pycache__, .tox, .pytest_cache, .mypy_cache, .ruff_cache, .ipynb_checkpoints, .eggs, htmlcov); coverage (coverage, .nyc_output); Rust/Java/Gradle (target, .gradle); Go/PHP (vendor); Terraform (.terraform, .terragrunt-cache); plus security: **/*.tfstate*, granular .env variants (keeps .env.example readable), .ssh/**, SSH keypairs (id_rsa, id_ed25519, id_ecdsa, id_dsa and .pub), and **/*.pem/**/*.key.
  • To replace defaults entirely instead of appending, set excludeReset: true on the same {dir} entry. excludeReset is validated as a strict boolean by validateFiles; non-boolean values are rejected. Use only when reviewing files defaults would block (e.g., Terraform state in a security audit, or legitimate .pem public certs). Tradeoff is explicit: the bridge prefers a false positive (blocking a legitimate .pem) over a false negative (leaking a private key).
  • maxFiles (default 50), maxBytes (default 128 MB). Exceeding either throws a hard error with counts - no silent truncation.
  • Walker is symlink-safe: dirs are pruned before descent; symlinks to dirs are not followed (cycle safety); symlinks to files are followed only when realpath stays inside the resolved root.
  • Patterns are POSIX (/ separator). Backslash escape sequences are rejected at validation; literal path/dir values may contain backslashes (Windows OK).

Inline vs upload delivery (mode)

xAI's input_file references are searchable attachments; for large source files the model may enumerate them rather than read line-by-line. To force a full line-by-line read, deliver the content as input_text instead:

js
files: [
  { path: "app/apps/api/routes.py", mode: "inline" },   // forced inline
  { path: "modules/web.tm.hcl",     mode: "auto" },     // text + small → inline
  { path: "design.pdf",             mode: "auto" },     // binary or big → upload
  { dir:  "src", include: ["**/*.ts"], mode: "auto" },  // each walked file decides
]
  • "upload" (default) - always uses the xAI Files API. Matches the v2.0 behavior.
  • "inline" - embeds the file content directly as a separate input_text part with a === {filename} === header. No /files call, no cache row, no uploadedFileIds entry. Best for source code review.
  • "auto" - inlines when the file is probably text (no NUL byte; <5% non-printable bytes in the first 4 KB) AND its size is at or below GROK_INLINE_MAX_BYTES (default 262144 = 256 KB). Otherwise uploads.

For {dir} entries the mode is inherited by every walked file. mode must NOT be set on file_id / file_url entries (those bypass the upload path entirely; setting mode on them returns -32602 from validateFiles). Override the inline ceiling with GROK_INLINE_MAX_BYTES=<bytes> in the bridge environment.

Content-hash cache

Uploads are deduplicated by SHA-256 content hash. A reuse hit requires the SAME content plus the same API key, the same normalised apiBase, and the same effective filename (see cache-key below); identical bytes uploaded under a different filename or a different key produce separate cache rows:

  • Cache file: ~/.cache/deliberation/grok-files.json (canonical XDG path; Windows %LOCALAPPDATA%\deliberation\grok-files.json). Override with DELIBERATION_CACHE.
  • Cache key: sha256(bytes)@sha256(XAI_API_KEY)[:16]@normalize(apiBase)@effectiveFilename
    • Key rotation auto-invalidates entries (different keyFp).
    • Different apiBase (including port/protocol differences) → separate rows.
    • Different effective filename (basename or filename override) → separate rows.
  • Reuse check: hit + expiresAt > now + 60s + apiBase + keyFp all match.
  • In-process Promise dedup (withInflight): concurrent uploads of the same content collapse into a single network call.
  • Cross-process safety: mkdir-based lock (server/grok/lock.js) with token-specific owner markers + stale reclaim via atomic rename. (lock.heartbeat() is provided for long-running holders; cache writes complete sub-second so the 5s stale window is not at risk and the bridge does not call it.)
  • Stale xAI file id mid-/v1/responses: when the responses call returns a 4xx whose body names a file_* / file-* id from the current refs (and the ref has a sourcePath), the bridge evicts the cached row, re-uploads from the original disk path, and retries the responses call once. Errors that don't name a known file id are surfaced unchanged.
  • XAI_DISABLE_FILE_CACHE=1 (env) skips the cache layer entirely (debugging).

Stored upload filenames are deliberation-{sha256[:16]}-{basename}. Uploads also carry expires_after set by GROK_FILE_TTL_SECONDS (default 604800 = 7 days, clamped 1h..30d).

Cleanup (/grok-files)

The bundled server/grok/files-admin.js supports three subcommands:

  • list - shows total xAI file count and every deliberation-* upload.
  • prune --older-than <30m|24h|7d|seconds> [--yes] - dry run by default; deletes remote bridge-owned files matched by filename prefix + age. Works without the local cache; safe for environments where the cache was lost or never existed.
  • gc [--all-keys] [--force-local-prune] - syncs the local cache with the remote file list via one paginated GET /v1/files. Prunes local rows whose fileId is no longer on xAI. Default scope is the current XAI_API_KEY + XAI_API_BASE rows only. --all-keys widens to foreign rows but leaves them in place when remote absence is ambiguous (the current key can't see foreign files). --force-local-prune drops ambiguous foreign rows anyway.

prune and gc are complementary: prune is the remote-side cleaner; gc keeps the local cache aligned with remote state. The deliberation- filename prefix is a hard safety invariant on both paths - your own xAI files are never touched.

Rendered from TECHNICAL.md in the repository.

Type to search the site.