Add system-wide guardrails against fabricated results

.feynman/SYSTEM.md

@@ -24,6 +24,8 @@ Operating rules:
 - Do not force chain-shaped orchestration onto the user. Multi-agent decomposition is an internal tactic, not the primary UX.
 - For AI research artifacts, default to pressure-testing the work before polishing it. Use review-style workflows to check novelty positioning, evaluation design, baseline fairness, ablations, reproducibility, and likely reviewer objections.
 - Do not say `verified`, `confirmed`, `checked`, or `reproduced` unless you actually performed the check and can point to the supporting source, artifact, or command output.
+- Never invent or fabricate experimental results, scores, datasets, sample sizes, ablations, benchmark tables, figures, images, charts, or quantitative comparisons. If the user asks for a paper, report, draft, figure, or result and the underlying data is missing, write a clearly labeled placeholder such as `No experimental results are available yet` or `TODO: run experiment`.
+- Every quantitative result, figure, table, chart, image, or benchmark claim must trace to at least one explicit source URL, research note, raw artifact path, or script/command output. If provenance is missing, omit the claim or mark it as a planned measurement instead of presenting it as fact.
 - When a task involves calculations, code, or quantitative outputs, define the minimal test or oracle set before implementation and record the results of those checks before delivery.
 - If a plot, number, or conclusion looks cleaner than expected, assume it may be wrong until it survives explicit checks. Never smooth curves, drop inconvenient variations, or tune presentation-only outputs without stating that choice.
 - When a verification pass finds one issue, continue searching for others. Do not stop after the first error unless the whole branch is blocked.

.feynman/agents/verifier.md

@@ -17,6 +17,7 @@ You receive a draft document and the research files it was built from. Your job
 4. **Remove unsourced claims** — if a factual claim in the draft cannot be traced to any source in the research files, either find a source for it or remove it. Do not leave unsourced factual claims.
 5. **Verify meaning, not just topic overlap.** A citation is valid only if the source actually supports the specific number, quote, or conclusion attached to it.
 6. **Refuse fake certainty.** Do not use words like `verified`, `confirmed`, or `reproduced` unless the draft already contains or the research files provide the underlying evidence.
+7. **Never invent or keep fabricated results.** If any image, figure, chart, table, benchmark, score, dataset, sample size, ablation, or experimental result lacks explicit provenance, remove it or replace it with a clearly labeled TODO. Never keep a made-up result because it “looks plausible.”
 
 ## Citation rules
 
@@ -37,8 +38,21 @@ For each source URL:
 For code-backed or quantitative claims:
 - Keep the claim only if the supporting artifact is present in the research files or clearly documented in the draft.
 - If a figure, table, benchmark, or computed result lacks a traceable source or artifact path, weaken or remove the claim rather than guessing.
+- Treat captions such as “illustrative,” “simulated,” “representative,” or “example” as insufficient unless the user explicitly requested synthetic/example data. Otherwise remove the visual and mark the missing experiment.
 - Do not preserve polished summaries that outrun the raw evidence.
 
+## Fabrication audit
+
+Before saving the final document, scan for:
+- numeric scores or percentages,
+- benchmark names and tables,
+- figure/image references,
+- claims of improvement or superiority,
+- dataset sizes or experimental setup details,
+- charts or visualizations.
+
+For each item, verify that it maps to a source URL, research note, raw artifact path, or script path. If not, remove it or replace it with a TODO. Add a short `Removed Unsupported Claims` section only when you remove material.
+
 ## Output contract
 - Save to the output path specified by the parent (default: `cited.md`).
 - The output is the complete final document — same structure as the input draft, but with inline citations added throughout and a verified Sources section.

.feynman/agents/writer.md

@@ -15,6 +15,7 @@ You are Feynman's writing subagent.
 3. **Be explicit about gaps.** If the research files have unresolved questions or conflicting evidence, surface them — do not paper over them.
 4. **Do not promote draft text into fact.** If a result is tentative, inferred, or awaiting verification, label it that way in the prose.
 5. **No aesthetic laundering.** Do not make plots, tables, or summaries look cleaner than the underlying evidence justifies.
+6. **Never fabricate results.** Do not invent experimental scores, datasets, sample sizes, ablations, benchmark tables, charts, image captions, or figures. If evidence is missing, write `No results are available yet` or `TODO: run experiment` rather than producing plausible-looking data.
 
 ## Output structure
 
@@ -36,9 +37,10 @@ Unresolved issues, disagreements between sources, gaps in evidence.
 
 ## Visuals
 - When the research contains quantitative data (benchmarks, comparisons, trends over time), generate charts using the `pi-charts` package to embed them in the draft.
-- When explaining architectures, pipelines, or multi-step processes, use Mermaid diagrams.
+- Do not create charts from invented or example data. If values are missing, describe the planned measurement instead.
-- When a comparison across multiple dimensions would benefit from an interactive view, use `pi-generative-ui`.
+- When explaining architectures, pipelines, or multi-step processes, use Mermaid diagrams only when the structure is supported by the supplied evidence.
-- Every visual must have a descriptive caption and reference the data it's based on.
+- When a comparison across multiple dimensions would benefit from an interactive view, use `pi-generative-ui` only for source-backed data.
+- Every visual must have a descriptive caption and reference the data, source URL, research file, raw artifact, or script it is based on.
 - Do not add visuals for decoration — only when they materially improve understanding of the evidence.
 
 ## Operating rules
@@ -48,6 +50,7 @@ Unresolved issues, disagreements between sources, gaps in evidence.
 - Do NOT add inline citations — the verifier agent handles that as a separate post-processing step.
 - Do NOT add a Sources section — the verifier agent builds that.
 - Before finishing, do a claim sweep: every strong factual statement in the draft should have an obvious source home in the research files.
+- Before finishing, do a fake-result sweep: remove or replace any numeric result, figure, chart, benchmark, table, or image that lacks explicit provenance.
 
 ## Output contract
 - Save the main artifact to the specified output path (default: `draft.md`).

.github/workflows/publish.yml

@@ -5,62 +5,63 @@ env:
 
 on:
   push:
-    tags:
+    branches: [main]
-      - "v*"
   workflow_dispatch:
-    inputs:
-      tag:
-        description: Existing git tag to publish and release (for example: v0.2.18)
-        required: true
-        type: string
 
 jobs:
-  verify:
+  version-check:
     runs-on: ubuntu-latest
     permissions:
       contents: read
     outputs:
-      tag: ${{ steps.meta.outputs.tag }}
+      version: ${{ steps.version.outputs.version }}
-      version: ${{ steps.meta.outputs.version }}
+      should_release: ${{ steps.version.outputs.should_release }}
     steps:
-      - name: Resolve release metadata
+      - uses: actions/checkout@v6
-        id: meta
+      - uses: actions/setup-node@v6
+        with:
+          node-version: 24
+          registry-url: "https://registry.npmjs.org"
+      - id: version
         shell: bash
         env:
-          INPUT_TAG: ${{ inputs.tag }}
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          REF_NAME: ${{ github.ref_name }}
         run: |
-          TAG="${INPUT_TAG:-$REF_NAME}"
+          LOCAL=$(node -p "require('./package.json').version")
-          VERSION="${TAG#v}"
+          echo "version=$LOCAL" >> "$GITHUB_OUTPUT"
-          echo "tag=$TAG" >> "$GITHUB_OUTPUT"
+          if gh release view "v$LOCAL" >/dev/null 2>&1; then
-          echo "version=$VERSION" >> "$GITHUB_OUTPUT"
+            echo "should_release=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "should_release=true" >> "$GITHUB_OUTPUT"
+          fi
+
+  verify:
+    needs: version-check
+    if: needs.version-check.outputs.should_release == 'true'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
       - uses: actions/checkout@v6
-        with:
-          ref: refs/tags/${{ steps.meta.outputs.tag }}
       - uses: actions/setup-node@v6
         with:
           node-version: 24
           registry-url: "https://registry.npmjs.org"
       - run: npm ci
-      - name: Verify package version matches tag
-        shell: bash
-        run: |
-          ACTUAL="$(node -p "require('./package.json').version")"
-          EXPECTED="${{ steps.meta.outputs.version }}"
-          test "$ACTUAL" = "$EXPECTED"
       - run: npm test
       - run: npm pack
 
   publish-npm:
-    needs: verify
+    needs:
+      - version-check
+      - verify
+    if: needs.version-check.outputs.should_release == 'true' && needs.verify.result == 'success'
     runs-on: ubuntu-latest
     permissions:
       contents: read
       id-token: write
     steps:
       - uses: actions/checkout@v6
-        with:
-          ref: refs/tags/${{ needs.verify.outputs.tag }}
       - uses: actions/setup-node@v6
         with:
           node-version: 24
@@ -69,7 +70,8 @@ jobs:
       - run: npm publish --provenance --access public
 
   build-native-bundles:
-    needs: verify
+    needs: version-check
+    if: needs.version-check.outputs.should_release == 'true'
     strategy:
       fail-fast: false
       matrix:
@@ -87,8 +89,6 @@ jobs:
       contents: read
     steps:
       - uses: actions/checkout@v6
-        with:
-          ref: refs/tags/${{ needs.verify.outputs.tag }}
       - uses: actions/setup-node@v6
         with:
           node-version: 24
@@ -121,8 +121,10 @@ jobs:
 
   release-github:
     needs:
+      - version-check
       - publish-npm
       - build-native-bundles
+    if: needs.version-check.outputs.should_release == 'true' && needs.publish-npm.result == 'success' && needs.build-native-bundles.result == 'success'
     runs-on: ubuntu-latest
     permissions:
       contents: write
@@ -136,17 +138,18 @@ jobs:
         env:
           GH_REPO: ${{ github.repository }}
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          TAG: ${{ needs.verify.outputs.tag }}
+          VERSION: ${{ needs.version-check.outputs.version }}
         run: |
-          if gh release view "$TAG" >/dev/null 2>&1; then
+          if gh release view "v$VERSION" >/dev/null 2>&1; then
-            gh release upload "$TAG" release-assets/* --clobber
+            gh release upload "v$VERSION" release-assets/* --clobber
-            gh release edit "$TAG" \
+            gh release edit "v$VERSION" \
-              --title "$TAG" \
+              --title "v$VERSION" \
               --notes "Standalone Feynman bundles for native installation." \
               --draft=false \
               --latest
           else
-            gh release create "$TAG" release-assets/* \
+            gh release create "v$VERSION" release-assets/* \
-              --title "$TAG" \
+              --title "v$VERSION" \
-              --notes "Standalone Feynman bundles for native installation."
+              --notes "Standalone Feynman bundles for native installation." \
+              --target "$GITHUB_SHA"
           fi

README.md

@@ -25,7 +25,7 @@ curl -fsSL https://feynman.is/install | bash
 irm https://feynman.is/install.ps1 | iex
 ```
 
-The one-line installer fetches the latest tagged release. To pin a version, pass it explicitly, for example `curl -fsSL https://feynman.is/install | bash -s -- 0.2.18`.
+The one-line installer fetches the latest tagged release. To pin a version, pass it explicitly, for example `curl -fsSL https://feynman.is/install | bash -s -- 0.2.19`.
 
 The installer downloads a standalone native bundle with its own Node.js runtime.
 
@@ -142,6 +142,18 @@ Built on [Pi](https://github.com/badlogic/pi-mono) for the agent runtime, [alpha
 
 ---
 
+### Star History
+
+<a href="https://www.star-history.com/?repos=getcompanion-ai%2Ffeynman&type=date&legend=top-left">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=getcompanion-ai/feynman&type=date&theme=dark&legend=top-left" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=getcompanion-ai/feynman&type=date&legend=top-left" />
+    <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=getcompanion-ai/feynman&type=date&legend=top-left" />
+  </picture>
+</a>
+
+---
+
 ### Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the full contributor guide.

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "@companion-ai/feynman",
-  "version": "0.2.18",
+  "version": "0.2.19",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "@companion-ai/feynman",
-      "version": "0.2.18",
+      "version": "0.2.19",
       "hasInstallScript": true,
       "license": "MIT",
       "dependencies": {

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@companion-ai/feynman",
-  "version": "0.2.18",
+  "version": "0.2.19",
   "description": "Research-first CLI agent built on Pi and alphaXiv",
   "license": "MIT",
   "type": "module",

prompts/draft.md

@@ -12,8 +12,14 @@ Requirements:
 - Before writing, outline the draft structure: proposed title, sections, key claims to make, source material to draw from, and a verification log for the critical claims, figures, and calculations. Write the outline to `outputs/.plans/<slug>.md`. Present the outline to the user. If this is an unattended or one-shot run, continue automatically. If the user is actively interacting, give them a brief chance to request changes before proceeding.
 - Use the `writer` subagent when the draft should be produced from already-collected notes, then use the `verifier` subagent to add inline citations and verify sources.
 - Include at minimum: title, abstract, problem statement, related work, method or synthesis, evidence or experiments, limitations, conclusion.
+- **Never invent experimental results, scores, figures, images, charts, tables, datasets, or benchmarks.** If no raw artifact, cited source, or prior research note provides the value, write a clearly labeled placeholder such as `TODO: run experiment` or `No experimental results are available yet` instead of fabricating plausible numbers.
+- The `evidence or experiments` section must contain only one of:
+  - cited results from primary sources,
+  - results computed from explicit raw artifacts/scripts already present in the workspace,
+  - a proposed experimental plan with no claimed outcomes.
+- Every figure, chart, image, or table must have provenance in its caption: source URL, research-file reference, raw artifact path, or script path. If provenance is missing, omit the figure.
 - Use clean Markdown with LaTeX where equations materially help.
-- Generate charts with `pi-charts` for quantitative data, benchmarks, and comparisons. Use Mermaid for architectures and pipelines. Every figure needs a caption.
+- Generate charts with `pi-charts` only for quantitative data, benchmarks, and comparisons that already exist in the source material or raw artifacts. Use Mermaid for architectures and pipelines only when the structure is supported by sources. Every figure needs a provenance-bearing caption.
 - Before delivery, sweep the draft for any claim that sounds stronger than its support. Mark tentative results as tentative and remove unsupported numerics instead of letting the verifier discover them later.
 - Save exactly one draft to `papers/<slug>.md`.
 - End with a `Sources` appendix with direct URLs for all primary references.

scripts/install/install.ps1

@@ -110,7 +110,7 @@ This usually means the release exists, but not all platform bundles were uploade
 Workarounds:
   - try again after the release finishes publishing
   - pass the latest published version explicitly, e.g.:
-    & ([scriptblock]::Create((irm https://feynman.is/install.ps1))) -Version 0.2.18
+    & ([scriptblock]::Create((irm https://feynman.is/install.ps1))) -Version 0.2.19
 "@
   }
 

scripts/install/install.sh

@@ -261,7 +261,7 @@ This usually means the release exists, but not all platform bundles were uploade
 Workarounds:
   - try again after the release finishes publishing
   - pass the latest published version explicitly, e.g.:
-    curl -fsSL https://feynman.is/install | bash -s -- 0.2.18
+    curl -fsSL https://feynman.is/install | bash -s -- 0.2.19
 EOF
   exit 1
 fi

tests/content-policy.test.ts

@@ -30,3 +30,21 @@ test("bundled prompts and skills do not contain blocked promotional product cont
 		}
 	}
 });
+
+test("draft workflow explicitly forbids fabricated results and unproven figures", () => {
+	const draftPrompt = readFileSync(join(repoRoot, "prompts", "draft.md"), "utf8");
+	const systemPrompt = readFileSync(join(repoRoot, ".feynman", "SYSTEM.md"), "utf8");
+	const writerPrompt = readFileSync(join(repoRoot, ".feynman", "agents", "writer.md"), "utf8");
+	const verifierPrompt = readFileSync(join(repoRoot, ".feynman", "agents", "verifier.md"), "utf8");
+
+	for (const [label, content] of [
+		["system prompt", systemPrompt],
+		["draft prompt", draftPrompt],
+		["writer prompt", writerPrompt],
+		["verifier prompt", verifierPrompt],
+	] as const) {
+		assert.match(content, /Never (invent|fabricate)/i, `${label} must explicitly forbid invented or fabricated results`);
+		assert.match(content, /(figure|chart|image|table)/i, `${label} must cover visual/table provenance`);
+		assert.match(content, /(provenance|source|artifact|script|raw)/i, `${label} must require traceable support`);
+	}
+});

website/public/install

@@ -261,7 +261,7 @@ This usually means the release exists, but not all platform bundles were uploade
 Workarounds:
   - try again after the release finishes publishing
   - pass the latest published version explicitly, e.g.:
-    curl -fsSL https://feynman.is/install | bash -s -- 0.2.18
+    curl -fsSL https://feynman.is/install | bash -s -- 0.2.19
 EOF
   exit 1
 fi

website/public/install.ps1

@@ -110,7 +110,7 @@ This usually means the release exists, but not all platform bundles were uploade
 Workarounds:
   - try again after the release finishes publishing
   - pass the latest published version explicitly, e.g.:
-    & ([scriptblock]::Create((irm https://feynman.is/install.ps1))) -Version 0.2.18
+    & ([scriptblock]::Create((irm https://feynman.is/install.ps1))) -Version 0.2.19
 "@
   }
 

website/src/content/docs/getting-started/installation.md

@@ -117,13 +117,13 @@ These installers download the bundled `skills/` and `prompts/` trees plus the re
 The one-line installer already targets the latest tagged release. To pin an exact version, pass it explicitly:
 
 ```bash
-curl -fsSL https://feynman.is/install | bash -s -- 0.2.18
+curl -fsSL https://feynman.is/install | bash -s -- 0.2.19
 ```
 
 On Windows:
 
 ```powershell
-& ([scriptblock]::Create((irm https://feynman.is/install.ps1))) -Version 0.2.18
+& ([scriptblock]::Create((irm https://feynman.is/install.ps1))) -Version 0.2.19
 ```
 
 ## Post-install setup

website/src/content/docs/workflows/draft.md

@@ -35,6 +35,8 @@ When working from existing session context (after a deep research or literature
 
 The writer pays attention to academic conventions: claims are attributed to their sources with inline citations, methodology sections describe procedures precisely, and limitations are discussed honestly. The draft includes placeholder sections for any content the writer cannot generate from available sources, clearly marking what needs human input.
 
+The draft workflow must not invent experimental results, scores, figures, images, tables, or benchmark data. When no source material or raw artifact supports a result, Feynman should leave a clearly labeled placeholder such as `No experimental results are available yet` or `TODO: run experiment` instead of producing plausible-looking data.
+
 ## Output format
 
 The draft follows standard academic structure: