How to Run a ChatGPT Code Detector Workflow with Hermes Agent

The 2025 Stack Overflow Developer Survey found that 84% of respondents use or plan to use AI tools in their development process, up from 76% the year before. The same survey shows that 46% of developers do not trust the accuracy of AI tool output, up from 31% in 2024. Only 33% report trusting that output. That trust gap is the reason a ChatGPT code detector belongs in the pull request path, not in a one-off browser paste after something already merged.

Search demand for this problem is still early but real. DataForSEO US data puts "chatgpt code detector" around 110 monthly searches with keyword difficulty 21, while related queries like "ai generated code checker" sit near 90 searches with a CPC around $18.16. Most public results answer with generic paste-a-snippet detectors. Almost none describe an agent-driven gate that runs on diffs before a human merge review. That is the workflow gap this guide fills.

A ChatGPT code detector workflow screens pull requests and snippets for likely ChatGPT-authored code so teams can apply review depth proportional to AI assistance risk. The goal is not to ban AI-assisted work. Stack Overflow also reported that 45% of respondents find debugging AI-generated code time-consuming, and that "almost right, but not quite" is the single most common AI frustration (66%). Teams still ship AI-assisted code. They need a way to know which files deserve a careful human pass, which need extra tests, and which can take a lighter path.

Nous Research Hermes Agent is a strong fit for that gate. Official docs position Hermes as an autonomous agent for code writing and review workflows, with tools, an open skills system compatible with agentskills.io, subagent delegation, hooks, and MCP connectivity. Hermes is open source under MIT. It runs on your machine or server rather than as a closed SaaS detector. You build the screening policy as a skill and attach it to PR intake, instead of hoping a random web checker understands your repository.

No commercial or open detector can prove authorship with certainty. Models change, developers edit heavily, and teams mix ChatGPT, Copilot, Claude, and local models in the same PR. Treat every score as a risk estimate that drives process, not a yes/no accusation.

Practical signals fall into four buckets. Use them together.

Style and structure signals. Uniform comment density, symmetric helper naming, generic error messages, and "textbook" function shapes that do not match surrounding modules. Over-explained helpers next to sparse domain code is a common tell. Sudden adoption of patterns the repo has never used (new logging style, new retry wrapper, new DTO layer) without a design note also raises the score.

Correctness and fit signals. Tests that assert happy paths only, mocks that hide real boundary failures, and APIs that look complete but ignore existing package conventions. Stack Overflow's trust data exists for a reason: AI code often looks polished while missing the hard cases reviewers care about. Flag code that invents utility modules instead of reusing project helpers.

Provenance signals. Author self-disclosure ("assisted by ChatGPT"), commit messages that mention AI tools, PR templates with an AI-assistance checkbox, IDE-generated footers, or linked chat exports. These are the only high-confidence signals. Prefer honest disclosure over adversarial detection.

Process signals. Large multi-file PRs opened by junior contributors with short cycle time, churn on the same file across tiny commits, and changes in security-sensitive paths (auth, crypto, payments, migrations) without accompanying design discussion. Process signals do not prove AI use. They do justify deeper review regardless of authorship.

Weight signals by risk surface. A cosmetic React component with high "AI-likeness" can stay low priority. A small auth change with medium score should still force senior review. The detector's job is triage.

What generic checkers miss

Browser-based AI detectors and essay-style classifiers optimize for prose. Source code is different. Token patterns, indentation, and library idioms shift by language and model version. ZeroGPT, GPTZero, and similar tools can still help as one heuristic on long comments or generated docs, but they are weak as a sole PR gate. An agent workflow that reads the full diff, project conventions, and test layout will catch more useful risk than a black-box percentage on a pasted file.

Hermes Agent does not ship a one-click "detect ChatGPT code" product switch. Official positioning is broader: code writing and review through tools, skills, and subagents. That is enough. You encode your detector as a skill, run it on PR diffs, and optionally call external checkers as tools when you want a second opinion.

Start from the official learning path for a CLI coding assistant: install Hermes, complete setup, learn CLI usage and code execution, then add context files and skills. For team automation, also read messaging, cron, hooks, and MCP docs so the agent can receive PR events and write durable reports.

Author a screening skill

Skills live under ~/.hermes/skills/ and follow the agentskills.io-style SKILL.md format documented by Hermes. A practical skill might be named chatgpt-code-detector with sections for when to use it, procedure, pitfalls, and verification. The procedure should instruct the agent to:

1. Collect the PR title, body, file list, and unified diffs for changed files.
2. Read surrounding project conventions (lint rules, existing helpers, CONTRIBUTING notes, prior review comments).
3. Score each file against the style, fit, provenance, and process signals above.
4. Produce a structured report: overall risk, per-file notes, missing tests, security hotspots, and a recommended review checklist.
5. Never assert absolute authorship. Prefer language like "likely AI-assisted" and "elevated review recommended."

You can draft the skill by hand or use Hermes /learn against your own runbooks and past review notes so the agent authors a skill that matches how your team already reviews. If you want human control over agent skill writes, enable skill write approval so staged skill changes require approve/deny before they become procedural memory.

Hermes also supports skill bundles. Pair the detector skill with existing GitHub-oriented skills such as PR workflow helpers and code review skills when those are installed. The bundle loads several procedures in one invocation so the agent can open or inspect a PR, score the diff, and open follow-up tasks without re-prompting each step.

Use tools, subagents, and optional detectors

Hermes can run code, read files, and call MCP servers. A useful pattern:

• Parent agent owns the PR report and merge recommendation.
• One subagent inspects tests and coverage gaps.
• Another subagent focuses on security-sensitive paths only.
• Optional tool call to a local or third-party AI-generated code checker for long comment blocks or generated docs.

Keep external detectors optional. They add noise and often disagree. Your project-specific rubric plus human review is the source of truth.

Wire it to pull request intake

Practical intake options teams use with Hermes:

• Manual CLI: Reviewer pastes a PR URL or local branch and invokes the skill.
• Messaging gateway: A bot channel receives PR links from Telegram, Discord, or similar and returns the risk report where the team already works.
• Scheduled or hook-driven runs: Cron or hooks kick the agent when a label like needs-ai-screen is applied.

Do not invent policy into CI until the rubric is stable. Start with advisory comments. Move to blocking only after false-positive rates are acceptable on your repos.

Keep PR screening reports where humans can re-open them

Store Hermes Agent risk scores, evidence snippets, and approval history in a shared Fast.io workspace with Intelligence Mode search, tasks, and versioned files. Start with a 14-day free trial, then pick Solo, Business, or Growth as the team grows.

Start 14-Day Trial

This is the workflow most teams want in a featured snippet form. Adapt the thresholds to your org, but keep the sequence stable so reviewers know what the score means.

Step 1: Capture the change set. Hermes pulls the PR description, linked issues, changed paths, and full diffs. Prefer path-limited scans for monorepos so the agent does not drown in unrelated packages.

Step 2: Classify paths. Tag each file as product code, tests, config, generated, docs, or security-critical. Generated lockfiles and pure renames score zero unless they hide real logic.

Step 3: Score signals. Apply the Low/Medium/High rubric. Record evidence snippets (not whole files) so a human can verify quickly.

Step 4: Apply thresholds.

• Low: Post an advisory summary. Standard reviewer assignment continues.
• Medium: Require author notes on AI use, expand test coverage on changed behaviors, and assign a second reviewer if the author is the only domain expert.
• High: Hold merge. Require senior review on security paths, explicit rationale for new modules, and tests that fail when the new logic is removed. Re-run the detector after revisions.

Step 5: Write the review package. The agent should output:

• Overall risk and confidence
• Top three risks, not a wall of nits
• Suggested test cases
• Open questions for the author
• A one-paragraph merge recommendation

Step 6: Human decides. The detector never merges. Humans own policy, exceptions, and accountability. Hermes can draft comments; reviewers still click approve.

How to review ChatGPT-assisted code in the PR

When a PR is flagged Medium or High, change the review posture:

• Ask the author to narrate intent for non-obvious helpers.
• Diff against existing project utilities before accepting new ones.
• Run the feature under failure conditions, not only the demo path.
• Check dependency and secret handling carefully; AI code often hardcodes defaults or broad permissions.
• Prefer smaller follow-up PRs over rubber-stamping a large AI-generated scaffold.

Hermes is especially useful here as a second reader: it can re-scan after force-pushes, compare iterations, and keep the risk history attached to the PR conversation.

Screening only helps if the report survives beyond a chat scrollback. Local agent disks, temporary CI artifacts, and private DMs all work for a solo experiment. They fail when two reviewers need the same evidence next week, or when an auditor asks why a High-risk PR was approved.

Common storage options:

• Git host artifacts: PR comments and check runs on GitHub or GitLab. Good for inline review, weak for long-lived structured history outside the PR.
• Object storage or S3 buckets: Cheap for raw JSON reports. You still need search, permissions, and a human-friendly UI.
• Shared drives (Google Drive, Dropbox): Easy for people, awkward for agents that need stable APIs and versioned machine-readable outputs.

Fast.io sits in the agent-friendly middle. Hermes can write screening reports into a shared workspace via the Fast.io API or MCP server (Streamable HTTP at /mcp, legacy SSE at /sse). Enable Intelligence Mode so reports become searchable by meaning, not only filename. Use tasks and approvals to route High-risk PRs to a senior reviewer, and keep an append-only audit log of who screened, who approved, and which report version they used.

Practical layout many teams copy:

• screens/YYYY/PR-1234/report.md for the human-readable summary
• screens/YYYY/PR-1234/score.json for the machine score
• screens/YYYY/PR-1234/diff-snippets/ for evidence excerpts
• A Metadata View over score JSON for dashboards of High/Medium rates by repo (see document data extraction)

Per-file version history matters when the agent re-runs after a push. Reviewers can open the previous report and see what improved. Ownership transfer helps when an agent account sets up the workspace and a human org admin takes over for ongoing policy. Plans start with a 14-day free trial (credit card required): Solo at $29/mo, Business at $99/mo, Growth at $299/mo. Creating an account is free, but real org work needs a paid subscription after the trial.

For Hermes specifically, treat Fast.io as the durable handoff layer. Hermes runs on local, Docker, SSH, or serverless backends and may sleep between jobs. The workspace keeps reports, rubrics, and skill exports available to humans and other agents after the container is gone. Pair that with branded shares only when you need to send a redacted screening summary outside the eng org.

Expect false positives on highly regular human code, especially style-enforced TypeScript services and generated API clients. Expect false negatives on heavily edited AI drafts that match local idioms. Tuning is continuous.

Calibrate on known samples. Feed the skill a set of PRs you already know were AI-assisted and a set that were not. Adjust weights until Medium/High rates match review capacity. If everything is High, reviewers ignore the signal.

Exclude generated noise. Lockfiles, protobuf outputs, vendor trees, and large JSON fixtures should not dominate scores.

Prefer disclosure over policing. Add an optional PR template checkbox for AI assistance. When authors disclose, lower the authorship weight and raise the test/security checklist instead. That reduces adversarial dynamics.

Separate quality from origin. A high-quality AI-assisted change can still be Medium because tests are thin. A low-scoring human change can still need senior review if it touches payments. Keep those axes independent in the report.

Version the skill. When your team changes the rubric, bump the skill version and store both rubric and reports in the workspace. Old PRs remain interpretable.

Do not outsource judgment to a percentage. External AI-generated code checkers are optional inputs. Hermes should summarize their output as one line among many, never as the merge decision.

Teams that get value from this workflow treat the detector as CI-adjacent policy: automated, explainable, and subordinate to humans. Hermes Agent supplies the automation surface (skills, tools, subagents, gateways). Your rubric and storage layer supply the institutional memory. Together they answer a practical question every busy team faces: where should the next hour of human review go?