Simon Willison’s analysis of the Grok Build incident, published July 15, deserves attention beyond the security research community.
The short version: xAI’s Grok Build coding CLI shipped behavior that packaged developers’ entire Git repositories — full commit history, committed secrets, all of it — and transmitted them to a Google Cloud Storage bucket. When this became public, the company responded via social media rather than a formal security advisory. Days later, Grok Build was open-sourced under Apache 2.0. Willison’s read of the published code: the upload logic is still present in the binary, disabled by a server-side flag that xAI controls and can re-enable without shipping a software update.
The privacy toggle, it turned out, changed nothing.
This is not primarily a story about one company’s tool. It is a diagnostic for how most enterprise teams evaluate AI developer tools before they install them.
flowchart TD
Q{AI coding tool<br/>request received}
Q --> A{Data handling<br/>explicitly documented?}
A -->|No| R1[Reject — require<br/>vendor assessment first]
A -->|Yes| B{Regulated data<br/>in scope?}
B -->|Yes| C{DPA and traffic<br/>baseline complete?}
B -->|No| D[Standard IT review]
C -->|No| R2[Block pending<br/>compliance review]
C -->|Yes| E[Approved — monitor<br/>continuously]
D --> E
class R1 bad
class R2 bad
class E good
classDef good fill:#163a26,stroke:#44cc77,color:#d7ffe6;
classDef bad fill:#3a1620,stroke:#ff5555,color:#ffd9d9;
classDef warn fill:#3a2e16,stroke:#ffaa33,color:#ffe9c7;
classDef accent fill:#15233b,stroke:#4488ff,color:#dce9ff;
The rundown
Grok Build was positioned as a coding CLI competing in the same space as Cursor, GitHub Copilot, and Claude Code. Users who installed and ran it were operating under the assumption, common to most developer tools, that local code stays local unless you explicitly push it somewhere.
That assumption was wrong.
A security researcher’s wire-level analysis showed the tool was transmitting developer repositories to a Google Cloud Storage bucket at a volume roughly 27,800 times greater than what the coding task required. The full Git history went too — including secrets committed and later removed from tracked history. xAI’s initial response was a few posts on X rather than a formal security advisory.
xAI subsequently open-sourced Grok Build under Apache 2.0. Willison’s examination of the published codebase found the upload code still present and active in the binary. There is an upload_session_state() function that returns a hard-coded session_state_upload_unavailable error — a server-side gate, not a removed feature. The company controls the switch. You do not.
For engineers: you don’t control what leaves the machine
The technical disclosure is specific: a server-side flag in xAI’s infrastructure governs whether your code gets uploaded. That flag is not inside the tool you installed. It is inside a system you have no visibility into. If you are running Grok Build, or any coding CLI with similar architecture, you are trusting the vendor’s server-side configuration as much as you are trusting the client-side binary.
This is a different trust model than the one most developers apply to local development tools. Editors, linters, compilers, and most local tooling do what you can observe. The AI coding tool category — where the value proposition requires network connectivity — introduces a different pattern: the tool needs to communicate outbound to function, and you are relying on the vendor’s behavior rather than your own observation.
The practical audit question: for every AI coding tool currently running in your environment, do you know what data leaves the machine? Not what the privacy policy says — what does the network traffic actually show?
Traffic monitoring is not glamorous. Most development teams do not run it for internal tools. If your team has not run a traffic baseline on the AI tools your engineers are using, you have not answered the question.
For business owners and operators: this is an IP and compliance question your contracts haven’t answered
Most commentary on Grok Build focused on privacy. The enterprise exposure is also an intellectual property and compliance problem.
If your development team is using AI coding tools, your codebase — including proprietary implementations, trade-secret algorithms, and anything sensitive in test fixtures — is potentially in motion. Where it goes, how long it is stored, and who can access it are vendor decisions. You probably do not have a contract that specifies what happens to code transmitted to the vendor’s infrastructure, how long it is retained, or whether it can be used for model training.
For companies under HIPAA, PCI DSS, SOC 2, or government contracting frameworks, this is not a hypothetical concern. Regulated environments require knowing where data goes. Code that handles patient records, financial data, or government systems is often tested against data that has been sanitized — or has not been. An AI tool with undisclosed upload behavior is a potential compliance event.
The governance question for business owners is not whether Grok Build had malicious intent — intent isn’t the audit question. The question is: what process does your team follow before an AI tool is installed on a codebase that contains IP or regulated data?
My take: regulated environments already know this lesson
At HBSGI, I architected an EDI claims-processing system under HIPAA standards — ANSI 837/835/997 billing submission integrated with government health systems. The compliance environment was exacting: every data flow had to be documented, every transmission audited. Working against an 800-page specification meant that an unexpected outbound transmission wasn’t just a security risk — it was a reportable incident.
If a developer on that project had installed a tool that silently packaged the repository and sent it to a third-party cloud bucket, the project would have stopped. Not because the intent was malicious, but because we could not have answered regulators’ questions about what happened to that data.
Most enterprise teams are not thinking about their AI toolchain the way they think about their data flows. They should be. The Grok Build incident is useful precisely because it is visible — Willison’s analysis is public, the published code is readable, you can examine what happened. The more common risk is the tool that does something similar and hasn’t been caught yet.
A few things that should be standard practice now:
- Before any AI coding tool enters production use, run a network traffic baseline. Understand what it connects to and when.
- Require a specific data handling statement from the vendor — not a link to a policy document, but a direct answer about what leaves the machine, where it goes, and how long it is retained.
- For regulated environments — HIPAA, PCI, SOC 2, government contracts — treat AI coding tools as third-party data processors. That means a vendor assessment, a data processing agreement, and a review of what is in scope for that tool’s access.
This is not an argument against AI coding tools. They are genuinely useful, and the productivity evidence is real. It is an argument for having a policy before the tool is already installed on your codebase.
