Somewhere in your organization there is AI you have not counted. That is not a guess about your diligence. It is what happened at one federal agency, twice.
The Small Business Administration was piloting AI to detect fraud in disbursed pandemic loans, but the pilot was missing from the 34 AI use cases on its public website as of March 2026. In a September 24, 2026, advisory, its inspector general disputed managers’ call that the pilot was not high-impact; SBA had produced no documentation to support it.¹
The agency had done this before, with a system it paid someone to build. From 2021 through 2023, SBA reported no qualifying AI use cases while using a contractor-developed model to score Paycheck Protection Program loan-forgiveness applications for fraud risk.²,³ For about 92,000 loans that already had fraud flags or alerts on them, AI cleared the loans and the contractor sent the recommendation “without conducting a human-led manual review.”³ The Government Accountability Office warned that loans scored as lower risk could still be fraudulent and receive no additional review or referral.³ SBA’s explanation for the missing report: officials had looked for AI only inside the Office of the Chief Information Officer.² The model was elsewhere.
When we ask an organization where its AI is, someone tells us to ask IT. That is a reasonable place to begin and a dangerous place to stop. SBA’s inventory missed a model the agency had paid for. It can just as easily miss a feature nobody asked for.
AI lives in four places: the systems an organization builds, the products and services it buys, the AI bundled inside software and services it already uses, and the tools employees bring to work. Bought, Built, Bundled, and Brought. Those places do not map neatly onto an org chart. A contractor-built model can sit outside the CIO’s office. An update can introduce AI without a new purchase.
Bought AI
The federal government’s latest AI census accounts for both stand-alone projects and features inside familiar products. Alongside individual use cases, the Office of Management and Budget published a separate consolidated list of COTS (commercial off-the-shelf) AI, including Copilot in Outlook, Face ID on government phones, and machine learning inside Concur.⁴ AI is not always a separate purchase. It can come with the software, the subscription, or the update.
Built AI
Even among deployed projects that agencies did report, fewer than half of them consisted of custom-developed code. In the 2025 inventory, 41 agencies reported 3,611 individual use cases.⁴ Of those, 1,040 were marked “Deployed”; we found that 542, or 52%, reported no custom-developed code.⁵ A search limited to the dev shop risks the same kind of omission SBA made: the place the organization looks determines which AI it sees.
Bundled AI
In a hospital, bundled AI can shape decisions about patients.
Epic’s electronic health record included a proprietary sepsis-prediction model already used at hundreds of hospitals when University of Michigan researchers published an external evaluation in 2021. At the threshold they studied, the model failed to identify 67% of sepsis cases while flagging 18% of hospitalizations, the researchers found.⁶ That was a retrospective evaluation, not proof that the model caused those patients harm. It showed why a familiar software supplier is not a substitute for testing the predictions its software makes.
Epic revised the model. A multicenter study published in February 2026 found that the new version was better at distinguishing patients with sepsis from those without it, but performance varied substantially between hospitals, many alerts did not correspond to sepsis, and the alert burden remained high.⁷ Hospitals need to know which model version they have and how it performs with their patients. The name of the software supplier cannot answer either question.
The bundle keeps growing. When New Jersey’s Inspira Health launched Epic in July 2026, it said it was deploying “more than 50 digital tools from Epic’s AI suite and cognitive computing models.”⁸ A hospital’s decision to adopt an electronic health record can contain dozens of decisions about AI. Each capability needs an owner and a reason to trust it.
The consequences extend beyond prediction. In June 2025, Aim Labs disclosed EchoLeak, a flaw that could let a malicious email cause Microsoft 365 Copilot to leak information without the victim clicking an attacker-supplied link.⁹ Microsoft tracked it as CVE-2025-32711 and said the vulnerability “requires no customer action to resolve.”¹⁰ The vulnerability was in a service the customer used, not in software the customer had written. Asking what the organization built would miss it.
The fourth place is harder to see.
Brought AI
In a 2024 survey of 31,000 knowledge workers across 31 markets, Microsoft and LinkedIn found that 78% of AI users brought their own AI tools to work.¹¹ Some of that use may be known and approved. But a personal account does not automatically produce a purchase order, a deployment ticket, or a review of what data the tool can receive.
An employee who pastes a client record into a consumer chatbot may expose the organization’s data without creating any of the records an inventory normally draws on.
Not everything brought in is a tool. An employee who asks a consumer chatbot a question at home and gets a confident wrong answer brings the answer to work, not the chatbot. No data left the organization and nothing was installed. There is no trace. The AI stays home; its output shows up in a decision, a draft, or a meeting, with no sign of where it came from.
There is no good answer for this one. A ban can push AI use further out of sight; approved tools can reduce it without ending it. An honest inventory records what it has found and treats the unseen portion as a known unknown. Policies may shrink that uncertainty. They cannot turn unreported use into verified absence.
For AI delivered by a supplier or contractor, the problem is different. Someone outside the organization knows what arrived, and when.
We have made AI easier to acquire than to account for. A decision to keep using familiar software can now be a decision to accept unfamiliar capabilities. The people affected may be patients, borrowers, employees, or customers who had no part in either decision.
An organization cannot outsource responsibility to its supplier or its contractor. But it should not have to discover a new AI capability after that capability has begun shaping its work. A supplier introducing an AI feature and a contractor choosing to use a model know something their customer needs to know. That knowledge should travel with the update or the work.
Disclosure should be the default. That does not mean a notice for every weekly release. Call a change material when it introduces or alters AI in ways that affect decisions about people, access to sensitive data, or the authority to act on a customer’s behalf. Activating such a capability by default should also trigger notice.
Before deployment, vendors should explain what changed, what will be enabled, what data the AI can access, and what decisions or actions it can influence. Contractors should disclose, before work begins, AI used to assess people, recommend decisions, or process their data, and report material changes during the contract. A customer buying a service deserves to know how the service is performed, not just what software is delivered.
Customers should have a meaningful opportunity to approve or refuse material changes, not merely an explanation of how to switch them off afterward. Responsibility should not disappear because the AI arrived inside a familiar product or behind a contractor’s invoice.
Federal policy already recognizes the problem, up to a point. In its April 2025 AI acquisition memo, M-25-22, OMB tells agencies to consider requiring notification before vendors integrate new AI features and to determine when contractors should disclose AI used in performing a contract.¹² The memo’s scope leaves out certain common commercial products that carry embedded AI, judged partly by whether the product does substantial work that is not AI. It also leaves out AI that a contractor happens to use while doing the work.¹²
OMB should make advance notification mandatory for material AI changes, rather than something agencies consider requiring. It should also bring embedded commercial AI within that disclosure requirement when its capabilities meet the same materiality threshold. Public agencies should make these disclosures a condition of buying software and services. Being part of a familiar product should not be an exemption.
But what about the enterprise? In the US, there is little federal regulation on AI, and a company cannot wait for a rule to require this disclosure. It has to ask, and the time to ask is at purchase and renewal, when the supplier most wants the business. What AI is in the product? How will the company hear about the next one?
ISO/IEC 42001, the international standard for AI management systems, already points here. It applies to any organization that adopts it, whatever its role. An organization buying or using AI must control the products and services it gets from outside (clause 8.1) and reassess AI risk when significant changes occur (clause 8.2), and control A.10.3 calls for a process that keeps suppliers aligned with its approach to responsible AI.¹³ A supplier switching on a new AI feature should count as a significant change. A provider with its own 42001 system has the other half: giving users the information they need about its AI (control A.8.2) and taking customer expectations into account (control A.10.4).
SBA looked for AI in one office and missed what was running outside it. An inventory built the same way will miss the same things. The fix is not a longer list of model names. It is a record of who changed what, when it changed, and who accepted responsibility. It covers what the organization built and what it bought, what arrived inside software it already had, and what it knows it cannot see.
Sources and method
- SBA Office of Inspector General: SBA’s Progress on Implementing Federal Artificial Intelligence Requirements, Report 26-15, September 24, 2026. Review findings cover information through April 30, 2026; the inventory count and omission are stated as of March 2026.
- Government Accountability Office: Artificial Intelligence: Uses and Risks for Small Business Contracting and Innovation Research, GAO-26-107828, May 4, 2026.
- Government Accountability Office: COVID-19 Relief: Improved Controls Needed for Referring Likely Fraud in SBA’s Pandemic Loan Programs, GAO-25-107267, March 24, 2025.
- Office of Management and Budget: 2025 Federal Agency AI Use Case Inventory, including the consolidated commercial off-the-shelf collection.
- Author’s calculation: Individual-use-case CSV at commit 06c7ebe. Filter development_stage to Deployed: 1,040 rows. Of these, has_custom_code is No for 542, Yes for 466, and blank for 32. Blanks remain in the denominator; 542/1,040 is 52.1%. The question asks whether the project includes custom-developed code; it does not establish whether the system was bought, built in-house, or adopted unknowingly.
- Wong A, Otles E, Donnelly JP, et al.: External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients, JAMA Internal Medicine, June 21, 2021. DOI: 10.1001/jamainternmed.2021.2626.
- Wong A, Currey D, Schwinne M, et al.: Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model, JAMA Network Open, February 27, 2026. DOI: 10.1001/jamanetworkopen.2026.0181.
- Inspira Health: Inspira Health Launches Epic, Advancing a More Connected Care Experience for South Jersey Patients, July 27, 2026.
- Aim Labs: EchoLeak technical disclosure (archived copy). Used for the attack mechanism and proof of concept.
- Microsoft Security Response Center: CVE-2025-32711: M365 Copilot Information Disclosure Vulnerability, released June 11, 2025. Microsoft states that the vulnerability requires no customer action to resolve.
- Microsoft and LinkedIn: 2024 Work Trend Index: AI at Work Is Here. Now Comes the Hard Part, May 8, 2024. Survey conducted by Edelman Data & Intelligence among 31,000 knowledge workers across 31 markets, February 15 to March 28, 2024. The 78% figure concerns AI users, not the entire workforce.
- Office of Management and Budget: M-25-22: Driving Efficient Acquisition of Artificial Intelligence in Government, April 3, 2025.
- ISO and IEC: ISO/IEC 42001:2023 Information technology: Artificial intelligence: Management system, December 2023. Clauses 8.1 and 8.2 and control A.10.3 are paraphrased, not quoted.