← Tools I Use

EY Canada published a cybersecurity report and 16 of its 27 citations were fake

2026-06-02 · ~6 min read · Conor Dobbs

A Big Four firm put out a cybersecurity report this year and more than half the citations in it were not real. That is the short version. The longer version is more interesting, and it says something about where automated tooling is heading for everyone who works in security or builds tools around it.

What happened

EY Canada published a study called "Points of Attack: Uncovering Cyber Threats and Fraud in Loyalty Systems." It looked at fraud risk in airline miles, hotel points, and retail rewards programs. Reasonable topic, real problem. Then GPTZero's investigations team pulled the report apart and found that 16 of its 27 references were hallucinated. Their AI detector flagged about 72% of the text as machine-generated. EY Canada quietly took the report off its website and said it was reviewing why the study was published in the first place.

How the citations failed

When you read GPTZero's breakdown closely, the failure pattern is the part worth studying. The citations were not sloppy in a human way. They were sloppy in a specific machine way. A McKinsey report titled "Loyalty Economics Report (2022)" was cited as a source. It does not exist. The reference appears to have been laundered out of a blog post on a financial IT site, which is how a confident-sounding but fake number ends up wearing a McKinsey badge. A Cisco Talos blog post that was cited returns a 404. Multiple Wired articles, a Gartner document, a Forbes piece by a named author, all either dead links or pointing at nothing that matches the claim. The report even contradicted itself on a headline figure, calling the same "$200 billion" number both the total size of the loyalty market and the value of unredeemed points, which cannot both be true.

A GPTZero engineer called this "vibe citing." I like the phrase because it names a real thing. The model is not lying on purpose. It is producing text that has the shape of a citation. Author, title, year, plausible publisher, a URL that looks structurally correct. Everything is there except the part where the source is real. If you are skimming, it reads as rigorous. If you click, it falls apart.

Why nobody caught it

I build small security and developer tools for a living, so my reaction to this is not "AI bad." My reaction is that this is exactly the failure mode the whole industry has been warned about, showing up in a deliverable from a firm whose entire value proposition is rigor. The interesting question is not how EY let it happen. The interesting question is why it took an outside three-person team running a detector to catch it, when the cost of catching it internally was close to zero.

Because that is the thing. Verifying a citation is one of the cheapest checks in existence. You take every URL in the document and you hit it. Dead link, fail. You take every "Report Title, Publisher, Year" and you search for it. No exact match, flag for human review. That is a script. It is not even a hard script. Any junior could write the link-checker in an afternoon, and the title-matcher is a search query in a loop. The report had 27 references. A machine could have validated all of them in under a minute and handed back a list that said "these 16 do not resolve." Nobody ran it.

The lesson for anyone shipping automated work

This is the part I want people in security to sit with, because we are about to make the same mistake at scale. The reflex right now is to point AI at the boring, high-volume, detail-heavy work. Pull these logs and summarize the anomalies. Read this vendor's SOC 2 and tell me the gaps. Draft the threat model. Write the incident report. All of that is genuinely useful, and I am not telling anyone to stop. But every one of those outputs has the same property the EY report had. It looks authoritative, it is full of specific claims, and the specifics are exactly where a language model is least trustworthy. A summary that invents a finding is worse than no summary, because it carries the credibility of the tool that produced it straight into a decision.

The lesson is not "do not use the tool." The lesson is that the output of an automated process is an input that still needs checking, and the check is usually cheap relative to the harm of skipping it. The EY report is now sitting in news archives, blog posts, and AI search overviews. The next model that scrapes the web to answer a question about loyalty fraud may well cite the fabricated McKinsey number, because it is published, it has EY's name on it, and it reads clean. One unverified deliverable poisons the well for everyone downstream. That is the compounding cost, and it is why "we will catch it later" does not work.

I think about this constantly when I build, because my own tools have the same shape. A scanner that grades a website's security, a script that flags a misconfiguration, a model that drafts a fix. Every one of them can be confidently wrong, and the most dangerous version of wrong is the one that looks right. So the design rule I keep coming back to is boring on purpose: make the cheap verification step mandatory, not optional. If a tool produces a citation, check the link before showing it. If it claims a header is missing, fetch the header and confirm. If it grades something an F, let the user see the raw evidence that earned the F. The point is never to trust the confident output. The point is to make the output checkable, and then check it.

The honest take

EY will be fine. They pulled the report, they will issue a process update, and in a year this will be a footnote. But the cheaper, more useful takeaway is for the rest of us. The tools are good enough now that the bottleneck is no longer generating the work. The bottleneck is verifying it, and verification is the step everyone is tempted to skip because the output already looks done. It looks done is precisely the trap. A fake citation looks exactly like a real one until you click it. So click it. Build the click into your process so a human does not have to remember to.

If you want a concrete version of this discipline, that "verify the automated output, do not trust it" idea is the spine of how I build the small security tools and cookbooks I publish. You can find them at tools.thesoundmethod.me. None of it is magic. It is mostly the unglamorous habit of checking the thing the machine handed you before you put your name on it, which, as EY demonstrated in public this year, is a habit worth more than it appears.

Sources

Related reading