Make Search Engines and LLMs Accountable for What They Hide

Exclusion map, regulatory comparison, auditing methodology, charter and publisher diagnostic specification · CognitiveLiberties.com · 11 October 2026

Executive finding

A missing result has many possible causes. Accountability requires forensic humility: distinguish a URL that was never crawled from one indexed but ranked poorly, a policy removal from a canonicalization error, and a model that omitted a source from one that lacked access to it. The most achievable first step is a reproducible, publisher-facing diagnostic process, paired with transparent appeal channels and independent cross-query measurements. A universal obligation to explain every commercial ranking decision does not currently exist; rights and responsibilities differ among jurisdictions and service classes.

1. The information-discovery chokepoint map

Publisher publishes → robots/access → crawler discovers → canonical/duplicate selection → indexing → retrieval/query expansion → relevance and policy ranking → displayed snippets and links → AI retrieval context → generation/summarization → citation selection → user's next click

At each arrow a viewpoint may become less visible without its text being erased from the open web. This is a diagnostic model, not a claim that filtering is always ideological. A site may be accessible directly yet absent from results because it is new, duplicated, low-quality under a service's criteria, incorrectly canonicalized, blocked by technical settings, or simply irrelevant to a query. Google Search Console URL Inspection and Bing Webmaster URL Inspection expose important parts of the crawl/index pipeline but do not supply a full internal explanation for every ranking [1, 2].

2. What law and existing tools provide

Under the EU Digital Services Act (DSA), obligations are service-specific. Article 17 imposes statement-of-reasons requirements for qualifying hosting restrictions; Article 20 concerns internal complaint handling by online platforms, subject to scope and exemptions. The same duties must not be generalized automatically to every general-purpose search result or language-model response. Very large online platforms and search engines have added systemic-risk and auditing duties, and Article 40 provides a path for vetted researchers to request qualifying data; Delegated Regulation (EU) 2025/2050 details procedures [3, 4, 5]. Competition law, privacy law and national speech doctrine add context, but do not establish a universal 'right to top-ten rank.'

Google's URL Inspection provides a view of indexed status and indexability, with limitations; Bing's inspection provides crawl/index/markup diagnostics, remedial guidance and webmaster support [1, 2]. These tools help publishers debug infrastructure but offer limited visibility into comparative ranking or generative citation decisions. NIST's voluntary AI Risk Management Framework gives a governance vocabulary for mapping, measuring, managing and governing relevant risks [6].

3. Search and AI Accountability Charter (proposal)

Disclosure: publish comprehensible categories of crawling, index, content-policy and answer-citation exclusions, without exposing security-sensitive anti-abuse detail. Traceability: make identifiable supporting sources accessible and enable claims to be matched to citations. User choice: provide at least one exploratory option for relevant credible alternatives. Diagnostic access: allow site owners to determine technical eligibility and correct errors. Public reporting: disclose benchmark definitions, error rates, corrections and significant policy changes. Review: a narrowly specified remedy for significant mistakes, not a guarantee of preferred placement. Independent scrutiny: researchers should have proportionate access to test mechanisms with data-protection controls. Editorial freedom: publishers and platforms retain justified content selection subject to applicable law. Abuse resistance: anti-spam and anti-fraud safeguards continue to operate. Accuracy: accountability claims must be supported by replicable evidence and must distinguish policy findings from conjecture.

4. Independent auditing protocol

Design: Build a preregistered 240-query benchmark: 60 exact-title navigational, 60 topical scholarly, 60 competing-theory and 60 multilingual translations. For each define ten eligible source URLs from diverse publishers and an independent relevance label. Compare search engines and AI assistants as different product classes, not as though rank 1 and a generated summary were interchangeable. Record outputs at three time points, search locale, logged-in state, query formulation, settings, indexability and source freshness. Annotators should not know which experimental condition generated a result.

Metrics: index coverage; eligible-source recall@10; source diversity by verified publisher category; omission from generated answers; citation fidelity; update lag; technical-diagnostic resolution; appeal turnaround; error-correction persistence. Publish overall and segment-specific measures, confidence intervals and a list of cases with unknown causes. No data have been collected for this proposed audit.

Causal caution: a ranking disparity is an observation, not evidence of discriminatory motive. Stronger causal designs hold page quality, topic, relevance and technical status constant; compare matched publisher pages across queries and languages; and test whether the same effect persists through time. Even then, proprietary internals may prevent attribution to a specific ranking feature.

5. Publisher-facing discoverability diagnostic: product specification

Required inputs

Verified domain; affected canonical URL; target query and region; publication/update date; sitemap entry; current robots and meta directives; structured metadata; captured SERP or AI answer; target source text; and optional Search Console/Bing export. Do not collect reader search histories or unnecessary personal data.

Decision tree and output

1. Delivery: check DNS, TLS, HTTP status, redirects and content accessibility for a normal visitor and, where allowed, a verified crawler.
2. Eligibility: inspect robots, meta robots, canonical references, sitemaps and duplicate content indicators.
3. Index evidence: import first-party webmaster tool result; label unobserved states as 'unknown'.
4. Query relevance: map user intent to source page sections and independent comparison URLs.
5. AI trace: if the URL appears in provided retrieved-source logs, test whether an answer correctly represents and cites it; otherwise mark source availability unverified.
6. Case label: Technical fault / Documented platform notice / Search relevance dispute / Citation error / Insufficient evidence.
7. Action: publish fix instructions, responsible owner, URL of any available platform appeal and a timestamped retest plan.

JSON field sketch: case_id, observed_at, url, query, locale, url_status, robots, index_status, canonical, platform_notice, ai_retrieval_evidence, hypotheses[], known_limitations[], appeal_status, resolution_date. The diagnostic should never output 'censorship proven' based only on absence from a top-ten query.

6. Policy and voluntary implementation choices

Voluntary: publish standardized reason categories, webmaster diagnostic APIs, independent source diversity reporting, and notification/correction paths. Open standard: common schemas for observable exclusion events and publisher appeals, with typed uncertainty and jurisdiction flags. Public policy: evaluate access to independent audit data, carefully scoped transparency obligations and judicially reviewable decisions where state action or regulated platform duties apply. Design decisions should guard trade secrets, security, privacy, due process and legitimate editorial independence.

7. Pilot and success criteria

Pilot with 20 independent publishers across languages and domains. Seek a reduction in the fraction of unexplained technical non-indexing incidents, measured time-to-resolution, and improved factual citation correction. Do not use increased traffic as the sole goal: relevance and public learning matter. Publish cases where diagnostic findings disprove participants' initial assumptions. This pilot is proposed only; no independent ranking tests or authenticated webmaster accounts were accessed for this report.

Sources

1. Google Search Console, URL Inspection Tool. https://support.google.com/webmasters/answer/9012289?hl=en
2. Bing Webmaster Tools, URL Inspection. https://www.bing.com/webmasters/help/url-inspection-55a30305
3. Regulation (EU) 2022/2065, Digital Services Act, especially Articles 17, 20, 34, 37, 40. https://eur-lex.europa.eu/eli/reg/2022/2065
4. European Commission, How the DSA enhances transparency. https://digital-strategy.ec.europa.eu/en/policies/dsa-brings-transparency
5. Commission Delegated Regulation (EU) 2025/2050, researcher access procedure. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32025R2050
6. NIST, AI RMF and Generative AI Profile (2023/2024). https://www.nist.gov/itl/ai-risk-management-framework