Power BI Copilot can give a wrong answer when the semantic model leaves an important choice unresolved: which measure represents the business term, which date controls the period, which grain is safe to aggregate, or which filter a user intended. A dashboard can still look correct because its author already made those choices in a visual. Copilot must infer them from the question and the model context. Start by tracing the answer through terminology, schema, calculation, security, and testing—not by endlessly rewriting the prompt.
Microsoft states that an unprepared model can produce less useful or inaccurate results, and that Copilot output is nondeterministic even when the prompt and grounding data are the same. Microsoft’s current preparation features can reduce ambiguity, but they cannot ensure one specific output every time. Prepare your data for AI
The failure path
A natural-language answer passes through more decisions than a fixed report visual:
- Interpret the user’s words.
- Select model objects that appear to match.
- Generate or select a semantic query.
- Apply relationships, filters, time logic, and the user’s permissions.
- Choose a visual and summarize its result in language.
Microsoft documents that Copilot grounds on semantic-model information such as tables, columns, measures, relationships, calculation groups, descriptions, data types, and format strings. For data questions it can return a generated visual plus a natural-language summary of the query result. Use Copilot with semantic models
That makes the model—not the prompt—the first place to investigate when an answer is plausible but wrong.
Diagnostic matrix: symptom to likely control
| Symptom | Likely failure mechanism | Evidence to inspect | First control to test |
|---|---|---|---|
| “Sales” returns a valid but unexpected number | Competing measures or columns can all plausibly mean sales | Measure list, descriptions, report definitions, owner-approved metric glossary | Expose one approved measure in the AI data schema; hide or rename unsafe alternatives |
| The value is right for a different period | Order date, invoice date, fiscal calendar, or rolling-period logic is ambiguous | Date relationships, calculation groups, fiscal calendar specification, expected-answer query | Make the calendar contract explicit in measures and terminology; add a focused instruction only after the model is sound |
| Totals are inflated | Facts at different grains are being combined or summarized through an unsafe relationship | Table grain, relationship cardinality and direction, DAX query, duplicate-key tests | Correct the model or measure; do not try to prompt around a grain defect |
| Answers vary when the wording changes slightly | Trigger phrases, terminology, or schema context do not cover normal user language | Prompt variants, field-selection traces, verified-answer triggers | Add synonyms/descriptions, then test an AI instruction or verified answer for the specific class of question |
| Copilot chooses the wrong field | Too many overlapping fields are available | AI data schema, hidden fields, duplicate business terms | Narrow the AI data schema to the fields that serve the target users |
| The visual is correct but the prose overstates it | The generated summary goes beyond the returned data | Query result, visual, generated narrative | Define an answer contract that separates returned facts from interpretation |
| One user receives an empty or different result | Permission context or row-level security changes the accessible data | Role mapping, “view as” tests, user identity, source permissions | Test each target role; do not validate only as an administrator |
| Copilot times out or returns a generic error | The model, DAX, or query is too complex or insufficiently optimized | Performance Analyzer, DAX query, model size and relationships | Simplify or optimize the model/query before adding more instructions |
The matrix is a Refinity diagnostic method. The underlying product behaviors are described in Microsoft’s semantic-model Copilot guidance and AI data schema documentation.
Worked example: “What was revenue last quarter by region?”
Suppose the executive dashboard shows Net Revenue by fiscal quarter. The model also exposes:
- a raw
SalesAmountcolumn that can be summed implicitly; - a
Gross Revenuemeasure used by finance; - order date and invoice date relationships;
- calendar and fiscal-quarter labels;
- a region on both customer and sales-territory tables.
The dashboard author chose Net Revenue, fiscal quarter, invoice date, and customer region when building the visual. The user’s sentence contains none of those decisions. Several generated queries could be technically valid while failing the business intent.
The durable repair is an answer contract:
| Contract field | Approved interpretation |
|---|---|
| Business term | Revenue means [Net Revenue] for this user group |
| Source | Certified priority semantic model |
| Grain | One result per fiscal quarter and customer region |
| Calendar | Corporate fiscal calendar |
| Date role | Invoice date |
| Filters | Exclude internal transfers; respect the viewer’s security role |
| Expected evidence | Total reconciles to the approved finance report for the same period |
Then make the model and AI configuration express that contract: expose the approved measure and dimension, remove confusing fields from the AI schema, create a verified answer if a maintained report visual already answers the high-value question, and use a concise instruction for organization-specific language that the model cannot otherwise express.
A five-step investigation that avoids prompt whack-a-mole
1. Freeze the expected answer
Write the expected measure, period, dimensions, filters, security role, and trusted comparison before testing Copilot. “The dashboard says 12.4” is not enough unless you can explain how that dashboard produced 12.4.
2. Reproduce with representative wording
Test the business user’s wording, not only a model author’s ideal prompt. Include a direct question, a common synonym, a follow-up, and an intentionally ambiguous version. Repeat important tests because Microsoft identifies Copilot output as nondeterministic.
3. Localize the first wrong decision
Determine whether Copilot misunderstood the term, selected the wrong field, generated the wrong calculation, applied the wrong time/filter context, or summarized a correct result incorrectly. Fix the earliest failing layer.
4. Change one control
Keep the expected answer and prompt set fixed. Change one of the model, AI data schema, verified answer, or instructions. Microsoft recommends implementing the preparation controls in sequence: AI data schema, verified answers, AI instructions, then descriptions. Prep data for AI FAQ
5. Turn the incident into a regression test
Add the question, variants, expected answer contract, security roles, and tolerance rule to a versioned evaluation set. Microsoft’s content lifecycle guidance recommends testing known benchmarks, comparing Copilot output with trusted reports or analyses, and comparing results before and after model changes. Validate Power BI content
What not to fix with AI instructions
AI instructions are useful for business context and interpretation rules, but Microsoft says they are stored at the semantic-model level, are not persona-specific, and are not guaranteed to be respected in every Copilot experience. AI instructions
Do not use instructions to conceal:
- an incorrect measure;
- a many-to-many or mixed-grain double count;
- an untested row-level security design;
- an unclear ownership decision;
- two teams using incompatible definitions under the same name.
Those are semantic-layer and governance problems. Fix them where they originate.
Microsoft product facts vs. Refinity recommendations
Microsoft product facts
- Copilot uses semantic-model metadata and query results as grounding context.
- Current Prep data for AI controls include AI data schemas, verified answers, and AI instructions.
- The controls are configured on the semantic model, not separately for each report.
- Copilot output remains nondeterministic.
- AI data schemas apply to certain schema-based data questions, not every Copilot capability.
Refinity recommendations
- Define an answer contract before changing prompts or AI configuration.
- Treat wrong answers as traceable failures across terminology, model, query, security, and presentation layers.
- Test important questions across prompt variants and permission roles.
- Require evidence for the “correct” comparison result.
- Promote production use only when failures become versioned regression tests.
When the problem needs a readiness review
A technical readiness review is warranted when the same term can resolve to several measures, leaders disagree about the benchmark, important answers depend on undocumented calendar logic, security has not been tested by role, or the team cannot show a repeatable evaluation set. Those conditions are broader than prompt design.
Primary CTA: Score your model across seven Power BI Copilot readiness dimensions.
Related reading: