A Data Factory pipeline reports success at 02:00, but a report still shows yesterday’s numbers. That is a common operations incident: the orchestrator may have completed while data quality, source freshness or a downstream refresh failed. DP-700’s monitoring and optimization objectives include pipeline, Dataflow Gen2, notebook, Eventstream, Eventhouse, T-SQL and OneLake shortcut troubleshooting. A good runbook distinguishes these stages instead of declaring the whole solution healthy because one activity turned green.<\/p>\n
Execution success<\/strong> means the relevant Fabric activity completed according to its runtime rules. Data success<\/strong> means the expected records and business invariants are present at the destination. Consumer success<\/strong> means downstream queries or semantic-model refreshes can read the accepted version. These statuses can disagree. A notebook can exit without throwing an exception while dropping rejected records; a dataflow can finish while loading a stale source snapshot; and a report refresh can fail long after the lakehouse has been updated.<\/p>\n For a daily sales dataset, track a source watermark, the latest accepted target batch ID and the last consumer refresh. Their timestamps and run IDs let operations locate the first broken boundary. Without this separation, dashboards may show a plausible but incorrect number.<\/p>\n Every production run should expose its orchestration ID, scheduled-versus-actual start time, source version, copied row count, transformed row count, rejected\/quarantined rows, target committed version, retries and completion reason. A failure alert should include a direct link to the affected Fabric item and run, not only \u201csomething failed.\u201d Avoid sending secrets or raw personal data into alert payloads.<\/p>\n Record dependencies explicitly. If a source lookup fails, downstream transformation should not start with yesterday’s watermark as though nothing happened. If transformation succeeds but quality validation fails, the publish or semantic refresh step should remain blocked. If the notification activity itself fails, the runbook needs a secondary monitoring signal or ownership process.<\/p>\n The reliable Fabric pipeline article<\/a> describes retry boundaries. Here the concern is turning those boundaries into useful evidence when an operator takes over.<\/p>\n For a stale dashboard, inspect the source export or event producer first. Next examine the Fabric ingestion job history and its last accepted watermark. Then inspect data transformations, quarantine tables and Delta or warehouse commit evidence. Finally check the semantic-model refresh and report query. Reversing this order can waste hours tuning a report whose source never arrived.<\/p>\n For a failed notebook, determine whether Spark failed to start, a library version changed, authentication expired, an input schema shifted, or a transformation threw an exception. A notebook that prints \u201cERROR\u201d but exits successfully can mask failures from orchestration. Ensure genuine terminal failures propagate a failed status while recoverable bad rows are quarantined under an agreed contract.<\/p>\n A Dataflow Gen2 can fail because a connector credential expired, source schema changed, a Power Query transformation has incompatible data types, or the destination write cannot complete. Inspect the refresh\/run details and the particular transformation step. Separate a failure to fetch source data from a failure to map columns and from a failure writing the result to a destination. When a source becomes much larger, consider whether query folding or transformation design changed and caused resource pressure.<\/p>\n For example, changing For a OneLake shortcut that suddenly fails, verify the source dataset still exists, the cloud connection credentials remain valid, item-level permissions allow access, and the referenced path\/schema has not changed. A shortcut failure is not necessarily a lakehouse compute problem. For a warehouse T-SQL error, inspect query text, object permissions, locks\/concurrency where relevant, transaction\/resource state and relevant engine diagnostics.<\/p>\n For an Eventstream that stops delivering, compare source events sent, events accepted, transformation rejects and destination receipts. An empty KQL query may be caused by stale Configure separate thresholds for failures, freshness and capacity. A daily warehouse build might have a 06:30 completion objective, a critical fact might require zero duplicate primary business keys, and an Eventstream may need near-real-time ingestion lag below an agreed duration. Thresholds should come from business tolerance, not copied default numbers. If the alert trips too often on normal variation, tune it against historical distributions and preserve its purpose.<\/p>\n Capacity contention is a cross-item factor. A long-running Spark job or concurrent warehouse query workload can slow otherwise healthy operations. Compare observed runtimes with workload concurrency and Fabric capacity metrics rather than blaming an individual notebook by default. Capacity scaling may help an appropriate bottleneck but does not correct a malformed schema or an unbounded many-to-many join.<\/p>\n Use a small operational test day. At 01:00, write a good CSV input and prove the target fact count. At 02:00, introduce a duplicate order and a changed decimal format. At 03:00, break a test source credential. The expected outputs are not three green checkmarks; they are differentiated error paths, useful alerts, preserved source evidence and an explicit recovery decision for each incident.<\/p>\n Write a runbook table with symptom, first evidence location, expected root causes, safe repair and replay validation. Include the exact timestamp and source boundary that operators would use after a failure. When the system recovers, compare both the business totals and item execution health before lifting the stale-data alert.<\/p>\n The DP-700 exam guide<\/a> and Microsoft’s current study guide<\/a> make clear why monitoring is one-third of the assessment. Treat troubleshooting as a design discipline from the first lab, not as a final checklist after pipelines have been built.<\/p>\n","protected":false},"excerpt":{"rendered":" A Data Factory pipeline reports success at 02:00, but a report still shows yesterday’s numbers. That is a common operations incident: the orchestrator may have completed.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12,8],"tags":[],"class_list":["post-4205","post","type-post","status-publish","format-standard","hentry","category-ai-data","category-certifications"],"_links":{"self":[{"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/posts\/4205","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/comments?post=4205"}],"version-history":[{"count":1,"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/posts\/4205\/revisions"}],"predecessor-version":[{"id":4217,"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/posts\/4205\/revisions\/4217"}],"wp:attachment":[{"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/media?parent=4205"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/categories?post=4205"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.examtopics.info\/blog\/wp-json\/wp\/v2\/tags?post=4205"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Build an observable pipeline contract<\/h2>\n
Troubleshoot from the producer toward the consumer<\/h2>\n
Dataflow Gen2 has distinct refresh and transformation errors<\/h2>\n
net_amount<\/code> from decimal to localized text can break a downstream type conversion. A robust fix documents the expected numeric format, quarantines invalid records and tests the corrected transform on both valid and malformed samples. Silently coercing every invalid value to zero changes the business result and is not a successful repair.<\/p>\nShortcuts, warehouses and Eventhouses need different diagnostics<\/h2>\n
event_time<\/code> values and an ago(5m)<\/code> filter, not by an empty Eventhouse table. Inspect ingestion timestamp separately. An Eventstream\/Eventhouse walkthrough<\/a> demonstrates this clock distinction.<\/p>\nAlerts should correspond to service objectives<\/h2>\n
Controlled incident-response exercise<\/h2>\n