Microsoft Fabric Data Factory offers several ways to move and transform data, but two components are frequently confused: pipelines and Dataflow Gen2. They work together, yet they solve different problems. Pipelines coordinate activities over time. Dataflow Gen2 uses the Power Query experience to connect, clean, combine, reshape, and load data. Treating them as interchangeable usually produces workflows that are harder to operate.<\/p>\n
The current DP-600<\/a> scope expects analytics engineers to prepare data, manage analytical assets, and understand dependencies across Fabric items, while DP-700<\/a> focuses more directly on engineering ingestion, transformation, orchestration, monitoring, and optimization. Together they reflect the real Fabric workflow: data movement, transformation, and orchestration should be designed as connected responsibilities.<\/p>\n The simplest mental model is that a dataflow answers \u201cHow should this dataset be transformed?\u201d while a pipeline answers \u201cWhen, in what order, and under which conditions should the work run?\u201d Complex solutions often need both.<\/p>\n There is also a team-design benefit to this separation. A data steward or analyst can own a Power Query transformation while a platform or engineering team owns the orchestration around it. Clear ownership reduces the temptation to duplicate a transformation merely because another team needs to schedule it differently.<\/p>\n Before building either artifact, write the dataset contract: source, expected schema, output grain, refresh objective, quality rules, destination, and consumers. Tool choice becomes easier once the data product is defined.<\/p>\n A Fabric pipeline is a logical collection of activities. Those activities can copy data, execute a Dataflow Gen2, run a notebook, call a stored procedure, execute SQL, invoke another pipeline, wait, branch, loop, or perform other control-flow work. Dependencies determine what runs after success, failure, or completion.<\/p>\n This makes pipelines appropriate for workflows that cross multiple technologies or stages. A nightly process might copy raw data to a lakehouse, run a validation notebook, execute a dataflow for business transformations, publish a curated table, and send a notification. The pipeline owns the sequence and recovery behavior rather than embedding all of that logic inside one transformation artifact.<\/p>\n Parameters also make pipelines reusable. A single design can process different dates, business units, files, or destinations without duplicating the whole canvas. Treat parameters as part of the execution contract and validate them before high-impact activities begin.<\/p>\n Control-flow activities should remain understandable. Deeply nested conditions and loops can turn a visual pipeline into code that is harder to review than code. Break large processes into child pipelines or reusable components when that creates a clearer ownership and failure boundary.<\/p>\n Event-driven starts can be useful when data arrives unpredictably, while schedules fit predictable batch windows. The trigger model should match how the source changes. Polling a source frequently with no new data wastes capacity and can complicate overlapping-run control.<\/p>\n Dataflow Gen2 provides a cloud-based Power Query authoring experience with a large set of connectors and transformations. It is well suited to filtering, type conversion, joins, column derivation, aggregation, deduplication, and other preparation tasks that can be expressed clearly in the query pipeline.<\/p>\n The interface is accessible to analysts as well as data engineers, which can be an advantage when domain specialists understand the transformations better than a centralized engineering team. The goal should still be production discipline: meaningful query names, clear steps, controlled connections, and an output contract that downstream users can depend on.<\/p>\n Dataflow Gen2 can write to supported destinations such as Fabric lakehouses, warehouses, and other services. Destination behavior matters. Some managed settings can replace target data during refresh and can recreate tables when schemas change, so architects should understand how a dataflow will affect downstream relationships, measures, and consumers before enabling automatic behavior.<\/p>\n Query folding deserves attention for large sources. When transformations can be pushed to the source, less data may need to move into the dataflow engine. A seemingly simple step can break folding and dramatically increase refresh cost. Inspect execution behavior for large or slow flows rather than assuming visual simplicity equals efficient processing.<\/p>\n Staging can improve transformation performance but creates additional managed artifacts and capacity usage. Understand when staging is used, how long intermediate data persists, and what permissions are required so operational teams are not surprised by storage or refresh behavior.<\/p>\n A pipeline can run a Dataflow Gen2 as an activity. This is usually cleaner than making every dataflow run independently on its own schedule when it participates in a larger process. The pipeline can ensure that upstream ingestion succeeded, pass execution context, handle failures, and trigger downstream steps only when the dataflow completes as expected.<\/p>\n Keep responsibilities separate. The dataflow should own transformation logic; the pipeline should own sequencing, dependency, scheduling, retry policy, and process-level status. If a dataflow contains hidden assumptions about whether another task ran first, or a pipeline duplicates transformation logic across many expressions, the boundary has become blurred.<\/p>\n The distinction resembles automation versus orchestration<\/a> in infrastructure work. A transformation can be automated on its own, but an enterprise process needs coordination across multiple automated steps.<\/p>\n Keep the boundary explicit. A pipeline should know that a transformation succeeded, failed, timed out, or produced a governed output; it should not need to understand every Power Query step inside the dataflow. The dataflow should transform a defined input into a defined destination and expose enough outcome metadata for orchestration. This separation creates clearer failure domains: a connectivity or transformation defect can be diagnosed inside the dataflow, while dependency sequencing and cross-system recovery remain pipeline concerns.<\/p>\n For larger solutions, publish data contracts between stages. Record expected schema, key fields, partition or watermark semantics, quality checks, and destination ownership. A pipeline run can then validate that the product delivered by one activity satisfies what the next activity expects instead of treating \u201cactivity succeeded\u201d as proof that the data is usable.<\/p>\n Not every source should enter Fabric through a dataflow. Copy activities and Copy jobs are designed for data movement, including large-volume transfers and common batch or incremental patterns. Mirroring can keep supported operational sources synchronized into OneLake with a different operational model. Eventstreams serve low-latency event scenarios.<\/p>\n If the task is primarily to move a large table unchanged into a bronze layer, a copy mechanism is often clearer than forcing the data through Power Query transformations. Dataflow Gen2 becomes valuable when the data needs shaping, cleansing, combining, or business logic before landing in its intended form.<\/p>\n Architecture should therefore distinguish ingestion from transformation even when one tool can technically do both. The broader data storage and processing<\/a> perspective helps teams select tools by data shape, volume, latency, and downstream use instead of by familiarity with a single interface.<\/p>\n A Dataflow Gen2 refresh executes its query transformations and writes the results to configured destinations. It can be run on demand, scheduled, or invoked from a pipeline. For simple independent datasets, a dataflow schedule may be sufficient. For a multi-stage business process, pipeline orchestration usually gives better control over upstream and downstream dependencies.<\/p>\n Incremental processing requires more than running more frequently. Decide how changed records are detected, whether late-arriving data is replayed, how deletions are represented, and whether destination writes are idempotent. A pipeline can maintain watermarks or pass processing windows, while the transformation layer applies the corresponding logic.<\/p>\n Avoid overlapping runs unless the design supports concurrency. Two executions updating the same destination can create inconsistent state or unnecessary capacity pressure. Use schedule, dependency, or control mechanisms to make the operational contract explicit.<\/p>\n Incremental processing needs a durable definition of progress. Timestamp watermarks are common, but they must account for late-arriving and corrected records. Where the source provides change tracking or sequence identifiers, those can be safer than relying only on ingestion time. Persist the last successfully committed boundary only after downstream writes and quality checks complete; advancing it too early can permanently skip data after a partial failure.<\/p>\n Backfills should be a first-class operating path rather than an improvised emergency. Parameterize date or partition ranges, isolate reprocessing from the normal schedule when necessary, and make destination writes idempotent. The goal is to allow operators to repair a missing window without copying a pipeline, editing production logic by hand, or duplicating records.<\/p>\n Transient network failures may justify retrying an activity. A deterministic schema error usually does not. Configure retries where they can recover a temporary condition and route unrecoverable failures into a clear operational path. A pipeline provides process-level control for retry, branching, and notification around the underlying transformation.<\/p>\n Inside a dataflow, distinguish rejected data from a failed process. A few malformed records may be quarantined while valid records continue, if that matches the business contract. A missing required source column might require the entire refresh to fail. Make those decisions explicit instead of allowing arbitrary connector behavior to define success.<\/p>\n Retries should also be safe. If a prior attempt partially wrote to a destination, rerunning the activity should not create duplicate business records. Stable keys, replace semantics, merges, staging, or transaction-aware patterns can make recovery predictable.<\/p>\n Fabric provides pipeline run history and monitoring as well as Dataflow Gen2 refresh history. Use them to answer different questions. Pipeline monitoring should reveal which activity failed, dependency paths, duration, and retry behavior. Dataflow monitoring should reveal refresh status, duration, and transformation execution.<\/p>\n Operational success should also include data-level checks. A pipeline can be green while a source delivered zero rows or a transformation filtered nearly everything. Capture row counts, rejected-record counts, watermark ranges, destination freshness, and business validation results so operators can identify silent data failures.<\/p>\n Define observability identifiers consistently across pipeline and dataflow runs. A business batch ID carried from ingestion through transformation to destination can make reconciliation far easier than relying only on platform run IDs. Persist it with output data where it adds audit value.<\/p>\n Alert on missed freshness objectives, not every minor warning. A dataflow that finishes five minutes slower may not matter, while a gold table that misses the morning reporting window does. Service objectives help monitoring distinguish symptoms from business-impacting failures.<\/p>\n Trend duration and capacity consumption. Gradual increases can reveal larger inputs, query-folding changes, staging behavior, source degradation, or inefficient transformations before users notice stale data.<\/p>\n Current Dataflow Gen2 items are created with CI\/CD and Git integration support, which makes their lifecycle increasingly similar to other development artifacts. Pipelines also participate in Fabric lifecycle-management patterns. Use source control, peer review, development workspaces, and deployment processes so changes are intentional and traceable.<\/p>\n Review environment-specific configuration carefully. Connections, workspace identifiers, destinations, schedules, and credentials should not be hard-coded in a way that makes promotion unsafe. Keep secrets outside source-controlled definitions and validate target resources during deployment.<\/p>\n The practices behind Git-based workflow<\/a> and AZ-400<\/a> become relevant when Fabric integration assets move from individual authoring into team-managed production. The tooling may be low-code, but the need for review, reproducibility, and rollback is the same.<\/p>\n Deployment design must include environment bindings. Repository changes can version pipeline definitions and dataflow assets, but connections, gateways, credentials, workspace identifiers, and destination endpoints differ across development, test, and production. Keep those values external to the logical definition where possible and verify them during promotion. A successful Git merge is not proof that the deployed workflow points to the correct production systems.<\/p>\n Release validation should execute a controlled end-to-end slice. Confirm that triggers are disabled or intentionally configured, connections resolve, data lands in the expected destination, quality checks run, and monitoring receives the correct metadata. This catches environment problems that static definition comparison cannot detect.<\/p>\n Use pipelines when the problem is orchestration: dependencies, schedules, control flow, multi-step execution, and recovery. Use Dataflow Gen2 when Power Query is a good expression of the data transformation and the team benefits from a managed low-code experience. Use notebooks, SQL, dbt, or other Fabric engines when their programming and performance model fits the transformation better.<\/p>\n A mature solution can combine all of them without becoming chaotic if the boundaries remain clear. Raw ingestion lands data, transformation components produce defined datasets, pipelines coordinate the sequence, monitoring validates both execution and data outcomes, and lifecycle tooling controls promotion.<\/p>\nUse pipelines to coordinate an end-to-end process<\/h3>\n
Use Dataflow Gen2 for low-code data preparation<\/h3>\n
Combine the two when orchestration and transformation are both needed<\/h3>\n
Choose the right ingestion mechanism before transforming<\/h3>\n
Design refresh and incremental behavior deliberately<\/h3>\n
Engineer failure and retry behavior at the correct layer<\/h3>\n
Monitor both process execution and data outcomes<\/h3>\n
Use CI\/CD and version control as the solution grows<\/h3>\n
Design a Fabric workflow around clear component ownership<\/h3>\n