An insurer receives overnight files from a partner, captures new claim events continuously, and must deliver audited financial reports each morning. The engineering team suggests Azure Data Factory for everything because it can move and transform data. Analytics wants Microsoft Fabric, while a legacy group prefers existing Synapse pipelines. The architect’s job is not to pick a product with the longest feature list. It is to determine whether the work is batch integration, streaming ingestion, relational transformation, warehouse analysis, or data governance, then assign each responsibility to a maintainable service.
Microsoft AZ-305 includes data integration and analysis as part of storage solution design. The existing high-scale Azure storage discussion identifies where transactional, object, and analytical data may live. This article follows the data between those stores: orchestration, transformation, schema contracts, failure handling, security and costs. Azure Data Factory, Synapse Analytics and Fabric overlap in some capabilities but have different operating and commercial models.
Describe movement, transformation, and analysis separately
Define each input source, data size, change frequency, transport, destination, freshness requirement, and business owner. A nightly 300-GB on-premises extract has a different failure profile from continuous device telemetry. A report that must be ready by 7 a.m. needs an observable completion condition, not merely a pipeline success indicator from one step. Separate transport success from semantic correctness of the transformed records.
Record which transformations are necessary before the destination is useful. Simple field mapping may fit a managed copy activity, whereas large distributed joins or aggregations may require Spark or a supported warehouse engine. A query serving an analyst is not the same as a pipeline writing curated tables; give each path its own latency, concurrency and governance expectations.
Use Azure Data Factory for owned data movement workflows
Azure Data Factory supports orchestrating activities and moving data among supported sources and sinks, with integration runtimes and connectors that depend on where data resides. It is useful for repeatable file or database ingestion, scheduled pipelines, copy jobs, and coordination of downstream processing. The Azure Data Factory introduction describes pipelines, activities, datasets, linked services, and integration runtime roles.
A pipeline that transfers partner records should record the source snapshot or watermark, job execution ID, destination staging path, row counts, and validation result. Treat partial copies and retries as first-class states. A simple schedule without checkpointing can duplicate a day’s data after a timeout. Choose incremental loading where appropriate and make the logical batch identity explicit.
Evaluate Fabric and Synapse from consumer needs
Microsoft Fabric brings integrated analytics experiences around OneLake and multiple workload types. Azure Synapse Analytics offers established dedicated and serverless SQL and Spark patterns, subject to service-specific lifecycle and deployment requirements. A current architecture must examine the actual Fabric capacity and governance model, existing Synapse investments, data residency, and migration obligations rather than assuming a new product instantly replaces every deployed Synapse feature.
Ask what the consumers do. A large reporting model with predictable SQL workload might require a warehouse; a data science transformation may need Spark notebooks; a near-real-time operational display may use a different ingestion and query path. Evaluate the supported capabilities and economics of the current service offering. Avoid migrating an operational pipeline purely because another platform now uses the same marketing terminology.
Choose batch or streaming from timeliness and replay needs
For batch ingestion, a watermark or immutable input identifier helps resume after partial failure. For streams, consider ordering, partitioning, retention, replay, and late or duplicate events. An event accepted by a transport is not guaranteed to appear exactly once in the final analytical table. Define the expected event key and deduplication logic near the authoritative write path.
A finance dashboard may tolerate an hourly batch, while fraud detection may need near-real-time processing. Costs, required operational maturity, and data correctness change with the latency target. Do not select stream processing because the source produces events if daily transformation is the actual business requirement. Similarly, do not force a 24-hour batch on a risk control that must act within seconds.
Connect hybrid inputs without overexposing networks
On-premises data sources may require an appropriate self-hosted integration runtime or supported hybrid connectivity depending on the service and source. A runtime with broad network permissions to every database is a high-value target. Limit source credentials and destination rights, secure data in transit, monitor job identity, and document who updates the runtime host and connector versions.
The hybrid connectivity architecture explains route and VPN/ExpressRoute choices. Applying that pattern to data integration requires a specific network and DNS test from the runtime to its source and destination. A working private circuit does not prove the connector can authenticate to the database, and a successful source query does not prove the destination transformation has valid permissions.
Establish schema and quality contracts before automation
Describe expected fields, units, nullability, key uniqueness, late arrivals, and acceptable source changes. A pipeline that processes every file successfully can still corrupt a report if one partner silently changes a currency unit or swaps identifiers. Validate schema and business constraints before merging into authoritative tables; send exceptions to a quarantined location with an owner and remediation procedure.
Data lineage should connect source datasets, transformations, targets and downstream reports. Microsoft Purview or supported native governance mechanisms may help with discovery and governance, but the business must still decide what data is sensitive and which team owns corrections. Governance tooling cannot infer every financial invariant from the shape of a JSON file.
Plan incremental state and idempotent recovery
A reliable integration pipeline knows where its last committed checkpoint is. It may copy data into a staging area, validate counts or checksums, then atomically or transactionally promote the curated result where supported. On retry, the same input should not create double-counted financial rows. A completion marker belongs to the final business state, not just the successful return of an upstream copy activity.
Rehearse an incomplete run: stop processing after half the data has arrived, resume, and verify that each logical claim exists exactly once. Then deliberately change one source schema and verify the pipeline fails visibly without replacing last night’s valid report. These tests explain more about reliability than a dashboard that only counts green scheduled runs.
Model compute and data-movement costs together
Estimate pipeline runs, connector and integration-runtime costs, storage requests, Spark or SQL processing, Fabric capacity usage, and cross-region or on-premises transfer. A cheap landing zone can become expensive if the same dataset is repeatedly copied between platforms for every consumer. Shared storage can reduce copies but may increase coordination and governance complexity. Review total data movement, not just line-item unit prices.
The cost-aware Azure architecture framework applies to pipelines: balance delivery latency, reliability, and spend per valid completed report. Shut down avoidable idle development capacity where supported, but do not impair the production deadline to save on one processor’s running time.
Close with a consumer-validating acceptance test
Build a bounded synthetic example from on-premises CSV to an analytical table. Include one duplicate ID, one late record, one schema change, and a forced retry. Confirm the final report totals, lineage and sensitive-data access. Document failure notifications, time to recover, data retention and cost. If the output is wrong despite every connector displaying success, the architecture has failed the business.
An AZ-305 data integration recommendation should say why a batch, stream, warehouse or Spark pattern fits the workload, which platform owns each stage, how the system resumes after errors, and what measures the quality of the result. The correct stack is the one whose data contract remains provable through normal operation, source change, and recovery—not the one with the most product logos.