A streaming notebook that successfully processes one batch is not the same as a recoverable streaming service. If you are new to Fabric notebook configuration and environment control, the existing Spark notebook operations article explains how a notebook becomes a repeatable production step. Microsoft Fabric’s Spark Structured Streaming must survive job restarts, repeated source messages, late events and changes to the query code. DP-700 covers streaming engine selection, Spark Structured Streaming, transformations and optimization; a reliable answer should explain checkpoint state and the chosen delivery semantics.
Microsoft’s Structured Streaming overview distinguishes continuously running queries from available-now runs. Both use the same readStream/writeStream style of programming, but the latter drains currently available data, stops and can be relaunched on a schedule. This makes it useful for periodic incremental ETL as well as genuinely continuous stream processing.
Start with an explicit stream contract
Imagine parcel scans arriving through a replayable event source. Every record has an event_id, parcel_id, source timestamp and location. Operations wants parcel-level history, with one record per unique scan. Write down which late events are acceptable, how long to retain state, whether a source can replay already processed messages, and which table receives invalid records. Without these definitions, “exactly once” is just a marketing phrase.
A checkpoint records progress, committed offsets and state required to restart a query. Keep each query’s checkpoint in durable OneLake storage, not on the temporary driver filesystem. Separate queries need separate checkpoint locations: two independent transformations must not share and corrupt the same progress state. For stateful queries, even seemingly small changes to grouping keys, operators or state schema may not be compatible with a saved checkpoint.
Write to Delta with an owned checkpoint
An illustrative PySpark streaming sink might look like this:
query = (
clean_events.writeStream
.format("delta")
.outputMode("append")
.option("checkpointLocation", "Files/checkpoints/parcel-events-to-bronze")
.toTable("bronze_parcel_events")
)
Here clean_events is a previously defined streaming DataFrame; the snippet is intentionally not a standalone end-to-end connector setup. The checkpoint path belongs to this query definition and destination. If the process stops and restarts with compatible code and the same checkpoint, Spark can resume after the last committed progress rather than scanning source history from scratch.
Do not assume every sink shares Delta’s transaction behavior. Custom foreachBatch logic can run more than once for a batch during recovery; any SQL upsert or API side effect inside it needs idempotent keys or transaction identifiers. Microsoft’s streaming best practices explicitly distinguish Delta/file sink guarantees from custom and at-least-once sinks.
Watermarks bound late-arrival state
Consider counting scans per five-minute event-time window. If late events are allowed forever, the engine could need to retain window state indefinitely. A watermark expresses an acceptable lateness bound and lets the engine eventually retire completed state. The engineering trade-off is unavoidable: a narrow watermark reduces memory and latency but can exclude valid late records; a generous bound increases state and delays finality. Choose it from measured source delay distribution and business tolerance.
A device that is offline for twenty minutes may upload a burst of events. Test how those timestamps interact with the watermark before promising that an hourly aggregate is final. Store rejected late records or reconciliation evidence when the business requires complete historical accounting. This is particularly important in financial or regulated pipelines where a fast provisional count and an audited final number may have different semantics.
Stateful joins and deduplication need bounded memory
A stream-stream join retains unmatched records until their join window expires. Without time bounds and watermarks, state can grow with every event. For deduplication, a stable event ID prevents a retried delivery from inflating counts; however, the deduplication state also needs a retention policy. If event IDs can recur legitimately months later, do not keep them in an unbounded forever-set. Decide what uniqueness means by source, time horizon and business process.
Microsoft’s stateful processing guide explains the checkpointed state store, event-time windows and deduplication. For large state, RocksDB can reduce pressure on JVM memory, but it does not remove the need to bound the number of keys and windows retained.
Reconcile streaming correctness after failures
Construct a controlled three-part failure test. First, process 100 source events and capture the last committed event IDs. Second, stop the query during a new batch of 20 and restart it using the same checkpoint. Third, replay ten already observed events and compare unique target IDs. A correct idempotent design reaches 120 unique accepted events without gaps or duplicates, subject to the deliberately injected malformed or late messages.
If the observed count differs, isolate the source event log, parsed records, checkpoint state, destination transactions and user-defined foreachBatch behavior separately. A successful restart does not prove correctness if the custom side effect silently executed twice. Record checkpoint location, application version and last committed progress in the runbook.
Separate always-on stream operations from scheduled batch orchestration
Available-now execution can be attractive for files that land several times daily: it processes what has arrived using the prior checkpoint and then stops, allowing orchestration to schedule the next run. An always-on query is more suitable for continuously incoming events with a tight freshness SLA. In both cases, operators should monitor throughput, processing duration, backlogs, error state and output Delta file growth.
Choosing a very short trigger interval can generate too many small Delta files. Longer intervals can improve batching and reduce overhead but delay availability. Optimize Write, compaction and maintenance schedules should follow measured file size and query workload, not be enabled blindly. The lakehouse Delta optimization article covers the downstream storage costs of streaming writes.
Compare Spark to the alternative Eventhouse path
Spark offers code-defined transformations, flexible stateful operators and Delta transactional sinks. An Eventstream and Eventhouse design may better fit low-code routing and rapid KQL investigation. The exam expects you to choose based on transformations, latency, skills, governance and operating burden. A technology choice is defensible when you can explain what the last safely processed event was, how the pipeline restarts and what counts as complete data.