Generative AI changes the operating model of machine learning. A conventional model release can often be described in terms of training data, code, model artifacts, deployment, and prediction monitoring. A generative AI application adds more moving parts: foundation-model versions, prompts, system instructions, retrieval indexes, safety controls, tool definitions, evaluation datasets, agent behaviors, and provider-side model updates. The result is not simply \u201cMLOps with a language model.\u201d It is a broader release and governance problem.<\/p>\n
Microsoft now treats these practices as a combined MLOps and GenAIOps discipline in the scope of AI-300<\/a>. The exam centers on operationalizing machine learning and generative AI solutions with Azure Machine Learning, Microsoft Foundry, deployment automation, evaluation, observability, and lifecycle controls. That framing is useful beyond exam preparation because it highlights the core production challenge: teams need a repeatable way to move an AI capability from experimentation to a controlled service without losing traceability.<\/p>\n The foundational ideas still overlap with established machine learning concepts<\/a>, but generative systems require additional evidence. A deployment can be technically healthy while its answers become less relevant, less grounded, or less safe. A prompt change can alter behavior without changing model weights. A retrieval update can change responses without a new application binary. Operations therefore have to track the whole AI system, not just the endpoint serving a model.<\/p>\n A release process is much easier to control when the team agrees on what constitutes a versioned AI solution. For a simple predictive model, that unit might include code, a model artifact, environment dependencies, and deployment configuration. For a generative application, it may include an application build, selected model and model version, system prompt, prompt templates, retrieval configuration, index or corpus version, content-safety settings, evaluation configuration, and infrastructure definitions.<\/p>\n These pieces should not be changed independently in production without a record of the resulting combination. If an incident begins after a prompt update but the deployed application version is unchanged, operators still need a way to identify exactly what changed. The same is true when a retrieval corpus is refreshed, a content filter threshold is adjusted, or a hosted model alias starts resolving to a newer underlying version.<\/p>\n A practical release manifest can tie those elements together. It does not need to be complicated, but it should answer: which code revision is running, which model configuration it uses, which prompt revision is active, which data or index version supports retrieval, which evaluation set was used, and which infrastructure release created the environment. Without that lineage, rollback becomes guesswork.<\/p>\n Generative AI development is naturally exploratory. Teams compare models, adjust instructions, test retrieval strategies, and experiment with evaluation criteria. That work needs speed. Production needs the opposite qualities: repeatability, review, and bounded change. The delivery process should therefore provide a clear boundary between experimentation and a candidate release.<\/p>\n Once an experiment is considered for deployment, its important configuration should move into version-controlled artifacts rather than remaining only inside an interactive playground or a developer’s local notebook. Prompts, application configuration, evaluation definitions, and infrastructure templates should be reviewable. Dependencies should be pinned sufficiently to reproduce the build. Secrets and connection details should remain outside source control and be injected through managed configuration.<\/p>\n This is where traditional DevOps practices remain highly relevant. The existing discussion of DevOps in cloud platforms<\/a> applies directly to AI delivery: source control, automated validation, controlled promotion, environment separation, and observable releases reduce the risk of unreviewed production changes. The difference is that an AI release needs tests for behavior as well as tests for software correctness.<\/p>\n Continuous integration for a generative AI application should begin with ordinary engineering checks: linting, unit tests, dependency scanning, configuration validation, and build reproducibility. It should then add AI-specific validation. A prompt template can be checked for required variables. Retrieval code can be tested against known documents. Tool schemas can be validated. Safety policies can be exercised with controlled test cases. Evaluation datasets can be checked for accidental leakage of secrets or regulated data.<\/p>\n Prompt assets deserve first-class treatment. If prompts affect production behavior, they should be versioned and reviewed like code. The release process should identify the prompt revision that was evaluated and prevent an untested prompt from bypassing the promotion path. This reduces a common form of configuration drift in which a team carefully validates an application build but later edits production instructions through a different interface.<\/p>\n Git-based workflows help because they produce a history of changes and make review explicit. Engineers preparing for the neighboring AZ-400<\/a> domain will recognize the same principles in release engineering: small reviewable changes, automated gates, traceable artifacts, and a promotion process that does not depend on one person’s workstation.<\/p>\n Generative outputs are probabilistic, so a production pipeline needs evaluation gates that reflect the application’s actual task. Depending on the workload, teams may measure relevance, groundedness, coherence, fluency, task completion, retrieval quality, tool-selection accuracy, or domain-specific correctness. Safety evaluation can examine harmful content, jailbreak susceptibility, prompt-injection behavior, data leakage, and policy violations.<\/p>\n Evaluation should use representative cases rather than a handful of ideal prompts. Include ordinary requests, ambiguous requests, difficult edge cases, adversarial inputs where appropriate, and examples that test the boundaries of the system’s allowed behavior. For retrieval-augmented generation, include cases where the supporting corpus contains the answer, cases where it does not, and cases where retrieved sources conflict or are stale.<\/p>\n Automatic evaluators can scale testing, but they should not be treated as an unquestionable oracle. Some quality dimensions require human review, especially for high-impact or domain-specific use cases. A practical gate can combine deterministic checks, model-based evaluators, safety tooling, and targeted human acceptance. The important point is that the release decision is supported by recorded evidence rather than by a developer saying the latest prompt \u201clooks better.\u201d<\/p>\n Environment drift is especially risky when an AI application spans multiple Azure resources. Development, test, and production may involve separate Foundry projects, model deployments, storage, search, monitoring, managed identities, private networking, and policy configuration. If those environments are built manually, subtle differences can invalidate test results.<\/p>\n Infrastructure as code makes the platform layer reviewable and repeatable. Deployment pipelines can create or update resources using declarative templates and controlled parameters, while secrets remain in managed stores. The same release can then bind application artifacts to the intended model deployment, index, and monitoring configuration for each environment.<\/p>\n Promotion should be staged. A candidate release can first enter an integration environment, run automated functional and evaluation suites, and then move to a production-like environment for load, security, and operational testing. High-impact systems may require a formal approval before production. Lower-risk internal tools may use a lighter gate, but the decision should still be explicit and traceable.<\/p>\n The practical techniques described in Azure DevOps tooling<\/a> can support this style of controlled delivery, but the pipeline design matters more than any one product. The objective is reproducibility: the team should be able to explain what is deployed and recreate it without a sequence of undocumented console clicks.<\/p>\n A generative AI deployment should assume that some problems will appear only under real traffic. Canary releases, staged rollouts, versioned endpoints, or other traffic-splitting strategies can limit the impact of a bad change. A small percentage of requests can be sent to the candidate while quality, latency, errors, safety signals, and cost are compared with the current version.<\/p>\n Rollback needs more than an old application image. If the release changed prompts, model selection, retrieval content, or policy settings, those elements must also be recoverable. That is another reason to treat the entire AI configuration as a versioned release unit. Restoring only code while leaving the new prompt or new retrieval index active may fail to restore prior behavior.<\/p>\n Teams should also distinguish rollback from roll-forward. A security or compliance issue may require immediate rollback. A small quality regression might be safer to correct with a new release if reverting would reintroduce another known defect. The operating procedure should define who can make that decision and what evidence is required.<\/p>\n Traditional service metrics remain essential: request volume, latency, error rates, resource saturation, availability, and cost. They reveal whether the service is functioning. GenAIOps adds another layer: whether the service is behaving acceptably. Monitor evaluation signals on sampled production interactions, retrieval quality where measurable, safety events, user feedback, tool failures, token consumption, and changes in the distribution of requests.<\/p>\n Tracing is particularly valuable for multi-step generative applications and agents. A trace can show which prompt was constructed, which retrieval call ran, what tools were invoked, where latency accumulated, and which step failed. Sensitive content must be handled carefully; observability should not become an uncontrolled store of prompts, responses, credentials, or personal data.<\/p>\n Model and data drift also take new forms. Input topics can shift as user behavior changes. A retrieval corpus can age. An upstream model provider can update behavior. A prompt that worked well on yesterday’s traffic may underperform on a new class of requests. Operations therefore need baselines, thresholds, and a process for deciding when a signal triggers investigation, reevaluation, retraining, re-indexing, or release.<\/p>\n Security cannot be bolted onto an AI deployment after the pipeline is built. Threat modeling should consider prompt injection, excessive tool permissions, insecure retrieval, sensitive-data exposure, malicious files, model endpoint abuse, compromised dependencies, and secrets leaking through logs or prompts. The article on AI security risks<\/a> provides additional context for why generative systems need controls beyond normal application hardening.<\/p>\n Identity and least privilege are central. Deployment identities should have only the rights needed to create or update their resources. Runtime identities should be narrower still, especially when an agent can act on external systems. Safety filters, tool allowlists, network controls, data-access policies, and human approval for high-impact actions can all become part of the release policy.<\/p>\nDefine the deployable unit before building the pipeline<\/h3>\n
Keep experiments separate from controlled releases<\/h3>\n
Build CI around code, prompts, and evaluation assets<\/h3>\n
Evaluate quality before deployment, not only after incidents<\/h3>\n
Promote infrastructure and AI configuration together<\/h3>\n
Release with rollback and traffic-control strategies<\/h3>\n
Monitor behavior, not just endpoint health<\/h3>\n
Make security and responsible AI release gates<\/h3>\n