Quality assurance · Production

LLMOps in restricted BYO networks using Azure Machine Learning with conditional continuous evaluation

The problem

The team needed to establish an LLMOps and continuous evaluation pipeline in a restricted bring-your-own network, facing service configuration complexity, SDK limitations in private networks, and a 6-hour end-to-end evaluation run that blocked deployments on every commit.

First attempt

The AML managed endpoint did not work with the private network and required a fallback to Docker packaging; the pfazure SDK had compute host resolution issues; and obtaining Service Principal permissions was a lengthy process.

Workflow diagram · grounded in source
1
PR creation triggers CI
trigger
“PR from feature to develop runs CI checks and merges on success, needing human intervention on failure”
2
CI code quality validation
validation
“Code Quality Checks - Unit Tests - End-to-end Flow (PromptFlow) syntactic validation on the Sample Data”
3
PR label gates E2E eval
routing
“when opt-out flag is ON – long running e2e eval flow is not executed as part of CE pipeline”
4
Continuous evaluation runs
validation
“Synthetic data preparation for evaluation - (PromptFlow) Node-level Evaluation - (PromptFlow) End-to-end Flow Simulation and Evaluation”
5
Data scientist reviews metrics
human_review
“The links to the metrics and AML jobs for end-to-end component validations will be provided in the summary, and they will be validated manually by the data scientist before promoting to the next stages”
6
CD build and deployment
output
“Versioned PromptFlow is containerized - Image pushed to Azure Container Registry - PromptFlow deployed to Azure Web App”
Reported outcome

The team implemented a CI/CE/CD pipeline with an opt-out mechanism for the long-running E2E evaluation, allowing the ~30-minute minimal dataset run to proceed before every dev deployment while the full ~6-hour evaluation runs in parallel.

Reported metrics
full E2E evaluation run duration~6hrs
minimal dataset E2E run duration~30 mins
Reported stack
Prompt FlowAzure Machine Learningpfazure SDKAzure Container RegistryAzure Web AppDockerGitHub
Source
https://devblogs.microsoft.com/ise/llmops-in-restricted-networks-and-continuous-evaluation-long-run-constraints/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The team implemented a CI/CE/CD pipeline with an opt-out mechanism for the long-running E2E evaluation, allowing the ~30-minute minimal dataset run to proceed before every dev deployment while the full ~6-hour evaluat…

What tools did this team use?

Prompt Flow, Azure Machine Learning, pfazure SDK, Azure Container Registry, Azure Web App, Docker, GitHub.

What results were reported?

full E2E evaluation run duration: ~6hrs; minimal dataset E2E run duration: ~30 mins (source-reported, not independently verified).

What failed first in this deployment?

The AML managed endpoint did not work with the private network and required a fallback to Docker packaging; the pfazure SDK had compute host resolution issues; and obtaining Service Principal permissions was a lengthy…

How is this quality assurance AI workflow structured?

PR creation triggers CI → CI code quality validation → PR label gates E2E eval → Continuous evaluation runs → Data scientist reviews metrics → CD build and deployment.

Related quality assurance cases