Machine Learning Operations Engineer Associate · 14% of the exam

Optimize generative AI systems and model performance: free practice questions

5 sample questions from our 50-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Microsoft exam.

1. A healthcare company is creating a fine-tuning dataset for a clinical documentation assistant. They want to use synthetic data generation to scale their dataset but must ensure the synthetic examples are safe and clinically accurate. Which approach best ensures quality and safety of the synthetic data?

  • A. Generate synthetic examples using a large frontier model, then have clinical experts review and filter the outputs before including them in the training set✓ Correct
  • B. Generate synthetic examples and include them all without review to maximize dataset size
  • C. Use a smaller domain-specific model to generate synthetics to avoid hallucinations from large models
  • D. Scrape publicly available medical forums and use those as synthetic training examples
Explanation

Using a large frontier model for generation combined with clinical expert review (human-in-the-loop validation) ensures both scale and accuracy, catching hallucinations or unsafe content before training. Including all outputs without review (B) risks training on incorrect or unsafe clinical information, which is a patient safety issue. Using a smaller model (C) does not reduce hallucinations; smaller models are generally more prone to errors in specialized domains. Scraping medical forums (D) produces noisy, unverified content that is not synthetic in the controlled sense and raises legal and quality concerns.

2. An AI team has deployed a RAG pipeline for a corporate knowledge base. During evaluation, they notice that the retrieved chunks are highly relevant to user queries, but the generated answers frequently include claims not supported by the retrieved context. Which RAG Triad metric is most directly failing, and what is the recommended corrective action?

  • A. Context relevance is failing; increase the number of retrieved chunks to provide more supporting evidence
  • B. Answer relevance is failing; rewrite the system prompt to instruct the model to stay on topic
  • C. Groundedness is failing; adjust the system prompt to instruct the model to base responses only on retrieved context and add a groundedness evaluation step✓ Correct
  • D. Context recall is failing; lower the similarity threshold to retrieve a broader set of documents
Explanation

Groundedness (Option C) measures whether the generated answer is supported by the retrieved context. When retrieved chunks are relevant but the answer contains unsupported claims, groundedness is the failing metric. The fix is to strengthen the prompt constraint and add a groundedness evaluation. Option A is incorrect because context relevance measures how well retrieved chunks match the query — it is already performing well in this scenario. Option B (answer relevance) measures whether the answer addresses the question, not whether it fabricates content. Option D (context recall) relates to how many relevant documents are retrieved, not to hallucination in generation.

3. An enterprise AI team is deploying a fine-tuned Azure OpenAI model for a regulated financial advisory application. Before moving from staging to production, they must satisfy THREE conditions: the model must meet accuracy thresholds on held-out evaluation data, it must pass safety/content policy checks, and it must have a documented rollback plan. Which deployment practice on Azure OpenAI best addresses all three requirements? (Select THREE.)

  • A. Run the fine-tuned model against a held-out golden evaluation dataset and confirm it meets the defined accuracy and quality metric thresholds before promotion✓ Correct
  • B. Deploy directly to production and rely on user feedback collected over the first 30 days to validate accuracy
  • C. Run Azure AI Content Safety evaluation on the fine-tuned model's outputs using adversarial test prompts to verify content policy compliance✓ Correct
  • D. Document a rollback procedure that identifies the previous production deployment and defines the conditions and steps to revert if production metrics degrade✓ Correct
  • E. Increase the fine-tuning dataset size by 10x and retrain as a precaution before every production deployment
  • F. Deploy the new model as the sole endpoint immediately to simplify traffic routing
Explanation

Option A (held-out evaluation against accuracy thresholds) satisfies the accuracy requirement with a controlled pre-deployment gate. Option C (content safety evaluation with adversarial prompts) satisfies the safety/content policy requirement by stress-testing the model before it serves real users. Option D (documented rollback plan) satisfies the third requirement by ensuring the team can revert quickly if production issues arise. Option B (relying on production user feedback) is insufficient for a regulated application — accuracy must be validated before deployment, not after. Option E (10x dataset retraining before every deployment) is not a deployment lifecycle practice; it is a training decision and would be prohibitively expensive. Option F (sole endpoint immediately) eliminates the ability to perform safe staged rollouts and removes the option for easy rollback.

4. An AI team has fine-tuned a GPT-4o mini model on Azure OpenAI for an internal compliance document Q&A assistant. During evaluation, groundedness scores are high (0.91), but answer relevance scores are low (0.54). Users report that responses are factually grounded in retrieved chunks but frequently include excessive regulatory background that does not address the specific question asked. What is the most likely root cause, and what is the correct remediation?

  • A. The similarity threshold is too low, causing irrelevant chunks to be retrieved; increase the minimum cosine similarity score to filter out low-confidence chunks.
  • B. The fine-tuning data contains many examples where the ideal response is a broad regulatory summary rather than a targeted answer, causing the model to over-explain; curate fine-tuning examples to include concise, question-focused ideal responses.✓ Correct
  • C. The embedding model is underperforming on compliance terminology; replace it with a domain-fine-tuned embedding model to improve chunk relevance.
  • D. The chunk size is too small, causing the model to lack sufficient context; increase chunk size to 1,024 tokens to provide more complete regulatory passages.
Explanation

Option B is correct because the diagnostic pattern—high groundedness but low answer relevance—indicates the model is faithfully reproducing retrieved content but not distilling it into focused answers. This is a generation behavior issue, not a retrieval issue. If fine-tuning examples used broad regulatory summaries as ideal outputs, the model learned to replicate that style. Curating the fine-tuning dataset so that target responses are concise and directly address the specific question will re-train the generation behavior. Option A is wrong because the groundedness score is already high (0.91), indicating retrieved chunks are relevant and factually useful; the problem is not noisy retrieval but verbose generation. Option C is wrong for the same reason: embedding quality affects what chunks are retrieved, and high groundedness already confirms retrieval is working well. Option D is wrong because larger chunks would give the model even more regulatory background to reproduce, likely worsening the verbosity problem rather than improving answer focus.

5. An AI team is designing a RAG pipeline for a technical support system. Documents range from short FAQ entries (50 words) to detailed troubleshooting guides (5,000 words). They want a chunking strategy that preserves semantic coherence across different document lengths. Which strategy is most appropriate?

  • A. Fixed-size chunking with 512 tokens for all documents regardless of structure
  • B. Recursive character text splitting that respects paragraph and section boundaries, with chunk sizes adapted to document structure✓ Correct
  • C. Splitting every document into exactly two equal halves
  • D. Using the entire document as a single chunk for all documents to preserve full context
Explanation

Recursive character text splitting respects natural document boundaries (paragraphs, sections, sentences) and can adapt to different document lengths, preserving semantic coherence better than arbitrary size limits. This is the appropriate strategy for heterogeneous document collections. Fixed-size chunking (A) ignores document structure and may split sentences or paragraphs mid-thought, harming coherence. Splitting into two halves (C) is arbitrary and unrelated to semantic structure. Using entire documents as single chunks (D) will exceed context windows for long documents and makes embedding quality poor for very long texts.

45 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free