Operationalizing ML and Generative AI Solutions · 20% of the exam

Optimize GenAI systems and model performance: free practice questions

5 sample questions from our 50-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Microsoft exam.

1. A team has completed fine-tuning a GPT-4o mini model on Azure OpenAI for a specialized internal HR assistant. They are now planning the model's production lifecycle. Which two practices should they implement to maintain model quality over time? (Select TWO)

A. Continuously monitor production outputs and collect user feedback to detect performance drift✓ Correct
B. Redeploy the same fine-tuned model indefinitely without monitoring since fine-tuning is a one-time improvement
C. Establish a periodic retraining schedule using newly collected production data and updated examples✓ Correct
D. Delete the fine-tuning dataset after training to reduce storage costs and privacy exposure
E. Increase the model's temperature setting in production to improve creative responses

Explanation

Monitoring production outputs (A) is essential for detecting drift as user behavior, language, and HR policies evolve over time. Periodic retraining with new data (C) ensures the model stays current with policy changes and new examples. Redeploying the same model indefinitely (B) ignores drift and policy changes, degrading quality over time. Deleting the fine-tuning dataset (D) prevents future retraining and auditing — the dataset should be archived securely, not deleted. Increasing temperature (E) makes outputs less consistent and is unrelated to maintaining fine-tuned model quality.

2. A developer is configuring chunk sizes for a RAG system that will answer questions from dense technical API documentation. The documentation contains many interdependent code snippets and parameter descriptions. Users report that answers are incomplete because responses are missing important contextual details. Which chunking strategy adjustment is MOST appropriate?

A. Reduce chunk size from 512 to 128 tokens to improve retrieval precision and reduce noise.
B. Increase chunk size to preserve more surrounding context within each retrieved passage.✓ Correct
C. Keep chunk size the same but reduce the similarity threshold to retrieve more low-relevance chunks.
D. Switch to character-level chunking instead of token-level chunking to align better with code boundaries.

Explanation

When answers are incomplete because context is fragmented across chunks, the appropriate fix is to increase chunk size so that related content (e.g., a function signature plus its parameters and description) is contained in a single retrievable unit. Option A (smaller chunks) would worsen the fragmentation problem. Option C (lower threshold) would bring in more chunks but many would be less relevant, increasing noise rather than improving completeness. Option D (character-level chunking) is a granularity change unrelated to the contextual completeness problem and could introduce its own alignment issues with token-based embedding models.

3. A machine learning engineer is designing a fine-tuning pipeline for a domain-specific model on Azure OpenAI. They want to use synthetic data generated by a larger GPT-4o model to fine-tune a smaller GPT-4o mini model. Which of the following risks is MOST important to mitigate when using this approach, and what is the recommended mitigation?

A. The fine-tuned model may exceed the token limits of the base model; mitigate by truncating all training examples to 512 tokens
B. The fine-tuned model may inherit hallucinations or inaccuracies present in the GPT-4o-generated synthetic data; mitigate by validating synthetic examples against ground-truth sources before inclusion in the training set✓ Correct
C. The synthetic data will cause the model to become too large to deploy on Azure OpenAI; mitigate by using quantization after fine-tuning
D. GPT-4o-generated data is always lower quality than human-authored data; mitigate by using only human-written examples for fine-tuning

Explanation

Option B is correct. The most significant risk of using LLM-generated synthetic data is that the teacher model (GPT-4o) may generate plausible-sounding but factually incorrect examples. Fine-tuning on these examples transfers the errors to the student model (GPT-4o mini). The mitigation is validation against authoritative ground-truth sources. Option A is a misunderstanding — token limits apply to inference, not model size, and arbitrary truncation destroys training examples. Option C is incorrect — fine-tuning on Azure OpenAI does not change model deployment size or eligibility. Option D is overly absolute — LLM-generated synthetic data can be high quality, especially when properly validated; the issue is not the source but the accuracy of the content.

4. An AI engineer is building a RAG pipeline for an e-commerce site. The product catalog uses highly specific product codes and brand names (e.g., 'Nike AIR-MAX-270-BLK-10') that embedding models tend to treat as semantically similar to unrelated products. Which retrieval configuration best handles this?

A. Pure semantic search with a general-purpose embedding model
B. Hybrid search with semantic embeddings and BM25 keyword matching, with higher weight on the keyword component for product code queries✓ Correct
C. Increasing the top-k retrieval count to retrieve more candidates and hoping the LLM filters correctly
D. Reducing chunk size to isolate individual product records for embedding

Explanation

Hybrid search with higher keyword weighting (B) ensures that exact product codes and brand names are matched precisely via BM25, while semantic search still handles natural language queries. Product codes are exact-match problems poorly suited to semantic embeddings. Pure semantic search (A) will struggle because embeddings smooth over precise token differences like color codes and sizes. Increasing top-k (C) adds noise and increases cost without solving the precision problem for exact codes. Reducing chunk size (D) may help isolate products but does not solve the fundamental issue of semantic embeddings poorly representing alphanumeric codes.

5. A machine learning engineer is applying LoRA (Low-Rank Adaptation) to fine-tune a large language model for a specialized coding assistant. They need to decide which layers to apply LoRA adapters to in order to maximize task performance within a fixed parameter budget. Which layers should they prioritize?

A. Only the embedding layers, since those represent token meaning
B. Only the final output projection layer, since that produces the final logits
C. The attention layers (query and value projection matrices), as these most directly influence how the model attends to and integrates context✓ Correct
D. The feedforward network (FFN) layers only, since those store factual knowledge

Explanation

LoRA was specifically designed to be applied to attention weight matrices (particularly Q and V projections), as these govern how the model attends to input context and are highly influential for task-specific adaptation. Research shows attention layers yield the best performance-per-parameter trade-off for LoRA. Embedding layers (A) represent token identity and are less task-adaptive; modifying them risks destabilizing the vocabulary space. The final output projection (B) is only one layer and insufficient alone for comprehensive task adaptation. FFN layers (D) do store knowledge but are computationally denser; LoRA's original formulation prioritizes attention layers for efficiency.

45 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free