AI Foundations: Foundation Models & Interfaces Practice Questions

HumanEval is a benchmark specifically designed to measure code generation capability — it presents programming problems and evaluates whether the model produces functionally correct code. MMLU measures broad academic knowledge across 57 subjects but is not code-focused. HELM is a meta-framework that aggregates many benchmarks for holistic evaluation and is not specialized for coding. BIG-Bench is a broad reasoning benchmark covering many tasks but is not specifically a coding benchmark.

Anthropic's Claude lineup follows the order Haiku (smallest, fastest, lowest cost) → Sonnet (mid-tier, balanced performance and cost) → Opus (largest, most capable, highest cost). This mirrors the naming convention — a haiku is a short poem, a sonnet is medium-length, and an opus is a large-scale work. The other orderings misplace one or more models in the hierarchy.

The task requires the model to process an image (photo of the damaged car) alongside text. This is precisely what multimodal vision input capability provides. Extended thinking improves reasoning quality on text tasks but does not add image understanding. TTS converts text to audio, which is unrelated. RAG enhances responses with retrieved documents but does not grant a model the ability to perceive images.

Open-weight models (e.g., Llama 3, Mistral) allow local fine-tuning without data leaving your infrastructure (A — correct), and they require self-managed inference hardware or cloud VMs (C — correct). Proprietary models do NOT always have larger context windows — Llama 3 and Mistral variants offer competitive context lengths (B — wrong). Open-weight models are not always cheaper; infrastructure and GPU costs can exceed API pricing at scale (D — wrong). Many open-weight models (including Llama 3 and Mistral) have permissive licenses that allow commercial use (E — wrong).

A (index 0): Correct — Gemini was built as a natively multimodal model from pretraining, unlike models that had vision bolted on afterward. C (index 2): Correct — Gemini Ultra, Pro, and Nano are the three size/deployment tiers of the Gemini family, analogous to Opus/Sonnet/Haiku for Claude. F (index 5): Correct — Gemini Nano is specifically optimized for on-device deployment on Android and edge hardware. B (index 1) is wrong: Gemini's multimodality is native, not achieved solely through post-training fine-tuning. D (index 3) is partially true but overstated as a general launch capability. E (index 4) is clearly wrong: Gemini supports audio and video, not just text and images.