Foundation Models & Interfaces: free practice questions

HumanEval is a benchmark specifically designed to measure code generation capability — it presents programming problems and evaluates whether the model produces functionally correct code. MMLU measures broad academic knowledge across 57 subjects but is not code-focused. HELM is a meta-framework that aggregates many benchmarks for holistic evaluation and is not specialized for coding. BIG-Bench is a broad reasoning benchmark covering many tasks but is not specifically a coding benchmark.

MMLU (Massive Multitask Language Understanding) is specifically designed to measure a model's knowledge and reasoning across 57 academic and professional subjects, making it ideal for assessing broad knowledge breadth. HumanEval measures code generation ability (writing Python functions). HELM is a broad evaluation framework covering many metrics (accuracy, robustness, fairness), not just subject-matter knowledge. SQuAD is a reading comprehension benchmark focused on extracting answers from given passages.

HELM (Holistic Evaluation of Language Models, from Stanford CRFM) is explicitly designed to evaluate models holistically across many NLP scenarios — including summarization, QA, and classification — while also capturing metrics like calibration, robustness, and fairness. This aligns directly with a general business assistant evaluation. Option A is incorrect because HumanEval is specifically a code generation benchmark (Python function completion), not a multi-task NLP benchmark. Option C is completely incorrect — HELM is a text-based multi-task NLP benchmark. Option D is incorrect because HumanEval is the standard specifically for coding benchmarks, not general-purpose evaluations.

Tokens are sub-word units — they can represent a whole word, a word fragment, punctuation, or whitespace, depending on the tokenizer (e.g., BPE or WordPiece). They are the fundamental unit LLMs use for both input and output. Tokens are NOT single characters (many words are single tokens, and rare words may be split into multiple tokens). They are not fixed-size byte chunks; tokenization is vocabulary-based, not byte-based. They are not cryptographic hashes.

A (index 0): Correct — Gemini was built as a natively multimodal model from pretraining, unlike models that had vision bolted on afterward. C (index 2): Correct — Gemini Ultra, Pro, and Nano are the three size/deployment tiers of the Gemini family, analogous to Opus/Sonnet/Haiku for Claude. F (index 5): Correct — Gemini Nano is specifically optimized for on-device deployment on Android and edge hardware. B (index 1) is wrong: Gemini's multimodality is native, not achieved solely through post-training fine-tuning. D (index 3) is partially true but overstated as a general launch capability. E (index 4) is clearly wrong: Gemini supports audio and video, not just text and images.