AI Engineer Interview Questions: RAG, Evals and Agent Design Rounds for OPT and H-1B Candidates

The AI engineer loop tests retrieval, evals and agent design rather than model training. Here are 12 prompts with answer outlines and the visa checks to run on an AI startup before the final round.

AI engineer interview questions in 2026 come in four kinds of rounds: practical coding against model APIs, LLM system design (usually a retrieval-augmented generation, or RAG, pipeline or a tool-using agent), evals and debugging, and a behavioral round that often asks about a production LLM failure. The loop tests the application layer far more than model training. On F-1 OPT, STEM OPT or H-1B, it also has a visa track, and at a small AI startup that track can decide the offer.

Status as of 2026-11-01: wage-weighted H-1B selection is in effect (since 2026-02-27), FY2028 registration is not announced, the $100,000 H-1B payment is blocked and not collected, and the $103,265 cap-subject fee and new OPT fees are only proposed. Details are in the status table below.

What questions are asked in an AI engineer interview?

An AI engineer interview asks you to build and defend systems on top of foundation models. Expect a practical coding round (API calls, streaming, retries, structured output), an LLM system design round on RAG or agents, an evals round on how you measure quality, and a behavioral round about shipping and failure. Training math comes up less than in ML loops.

Ask the recruiter in writing which AI tools each round allows (see AI-assisted coding interviews).

Demand figures measure different things. Indeed's Hiring Lab put AI-related postings at 7.3% of all US postings (October 2026 snapshot, 2026-10-22); Lightcast (2026-10-19) found almost 3% of US postings mention AI skills, with roughly one junior role per four senior roles; and employers told NACE more than one-third of entry-level jobs require AI skills (employer-reported). None tells you who sponsors.

How is an AI engineer interview different from an ML engineer interview?

An ML engineer trains, tunes and serves models. An AI engineer builds products on models someone else trained, so the interview asks about retrieval, evals, agents, cost and latency instead of loss functions. For training-side roles, use our ML engineer interview loop guide instead.

Still choosing a path? See our guide to becoming an AI engineer as an international student.

AI engineer interview questions on LLM system design and RAG

What is an LLM system design question?

It asks you to design a feature built on a large language model, such as a support assistant over company documents or an agent that acts through tools. Interviewers want requirements, then the pipeline, then proof it works. The method is the one in our system design interview prep guide; only the building blocks change.

1. "Design a support assistant that answers from 2 million internal documents." Clarify users, freshness and who may see which documents. Then walk the pipeline: structure-aware chunking, hybrid (embedding plus keyword) search, a reranker, a prompt that forces citations, and permission filtering at retrieval time rather than after generation. Finish with evaluation and cost per query.

2. "The bot answers confidently and wrongly. Is retrieval or generation at fault?" Measure retrieval on a labeled set (was the right passage in the top results?) separately from generation (did the answer stay faithful to it?). A missing passage points to chunking, embeddings or query rewriting; a present passage with a wrong answer points to the prompt, context order or model.

3. "How do you pick a chunk size?" Stating a magic number is a red flag. It depends on document structure and question type, so compare options against a retrieval eval, and consider retrieving small chunks while passing the parent section to the model.

4. "When would you not use RAG?" When the knowledge fits in the context window, when the problem is behavior or format rather than knowledge, when the data is structured (let the model call SQL or an API), or when latency cannot absorb a retrieval step.

5. "Design an agent that issues customer refunds." Strict tool schemas, least-privilege credentials, human confirmation for irreversible or high-value actions, a step cap, idempotent tool calls and full traces. Evaluate on scripted scenarios by task success, not transcript fluency.

6. "How do you defend a tool-using agent against prompt injection?" Treat retrieved text, emails and tool outputs as untrusted data, never instructions. Layer least privilege, confirmation for sensitive actions, input and output filters, and separation between what the model reads and what it may do. Say plainly that no single defense is complete.

Evals interview questions

7. "What's wrong with using an LLM as a judge?" Judges can favor longer answers, the first option shown, or their own model family's text. Earn trust with a narrow rubric, a pinned judge model and prompt, and measured agreement with human labels before you rely on its scores.

8. "Offline evals look great, but users complain. What now?" Assume the test set has drifted from production. Sample real traces, sort failures into a taxonomy, add them to a regression set, and pair offline scores with online signals such as edits, retries and escalations.

9. "Your new prompt scores two points higher. Do you ship it?" Ask how many examples, how much run-to-run variance, and which slices moved. An average gain can hide a regression on a critical slice, and cost and latency count too.

How do I talk about evals in an interview? A worked example

Suppose the prompt is: "We're launching an LLM that drafts replies to refund emails for human support agents. How do you know it's ready?" Answer in this order:

1. Define failure first. Agree with the product owner on blockers: a promise outside refund policy, a wrong order amount, exposed personal data, or no escalation when a case needs a human.
2. Build a golden set from real, de-identified tickets, sliced by category and seeded with hard cases: partial refunds, angry customers, non-English emails, and an email containing injected instructions. Name a starting size (say, a few hundred cases) that grows from production failures.
3. Match each check to a grader. Code checks facts the system knows (order ID, refund amount, valid JSON); a rubric-based LLM judge, calibrated against human labels, scores tone and policy wording; humans review the riskiest slice.
4. Set the launch bar per slice. Zero blocker failures, plus thresholds for the rest. A strong average with one policy violation is not ready.
5. Compare candidates on the same set, prompt v1 against v2 or model A against B, with cost per ticket and p95 latency beside quality.
6. Plan the online loop. Track how often agents edit or discard drafts and how often tickets escalate; feed new failures into the regression set.

The interviewer grades whether you defined failure first, sliced the data, calibrated the judge, and tied offline scores to something users feel.

Cost, latency and coding questions

10. "p95 latency is 9 seconds and the bill doubled. Where do you start?" Measure first: split time into retrieval, time to first token and generation, and cost into input and output tokens. Then try streaming, trimming context, caching repeated prompts, routing easy requests to a smaller model, and parallel tool calls, with the eval set guarding quality.

11. "Write a client that calls a model API, retries on rate limits and returns validated JSON." Show backoff with jitter, timeouts, a retry cap, schema validation with a re-ask path, and clear errors. Then raise the harder case yourself: valid JSON with wrong content.

12. "Build a small RAG app for this take-home." Ship something small that works end to end, with a tiny eval set and a README of trade-offs. A modest system with honest measurements beats an ambitious one you cannot defend.

The production LLM failure behavioral question

"Tell me about an LLM feature that failed in production" shows whether you monitor, diagnose and prevent. Structure it as:

• Situation: what the feature did and for whom.
• Detection: how you found out. If a user noticed before your monitoring did, say so and what you changed.
• Diagnosis: the real root cause, such as a stale index, an unannounced model version change, a prompt edit that broke one slice, or a tool timeout the agent papered over.
• Fix: the short-term step (rollback, flag, guardrail) and the long-term one (a new eval case, an alert).
• Result: a number you measured yourself.

No production experience? Use an internship, research or capstone project with real users, never someone else's incident; "what did the trace show?" exposes borrowed stories.

Do AI startups sponsor H-1B?

Some do, and many have never filed a petition, because sponsorship is the employer's choice. Large tech employers usually have immigration counsel, E-Verify enrollment, Form I-983 experience and a routine for registering new hires each cap season. A seed-stage AI company may have none of these and decide case by case, so get its answer before you stop interviewing.

Postings rarely settle it. According to one vendor's read of 2,718 engineering postings at 136 companies (Match.dev, 2026-10-13), 88.1% said nothing about sponsorship and 0.4% explicitly offered it. Run our startup sponsorship vetting checklist before the final round.

Ask two startup questions early. An H-1B petition rests on a Labor Condition Application committing the employer to at least the required wage, so a mostly-equity offer with thin cash pay can be hard to sponsor; ask the company's immigration counsel how the salary compares with the prevailing wage. And STEM OPT requires a "bona fide employer-employee relationship" (USCIS), so check contractor or staffing setups with your DSO.

Already on H-1B? The new employer files its own Form I-129, and USCIS says "you may begin working for the new employer as soon as they properly file a non-frivolous Form I-129 petition," subject to the conditions on its H-1B page. Before you resign, ask the employer's attorney whether your cap history means no new registration.

Where does an "AI engineer" title land on H-1B wage levels?

Nowhere by itself. DOL wage levels attach to an occupation (a SOC code) and a work location, and the employer, usually through its immigration attorney, picks the occupation from the job's duties, not the title. An application-focused AI engineer might be a Software Developer (15-1252), a modeling-heavy one a Data Scientist (15-2051), and a research role a Computer and Information Research Scientist (15-1221) (BLS SOC list).

Each occupation has its own prevailing wages by area (OFLC Wage Search), so one salary can fall at different levels. Under the weighted selection final rule, in effect since 2026-02-27, a Level IV registration gets four lottery entries and Level I gets one. You do not pick a level: the job description must match the real work, and inflated duties are a misrepresentation risk. See how your title and duties affect the wage level.

What E-Verify check does STEM OPT require at a startup?

On its STEM OPT page (updated 2026-01-30), USCIS says the employer must be enrolled in E-Verify and "remain a participant in good standing with E-Verify, as determined by USCIS." It also needs a valid IRS Employer Identification Number, must sign the training plan on Form I-983, and must offer an opportunity "commensurate with those of similarly situated U.S. workers." A startup new to STEM OPT may need to enroll first; here is how to ask an employer to enroll in E-Verify.

One timing risk: E-Verify is funded only through 2026-12-11 under the current continuing resolution, and in the 2025 lapse it was offline from 2025-10-01 to about October 8-9. If enrollment or your start date falls after 2026-12-11, ask HR for its plan, unless Congress passes funding by then.

How should I answer the sponsorship question in the recruiter screen?

Truthfully, with the end date of your current work authorization: "I'm authorized to work on STEM OPT through [date], and I'll need H-1B sponsorship to continue after that." On post-completion OPT, mention a STEM OPT extension only after your DSO confirms your degree qualifies. Long loops also cost time: post-completion OPT allows 90 days of unemployment, and STEM OPT 150 days "which includes any days accrued while on post-completion OPT" (in effect, unchanged). Track your count with the SEVP unemployment counter guidance.

What you may hear about visa costs, and the status

Common misconception: "A startup has to pay $100,000 to sponsor me off OPT." Correction: as of 2026-11-01 the payment is not collected; courts blocked the agency policies behind Proclamation 11069 (in effect on paper until 2027-09-21), though DHS "still plans to collect the payment" if that order is lifted (USCIS I-129 page, updated 2026-10-30). If collection resumes, USCIS has said it does not apply to an F-1 to H-1B change of status granted inside the US; it would apply if that change is denied or the petition requests consular notification (USCIS H-1B page, updated 2026-09-21).

H-1B and STEM OPT claims an AI employer may raise, as of 2026-11-01

Vet the employer before the final round

1. Search the company in the E-Verify Employer Search; if it is missing, ask HR whether it enrolled under another legal or trade name.
2. Look up the legal entity in the USCIS H-1B Employer Data Hub (FY2009 through FY2026 Q4) for approvals and denials; no record means no decisions on file, not a refusal.
3. Ask the recruiter in writing whether the company will sign Form I-983 and register you in the next H-1B cap season.
4. Check the likely occupation and your metro in the OFLC Wage Search on flag.dol.gov.
5. Consult the company's immigration attorney, or your own, about the petition and its timing before you sign; questions about your OPT or STEM OPT status go to your DSO.

After you accept, report the new employer through the SEVP Portal or your DSO, as your DSO instructs; on STEM OPT a new employer also means a new Form I-983.

Common mistakes

• Prepping like an ML engineer. Derivations will not save an evals round you never practiced.
• Naming one right answer. "512 tokens" or "always use model X" without trade-offs reads as inexperience.
• Telling an incident that wasn't yours. Follow-ups find it.
• Hearing "we sponsor" as "we're ready." Confirm E-Verify enrollment and I-983 willingness separately.
• Repeating headlines. As of 2026-11-01 the $100,000 payment is blocked and the OPT fees are only proposed.
• Ignoring the unemployment clock. Every day without employer information counts toward the 90 or 150.

Frequently asked questions

What questions are asked in an AI engineer interview?

Most AI engineer loops mix practical coding around model APIs, an LLM system design round on a RAG pipeline or tool-using agent, an evals round on measuring quality, and a behavioral question about an LLM feature that failed in production. Training math comes up less than in an ML engineer loop. On OPT, STEM OPT or H-1B, expect a work-authorization question in the recruiter screen.

How is an AI engineer interview different from an ML engineer interview?

An ML engineer interview focuses on training and serving models, so it leans on ML math, feature pipelines and offline metrics. An AI engineer interview focuses on products built on foundation models, so it leans on retrieval, tools, evals, guardrails, and cost and latency. Many roles blend the two, so ask the recruiter which rounds are in the loop.

Do AI startups sponsor H-1B?

Some do, and many have never filed a petition, because sponsorship is the employer's choice. Check the USCIS H-1B Employer Data Hub for the company's approval and denial history and the E-Verify Employer Search for the enrollment STEM OPT requires. As of 2026-11-01, H-1B cap selection is wage-weighted (in effect since 2026-02-27), the $100,000 payment is blocked and not collected, and FY2028 registration has not been announced.

What is an LLM system design question?

It asks you to design a feature built on a large language model, such as a support assistant over company documents or an agent that acts through tools. Interviewers want requirements first, then retrieval, prompting, tool permissions, guardrails, evaluation, and cost and latency budgets. The strongest answers explain how you would know the system works.

How do I talk about evals in an interview?

Define failure with the product owner first, then describe a golden test set drawn from real cases and sliced by category, the right grader for each check (code, a calibrated LLM judge, or a human), and a launch bar per slice. Close with the online signals you would watch after launch and how new failures feed your regression set.

Sources

• USCIS, STEM OPT extension page (updated 2026-01-30)
• USCIS, H-1B Specialty Occupations (updated 2026-09-21) and Form I-129 (updated 2026-10-30)
• USCIS, H-1B Employer Data Hub and H-1B Electronic Registration Process
• E-Verify, Employer Search; DOL OFLC, Wage Search; BLS, 2018 SOC list
• Federal Register, weighted selection final rule, Proclamation 11069, H-1B fee NPRM, OPT fees NPRM
• CourtListener, California v. Mullin; Study in the States, unemployment counter
• Indeed Hiring Lab, Lightcast, NACE, Match.dev (single vendor)

How we checked this: on 2026-11-01 we read the USCIS, Study in the States, E-Verify, DOL OFLC and BLS pages above, confirmed each Federal Register document's type and dates through the Federal Register's public API, read the Indeed, Lightcast, NACE and Match.dev pages, and took case status from the court dockets as reviewed that day.

This is general information, not legal or tax advice; your own I-20, EAD, I-94 and notices, and your DSO or immigration attorney, govern your situation.

If you are working through AI engineer loops on OPT or STEM OPT and want help finding employers set up to sponsor, F1Jobs can talk it through with you.