What guardrails and human-in-the-loop controls do production agents need?
Guardrails constrain what an agent can do and catch bad behavior. Key layers: scoped permissions (least-privilege tools, read vs write separation), input/output validation (schema checks, content filters), and action approval, requiring human confirmation before high-stakes or irreversible actions (sending money, deleting data, emailing customers). Human-in-the-loop means inserting checkpoints where a person reviews or approves before execution; human-on-the-loop means monitoring with the ability to intervene. Other controls: iteration and budget caps to stop runaway loops, allowlists/denylists for tools and domains, rate limits, audit logging of every action, and sandboxed execution. The design principle is to match the level of autonomy to the reversibility and cost of actions: let agents act freely on cheap, reversible steps, but gate anything consequential behind a human or a hard rule.