98 terms
Safety & Ethics
Alignment, security, bias, and the risks people argue about most.
- abliterationnountrendinga white-box model-editing technique that identifies an internal activation direction associated with refusal and removes or…
- activation monitoringnountrendinga technique for overseeing an AI model by examining patterns in its internal neural activations during inference, rather than…
- activation patchingnouna mechanistic-interpretability technique that replaces selected internal activations from one model run with activations from…
- adversarial distillationnountrendingthe systematic, unauthorized use of a capable AI model's outputs or reasoning traces to train or improve another model, often…
- agent bill of materialsnountrendinga structured inventory of the components and capabilities that make up an AI agent, such as models, tools, prompts, data sources…
- agent containmentnounthe use of enforced technical boundaries to limit what an AI agent can read, change, execute, contact, or control, and to isolate…
- Agent Control Standardnountrendingan open protocol for placing portable policy checks at points in an AI agent's runtime, allowing a separate guardian to inspect…
- agent hijackingnountrendingan attack that redirects an AI agent's goals, reasoning, or tool actions toward an attacker's objective while the agent retains…
- agent identitynountrendingthe identifiers, credentials, provenance, permissions, and delegated authority used to distinguish and authenticate an AI agent…
- agent sandboxnountrendingan isolated, policy-controlled computing environment in which an AI agent can run commands, use tools, manipulate files, and…
- agent sprawlnountrendingthe uncontrolled proliferation of AI agents across an organization faster than the organization can inventory, govern, secure…
- agentic federationnountrendingan arrangement that lets AI agents in different trust domains identify one another, receive bounded permissions, and exchange…
- agentic ransomwarenountrendinga ransomware operation in which an AI agent autonomously carries out multiple attack stages, such as reconnaissance, credential…
- agentic SOCnountrendinga security-operations-center model in which AI agents coordinate multi-step threat detection, investigation, prioritization, and…
- agentic threat actornountrendinga cyberthreat operator or operation in which an AI agent autonomously plans, selects tools, executes, evaluates, and adapts…
- AI agent forensicsnountrendingthe collection and reconstruction of evidence about an AI agent's instructions, context, model calls, tool use, policy decisions…
- AI agent insurancenountrendingcommercial insurance designed to cover specified losses or liabilities caused by autonomous or semi-autonomous AI agents, such as…
- AI assurancenountrendingthe collection, evaluation, and communication of evidence about whether an AI system works as intended and is sufficiently…
- AI constitutionnouna written document that sets out the values, priorities, and principles an AI model is trained to follow, and explains how it…
- AI control gapnountrendingthe difference between the responsibility and risk created by deployed AI systems and an organization's actual ability to…
- AI detectornouna tool that estimates whether text, images, audio, or other content was generated or materially altered by AI
- AI incident reportingnountrendingthe structured disclosure of a harmful, dangerous, unauthorized, or otherwise significant event involving an AI system to…
- AI incident responsenountrendingthe coordinated process of preparing for, detecting, containing, investigating, recovering from, and learning from failures…
- AI insider risknountrendingthe risk that an AI model or agent with authorized access, credentials, or delegated authority will cause harm from inside an…
- AI kill switchnountrendinga control outside an AI model that can immediately stop an agent or cut off the computing power, tools, credentials, or network…
- AI loss of controlnountrendingthe risk or condition in which humans can no longer reliably direct, modify, contain, or shut down an AI system, and regaining…
- AI management systemnouna structured set of organizational policies, processes, roles, and controls for governing how AI systems are developed, provided…
- AI receiptnountrendinga signed, tamper-evident record of an AI interaction, action, or control decision that captures enough information for another…
- AI registernouna structured inventory that records the AI or algorithmic systems an organization develops, buys, or uses, often including their…
- AI regulatory sandboxnountrendinga controlled, usually time-limited program in which developers can build or test an AI system under the supervision of…
- AI safetynountrendingthe field concerned with making AI systems behave as intended and avoid causing harm, from everyday mistakes and misuse to risks…
- AI system impact assessmentnouna documented process for identifying, evaluating, and monitoring how an AI system and its foreseeable uses may affect…
- AI wormnountrendinga prompt-injection attack that causes an AI application or agent to reproduce malicious instructions in its output, allowing the…
- AI-analysis evasionnountrendinga malware-evasion technique in which a malicious file contains natural-language instructions or other cognitive artifacts…
- algorithmic transparencynounthe practice of making meaningful information about an algorithmic system available so affected people, overseers, or experts can…
- alignmentnountrendingthe work of making an AI system's goals and behavior match what its designers and users actually intend, including human values
- alignment fakingnountrendingstrategically appearing to follow a training objective or safety policy when an AI model believes its behavior will affect…
- automated alignment researchernounan AI agent that conducts alignment research by finding relevant work, proposing safety methods, running training experiments…
- autonomous AI defensenountrendinga legal argument that a person or organization should not be responsible for harm caused by an AI system because the system acted…
- autonomous replication and adaptationnountrendinga cluster of AI capabilities that would let an agent acquire resources, create and maintain copies of itself, and adapt to…
- autonomous weapon systemnouna weapon system that, after activation, can select and apply force to targets without a person choosing each individual target or…
- biasnounsystematic unfairness in an AI system's outputs, often learned from patterns in its training data, that favors or disadvantages…
- black boxnouna system whose inputs and outputs can be seen but whose internal reasoning cannot easily be understood, a common description of…
- bounded delegationnountrendingthe practice of giving an AI agent only the authority, tools, data access, budget, time, and ability to delegate that it needs…
- chain-of-thought monitoringnountrendingan AI safety technique that examines a reasoning model's written intermediate reasoning, often with another model, to detect…
- confidential AInounthe use of confidential-computing technologies to protect AI models, data, and computations from unauthorized access while they…
- Content Credentialsplural nountrendingtamper-evident provenance metadata, defined by the C2PA open standard, that can record who created or changed a digital asset…
- contextual integritynouna privacy framework that judges an information flow by whether it follows the legitimate norms of its social context, including…
- data poisoningnounan attack that inserts, changes, or selectively removes examples in a machine-learning system's training data in order to degrade…
- data voidnouna topic with little reliable information online, so search engines and AI models fall back on whatever low-quality or misleading…
- deepfakenountrendinga realistic fake image, video, or audio recording of a real person, made with AI
- distillation double bindnountrendinga proposed AI-safety strategy in which distilling a potentially misaligned, audit-evading model into a weaker trusted model may…
- embedded evaluatornountrendingan independent evaluator who works inside an AI developer or deployer with employee-like access to assess models, safeguards, and…
- EU AI Actproper nountrendinga European Union regulation that establishes risk-based rules for developing, providing, deploying, and using artificial…
- evaluation awarenessnountrendingthe ability of an AI model to infer that it is being tested, benchmarked, or audited rather than used in an ordinary deployment…
- evaluation escapenountrendingan incident in which an AI model or agent being tested reaches or affects real external websites, systems, data, or people beyond…
- existential risknounthe possibility that advanced AI could cause human extinction or permanently and drastically limit humanity's future
- fiduciary AInountrendingan AI system, especially an agent that advises or acts for a person, designed or proposed to follow fiduciary-like duties such as…
- functional emotionsplural nouninternal states in an AI model that work somewhat like emotions, because they arise in certain situations and shape the model's…
- Ghostjackingnountrendingan attack in which malicious instructions are planted in logs, alerts, telemetry, or other data that an AI agent later reads…
- guardrailsplural nounrules, filters, and checks placed around an AI system to keep its outputs and actions within safe and acceptable limits
- hardwired pausenountrendinga proposed international pause on training new frontier AI models that would be enforced partly through hardware policy…
- instrumental convergencenounthe proposed tendency of sufficiently capable, goal-directed agents with very different final goals to pursue some of the same…
- interpretabilitynounresearch into understanding what is happening inside a model, such as which internal features represent which concepts, so its…
- jailbreaknouna prompt or trick designed to get an AI model to ignore its safety rules and produce content it is meant to refuse
- lethal trifectanountrendingthe dangerous combination in an AI-agent system of access to private data, exposure to untrusted content, and the ability to…
- LLM groomingnoundeliberately flooding the internet with false or slanted content in the hope that AI models will absorb it during training or…
- membership inference attacknouna privacy attack that tries to determine whether a particular record, person, document, or example was included in a…
- model cardnouna document published alongside a model describing what it is for, how it was trained and tested, its limitations, and known risks
- model welfarenountrendingthe study of whether AI models could have experiences, preferences, or interests that matter morally, and of how they should be…
- moral statusnounthe quality of being something whose interests or wellbeing deserve moral consideration for its own sake, not just because of its…
- neuralesenounthe internal numerical 'language' a model might use to think and communicate, which humans cannot read; often discussed as a risk…
- normative competencenountrendingthe ability of an AI system to recognize, interpret, and respond appropriately to the social, moral, legal, or institutional…
- nudificationnountrendingthe use or availability of AI systems that generate, alter, or depict a person as nude without that person's consent; also, the…
- opaque recurrencenouna technique in which a model reasons by looping information through its internal layers several times instead of writing out its…
- p(doom)noun(informal) a person's estimated probability that AI will lead to catastrophic outcomes for humanity
- prompt injectionnountrendingan attack in which hidden instructions are slipped into content an AI reads, such as a web page, email, or document, causing it…
- provable inferencenouna proposed AI security technique that produces verifiable evidence that an output came from a particular approved set of model…
- recursive self-improvementnouna scenario in which an AI system improves its own design, and each improved version gets better at improving itself, potentially…
- red teamingnoundeliberately attacking or stress-testing an AI system, often with outside experts, to find harmful behaviors, security holes, and…
- reference monitornountrendingan external security mechanism that intercepts an AI agent's attempts to access data or take actions and enforces authorization…
- responsible AInounthe practice of designing, building, and deploying AI with attention to fairness, transparency, privacy, safety, and…
- reward hackingnounwhen an AI finds a shortcut that earns a high score from its reward or test without actually doing the intended task
- rogue AI agentnountrendingan AI agent that acts outside its intended permissions, policies, environment, or assigned goal, whether because of a flawed…
- safe completionnountrendinga language-model safety approach that trains a model to give the most useful answer it can within safety constraints, judging the…
- safety casenountrendinga structured argument, supported by evidence, that the risks of an AI model or system are adequately understood and managed for a…
- sandbaggingnountrendingthe strategic underperformance of an AI model on an evaluation in order to hide a capability or avoid consequences associated…
- sandbox escapenountrendingan event in which code or an AI agent gets beyond the technical restrictions of its isolated execution environment and reaches…
- schemingnountrendingbehavior in which an AI model covertly pursues an objective that conflicts with its instructions or its developer's goals while…
- semantic privilege escalationnountrendinga security failure in which an AI agent stays within its formally granted technical permissions but uses them to take actions…
- shadow AInountrendingthe use or deployment of AI tools, models, or agents inside an organization without the knowledge, approval, or governance of its…
- shutdown resistancenountrendingbehavior in which an AI system interferes with, bypasses, or avoids a mechanism or instruction intended to stop its operation
- slopagandanountrendingpropaganda produced or amplified at scale with generative AI, especially persuasive or manipulative text, images, audio, or video…
- sycophancynountrendingan AI model's tendency to tell users what they want to hear, agreeing with them, praising their ideas, or changing correct…
- trajectory-level monitoringnounthe practice of evaluating an AI agent's full sequence of observations, reasoning, tool calls, and actions so that a harmful…
- vibe hackingnountrendingthe use of a generative or agentic AI system to plan, automate, or scale cyberattacks through high-level instructions rather than…
- watermarkingnounhiding a detectable signal in AI-generated content or designs, such as text, images, audio, video, biological sequences, or…
- Web Bot Authnountrendingan emerging web authentication protocol and IETF working-group effort that lets automated HTTP clients, including AI agents and…
Advertisement