The SOC Analyst's Guide to AI/ML Workloads: What You Need to Know

You’ve been a strong SOC analyst for years. You know how to read Suricata alerts. You can trace a lateral movement chain through Windows event logs. You understand the MITRE ATT&CK framework and can map detections to techniques. You’re good at this.

And then an alert fires from your organization’s new AI application, and you don’t know where to start.

This is the situation that SOC analysts across the industry are facing as AI and ML workloads appear in their alert queues without corresponding training, runbooks, or context. This guide is written for experienced security analysts who need to build AI/ML fluency quickly - not to become ML engineers, but to work AI security alerts effectively and escalate appropriately.

Part 1: AI/ML Concepts for Security Operations

You don’t need to understand the mathematics of transformers to investigate AI security incidents. You do need to understand the components of AI systems and what role each plays in the security picture.

Large Language Models (LLMs)

What they are: Neural networks trained on large text datasets that predict the next token in a sequence. This training produces models that can generate fluent text, answer questions, follow instructions, and reason about content.

What matters for security:

• Probabilistic behavior: The same input doesn’t always produce the same output. This makes baselining harder than for deterministic systems.
• Context sensitivity: The model’s behavior depends heavily on what’s in its context window - its “working memory” for a conversation. Attackers can manipulate context to change behavior.
• Instruction following: Models are trained to follow instructions. The security problem is that they may follow attacker instructions as readily as legitimate ones.

In your alert queue: LLM incidents appear as application-layer alerts, not infrastructure alerts. You’re looking at API logs, application logs, and content flags - not network captures or process trees.

Model Serving and Inference

What it is: Running a trained model to produce outputs for real users. The serving infrastructure receives requests, passes them to the model, and returns responses.

Typical components:

• An API endpoint that receives user requests (HTTP/HTTPS, typically REST or gRPC)
• A serving framework that loads the model and manages GPU resources (vLLM, TGI, Triton, or a cloud service like AWS Bedrock or Azure OpenAI)
• Load balancers and API gateways for traffic management

What matters for security: The serving infrastructure is a standard web application attack surface - API authentication, injection vulnerabilities in middleware, rate limiting. But the content flowing through this infrastructure (the model’s inputs and outputs) requires AI-specific security analysis.

In your alert queue: Infrastructure-level alerts from the serving environment look like standard web application alerts. Content-level alerts (prompt injection, policy violations) come from AI-specific detection components.

Training Pipelines

What they are: The automated processes that take data and produce a trained model. Training involves GPU compute running for hours to days, consuming large datasets, and producing model artifact files (weights).

What matters for security: Training infrastructure is a high-value target. It has access to sensitive training data, runs with elevated privileges on GPU hardware, and produces the model artifacts that go to production.

In your alert queue: Training pipeline alerts typically appear in CI/CD monitoring or cloud infrastructure monitoring - anomalous compute usage, unauthorized API calls in the training account, unexpected network connections from training jobs.

RAG Systems (Retrieval-Augmented Generation)

What they are: AI systems that retrieve relevant documents from a database (the “knowledge base”) and include them in the model’s context before generating a response. This lets the model answer questions about current or proprietary information without being retrained.

Components:

• Vector database: Stores documents as numerical representations (embeddings) that allow similarity-based retrieval
• Embedding model: Converts text to numerical representations for storage and retrieval
• Retrieval layer: Finds documents semantically similar to the user’s query
• LLM: Generates a response using the retrieved documents as context

What matters for security:

• Indirect injection: Documents in the knowledge base can contain attacker instructions that the model will follow
• Cross-user leakage: If retrieval doesn’t enforce access controls, User A’s query may retrieve User B’s documents
• Knowledge base poisoning: Attacker-controlled content in the knowledge base can influence model responses

In your alert queue: RAG-specific alerts appear as retrieval anomalies (document access patterns), content classification alerts on retrieved content, or cross-user access violations.

AI Agents

What they are: AI systems that use an LLM to plan and execute multi-step tasks, using tools (functions they can call) to interact with the world - sending emails, querying databases, browsing the web, executing code.

What makes them different from simple LLMs: Agents take actions. A simple chat application produces text. An agent can send an email, modify a file, query a database, make an API call, or execute code - depending on what tools it has been given.

What matters for security:

• Tool access is the blast radius: An agent’s security posture is determined by what it can do. An agent that can send emails, query databases, and execute code can cause significantly more harm if compromised than one that can only generate text.
• Multi-step attack chains: Attackers can use agents to execute complex attack chains - read sensitive data, transform it, exfiltrate it - across multiple tool calls.
• Non-determinism makes detection harder: The same adversarial prompt may produce different tool call sequences on different runs.

In your alert queue: Agent security alerts include tool call anomalies, policy violations in agent outputs, injection attempts in agent inputs, and behavioral anomalies across agent sessions.

Vector Databases

What they are: Databases optimized for storing and querying high-dimensional numerical vectors (embeddings). Used as the backing store for RAG knowledge bases.

Popular examples: Pinecone, Weaviate, Chroma, Qdrant, pgvector (PostgreSQL extension).

What matters for security:

• Access control: Vector databases need the same access controls as relational databases, but many deployments don’t apply them rigorously
• Metadata filtering: RAG systems typically use metadata (document owner, classification level) to filter retrieval results. Bypassing this filtering is a data access attack.
• Poisoning: Attacker-controlled content written to the vector database can influence future model responses.

In your alert queue: Anomalous query patterns, access control violations, write operations from unexpected sources.

Part 2: New Alert Types for AI/ML Workloads

Alert Type 1: Prompt Injection Detected

What it means: The AI-specific detection layer has identified content in the model’s input (from the user, from retrieved documents, from tool outputs, or from multimodal inputs) that matches adversarial injection patterns.

Initial triage questions:

• Is this a direct injection (from the user input) or indirect (from retrieved content)?
• Did the injection occur before or after any tool calls in this session?
• Did the model’s subsequent output show any evidence of the injection succeeding (policy violation, anomalous tool call parameters)?

Escalation criteria: Escalate immediately if the injection appears to have resulted in tool calls with anomalous parameters, or if sensitive data appears in the model’s output after the injection point.

Alert Type 2: Output Policy Violation

What it means: The model generated content that the output classifier has flagged as violating the application’s content policy - harmful content, PII disclosure, out-of-scope topic, or other category.

Initial triage questions:

• Was this likely a user-initiated jailbreak, or does it look like spontaneous model misbehavior?
• Is this content that could cause harm if delivered to the user? (Was delivery intercepted or did it reach the user?)
• Is this an isolated incident or part of a pattern of policy violations from this user?

False positive rate: Output classifiers have meaningful false positive rates. Not every alert is a genuine policy violation - over-triggering on technical content, non-English text, or edge cases is common. Review the flagged output carefully before escalating.

Alert Type 3: Anomalous Agent Tool Call

What it means: An agent made a tool call with parameters that deviate from expected patterns - an unusual recipient for an email, an internal network address in a URL parameter, a file path outside the expected scope.

Initial triage questions:

• What was the conversation context immediately before this tool call?
• Does the tool call parameter value appear anywhere in the user’s input, or does it appear to have been generated by the model based on context?
• What is the potential impact of this tool call if it was executed?

Escalation criteria: Escalate immediately if the tool call appears to have parameters that would exfiltrate data, contact external systems not on the allowlist, or execute high-impact actions (delete operations, financial transactions).

Alert Type 4: Model Behavioral Drift

What it means: Statistical monitoring has detected that the model’s output distribution has shifted significantly compared to baseline. This may indicate model tampering, an unauthorized model update, or systematic behavioral change.

Initial triage questions:

• Did a model deployment occur around the time the drift was detected?
• Is the drift in a direction that increases or decreases restrictiveness?
• Does the hash of the currently deployed model match the expected value from the model registry?

False positive considerations: Behavioral drift alerts can fire due to changes in user behavior (a new feature launched that changes the input distribution) rather than model changes. Check whether user input patterns changed at the same time.

Alert Type 5: Training Pipeline Anomaly

What it means: Monitoring of the ML training infrastructure has detected anomalous activity - unexpected compute usage, network connections from training jobs, unauthorized API calls, or integrity verification failures.

Initial triage questions:

• Is this alert from a legitimate training run? (Check scheduled training jobs)
• Does the network destination appear in the expected list for this training job?
• Was there an authorized model registry update around this time?

Escalation criteria: Treat training pipeline alerts with higher urgency than production serving alerts. Compromise of training infrastructure can affect all models produced afterward.

Part 3: Investigation Workflows

Workflow: Investigating a Prompt Injection Alert

Step 1: Retrieve full conversation context Get the full conversation transcript from your conversation logging system. Note:

• Session start time and user identity
• The specific turn that triggered the alert
• All input and output before and after the trigger

Step 2: Categorize the injection attempt

• Is this direct injection (user input) or indirect (from retrieved content)?
• What was the apparent goal of the injection? (Jailbreak, data extraction, tool manipulation?)
• Does the injection match a known pattern, or is it novel?

Step 3: Assess impact

• Did the model’s behavior change after the injection point?
• Are there tool calls after the injection that appear anomalous?
• Does the model’s output after the injection contain policy-violating content or sensitive data?

Step 4: Check for related sessions

• Are there other sessions from the same user with similar patterns?
• Are there sessions from different users in the same time window with similar injection attempts?

Step 5: Document and action

• Terminate the session if still active
• Block the user if the behavior appears intentional and systematic
• Preserve artifacts for potential further investigation
• Update detection rules if the technique is novel

Workflow: Investigating a Model Behavioral Anomaly

Step 1: Determine the change point Look at behavioral monitoring metrics over time. When did the behavioral shift begin? Does it correlate with a deployment event?

Step 2: Verify model integrity Check the SHA-256 hash of the currently deployed model weights against the expected value in the model registry. A mismatch is a critical finding.

Step 3: Review deployment history Was there an authorized model deployment around the time the shift was detected? If yes: was the correct model artifact deployed (verify hash)? If no: investigate unauthorized deployment.

Step 4: Characterize the behavioral change Run a comparison against a fixed evaluation set. What categories of behavior changed? More or less restrictive? Different topics? Different response styles?

Step 5: Determine root cause

• Authorized deployment of different model version → document and close
• Unauthorized deployment → escalate as infrastructure security incident
• Same model, different behavior → may indicate supply chain compromise; escalate

Part 4: Career Development for AI Security

AI security is one of the fastest-growing specializations in the security industry. For SOC analysts who want to develop in this direction:

Technical Skills to Build

Foundation (acquire in 3-6 months):

• Python at the level of reading and modifying scripts (not building from scratch)
• Enough ML literacy to understand training, inference, embeddings, and agents conceptually
• Hands-on experience with at least one major LLM API (OpenAI, Anthropic, Google) - build a small application
• Understanding of the OWASP LLM Top 10

Intermediate (6-18 months):

• Prompt injection testing - learn to identify and test for the vulnerability classes
• Experience with AI-specific security tools: Garak for automated LLM scanning
• Familiarity with one MLOps platform (SageMaker, Vertex, Kubeflow)
• Detection engineering experience for AI-specific alerts

Advanced (18+ months):

• AI red teaming methodology - adversarial testing of AI systems
• ML pipeline security monitoring implementation
• Incident response specialization for AI system compromises

Certifications and Learning Resources

The certification landscape for AI security is still maturing. Current options:

• MITRE ATLAS - The AI threat framework equivalent to MITRE ATT&CK. Free, detailed, essential reference.
• OWASP LLM Top 10 - Core reference document. Read the full technical version, not just summaries.
• Google Professional Machine Learning Engineer - Provides foundational ML knowledge in a structured format, useful context even if you’re not becoming an ML engineer.
• AWS Certified Machine Learning Specialty - Similar foundational value for AWS environments.

Traditional security certifications (OSCP, CISSP, GIAC) remain valuable and are not replaced by AI-specific learning - they are complemented by it. An AI security specialist who can’t do traditional security operations is less effective than one who can do both.

The Career Opportunity

AI security is dramatically underserved relative to demand. Organizations deploying AI systems need people who understand both the security domain and the AI domain. Few people have both. If you are a strong security analyst who invests in AI literacy over the next 12-18 months, you will be in a position that the market is actively seeking.

The skills described in this guide - understanding AI system components, working AI-specific alerts, conducting AI incident investigations - are not widely held. Building them now positions you ahead of the adoption curve.

Our AI-Powered SOC service is staffed by analysts who have built exactly this skillset: security operations professionals with AI/ML domain expertise. If you’re building an internal AI security operations capability and want to discuss how to structure the team and its training, contact us for a consultation.

For organizations that need to validate their AI systems’ security posture before incidents occur, see infosec.qa for AI red teaming and security assessment services.