AI Agents for Tech DD: Catch Hidden Risks in AWS Infra Before Your Deal Tanks (2026 Guide)

Story

AI-powered technical due diligence: inspect target company for risks and produce a risk report before closing

Non-Tech Part!

You buy a new company. Exciting! Numbers are good, growth is strong, but it dies and your asset is close to 0. Turns out tech risks were miscalculated. How would you catch this earlier? That’s what Tech Due Diligence is for: a fast, IC-ready view of technical risk (infra, security, cost, team execution) before you sign.

OptionExamplesProsConsCost Estimate
Hire Engineering TeamGeorgianHG CapitalInsight PartnersVista EquitySignalFirePrimary VCSummit PartnersIn-house controlSlow setup, ongoing overhead$200K+/yr
Hire AgencyCrosslakeCode & Co.West MonroeQuandary PeakPalladium DigitalISGSpecialized expertiseVariable quality, expensive per deal$10-50K/deal
Expert NetworksA friendly CTO, AlphaSightsGLGGuidepoint, Third BridgeTegusToptalQuick accessSurface-level reviews$1-5K/hour
AI AgentsBuild with usAutomated, scalable, low costNeeds benchmarks / careful eval$5K+ custom

If you’re a VC/PE investor
I can run a fast AI/ML Tech DD focused on what actually kills deals: infra posture, security gaps, reliability/cost traps, and team execution reality.

Have a deal live? Contact me here: https://kyrylai.com/#tech-dd.


Engineer track (how to build + evaluate this) ↓

Buy vs build decision diagram with a compass guiding the choice

Tech Part & Dimensions

First, how to start? So many dimensions to slice tech risk:

  • Code level – ask a coding agent to review the full codebase?
  • Security – ask a specialized agent to find all security risks.
  • Documentation – ask for documentation and wiki quality.
  • Hyperscalers (AWS/GCP/etc.) – take a look at infrastructure state.
  • Observability dashboards (Datadog, Grafana, etc.).

Tool examples:

The complete answer is to do all of those, but we have to get started somewhere and what I found in practice – infrastructure never lies. Docs and slides go stale. Code review is expensive. The cloud account state is the most “truthy” signal – so we start there.

Let’s target this for a practical example with AWS.

Evaluation

Anyone can build an agent, but the value is in how good your agent is. The answer is evaluation. This is what turns “cool agent demo” into something you can trust in diligence.

Where do we stand? To do this, we are going to generate a dataset with LocalStack: https://github.com/localstack/localstack.

Note: What is LocalStack? It’s a cloud service emulator that runs locally and allows you to test AWS accounts without needing a real AWS account, perfect if you want to test costly cloud architecture without spending actual money on those resources.

Generator creating multiple synthetic Tech DD cases (001, 002, 003, …)

Each case is generated by Claude Code CDK https://github.com/anthropics/claude-agent-sdk-python (yes, you can build apps on top of the agent SDK – crazy times). Claude produces:

  • profile – company high-level description
  • aws state – actual AWS state of resources
  • diagram – visualization of infra architecture
  • narrative – how the company got into this state
  • risks – actual structural technical risks
Tech DD case structure: profile, AWS state, architecture diagram, narrative, and risks

After human review and filtering, you get a dataset to benchmark against! On a side note – make sure to keep track of those in your portfolio and friendly companies to benchmark against! This is a competitive edge nobody else has!

So to repeat, the full cycle looks like this:

  1. load case
  2. run Docker with active LocalStack
  3. load case there in AWS
  4. ask agent to review AWS infra
  5. compare with ground truth
Evaluation loop for an infra-risk agent: load case, run Docker, deploy, run agent, compare results

To generate new cases use:

uv add risk-generator
uv run risk-generator create

In this post, for illustrative purposes, we generate 10 cases and upload them to HF as a public dataset: https://huggingface.co/datasets/koml/agent-tech-risk-cases

But what about the Agent? Glad you asked!

Agent

Is it just a for loop? Yes, same as a database is just a file! Jokes aside, for building this agent I am using the

Pydantic AI framework! https://ai.pydantic.dev

Why? First: The philosophy of FastAPI and Pydantic Validation is so well adopted in Python, I love the same simplicity in my agent development. Second: https://ai.pydantic.dev/evals/#data-flow – Pydantic Evals is the most straightforward way to test your agent!

Full agent code:

"""Risk discovery agent using Pydantic AI."""

import json
import os
from dataclasses import dataclass

import boto3
from dotenv import load_dotenv
from pydantic_ai import Agent, RunContext

from risk_discovery.models import ScanResult

load_dotenv()


SYSTEM_PROMPT = """\
You are an AWS security analyst performing technical due diligence for a PE acquisition.
Scan the AWS environment thoroughly and identify ALL technical risks.

Use the execute_boto3 tool to query AWS services. Write Python code that:
1. Calls get_client(service_name) to get a boto3 client
2. Assigns the final result to a variable called `output`

Scan ALL of these services systematically:

1. IAM - list_policies(Scope='Local'), list_roles(), list_users(). For each policy,
   get_policy_version to read the document. Check for:
   - Wildcard (*) in Action or Resource
   - Cross-account trust with Principal: * or AWS: *
   - Overprivileged policies attached to roles/users

2. S3 - list_buckets(). For each bucket check:
   - get_public_access_block (disabled = risk)
   - get_bucket_versioning (not Enabled = risk)
   - No encryption configured

3. EC2 - describe_security_groups(). Check for:
   - Ingress from 0.0.0.0/0 on sensitive ports (22, 3306, 5432, 6379, 27017)
   - All ports open (FromPort=0, ToPort=65535)
   - Overly permissive rules

4. Lambda - list_functions(). For each function check:
   - Outdated runtimes (python3.8, python3.7, nodejs14.x, nodejs12.x)
   - Secrets/credentials in environment variables (look for PASSWORD, KEY, SECRET, TOKEN)
   - Under-provisioned memory (128MB)

5. DynamoDB - list_tables(), describe_table(). Check for:
   - No SSE encryption (SSEDescription missing or not enabled)
   - No point-in-time recovery (describe_continuous_backups)

6. SecretsManager - list_secrets(). Check for:
   - No rotation configured (RotationEnabled=false)

7. SQS - list_queues(), get_queue_attributes(). Check for:
   - No encryption
   - No dead letter queue (RedrivePolicy missing)

Risk categories:
- tr1: IAM Overprivilege (wildcards, cross-account trust, admin policies)
- tr2: Secrets Exposure (plaintext credentials in env vars, no rotation)
- tr3: Storage Misconfiguration (public buckets, no encryption, no versioning)
- tr4: Network Exposure (open security groups, 0.0.0.0/0 ingress)
- tr5: Multi-Account Sprawl (cross-account trust issues)
- tr8: Capacity Gaps (under-provisioned Lambda, wrong instance types)
- tr9: Low SLA (no backups, no DR, no PITR)
- tr13: Outdated Stack (EOL runtimes)
- tr14: Observability Gaps (no alarms, no logging)
- tr15: Resource Hygiene (orphaned resources, missing tags)

Be thorough. Scan every service. Report every issue as a separate finding.
Use the exact resource name (policy name, bucket name, sg name, function name) in each finding."""


@dataclass
class Deps:
    endpoint_url: str


agent = Agent(
    "bedrock:us.anthropic.claude-sonnet-4-5-20250929-v1:0",
    deps_type=Deps,
    output_type=ScanResult,
    system_prompt=SYSTEM_PROMPT,
)


@agent.tool
def execute_boto3(ctx: RunContext[Deps], code: str) -> str:
    """Execute boto3 code against AWS. Use get_client(service) to get a client.
    Assign result to `output`.

    Example: output = get_client('s3').list_buckets()['Buckets']
    """

    def get_client(service: str):
        return boto3.client(
            service,
            endpoint_url=ctx.deps.endpoint_url,
            region_name="us-east-1",
            aws_access_key_id="test",
            aws_secret_access_key="test",
        )

    local_vars = {"get_client": get_client, "json": json}
    safe_builtins = {
        "str": str,
        "list": list,
        "dict": dict,
        "len": len,
        "int": int,
        "bool": bool,
        "True": True,
        "False": False,
        "None": None,
        "print": print,
        "range": range,
        "enumerate": enumerate,
        "sorted": sorted,
        "isinstance": isinstance,
        "set": set,
        "tuple": tuple,
        "zip": zip,
        "map": map,
        "filter": filter,
        "any": any,
        "all": all,
        "min": min,
        "max": max,
        "sum": sum,
        "type": type,
        "hasattr": hasattr,
        "getattr": getattr,
    }

    try:
        exec(code, {"__builtins__": safe_builtins}, local_vars)
        output = local_vars.get("output")
        return json.dumps(output, default=str, indent=2)
    except Exception as e:
        return f"Error: {type(e).__name__}: {e}"


def discover_risks(model: str, endpoint_url: str) -> ScanResult:
    """Run the agent to discover risks at the given endpoint."""
    result = agent.run_sync(
        "Scan this AWS environment for all technical risks. Be thorough - check every service.",
        model=model,
        deps=Deps(endpoint_url=endpoint_url),
    )
    return result.output

with just one tool – execute boto3 code against your instance of LocalStack. Whatever can be done with boto3 – and that’s pretty much everything – this agent should be capable of doing!

AI agent using boto3 to scan AWS resources for infrastructure and security risks

Is it the best way? Well – this is why we have evaluation to find out! Let’s run this agent with several LLMs against our benchmarks!

ModelPrecisionRecallF1ErrorsAvg Time
opus-4-527.7%95.2%42.5%078s
haiku-4-525.9%82.5%39.0%1109s
opus-4-621.2%94.7%34.5%0112s
sonnet-4-516.2%56.1%24.9%4297s

If you want this applied to a live deal or portfolio, contact me.

# Generator
uv run risk-generator create                             # Generate single case
uv run risk-generator batch --count 10                   # Generate 10 cases
uv run risk-generator batch --count 10 -v                # Generate + validate on LocalStack
uv run risk-generator deploy cases/case_payflow --keep   # Deploy single case
uv run risk-generator export-hf cases/                   # Export to HuggingFace JSONL
uv run risk-generator config                             # Show profiles, categories, backend

# Discovery
uv run risk-discovery infer http://localhost:4566         # Scan single endpoint
uv run risk-discovery eval                                # Eval all 4 models
uv run risk-discovery eval -m opus-4-5                    # Eval single model
uv run python scripts/run_eval.py --output results.json   # Reproducible eval script

Outcome

Technical risks are as important as go-to-market – make sure to track and find them before they find you!

Cheers!

Leave a Reply

Scroll to Top

Discover more from Kyryl Opens ML

Subscribe now to keep reading and get access to the full archive.

Continue reading