Competitor Intelligence Agent: 8 Hours to 5 Minutes

The Problem

Manual competitive analysis takes 8 hours and is stale on arrival

Competitive analysis across pricing, features, and positioning required a full business day of manual research. An analyst would visit each competitor’s website, extract pricing tiers, catalog feature lists, read positioning copy, and compile findings into a document.

By the time the analysis was compiled, competitor pages had already changed. The process was too slow to run regularly, which meant competitive intelligence was always 2-4 weeks out of date.

• 8 hours per analysis: manually visiting 8-12 competitor sites, extracting data from each
• Stale on delivery: competitors update pricing and features weekly; by the time analysis is done, some data is already outdated
• Inconsistent methodology: different analysts captured different data points, making cross-period comparison unreliable
• No structured output: free-form documents that couldn’t be diffed against previous analyses to spot changes
• Low frequency: too expensive to run more than monthly, missing rapid competitor moves

The Architecture

Multi-agent CrewAI pipeline with parallel execution

We built a CrewAI multi-agent system with three specialized agent roles, each responsible for one dimension of competitive analysis. Agents run in parallel across all competitors, with Pydantic schemas enforcing output structure at every step.

Agent 1: Pricing Extractor

The Pricing Extractor navigates competitor pricing pages, identifies tier structures, extracts per-unit costs, and normalizes pricing into a comparable format. It handles common obstacles:

• Dynamic pricing pages: uses headless browser rendering for JavaScript-heavy sites
• Enterprise “Contact Sales” tiers: flags these as null with a confidence annotation
• Free tier detection: identifies trial periods, feature gates, and usage limits
• Currency normalization: converts all pricing to USD monthly equivalent

The agent outputs a PricingReport Pydantic model with fields for each tier: name, monthly_price_usd, annual_discount_pct, feature_gates, and usage_limits.

Agent 2: Feature Analyzer

The Feature Analyzer catalogs each competitor’s feature set and maps features to a normalized taxonomy. This enables apples-to-apples comparison even when competitors use different terminology for equivalent capabilities.

The taxonomy covers 6 feature categories: core platform, integrations, security/compliance, analytics, support, and API/developer tools. Each feature is scored on availability (yes/no/partial) and maturity (beta/GA/enterprise-only).

Agent 3: Positioning Analyst

The Positioning Analyst reads marketing copy, case studies, and blog content to extract messaging patterns: target persona, value proposition, differentiators, and proof points. It uses Claude to classify positioning along three axes:

• Market segment: SMB / mid-market / enterprise
• Technical depth: developer-first / business-user / hybrid
• Competitive angle: cost leader / feature leader / vertical specialist

Output Validation

All three agents write their results to Pydantic models. A final merge step combines the three reports into a CompetitiveIntelligenceReport with per-competitor sections and a cross-competitor comparison matrix.

class CompetitorProfile(BaseModel):
    name: str
    website: str
    pricing: PricingReport
    features: FeatureReport
    positioning: PositioningReport
    last_analyzed: datetime
    changes_since_last: list[str] = []

The changes_since_last field diffs against the previous analysis run, surfacing pricing changes, new features, or messaging shifts without manual comparison.

Results

Quantified impact on competitive intelligence

• >95% time reduction: from 8 hours of manual research to under 5 minutes for a complete competitive landscape analysis
• Structured, diffable output: Pydantic-validated JSON reports enable automated change detection between runs
• 92% first-pass validation rate: Pydantic schemas catch extraction errors before they reach the final report
• Parallel execution: all three agents run concurrently across all competitors, with CrewAI managing task coordination
• Weekly cadence: analysis now runs weekly instead of monthly, catching competitor moves within 7 days
• Cross-period trend tracking: structured data enables pricing trend charts and feature parity tracking over time

Limitations & Edge Cases

• Login-gated content: competitors behind authentication walls require manual credential setup per run
• Dynamic pricing: A/B tested pricing pages may return different results on successive runs; the agent flags variance and averages across 3 page loads
• Rate limiting: aggressive crawling triggers bot detection on some sites; configurable delays and proxy rotation mitigate this
• Positioning subjectivity: Claude’s positioning analysis is directionally accurate but not a substitute for strategic interpretation

Architecture Trade-offs

Technology Stack

• Orchestration: CrewAI (multi-agent coordination, parallel task execution)
• LLM: Claude Sonnet for reasoning, classification, and copy analysis
• Validation: Pydantic v2 for structured output enforcement
• Web Scraping: BeautifulSoup + Playwright (for JS-rendered pages)
• Storage: JSON reports with git-tracked history for change diffing
• Reporting: Markdown summary auto-generated from structured data