Democratizing Detection Engineering at Block: Taking Flight with Goose and Panther MCP

Detection engineering stands at the forefront of cybersecurity, yet it’s often a tangled web of complexity. Traditional detection writing involves painstaking manual processes encompassing log format and schema comprehension, intricate query creation, threat modeling, and iterative manual detection testing and refinement, leading to time expenditure and reliance on specialized expertise. This can lead to gaps in threat coverage and an overwhelming number of alerts. At Block, we face the relentless challenge of evolving threats and intricate system complexities. To stay ahead, we've embraced AI-driven solutions, notably Goose, Block’s open-source AI agent, and Panther MCP, to allow the broader organization to contribute high-quality rules that are contextual to their area of expertise. This post delves into how we're transforming complicated detection workflows into streamlined, AI-powered, accessible processes for all stakeholders.

The Detection Engineering Challenge

Historically, creating effective detections has been a niche skill, requiring deep technical knowledge and coding proficiency. This has created significant obstacles such as:

• Steep Learning Curve: Crafting detections typically requires extensive technical expertise, often limiting participation.
• Resources Constraints: Even expert security teams often struggle with bandwidth, hindering their ability to develop and deploy new detections quickly.
• Evolving Threat Landscape: Advanced threats, particularly those from sophisticated nation-states actors, continuously evolve, outpacing traditional detection development processes.

Vision

We envision a future where anyone at Block can effortlessly create and deploy security detections, revolutionizing our defenses through intelligent automation and empowering a democratized security posture.

Introducing Panther MCP

What is Panther MCP?

Panther MCP is an open-source model context protocol server born from the collaboration between Panther and Block to democratize security operations workflows. By tightly integrating with Goose as an extension, Panther MCP allows security teams at Block to translate natural language instructions into precise, executable SIEM detection logic, making threat detection contributions easier and faster than ever.

This integration empowers analysts and engineers across Block to interact with Panther’s security analytics platform seamlessly. It shifts detection development from a coding-heavy process into an intuitive workflow accessible to everyone, regardless of technical background. Goose serves as an intermediary agent, coordinating calls to Panther MCP, reviewing the output, creating rule content, testing it, and making necessary edits for correctness or style. This AI-driven feedback loop saves countless hours of time.

Key Features

Panther MCP offers dozens of tools that enhance and accelerate detection engineering workflows powered by Goose:

1. Natural Language to Detection Logic
   Engineers define detections using plain English prompts, which Panther MCP translates directly into Panther-compatible detection rules that can be checked into their panther-analysis repository.
2. Interactive Data Exploration and Usage
   Engineers can rapidly explore log sources and perform searches on data and previously generated alerts through quick, natural-language driven interactions.
3. Unified Alert Triage and Response
   Enables AI-led alert triage with insights drawn from historical data and existing detections.

Accelerating Detection Creation with Goose

Goose significantly accelerates security detection creation by using AI to automate traditionally manual tasks like log analysis and rule generation. This drastically reduces effort, improves the speed of developing and deploying threat coverage, and enhances agility against evolving threats.

Integrating Panther MCP as a Goose Extension

Panther MCP functions as a Goose extension, seamlessly embedding its capabilities within the Goose environment through the following process:

1. Extension Registration: Panther MCP is registered within Goose, making its suite of tools readily accessible via the Goose interface.
2. API Connectivity: The extension establishes a connection to Panther's backend API, enabling seamless context retrieval.
3. Available Tools: Panther MCP provides Goose with a range of tools designed for efficient detection creation, intuitive data interaction, and streamlined alert management.

Leveraging Enhanced Context with .goosehints

The integration between Panther MCP and Goose is enhanced through the use of the .goosehints file—a Goose feature that supplies additional context like rule examples and best practices. This enriched context enables Goose to generate more accurate and efficient detections, aligned with Block’s standards and requirements.

Let's illustrate this with an example: creating a rule to detect users adding themselves to privileged Okta groups, a common privilege escalation technique.

Breaking Down the Barriers

Traditionally, creating this detection would require:

1. Deep knowledge of Okta and its log structure
2. Understanding of Panther’s detection framework
3. Python programming skills
4. Familiarity with different testing frameworks

With Goose and Panther MCP, this becomes as simple as:

“Write a detection rule for users adding themselves to privileged Okta groups.”

The Intelligence Behind the Simplicity

When a natural language request like "Write a detection rule for users adding themselves to privileged Okta groups" is received, Goose leverages a sophisticated, multi-stage process powered by Panther MCP to generate production-ready detection logic. This automated approach mirrors the workflow of an experienced detection engineer, encompassing threat research, relevant log identification, detection goal definition, logic outlining, sample log analysis, rule development, false positive consideration, severity/context assignment, thorough testing, refinement/optimization, and documentation. However, Goose executes these steps with the speed and scalability afforded by AI and automation.

Goose first parses the natural language input to understand the core intent and requirements. It identifies key entities like "users", "privileged Okta groups", and the action "adding themselves". This understanding forms the basis for outlining the detection's objective, the necessary log source (Okta.SystemLog), and the fundamental logic: identifying events where the actor (user initiating the action) is the same as the target user (the user being added to the group), and the group being joined is designated as privileged. Goose also considers potential false positives (e.g., legitimate automated processes) and assigns a preliminary severity level based on the potential impact of the detected activity (privilege escalation).

To ensure the generated logic is accurate and operates on valid data, Goose interacts with Panther MCP to retrieve the schema of the specified log source (Okta.SystemLog). This provides Goose with a structured understanding of the available fields and their data types within Okta logs. Furthermore, Goose utilizes Panther MCP's querying capabilities to fetch sample log events related to group membership changes. This step is crucial for:

• Identifying Common Event Patterns: Analyzing real-world logs allows Goose to understand the typical structure and values associated with relevant events (e.g., group.user_membership.add).
• Inferring Privileged Group Naming Conventions: By examining historical data, Goose can identify patterns and keywords commonly used in the naming of privileged groups within the organization's Okta instance (e.g., "admin", "administrator", "security-admin").
• Discovering Edge Cases: Examining diverse log samples helps uncover potential variations in event data or less common scenarios that the detection logic needs to accommodate.
• Mapping Typical User Behavior: Understanding baseline user behavior around group membership changes helps refine the detection logic and reduce the likelihood of false positives.

The interaction with Panther MCP at this stage involves API calls to retrieve schema information and execute analytical queries, enabling Goose to ground its reasoning in actual log data.

Goose doesn't operate in isolation; it accesses a repository of existing Panther detection rules to identify similar logic or reusable components. This promotes consistency across the detection landscape, encourages the reuse of well-tested helper functions (like okta_alert_context), and ensures adherence to established rule standards within our security ecosystem. Learning from existing detections is a core component of Goose’s intelligence, allowing it to build upon prior knowledge and avoid reinventing the wheel.

Based on the understanding of the detection goal, the analysis of log data, and the knowledge gleaned from existing detections facilitated by Panther MCP, Goose generates the complete Panther detection rule in Python. This includes:

• Rule Function (rule()): This function contains the core logic for evaluating each log event. In the example, it checks for the group.user_membership.add event type, verifies that the actor and target user IDs (or emails) are the same, and confirms that the target group's display name contains keywords indicative of a privileged group (defined in the PRIVILEGED_GROUPS set).
• Metadata Functions (title(), alert_context(), severity(), destinations()): These functions provide crucial context and operational information for triggered alerts.

from panther_okta_helpers import okta_alert_context

# Define privileged Okta groups - customize this list based on your organization's needs
PRIVILEGED_GROUPS = {
    "_group_admin",  # Administrator roles
    "admin",
    "administrator",
    "application-admin",   
    "aws_",  # AWS roles can be privileged
    "cicd_corp_system",  # CI/CD admin access 
    "grc-okta",
    "okta-administrators",
    "okta_admin",
    "okta_admin_svc_accounts", # Admin roles
    "okta_resource-set_",      # Resource sets are typically privileged
    "security-admin",
    "superadministrators",
}

def rule(event):
    """Determine if a user added themselves to a privileged group"""
    # Only focus on group membership addition events
    if event.get("eventType") != "group.user_membership.add":
        return False
    # Ensure both actor and target exist in the event
    actor = event.get("actor", {})
    targets = event.get("target", [])
    if not actor or len(targets) < 2:
        return False
    actor_id = actor.get("alternateId", "").lower()
    actor_user_id = actor.get("id")
    # Extract target user and group
    target_user = targets[0]
    target_group = targets[1] if len(targets) > 1 else {}
    # The first target should be a user and the second should be a group
    if target_user.get("type") != "User" or target_group.get("type") != "UserGroup":
        return False
    target_user_id = target_user.get("id")
    target_user_email = target_user.get("alternateId", "").lower()
    group_name = target_group.get("displayName", "").lower()
    # Check if the actor added themselves to the group
    is_self_add = (actor_user_id == target_user_id) or (actor_id == target_user_email)
    # Check if the group is privileged
    is_privileged_group = any(priv_group in group_name for priv_group in PRIVILEGED_GROUPS)
    return is_self_add and is_privileged_group

def title(event):
    """Generate a descriptive title for the alert"""
    actor = event.get("actor", {})
    targets = event.get("target", [])
    actor_name = actor.get("displayName", "Unknown User")
    actor_email = actor.get("alternateId", "[email protected]")
    target_group = targets[1] if len(targets) > 1 else {}
    group_name = target_group.get("displayName", "Unknown Group")
    return (f"User [{actor_name} ({actor_email})] added themselves "
            f"to privileged Okta group [{group_name}]")

def alert_context(event):
    """Return additional context for the alert"""
    context = okta_alert_context(event)
    # Add specific information about the privileged group
    targets = event.get("target", [])
    if len(targets) > 1:
        target_group = targets[1]
        context["privileged_group"] = {
            "id": target_group.get("id", ""),
            "name": target_group.get("displayName", ""),
        }
    return context

def severity(event):
    """Calculate severity based on group name - more sensitive groups get higher severity"""
    targets = event.get("target", [])
    if len(targets) <= 1:
        return "Medium"
    target_group = targets[1]
    group_name = target_group.get("displayName", "").lower()
    # Higher severity for direct admin groups
    if any(name in group_name for name in ["admin", "administrator", "superadministrators"]):
        return "Critical"
    return "High"

def destinations(_event):
    """Send to staging destination for review"""
    return ["staging_destination"]

Beyond the Python code, Goose also generates the corresponding YAML-based rule configuration file. This file contains essential metadata about the detection:

AnalysisType: rule
Description: Detects when a user adds themselves to a privileged Okta group, which could indicate privilege escalation attempts or unauthorized access.
DisplayName: "Users Adding Themselves to Privileged Okta Groups"
Enabled: true
DedupPeriodMinutes: 60
LogTypes:
  - Okta.SystemLog
RuleID: "goose.Okta.Self.Privileged.Group.Add"
Threshold: 1
Filename: goose_okta_self_privileged_group_add.py
Reference: >
  https://developer.okta.com/docs/reference/api/system-log/
  https://attack.mitre.org/techniques/T1078/004/
  https://attack.mitre.org/techniques/T1484/001/
Runbook: >
  1. Verify if the user should have access to the privileged group they added themselves to
  2. If unauthorized, revoke the group membership immediately
  3. Check for other group membership changes made by the same user
  4. Review the authentication context and security context for suspicious indicators
  5. Interview the user to determine intent
Reports:
  MITRE ATT&CK:
    - TA0004:T1078.004  # Privileged Accounts: Cloud Accounts
    - TA0004:T1484.001  # Domain Policy Modification: Group Policy Modification
Severity: High
Tags:
  - author:tomasz
  - coauthor:goose
Tests:
  - Name: User adds themselves to privileged group
    ExpectedResult: true
    Log:
      actor:
        alternateId: [email protected]
        displayName: Jane Doe
        id: 00u1234abcd5678
        type: User
      authenticationContext:
        authenticationStep: 0
        externalSessionId: xyz1234
      client:
        device: Computer
        geographicalContext:
          city: San Francisco
          country: United States
          geolocation:
            lat: 37.7749
            lon: -122.4194
          postalCode: "94105"
          state: California
        ipAddress: 192.168.1.100
        userAgent:
          browser: CHROME
          os: Mac OS X
          rawUserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
        zone: "null"
      debugContext:
        debugData:
          requestId: req123456
          requestUri: /api/v1/groups/00g123456/users/00u1234abcd5678
          url: /api/v1/groups/00g123456/users/00u1234abcd5678
      displayMessage: Add user to group membership
      eventType: group.user_membership.add
      legacyEventType: group.user_membership.add
      outcome:
        result: SUCCESS
      published: "2023-07-15 14:25:30.811"
      request:
        ipChain:
          - geographicalContext:
              city: San Francisco
              country: United States
              geolocation:
                lat: 37.7749
                lon: -122.4194
              postalCode: "94105"
              state: California
            ip: 192.168.1.100
            version: V4
      securityContext:
        asNumber: 12345
        asOrg: Example ISP
        domain: example.com
        isProxy: false
        isp: Example ISP
      severity: INFO
      target:
        - alternateId: [email protected]
          displayName: Jane Doe
          id: 00u1234abcd5678
          type: User
        - alternateId: unknown
          displayName: okta_admin_person_role_super_admin
          id: 00g5678abcd1234
          type: UserGroup
      transaction:
        detail: {}
        id: transaction123
        type: WEB
      uuid: event-uuid-123
      version: "0"
      p_event_time: "2023-07-15 14:25:30.811"
      p_parse_time: "2023-07-15 14:26:00.000"
      p_log_type: "Okta.SystemLog"
  - Name: User adds another user to privileged group
    ExpectedResult: false
    Log:
      actor:
        alternateId: [email protected]
        displayName: Admin User
        id: 00u5678abcd1234
        type: User
      authenticationContext:
        authenticationStep: 0
        externalSessionId: xyz5678
      client:
        device: Computer
        geographicalContext:
          city: San Francisco
          country: United States
          geolocation:
            lat: 37.7749
            lon: -122.4194
          postalCode: "94105"
          state: California
        ipAddress: 192.168.1.100
        userAgent:
          browser: CHROME
          os: Mac OS X
          rawUserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
        zone: "null"
      debugContext:
        debugData:
          requestId: req789012
          requestUri: /api/v1/groups/00g123456/users/00u9876fedc4321
          url: /api/v1/groups/00g123456/users/00u9876fedc4321
      displayMessage: Add user to group membership
      eventType: group.user_membership.add
      legacyEventType: group.user_membership.add
      outcome:
        result: SUCCESS
      published: "2023-07-15 14:30:45.123"
      request:
        ipChain:
          - geographicalContext:
              city: San Francisco
              country: United States
              geolocation:
                lat: 37.7749
                lon: -122.4194
              postalCode: "94105"
              state: California
            ip: 192.168.1.100
            version: V4
      securityContext:
        asNumber: 12345
        asOrg: Example ISP
        domain: example.com
        isProxy: false
        isp: Example ISP
      severity: INFO
      target:
        - alternateId: [email protected]
          displayName: Regular User
          id: 00u9876fedc4321
          type: User
        - alternateId: unknown
          displayName: okta_admin_person_role_super_admin
          id: 00g5678abcd1234
          type: UserGroup
      transaction:
        detail: {}
        id: transaction456
        type: WEB
      uuid: event-uuid-456
      version: "0"
      p_event_time: "2023-07-15 14:30:45.123"
      p_parse_time: "2023-07-15 14:31:00.000"
      p_log_type: "Okta.SystemLog"

Every detection rule generated by Goose undergoes rigorous automated testing and validation. This includes:

• Unit Testing: Using the test cases defined in the rule configuration, the Panther Analysis Tool is executed to verify that the rule logic correctly identifies true positives and avoids false negatives against simulated log data.
• Linting: Code linting tools (like Pylint) are automatically run to ensure the generated Python code adheres to established coding standards, including proper formatting, style conventions, and best practices. This contributes to code maintainability and reduces the risk of errors.

The seamless integration of Goose with Panther MCP automates these intricate steps, significantly reducing the time and specialized knowledge required to create and deploy security detections. This democratization empowers more individuals to contribute to Block's security posture, leading to more comprehensive threat coverage and a more resilient security environment.

Democratization in Practice

A typical detection creation workflow now looks like:

1. Proposal: A user describes a malicious behavior in natural language.
2. Generation: Goose transforms this description into detection logic with Panther MCP.
3. Review: The detection team reviews each detection against defined quality benchmarks.
4. Deployment: Approved detections are deployed to staging/production.

Early Impact & Lessons Learned

Expanding Collaboration to Enhance Coverage and Enable Self-Service

• Lowering the Technical Barrier: Goose and Panther MCP empower subject matter experts (SMEs) to easily understand their logs in Panther, enabling a self-service model where teams can create their own detections without extensive security engineering expertise, thus distributing the workload.
• Reduced Dependency on the Detection Team: Panther MCP reduces security team dependency by enabling users to independently resolve inquiries autonomously. This includes threat intelligence teams assessing MITRE ATT&CK coverage, compliance teams identifying relevant detections, and helping service SMEs create their own detections.
• Cross-Functional Detection Development: Democratizing detection engineering allows specialized teams to create detections that security teams might miss, leading to a more diverse detection ecosystem covering niche use cases. This fosters two-way knowledge transfer, enhancing overall security awareness and capabilities.

Accelerating the Detection Development Lifecycle

• Contextual Understanding: Detection engineering is becoming more efficient and consistent through tools that embed organizational context, provide guided best practices, understand existing log schemas and detections, and align with validation frameworks such as pytest. This approach enables broader participation and supports high-quality development across teams.
• Streamlined Development Process: Natural language interfaces are simplifying detection engineering by allowing users to interact with the system conversationally. This enables automated retrieval of example logs, analysis of log schemas, interpretation of detection goals or required changes, and generation of initial detection code—significantly accelerating development.
• Automated Technical Steps: Intelligent code generation incorporates error handling and best practices, while seamlessly generating test cases from data and producing comprehensive documentation—including descriptions, runbooks, and references.

Driving Consistency via Standardized Practices

• Code Style and Structure: Newly created detections adhere to consistent stylistic patterns, utilizing dedicated functions for specific checks instead of overloaded rule() checks. Standardized formatting, including brackets for dynamic alert title text, enhances readability and consistency.
• Code Reuse and Efficiency: Promote code reuse and efficiency through global helpers/filters, explicit typing in function signatures, and detailed docstrings for better function understanding and LLM code generation.
• Maintainability Improvements: Detections are designed with a consistent structure and standardized patterns, making them easier to understand, maintain, and update. This uniformity ensures predictable behavior across the detection code base and simplifies bulk changes when needed.
• Comprehensive Testing Requirements: For our team, each detection is required to include at least two unit tests: one positive case that triggers the detection and one negative case that does not. Test names are descriptive and aligned with expected outcomes to enhance readability and maintainability.
• Metadata and Documentation Standards: Metadata and documentation standards are being strengthened through structured definitions within pytests, helping to codify detection ownership and context. This includes clearly defined author and coauthor tags (e.g., for Goose-generated content), environment references such as staging or production, and accurate mapping of alert destinations.
• Structural Validation: This supports compliance with organizational standards by enforcing filename conventions (e.g., prefixing, length, lowercase formatting), ensuring Python rules include all required functions, and verifying that YAML files contain the necessary fields for proper functionality and processing.
• Security Framework Alignment: Relevant rules are mapped to applicable MITRE ATT&CK techniques to highlight coverage gaps, inform detection development, prioritize research efforts, and establish a common language for discussing threats.

Best Practices and Safeguards

• Platform-Conformant Development: Detections are developed in alignment with Panther’s recommended practices, such as using built-in event object methods like event.deep_get() and event.deep_walk() instead of importing them manually, ensuring consistency and maintainability within the platform.
• Proactive Error Prevention: We implement local validation checks through pre-commit and pre-push hooks to proactively catch and resolve errors before they reach upstream builds. These checks include validating alert destination names, verifying log types, and flagging grammatical issues to ensure quality and consistency.
• Continuous Improvement: Detection quality continuously improves by incorporating feedback, performance data, and analysis of detection trends. Panther MCP, along with other ticket tracking MCPs, provides insights from analyst feedback and alert dispositions, which facilitates automated adjustments, streamlines pull request development, and lowers operational overhead.

What’s Next?

Block is dedicated to improving its security defenses and supporting its team by leveraging AI. We believe AI holds significant promise for the future of detection and response at Block and are committed to making security more accessible.