Site Reliability Engineer, AI & Agentic Systems

Overview

As our SRE charter continues to evolve, this role demands strong hands-on ownership of production reliability and troubleshooting, coupled with advanced capabilities in AI- and agentic-driven automation and performance engineering.

The Site Reliability Engineer will play a critical role in ensuring reliability, scalability, performance, and operational excellence of our platforms. The ideal candidate will leverage Azure-native AI services and agentic systems to reduce toil, improve incident response, and enable intelligent operations—while also driving performance testing practices to validate system resilience under load.

**This is a hybrid role, located at our Plano, TX office.  Candidates must be willing and able to work in-office 3 days per week in Plano, TX.

Applicants must be currently authorized to work in the United States on a full-time basis and must not require sponsorship for employment visa status now or in the future

A DAY IN THE LIFE

In this role, you will…

• Own end-to-end reliability of large-scale, Azure-hosted production systems, ensuring high availability, fault tolerance, and graceful degradation
• Lead hands-on incident troubleshooting, root cause analysis (RCA), and post-incident reviews with actionable follow-ups
• Build and operate resilient, scalable services on Microsoft Azure (AKS, App Services, Functions, Event Hubs, etc.)
• Design and maintain comprehensive observability platforms using Prometheus for metrics, Loki for log aggregation, Tempo for distributed tracing, and Grafana for dashboarding and alerting
• Design, develop, and execute performance testing strategies for distributed systems and microservices, including load testing, stress testing, soak testing, and capacity planning
• Integrate AI agents with Azure monitoring stack, CI/CD tooling, and incident management platforms
• Contribute to evolving SRE standards, tooling, operational processes, and knowledge base

Responsibilities

Reliability Engineering & Production Ownership

• Own end-to-end reliability of large-scale, Azure-hosted production systems, ensuring high availability, fault tolerance, and graceful degradation
• Lead hands-on incident troubleshooting, root cause analysis (RCA), and post-incident reviews with actionable follow-ups
• Define, measure, and enforce Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets aligned with business outcomes
• Drive proactive reliability improvements based on operational insights, failure mode analysis, and capacity planning
• Participate in on-call rotations and take real-time ownership during production incidents

Platform & Automation Engineering

• Build and operate resilient, scalable services on Microsoft Azure (AKS, App Services, Functions, Event Hubs, etc.)
• Design and maintain comprehensive observability platforms using Prometheus for metrics, Loki for log aggregation, Tempo for distributed tracing, and Grafana for dashboarding and alerting
• Create automation to eliminate manual operational tasks and reduce Mean Time to Recovery (MTTR)
• Implement self-healing mechanisms, automated remediation workflows, and runbook automation
• Manage and optimize API lifecycle and traffic management using Gravitee API Gateway
• Design and implement durable, fault-tolerant workflows and microservice orchestration patterns using Temporal
• Administer and tune PostgreSQL databases for reliability, performance, and high availability
• Partner with application and platform teams to improve service operability, deployment safety, and change management

Performance Testing & Load Engineering

• Design, develop, and execute performance testing strategies for distributed systems and microservices, including load testing, stress testing, soak testing, and capacity planning
• Build and maintain performance test scripts and virtual user scenarios using Micro Focus LoadRunner and VuGen (Virtual User Generator)
• Analyze performance test results to identify bottlenecks, regressions, and scalability limits; produce clear reports with actionable recommendations
• Integrate performance testing into CI/CD pipelines to enable continuous performance validation and shift-left testing practices
• Establish and monitor performance baselines, benchmarks, and SLAs across critical service endpoints and user journeys
• Collaborate with development and architecture teams to resolve performance issues and optimize system throughput, latency, and resource utilization

AI / Agentic Engineering (Azure Focus)

• Design and implement AI-driven and agentic systems to enhance operational workflows and intelligent decision-making
• Build intelligent automation for operational use cases, including:
  • Incident triage, enrichment, and automated escalation
  • Alert correlation, deduplication, and noise reduction
  • Automated diagnosis and remediation of recurring failures
• Leverage Azure AI services (Azure OpenAI, Cognitive Services, Azure ML) for operational intelligence and predictive insights
• Integrate AI agents with Azure monitoring stack, CI/CD tooling, and incident management platforms
• Ensure safe, reliable, and observable operation of AI-powered systems in production, including guardrails, fallback mechanisms, and audit trails

Collaboration & Technical Leadership

• Act as a reliability, performance, and automation champion across engineering teams
• Mentor junior SREs and influence adoption of best practices in reliability, observability, and performance engineering
• Contribute to evolving SRE standards, tooling, operational processes, and knowledge base
• Participate in architecture reviews and provide guidance on non-functional requirements (reliability, scalability, performance)

Qualifications

Core SRE Skills

• 5+ years of experience in Site Reliability Engineering, DevOps, or Production Engineering roles
• Strong hands-on experience in production troubleshooting of distributed systems at scale
• Solid understanding of Linux internals, networking (TCP/IP, DNS, HTTP, TLS), and system performance tuning
• Deep hands-on experience with Microsoft Azure (compute, networking, storage, managed services, AKS)
• Strong knowledge of Kubernetes, container orchestration, Helm charts, and microservices architectures
• Proficiency in one or more programming languages: Python, Go, Java, or equivalent
• Experience with CI/CD pipelines (Azure DevOps, GitHub Actions) and Infrastructure as Code (Terraform, ARM Templates, Bicep)

Observability & Monitoring

• Hands-on experience building and operating observability stacks using Prometheus, Grafana, Loki, and Tempo
• Experience with alerting strategies, SLI/SLO-based monitoring, and on-call incident management

Performance Testing & Load Engineering

• Proven experience designing and executing performance and load testing for large-scale distributed applications
• Hands-on proficiency with Micro Focus LoadRunner and VuGen for scripting virtual user scenarios, parameterization, correlation, and result analysis
• Strong understanding of performance testing methodologies: load testing, stress testing, endurance/soak testing, spike testing, and capacity planning
• Ability to analyze performance metrics (throughput, response time, error rate, resource utilization) and translate findings into engineering actions
• Experience integrating performance tests into automated CI/CD pipelines

Platform & Middleware

• Experience with Gravitee or equivalent API gateway platforms for traffic management, rate limiting, and API lifecycle governance
• Hands-on experience with Temporal for workflow orchestration, durable execution, and distributed task management
• Strong PostgreSQL administration skills, including query optimization, replication, backup/recovery, and performance tuning

AI / Agentic Systems

• Hands-on experience building or integrating AI-powered automation in production environments
• Experience with agent-based systems, LLM-powered workflows, Retrieval-Augmented Generation (RAG), or intelligent assistants
• Familiarity with Azure-based AI and ML services (Azure OpenAI, Cognitive Services, Azure ML)
• Understanding of reliability, safety, observability, and operational challenges of AI systems in production