Exercise: Circuit Breaker Patterns

Difficulty - Advanced

Learning Objectives

• Apply circuit breaker concepts from distributed systems
• Implement advanced circuit breaker configurations
• Practice with monitoring and observability integration
• Understand production circuit breaker patterns

Problem Statement

This exercise builds upon the comprehensive circuit breaker coverage in Distributed Systems Patterns. Instead of reimplementing the core circuit breaker, you'll apply it to advanced scenarios and integrate it with production systems.

📖 Background: For detailed explanations of circuit breaker concepts, state transitions, and production considerations, see the Circuit Breaker section in the Distributed Systems tutorial.

Exercise Tasks

Task 1: Integrate Circuit Breaker with External Services

Using the circuit breaker implementation from the Distributed Systems article, implement a resilient HTTP client:

 1package main
 2
 3import (
 4    "context"
 5    "fmt"
 6    "net/http"
 7    "time"
 8)
 9
10// Exercise: Build a resilient HTTP client using the CircuitBreaker from
11// /03-advanced-topics/15-distributed-systems
12
13type ResilientHTTPClient struct {
14    client *http.Client
15    cb     *CircuitBreaker
16}
17
18func NewResilientHTTPClient(timeout time.Duration) *ResilientHTTPClient {
19    // TODO: Create circuit breaker with appropriate settings
20    // TODO: Configure HTTP client with timeouts
21    return &ResilientHTTPClient{}
22}
23
24// TODO: Implement HTTP call with circuit breaker protection
25func Get(ctx context.Context, url string) {
26    // 1. Check circuit breaker state
27    // 2. Execute HTTP call through circuit breaker
28    // 3. Handle errors and fallback responses
29    // 4. Return appropriate responses for open circuit
30}
31
32func main() {
33    client := NewResilientHTTPClient(10 * time.Second)
34
35    // TODO: Test the client with various scenarios
36    // - Normal operation
37    // - Service failures
38    // - Circuit opening
39    // - Recovery scenarios
40
41    fmt.Println("Resilient HTTP client implementation")
42}

Task 2: Multi-Service Circuit Breaker Management

Implement a circuit breaker manager for multiple services:

 1// Exercise: Build a manager for multiple circuit breakers
 2type CircuitBreakerManager struct {
 3    breakers map[string]*CircuitBreaker
 4    mu       sync.RWMutex
 5}
 6
 7// TODO: Implement methods for:
 8// - Creating circuit breakers per service
 9// - Monitoring all circuit breakers
10// - Handling cascading failures
11// - Providing dashboards/alerts
12func CallService(ctx context.Context, serviceName string, fn func() error) error {
13    // TODO: Get or create circuit breaker for service
14    // TODO: Execute function through circuit breaker
15    // TODO: Handle service-specific configurations
16}

Task 3: Observability Integration

Extend the circuit breaker with comprehensive monitoring:

 1// Exercise: Add metrics and distributed tracing
 2type ObservableCircuitBreaker struct {
 3    *CircuitBreaker
 4    metrics CircuitBreakerMetrics
 5    tracer  opentelemetry.Tracer
 6}
 7
 8// TODO: Implement:
 9// - Prometheus metrics for state changes
10// - OpenTelemetry tracing for calls
11// - Health check endpoints
12// - Alerting on state transitions

Task 4: Production Scenarios

Apply the circuit breaker patterns to these scenarios:

1. Database Connection Pooling: Prevent cascade when database fails
2. External API Calls: Handle third-party service outages
3. Microservices Communication: Protect against service failures
4. Message Queue Processing: Handle broker connectivity issues

 1// From /03-advanced-topics/15-distributed-systems
 2type CircuitBreaker struct {
 3    mu              sync.Mutex
 4    state           State
 5    failureCount    int
 6    maxFailures     int
 7    timeout         time.Duration
 8    lastFailureTime time.Time
 9}
10
11func Call(fn func() error) error
12func State() State

Solution Approach

 1func NewResilientHTTPClient(timeout time.Duration) *ResilientHTTPClient {
 2    settings := Settings{
 3        MaxRequests: 5,
 4        Interval:    time.Minute,
 5        Timeout:     30 * time.Second,
 6        ReadyToTrip: func(counts Counts) bool {
 7            failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
 8            return counts.Requests >= 5 && failureRatio >= 0.5
 9        },
10        OnStateChange: func(from, to State) {
11            log.Printf("Circuit breaker %s -> %s", from, to)
12        },
13    }
14
15    return &ResilientHTTPClient{
16        client: &http.Client{Timeout: timeout},
17        cb:     New("http-client", settings),
18    }
19}
20
21func Get(ctx context.Context, url string) {
22    var resp *http.Response
23    err := c.cb.Call(ctx, func() error {
24        req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
25        if err != nil {
26            return err
27        }
28
29        resp, err = c.client.Do(req)
30        return err
31    })
32
33    if err != nil {
34        return nil, fmt.Errorf("circuit breaker error: %w", err)
35    }
36
37    return resp, nil
38}

 1func CallService(ctx context.Context, serviceName string, fn func() error) error {
 2    m.mu.RLock()
 3    cb, exists := m.breakers[serviceName]
 4    m.mu.RUnlock()
 5
 6    if !exists {
 7        m.mu.Lock()
 8        cb, exists = m.breakers[serviceName]
 9        if !exists {
10            cb = m.createCircuitBreaker(serviceName)
11            m.breakers[serviceName] = cb
12        }
13        m.mu.Unlock()
14    }
15
16    return cb.Call(ctx, fn)
17}
18
19func createCircuitBreaker(serviceName string) *CircuitBreaker {
20    settings := Settings{
21        MaxRequests: 3,
22        Interval:    time.Minute,
23        Timeout:     15 * time.Second,
24        ReadyToTrip: func(counts Counts) bool {
25            return counts.ConsecutiveFailures >= 3
26        },
27        OnStateChange: func(from, to State) {
28            log.Printf("Service %s circuit breaker: %s -> %s", serviceName, from, to)
29        },
30    }
31
32    return New(serviceName, settings)
33}

Key Takeaways

• Circuit breakers prevent cascading failures in distributed systems
• Configuration must be tuned for each service's characteristics
• Observability is crucial for monitoring circuit breaker health
• Multiple circuit breakers require careful management to avoid coordination failures
• Production systems need sophisticated fallback and recovery strategies

Further Learning

For comprehensive coverage of circuit breaker patterns including:

• Detailed state transition logic
• Advanced patterns
• Production best practices and pitfalls
• Integration with observability platforms
• Benchmarking and performance optimization

See: Distributed Systems Patterns → Circuit Breaker

For cloud-native circuit breaker implementations, see:

• Service Mesh Circuit Breaking
• API Gateway Patterns

Additional Challenges

1. Adaptive Thresholds: Implement circuit breakers that adjust thresholds based on service performance
2. Distributed Coordination: Build circuit breakers that coordinate across multiple instances
3. Machine Learning Integration: Use ML to predict optimal circuit breaker configurations
4. Chaos Engineering: Design experiments to test circuit breaker effectiveness under failure conditions