Advanced Testing Strategies
Exercise Overview
Build a comprehensive testing suite for a microservices application. You'll implement integration tests, contract testing, load testing, and chaos engineering to ensure system reliability and performance.
Learning Objectives
- Design and implement integration tests for microservices
- Create contract tests for API compatibility
- Build automated load testing scenarios
- Implement chaos engineering experiments
- Add property-based testing with fuzzing
- Create test fixtures and test data management
- Implement test environment orchestration
- Add test reporting and monitoring
The System - E-Commerce Microservices
You have an e-commerce system with multiple microservices that need comprehensive testing:
- User Service - User management and authentication
- Product Service - Product catalog management
- Order Service - Order processing and management
- Payment Service - Payment processing
- Notification Service - Email and SMS notifications
Initial Testing Structure
1// test/integration/user_service_test.go
2package integration
3
4import (
5 "context"
6 "testing"
7 "time"
8)
9
10// TODO: Implement integration test framework
11type IntegrationTestSuite struct {
12 // Add test suite implementation
13}
14
15// TODO: Implement database test setup
16type TestDatabase struct {
17 // Add test database setup
18}
19
20// TODO: Implement service containers
21type TestContainers struct {
22 // Add Docker containers for testing
23}
24
25// TODO: Implement integration tests
26func TestUserServiceIntegration(t *testing.T) {
27 // Implement comprehensive integration tests
28}
29
30// TODO: Implement contract tests
31func TestAPIContracts(t *testing.T) {
32 // Implement API contract testing
33}
34
35// TODO: Implement load tests
36func TestSystemLoad(t *testing.T) {
37 // Implement load testing scenarios
38}
39
40// TODO: Implement chaos tests
41func TestSystemResilience(t *testing.T) {
42 // Implement chaos engineering experiments
43}
Tasks
Task 1: Build Integration Test Framework
Create a comprehensive integration testing framework:
1// test/framework/integration_test.go
2package framework
3
4import (
5 "context"
6 "fmt"
7 "log"
8 "sync"
9 "testing"
10 "time"
11
12 "github.com/testcontainers/testcontainers-go"
13 "github.com/testcontainers/testcontainers-go/wait"
14)
15
16type IntegrationTestEnvironment struct {
17 ctx context.Context
18 cancel context.CancelFunc
19 containers map[string]testcontainers.Container
20 services map[string]ServiceInstance
21 databases map[string]DatabaseInstance
22 config TestConfig
23 mu sync.RWMutex
24}
25
26type TestConfig struct {
27 ProjectName string
28 NetworkName string
29 Timeout time.Duration
30 CleanupEnabled bool
31 LogEnabled bool
32}
33
34type ServiceInstance struct {
35 Name string
36 Host string
37 Port int
38 HealthURL string
39 Ready bool
40}
41
42type DatabaseInstance struct {
43 Name string
44 Host string
45 Port int
46 Database string
47 Username string
48 Password string
49}
50
51func NewIntegrationTestEnvironment(config TestConfig) *IntegrationTestEnvironment {
52 ctx, cancel := context.WithCancel(context.Background())
53
54 return &IntegrationTestEnvironment{
55 ctx: ctx,
56 cancel: cancel,
57 containers: make(map[string]testcontainers.Container),
58 services: make(map[string]ServiceInstance),
59 databases: make(map[string]DatabaseInstance),
60 config: config,
61 }
62}
63
64func Setup(t *testing.T) error {
65 // Create Docker network
66 network, err := testcontainers.GenericContainer(ite.ctx, testcontainers.GenericContainerRequest{
67 ContainerRequest: testcontainers.ContainerRequest{
68 Image: "alpine:latest",
69 Cmd: []string{"tail", "-f", "/dev/null"},
70 Name: ite.config.NetworkName,
71 Labels: map[string]string{"com.github.testcontainers.integration": "true"},
72 },
73 Started: true,
74 })
75 if err != nil {
76 return fmt.Errorf("failed to create network: %w", err)
77 }
78
79 ite.containers["network"] = network
80
81 // Start databases
82 if err := ite.setupDatabases(); err != nil {
83 return fmt.Errorf("failed to setup databases: %w", err)
84 }
85
86 // Start services
87 if err := ite.setupServices(); err != nil {
88 return fmt.Errorf("failed to setup services: %w", err)
89 }
90
91 // Wait for all services to be healthy
92 return ite.waitForHealthyServices()
93}
94
95func setupDatabases() error {
96 // PostgreSQL for user service
97 postgresReq := testcontainers.ContainerRequest{
98 Image: "postgres:15-alpine",
99 ExposedPorts: []string{"5432/tcp"},
100 Env: map[string]string{
101 "POSTGRES_DB": "ecommerce_test",
102 "POSTGRES_USER": "test_user",
103 "POSTGRES_PASSWORD": "test_password",
104 },
105 WaitingFor: wait.ForLog("database system is ready to accept connections").
106 WithOccurrence(2).WithStartupTimeout(30 * time.Second),
107 }
108
109 postgresContainer, err := testcontainers.GenericContainer(ite.ctx, testcontainers.GenericContainerRequest{
110 ContainerRequest: postgresReq,
111 Started: true,
112 })
113 if err != nil {
114 return err
115 }
116
117 ite.containers["postgres"] = postgresContainer
118
119 // Get connection details
120 host, err := postgresContainer.Host(ite.ctx)
121 if err != nil {
122 return err
123 }
124
125 port, err := postgresContainer.MappedPort(ite.ctx, "5432")
126 if err != nil {
127 return err
128 }
129
130 ite.databases["postgres"] = DatabaseInstance{
131 Name: "postgres",
132 Host: host,
133 Port: port.Int(),
134 Database: "ecommerce_test",
135 Username: "test_user",
136 Password: "test_password",
137 }
138
139 // Redis for caching
140 redisReq := testcontainers.ContainerRequest{
141 Image: "redis:7-alpine",
142 ExposedPorts: []string{"6379/tcp"},
143 WaitingFor: wait.ForLog("Ready to accept connections"),
144 }
145
146 redisContainer, err := testcontainers.GenericContainer(ite.ctx, testcontainers.GenericContainerRequest{
147 ContainerRequest: redisReq,
148 Started: true,
149 })
150 if err != nil {
151 return err
152 }
153
154 ite.containers["redis"] = redisContainer
155
156 redisHost, _ := redisContainer.Host(ite.ctx)
157 redisPort, _ := redisContainer.MappedPort(ite.ctx, "6379")
158
159 ite.databases["redis"] = DatabaseInstance{
160 Name: "redis",
161 Host: redisHost,
162 Port: redisPort.Int(),
163 }
164
165 return nil
166}
167
168func setupServices() error {
169 // User service
170 userSvcContainer, err := testcontainers.GenericContainer(ite.ctx, testcontainers.GenericContainerRequest{
171 ContainerRequest: testcontainers.ContainerRequest{
172 Image: "ecommerce/user-service:test",
173 ExposedPorts: []string{"8080/tcp"},
174 Env: map[string]string{
175 "DB_HOST": ite.databases["postgres"].Host,
176 "DB_PORT": fmt.Sprintf("%d", ite.databases["postgres"].Port),
177 "DB_NAME": ite.databases["postgres"].Database,
178 "DB_USER": ite.databases["postgres"].Username,
179 "DB_PASSWORD": ite.databases["postgres"].Password,
180 "REDIS_HOST": ite.databases["redis"].Host,
181 "REDIS_PORT": fmt.Sprintf("%d", ite.databases["redis"].Port),
182 },
183 WaitingFor: wait.ForHTTP("/health").WithPort("8080"),
184 },
185 Started: true,
186 })
187 if err != nil {
188 return err
189 }
190
191 ite.containers["user-service"] = userSvcContainer
192
193 userHost, _ := userSvcContainer.Host(ite.ctx)
194 userPort, _ := userSvcContainer.MappedPort(ite.ctx, "8080")
195
196 ite.services["user-service"] = ServiceInstance{
197 Name: "user-service",
198 Host: userHost,
199 Port: userPort.Int(),
200 HealthURL: fmt.Sprintf("http://%s:%d/health", userHost, userPort.Int()),
201 Ready: false,
202 }
203
204 // Similar setup for other services...
205 return nil
206}
207
208func waitForHealthyServices() error {
209 ctx, cancel := context.WithTimeout(ite.ctx, 2*time.Minute)
210 defer cancel()
211
212 ticker := time.NewTicker(5 * time.Second)
213 defer ticker.Stop()
214
215 for {
216 select {
217 case <-ctx.Done():
218 return fmt.Errorf("timeout waiting for services to become healthy")
219 case <-ticker.C:
220 allHealthy := true
221 for name, service := range ite.services {
222 if !service.Ready {
223 healthy := ite.checkServiceHealth(service)
224 ite.mu.Lock()
225 ite.services[name].Ready = healthy
226 ite.mu.Unlock()
227 if !healthy {
228 allHealthy = false
229 }
230 }
231 }
232 if allHealthy {
233 return nil
234 }
235 }
236 }
237}
238
239func checkServiceHealth(service ServiceInstance) bool {
240 // Simple HTTP health check
241 client := &http.Client{Timeout: 5 * time.Second}
242 resp, err := client.Get(service.HealthURL)
243 if err != nil {
244 return false
245 }
246 defer resp.Body.Close()
247 return resp.StatusCode == http.StatusOK
248}
249
250func Cleanup() error {
251 if !ite.config.CleanupEnabled {
252 return nil
253 }
254
255 ite.cancel()
256
257 for name, container := range ite.containers {
258 if err := container.Terminate(ite.ctx); err != nil {
259 log.Printf("Failed to terminate container %s: %v", name, err)
260 }
261 }
262
263 return nil
264}
265
266func GetServiceURL(serviceName string) string {
267 ite.mu.RLock()
268 defer ite.mu.RUnlock()
269
270 service, exists := ite.services[serviceName]
271 if !exists {
272 return ""
273 }
274
275 return fmt.Sprintf("http://%s:%d", service.Host, service.Port)
276}
277
278func GetDatabaseConnection(dbName string) {
279 ite.mu.RLock()
280 defer ite.mu.RUnlock()
281
282 db, exists := ite.databases[dbName]
283 if !exists {
284 return "", fmt.Errorf("database %s not found", dbName)
285 }
286
287 if db.Name == "postgres" {
288 return fmt.Sprintf("postgres://%s:%s@%s:%d/%s?sslmode=disable",
289 db.Username, db.Password, db.Host, db.Port, db.Database), nil
290 }
291
292 if db.Name == "redis" {
293 return fmt.Sprintf("%s:%d", db.Host, db.Port), nil
294 }
295
296 return "", fmt.Errorf("unsupported database type: %s", db.Name)
297}
Task 2: Implement Contract Testing
Create contract tests to ensure API compatibility:
1// test/contracts/user_service_contract_test.go
2package contracts
3
4import (
5 "bytes"
6 "encoding/json"
7 "net/http"
8 "testing"
9
10 "github.com/stretchr/testify/assert"
11 "github.com/stretchr/testify/require"
12)
13
14type UserContractTestSuite struct {
15 baseURL string
16 httpClient *http.Client
17}
18
19type UserRequest struct {
20 Username string `json:"username"`
21 Email string `json:"email"`
22 Password string `json:"password"`
23}
24
25type UserResponse struct {
26 ID string `json:"id"`
27 Username string `json:"username"`
28 Email string `json:"email"`
29 Created string `json:"created"`
30}
31
32type ErrorResponse struct {
33 Error string `json:"error"`
34 Message string `json:"message"`
35}
36
37func NewUserContractTestSuite(baseURL string) *UserContractTestSuite {
38 return &UserContractTestSuite{
39 baseURL: baseURL,
40 httpClient: &http.Client{Timeout: 30 * time.Second},
41 }
42}
43
44func TestCreateUserContract(t *testing.T) {
45 // Test valid user creation
46 validUser := UserRequest{
47 Username: "testuser",
48 Email: "test@example.com",
49 Password: "SecurePass123!",
50 }
51
52 body, _ := json.Marshal(validUser)
53 resp, err := ucts.httpClient.Post(
54 ucts.baseURL+"/users",
55 "application/json",
56 bytes.NewBuffer(body),
57 )
58 require.NoError(t, err)
59 defer resp.Body.Close()
60
61 // Contract: Should return 201 Created
62 assert.Equal(t, http.StatusCreated, resp.StatusCode)
63
64 // Contract: Should return proper Content-Type
65 assert.Equal(t, "application/json", resp.Header.Get("Content-Type"))
66
67 // Contract: Response should match expected schema
68 var userResp UserResponse
69 err = json.NewDecoder(resp.Body).Decode(&userResp)
70 require.NoError(t, err)
71
72 assert.NotEmpty(t, userResp.ID)
73 assert.Equal(t, validUser.Username, userResp.Username)
74 assert.Equal(t, validUser.Email, userResp.Email)
75 assert.NotEmpty(t, userResp.Created)
76}
77
78func TestCreateUserValidationContract(t *testing.T) {
79 testCases := []struct {
80 name string
81 user UserRequest
82 expectedCode int
83 expectedErr string
84 }{
85 {
86 name: "missing_username",
87 user: UserRequest{
88 Email: "test@example.com",
89 Password: "SecurePass123!",
90 },
91 expectedCode: http.StatusBadRequest,
92 expectedErr: "username is required",
93 },
94 {
95 name: "invalid_email",
96 user: UserRequest{
97 Username: "testuser",
98 Email: "invalid-email",
99 Password: "SecurePass123!",
100 },
101 expectedCode: http.StatusBadRequest,
102 expectedErr: "invalid email format",
103 },
104 {
105 name: "weak_password",
106 user: UserRequest{
107 Username: "testuser",
108 Email: "test@example.com",
109 Password: "weak",
110 },
111 expectedCode: http.StatusBadRequest,
112 expectedErr: "password must be at least 8 characters",
113 },
114 }
115
116 for _, tc := range testCases {
117 t.Run(tc.name, func(t *testing.T) {
118 body, _ := json.Marshal(tc.user)
119 resp, err := ucts.httpClient.Post(
120 ucts.baseURL+"/users",
121 "application/json",
122 bytes.NewBuffer(body),
123 )
124 require.NoError(t, err)
125 defer resp.Body.Close()
126
127 assert.Equal(t, tc.expectedCode, resp.StatusCode)
128
129 var errorResp ErrorResponse
130 err = json.NewDecoder(resp.Body).Decode(&errorResp)
131 require.NoError(t, err)
132
133 assert.Contains(t, errorResp.Message, tc.expectedErr)
134 })
135 }
136}
137
138func TestGetUserContract(t *testing.T) {
139 // First create a user
140 user := UserRequest{
141 Username: "gettest",
142 Email: "gettest@example.com",
143 Password: "SecurePass123!",
144 }
145
146 body, _ := json.Marshal(user)
147 createResp, err := ucts.httpClient.Post(
148 ucts.baseURL+"/users",
149 "application/json",
150 bytes.NewBuffer(body),
151 )
152 require.NoError(t, err)
153 defer createResp.Body.Close()
154
155 var createdUser UserResponse
156 json.NewDecoder(createResp.Body).Decode(&createdUser)
157
158 // Contract: Get user by ID
159 getResp, err := ucts.httpClient.Get(ucts.baseURL + "/users/" + createdUser.ID)
160 require.NoError(t, err)
161 defer getResp.Body.Close()
162
163 assert.Equal(t, http.StatusOK, getResp.StatusCode)
164
165 var getUser UserResponse
166 err = json.NewDecoder(getResp.Body).Decode(&getUser)
167 require.NoError(t, err)
168
169 assert.Equal(t, createdUser.ID, getUser.ID)
170 assert.Equal(t, createdUser.Username, getUser.Username)
171 assert.Equal(t, createdUser.Email, getUser.Email)
172}
173
174func TestGetNotFoundContract(t *testing.T) {
175 resp, err := ucts.httpClient.Get(ucts.baseURL + "/users/nonexistent")
176 require.NoError(t, err)
177 defer resp.Body.Close()
178
179 // Contract: Should return 404 for non-existent user
180 assert.Equal(t, http.StatusNotFound, resp.StatusCode)
181
182 var errorResp ErrorResponse
183 err = json.NewDecoder(resp.Body).Decode(&errorResp)
184 require.NoError(t, err)
185
186 assert.Equal(t, "user_not_found", errorResp.Error)
187}
Task 3: Implement Load Testing
Create comprehensive load testing scenarios:
1// test/load/user_service_load_test.go
2package load
3
4import (
5 "context"
6 "fmt"
7 "net/http"
8 "sync"
9 "sync/atomic"
10 "testing"
11 "time"
12
13 "github.com/prometheus/client_golang/prometheus"
14 "github.com/prometheus/client_golang/prometheus/promauto"
15)
16
17type LoadTestConfig struct {
18 ConcurrentUsers int
19 Duration time.Duration
20 RampUpTime time.Duration
21 BaseURL string
22}
23
24type LoadTestResults struct {
25 TotalRequests int64
26 SuccessRequests int64
27 FailedRequests int64
28 TotalDuration time.Duration
29 MinResponseTime time.Duration
30 MaxResponseTime time.Duration
31 AvgResponseTime time.Duration
32 P95ResponseTime time.Duration
33 P99ResponseTime time.Duration
34 Errors map[string]int64
35}
36
37var (
38 requestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
39 Name: "load_test_requests_total",
40 Help: "Total number of requests in load test",
41 }, []string{"method", "status"})
42
43 requestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
44 Name: "load_test_request_duration_seconds",
45 Help: "Request duration in load test",
46 Buckets: prometheus.DefBuckets,
47 }, []string{"method", "endpoint"})
48)
49
50type LoadTester struct {
51 config LoadTestConfig
52 client *http.Client
53}
54
55func NewLoadTester(config LoadTestConfig) *LoadTester {
56 return &LoadTester{
57 config: config,
58 client: &http.Client{
59 Timeout: 30 * time.Second,
60 },
61 }
62}
63
64func RunUserCreationLoadTest(t *testing.T) *LoadTestResults {
65 ctx, cancel := context.WithTimeout(context.Background(), lt.config.Duration+lt.config.RampUpTime)
66 defer cancel()
67
68 results := &LoadTestResults{
69 Errors: make(map[string]int64),
70 }
71
72 var responseTimes []int64
73 var mu sync.Mutex
74 var wg sync.WaitGroup
75
76 start := time.Now()
77
78 // Ramp up users gradually
79 userInterval := lt.config.RampUpTime / time.Duration(lt.config.ConcurrentUsers)
80
81 for i := 0; i < lt.config.ConcurrentUsers; i++ {
82 wg.Add(1)
83 go func(userID int) {
84 defer wg.Done()
85
86 // Stagger user start times
87 time.Sleep(time.Duration(userID) * userInterval)
88
89 lt.runUserLoop(ctx, userID, results, &mu, &responseTimes)
90 }(i)
91 }
92
93 wg.Wait()
94
95 results.TotalDuration = time.Since(start)
96 lt.calculateMetrics(results, responseTimes)
97
98 return results
99}
100
101func runUserLoop(ctx context.Context, userID int, results *LoadTestResults, mu *sync.Mutex, responseTimes *[]int64) {
102 ticker := time.NewTicker(100 * time.Millisecond) // 10 requests per second per user
103 defer ticker.Stop()
104
105 for {
106 select {
107 case <-ctx.Done():
108 return
109 case <-ticker.C:
110 startTime := time.Now()
111 success := lt.createUser(userID)
112 duration := time.Since(startTime)
113
114 atomic.AddInt64(&results.TotalRequests, 1)
115
116 mu.Lock()
117 *responseTimes = append(*responseTimes, duration.Nanoseconds())
118 if success {
119 atomic.AddInt64(&results.SuccessRequests, 1)
120 } else {
121 atomic.AddInt64(&results.FailedRequests, 1)
122 }
123 mu.Unlock()
124
125 // Record metrics
126 status := "200"
127 if !success {
128 status = "error"
129 }
130 requestsTotal.WithLabelValues("POST", status).Inc()
131 requestDuration.WithLabelValues("POST", "/users").Observe(duration.Seconds())
132 }
133 }
134}
135
136func createUser(userID int) bool {
137 user := map[string]interface{}{
138 "username": fmt.Sprintf("loaduser_%d_%d", userID, time.Now().UnixNano()),
139 "email": fmt.Sprintf("loaduser_%d@test.com", userID),
140 "password": "LoadTestPass123!",
141 }
142
143 resp, err := lt.client.Post(
144 lt.config.BaseURL+"/users",
145 "application/json",
146 bytes.NewBuffer(jsonMarshal(user)),
147 )
148
149 if err != nil {
150 return false
151 }
152 defer resp.Body.Close()
153
154 return resp.StatusCode == http.StatusCreated
155}
156
157func RunMixedWorkloadTest(t *testing.T) *LoadTestResults {
158 // Test combination of operations: create, get, update, delete
159 ctx, cancel := context.WithTimeout(context.Background(), lt.config.Duration)
160 defer cancel()
161
162 results := &LoadTestResults{
163 Errors: make(map[string]int64),
164 }
165
166 var responseTimes []int64
167 var mu sync.Mutex
168 var wg sync.WaitGroup
169
170 start := time.Now()
171
172 for i := 0; i < lt.config.ConcurrentUsers; i++ {
173 wg.Add(1)
174 go func(userID int) {
175 defer wg.Done()
176 lt.runMixedWorkload(ctx, userID, results, &mu, &responseTimes)
177 }(i)
178 }
179
180 wg.Wait()
181 results.TotalDuration = time.Since(start)
182 lt.calculateMetrics(results, responseTimes)
183
184 return results
185}
186
187func runMixedWorkload(ctx context.Context, userID int, results *LoadTestResults, mu *sync.Mutex, responseTimes *[]int64) {
188 var createdUserIDs []string
189
190 ticker := time.NewTicker(50 * time.Millisecond) // 20 operations per second per user
191 defer ticker.Stop()
192
193 operationCounter := 0
194
195 for {
196 select {
197 case <-ctx.Done():
198 return
199 case <-ticker.C:
200 startTime := time.Now()
201 var success bool
202
203 switch operationCounter % 4 {
204 case 0: // Create user
205 if userID := lt.createUserForMixed(userID); userID != "" {
206 createdUserIDs = append(createdUserIDs, userID)
207 success = true
208 }
209 case 1: // Get user
210 if len(createdUserIDs) > 0 {
211 success = lt.getUser(createdUserIDs[len(createdUserIDs)-1])
212 }
213 case 2: // Update user
214 if len(createdUserIDs) > 0 {
215 success = lt.updateUser(createdUserIDs[len(createdUserIDs)-1])
216 }
217 case 3: // Delete user
218 if len(createdUserIDs) > 0 {
219 userID := createdUserIDs[0]
220 success = lt.deleteUser(userID)
221 if success {
222 createdUserIDs = createdUserIDs[1:]
223 }
224 }
225 }
226
227 duration := time.Since(startTime)
228 atomic.AddInt64(&results.TotalRequests, 1)
229
230 mu.Lock()
231 *responseTimes = append(*responseTimes, duration.Nanoseconds())
232 if success {
233 atomic.AddInt64(&results.SuccessRequests, 1)
234 } else {
235 atomic.AddInt64(&results.FailedRequests, 1)
236 }
237 mu.Unlock()
238
239 operationCounter++
240 }
241 }
242}
243
244func calculateMetrics(results *LoadTestResults, responseTimes []int64) {
245 if len(responseTimes) == 0 {
246 return
247 }
248
249 // Convert to time.Duration
250 durations := make([]time.Duration, len(responseTimes))
251 for i, ns := range responseTimes {
252 durations[i] = time.Duration(ns)
253 }
254
255 // Sort for percentile calculations
256 sort.Slice(durations, func(i, j int) bool {
257 return durations[i] < durations[j]
258 })
259
260 results.MinResponseTime = durations[0]
261 results.MaxResponseTime = durations[len(durations)-1]
262
263 var total time.Duration
264 for _, d := range durations {
265 total += d
266 }
267 results.AvgResponseTime = total / time.Duration(len(durations))
268
269 // Calculate percentiles
270 p95Index := int(float64(len(durations)) * 0.95)
271 p99Index := int(float64(len(durations)) * 0.99)
272
273 if p95Index < len(durations) {
274 results.P95ResponseTime = durations[p95Index]
275 }
276 if p99Index < len(durations) {
277 results.P99ResponseTime = durations[p99Index]
278 }
279}
280
281func TestUserServiceLoad(t *testing.T) {
282 if testing.Short() {
283 t.Skip("Skipping load test in short mode")
284 }
285
286 config := LoadTestConfig{
287 ConcurrentUsers: 50,
288 Duration: 2 * time.Minute,
289 RampUpTime: 30 * time.Second,
290 BaseURL: "http://localhost:8080",
291 }
292
293 tester := NewLoadTester(config)
294
295 t.Run("UserCreationLoad", func(t *testing.T) {
296 results := tester.RunUserCreationLoadTest(t)
297
298 t.Logf("Load Test Results:")
299 t.Logf(" Total Requests: %d", results.TotalRequests)
300 t.Logf(" Success Rate: %.2f%%", float64(results.SuccessRequests)/float64(results.TotalRequests)*100)
301 t.Logf(" Requests/sec: %.2f", float64(results.TotalRequests)/results.TotalDuration.Seconds())
302 t.Logf(" Avg Response Time: %v", results.AvgResponseTime)
303 t.Logf(" P95 Response Time: %v", results.P95ResponseTime)
304 t.Logf(" P99 Response Time: %v", results.P99ResponseTime)
305
306 // Assertions
307 assert.Greater(t, results.SuccessRequests, int64(0))
308 assert.Greater(t, float64(results.SuccessRequests)/float64(results.TotalRequests), 0.95) // 95% success rate
309 assert.Less(t, results.P95ResponseTime, 1*time.Second) // P95 under 1 second
310 })
311
312 t.Run("MixedWorkloadLoad", func(t *testing.T) {
313 results := tester.RunMixedWorkloadTest(t)
314
315 t.Logf("Mixed Workload Results:")
316 t.Logf(" Total Requests: %d", results.TotalRequests)
317 t.Logf(" Success Rate: %.2f%%", float64(results.SuccessRequests)/float64(results.TotalRequests)*100)
318 t.Logf(" Requests/sec: %.2f", float64(results.TotalRequests)/results.TotalDuration.Seconds())
319 t.Logf(" Avg Response Time: %v", results.AvgResponseTime)
320
321 assert.Greater(t, results.SuccessRequests, int64(0))
322 assert.Greater(t, float64(results.SuccessRequests)/float64(results.TotalRequests), 0.90) // 90% success rate
323 })
324}
Task 4: Implement Chaos Engineering
Create chaos experiments to test system resilience:
1// test/chaos/chaos_test.go
2package chaos
3
4import (
5 "context"
6 "fmt"
7 "testing"
8 "time"
9
10 "github.com/stretchr/testify/assert"
11 "github.com/stretchr/testify/require"
12)
13
14type ChaosExperiment struct {
15 Name string
16 Hypothesis string
17 SteadyState func() error
18 Action func() error
19 Rollback func() error
20 Verify func() error
21}
22
23type ChaosTester struct {
24 experiments []ChaosExperiment
25 baseURL string
26}
27
28func NewChaosTester(baseURL string) *ChaosTester {
29 return &ChaosTester{
30 experiments: make([]ChaosExperiment, 0),
31 baseURL: baseURL,
32 }
33}
34
35func AddExperiment(experiment ChaosExperiment) {
36 ct.experiments = append(ct.experiments, experiment)
37}
38
39func RunExperiment(t *testing.T, experimentName string) {
40 var targetExperiment *ChaosExperiment
41 for _, exp := range ct.experiments {
42 if exp.Name == experimentName {
43 targetExperiment = &exp
44 break
45 }
46 }
47
48 require.NotNil(t, targetExperiment, "Experiment not found: %s", experimentName)
49
50 t.Logf("Starting chaos experiment: %s", targetExperiment.Name)
51 t.Logf("Hypothesis: %s", targetExperiment.Hypothesis)
52
53 // 1. Establish steady state
54 t.Log("Establishing steady state...")
55 err := targetExperiment.SteadyState()
56 require.NoError(t, err, "Failed to establish steady state")
57
58 // 2. Introduce chaos
59 t.Log("Introducing chaos...")
60 chaosStart := time.Now()
61 err = targetExperiment.Action()
62 require.NoError(t, err, "Failed to introduce chaos")
63
64 // 3. Wait for effects to propagate
65 time.Sleep(10 * time.Second)
66
67 // 4. Verify system still behaves as expected
68 t.Log("Verifying system behavior...")
69 err = targetExperiment.Verify()
70 chaosDuration := time.Since(chaosStart)
71
72 if err != nil {
73 t.Errorf("Chaos experiment failed: %v", err)
74 t.Logf("System failed hypothesis after %v of chaos", chaosDuration)
75 } else {
76 t.Logf("System withstood chaos for %v", chaosDuration)
77 }
78
79 // 5. Rollback chaos
80 t.Log("Rolling back chaos...")
81 err = targetExperiment.Rollback()
82 require.NoError(t, err, "Failed to rollback chaos")
83
84 // 6. Verify steady state recovery
85 t.Log("Verifying steady state recovery...")
86 time.Sleep(5 * time.Second)
87 err = targetExperiment.SteadyState()
88 assert.NoError(t, err, "System failed to recover to steady state")
89
90 t.Logf("Chaos experiment completed: %s", targetExperiment.Name)
91}
92
93func TestUserServiceResilience(t *testing.T) {
94 if testing.Short() {
95 t.Skip("Skipping chaos test in short mode")
96 }
97
98 ct := NewChaosTester("http://localhost:8080")
99
100 // Experiment 1: Database connection failure
101 ct.AddExperiment(ChaosExperiment{
102 Name: "Database Connection Failure",
103 Hypothesis: "System remains available when database is temporarily unavailable",
104 SteadyState: func() error {
105 // Verify normal operation
106 return ct.verifyServiceHealth()
107 },
108 Action: func() error {
109 // Simulate database failure
110 return ct.simulateDatabaseFailure()
111 },
112 Rollback: func() error {
113 // Restore database connection
114 return ct.restoreDatabaseConnection()
115 },
116 Verify: func() error {
117 // Verify service responds with degraded functionality
118 return ct.verifyDegradedService()
119 },
120 })
121
122 // Experiment 2: High memory pressure
123 ct.AddExperiment(ChaosExperiment{
124 Name: "High Memory Pressure",
125 Hypothesis: "System maintains functionality under memory pressure",
126 SteadyState: func() error {
127 return ct.verifyServiceHealth()
128 },
129 Action: func() error {
130 return ct.induceMemoryPressure()
131 },
132 Rollback: func() error {
133 return ct.releaseMemoryPressure()
134 },
135 Verify: func() error {
136 return ct.verifyServiceUnderStress()
137 },
138 })
139
140 // Experiment 3: Network latency injection
141 ct.AddExperiment(ChaosExperiment{
142 Name: "Network Latency Injection",
143 Hypothesis: "System handles increased network latency gracefully",
144 SteadyState: func() error {
145 return ct.verifyServiceHealth()
146 },
147 Action: func() error {
148 return ct.injectNetworkLatency()
149 },
150 Rollback: func() error {
151 return ct.removeNetworkLatency()
152 },
153 Verify: func() error {
154 return ct.verifyServiceWithLatency()
155 },
156 })
157
158 // Run experiments
159 experiments := []string{
160 "Database Connection Failure",
161 "High Memory Pressure",
162 "Network Latency Injection",
163 }
164
165 for _, expName := range experiments {
166 t.Run(expName, func(t *testing.T) {
167 ct.RunExperiment(t, expName)
168 })
169 }
170}
171
172// Helper methods for chaos experiments
173func verifyServiceHealth() error {
174 // Check if service responds normally
175 resp, err := http.Get(ct.baseURL + "/health")
176 if err != nil {
177 return err
178 }
179 defer resp.Body.Close()
180
181 if resp.StatusCode != http.StatusOK {
182 return fmt.Errorf("service not healthy: status %d", resp.StatusCode)
183 }
184
185 return nil
186}
187
188func simulateDatabaseFailure() error {
189 // This would typically use chaos mesh or similar tools
190 // For demonstration, we'll simulate by calling a chaos endpoint
191 resp, err := http.Post(ct.baseURL+"/admin/chaos/database-failure", "application/json", nil)
192 if err != nil {
193 return err
194 }
195 defer resp.Body.Close()
196
197 if resp.StatusCode != http.StatusOK {
198 return fmt.Errorf("failed to simulate database failure")
199 }
200
201 return nil
202}
203
204func restoreDatabaseConnection() error {
205 resp, err := http.Post(ct.baseURL+"/admin/chaos/database-recovery", "application/json", nil)
206 if err != nil {
207 return err
208 }
209 defer resp.Body.Close()
210
211 if resp.StatusCode != http.StatusOK {
212 return fmt.Errorf("failed to restore database connection")
213 }
214
215 return nil
216}
217
218func verifyDegradedService() error {
219 // Service should respond but may have limited functionality
220 ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
221 defer cancel()
222
223 req, _ := http.NewRequestWithContext(ctx, "GET", ct.baseURL+"/health", nil)
224 resp, err := http.DefaultClient.Do(req)
225 if err != nil {
226 return err
227 }
228 defer resp.Body.Close()
229
230 // Should respond but may indicate degraded state
231 if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusServiceUnavailable {
232 return fmt.Errorf("unexpected response code during degradation: %d", resp.StatusCode)
233 }
234
235 return nil
236}
237
238func induceMemoryPressure() error {
239 resp, err := http.Post(ct.baseURL+"/admin/chaos/memory-pressure", "application/json", nil)
240 if err != nil {
241 return err
242 }
243 defer resp.Body.Close()
244
245 return nil
246}
247
248func releaseMemoryPressure() error {
249 resp, err := http.Post(ct.baseURL+"/admin/chaos/memory-recovery", "application/json", nil)
250 if err != nil {
251 return err
252 }
253 defer resp.Body.Close()
254
255 return nil
256}
257
258func verifyServiceUnderStress() error {
259 // Service should still respond but may be slower
260 start := time.Now()
261 resp, err := http.Get(ct.baseURL + "/health")
262 duration := time.Since(start)
263
264 if err != nil {
265 return err
266 }
267 defer resp.Body.Close()
268
269 if resp.StatusCode != http.StatusOK {
270 return fmt.Errorf("service failed under stress: status %d", resp.StatusCode)
271 }
272
273 // Allow for slower response under stress
274 if duration > 5*time.Second {
275 return fmt.Errorf("response time too high under stress: %v", duration)
276 }
277
278 return nil
279}
280
281func injectNetworkLatency() error {
282 resp, err := http.Post(ct.baseURL+"/admin/chaos/network-latency", "application/json", nil)
283 if err != nil {
284 return err
285 }
286 defer resp.Body.Close()
287
288 return nil
289}
290
291func removeNetworkLatency() error {
292 resp, err := http.Post(ct.baseURL+"/admin/chaos/network-recovery", "application/json", nil)
293 if err != nil {
294 return err
295 }
296 defer resp.Body.Close()
297
298 return nil
299}
300
301func verifyServiceWithLatency() error {
302 // Service should handle latency gracefully
303 start := time.Now()
304 resp, err := http.Get(ct.baseURL + "/health")
305 duration := time.Since(start)
306
307 if err != nil {
308 return err
309 }
310 defer resp.Body.Close()
311
312 if resp.StatusCode != http.StatusOK {
313 return fmt.Errorf("service failed with network latency: status %d", resp.StatusCode)
314 }
315
316 // Should account for injected latency
317 if duration > 10*time.Second {
318 return fmt.Errorf("response time too high with latency: %v", duration)
319 }
320
321 return nil
322}
Solution Approach
Click to see detailed solution
This would include complete implementations of all testing strategies, test utilities, and comprehensive test suites for ensuring system reliability and performance.
Running Tests
1. Integration Tests
1# Run all integration tests
2go test -v ./test/integration/...
3
4# Run specific integration test
5go test -v ./test/integration/ -run TestUserServiceIntegration
6
7# Run with coverage
8go test -v -cover ./test/integration/...
2. Contract Tests
1# Run contract tests
2go test -v ./test/contracts/...
3
4# Generate contract documentation
5go test -v ./test/contracts/ -contract-report
3. Load Tests
1# Run quick load test
2go test -v ./test/load/ -short
3
4# Run full load test suite
5go test -v ./test/load/ -timeout 30m
6
7# Run with different configurations
8go test -v ./test/load/ -users=100 -duration=10m
4. Chaos Tests
1# Run chaos experiments
2go test -v ./test/chaos/...
3
4# Run specific chaos experiment
5go test -v ./test/chaos/ -run TestDatabaseFailure
6
7# Run with monitoring
8go test -v ./test/chaos/ -monitor
Extension Challenges
- Add property-based testing - Use Go's fuzzing for comprehensive testing
- Add visual regression testing - Test UI components across browsers
- Add security testing - Implement automated security scanning
- Add compliance testing - Test for regulatory compliance
- Add cross-service testing - Test entire workflows across services
Key Takeaways
- Integration tests verify service interactions and end-to-end functionality
- Contract tests ensure API compatibility between services
- Load tests validate system performance under expected and peak loads
- Chaos engineering builds confidence in system resilience
- Test automation is essential for continuous delivery
- Comprehensive testing reduces production issues and improves reliability
- Test environments should closely mirror production configurations
This exercise demonstrates advanced testing strategies that are essential for building reliable, scalable microservices applications with confidence in their behavior under various conditions.