Overview

This PR implements Step 1 of the plan to eliminate unnecessary goroutines from the Lambda layer (issue #217).

Currently, each Lambda Function maintains a pool of Lambda Instances where each instance runs in its own goroutine waiting on channels. This creates significant memory overhead and unnecessary complexity.

SandboxSet provides a simple, thread-safe pool of sandboxes using mutex-based synchronization instead of goroutines and channels.

Changes

New Files

• go/worker/sandbox/sandboxSet.go - Core SandboxSet implementation (920 LOC)
• go/worker/sandbox/sandboxSetAPI.go - Enhanced API with options pattern, metrics, and events (449 LOC)
• go/worker/sandbox/mock_test.go - MockSandbox for testing without real containers (247 LOC)
• go/worker/sandbox/sandboxSet_unit_test.go - 22 comprehensive unit tests (645 LOC)
• go/worker/sandbox/sandboxSet_test.go - Integration tests with real Docker containers (450 LOC)

Architecture

Current Flow (with Lambda Instance goroutines):

HTTP Request
    ↓
Lambda Function
    ↓ [send via channel]
Lambda Instance [goroutine waiting] ← 8KB per instance
    ↓
Sandbox

New Flow (with SandboxSet - Steps 2&3):

HTTP Request
    ↓
Lambda Function
    ↓ [direct call]
SandboxSet [mutex-protected pool] ← ~500 bytes total
    ↓
Sandbox

Core API

type SandboxSet struct {
    mutex        sync.RWMutex
    sandboxes    []*sandboxWrapper
    config       *SandboxSetConfig
}

func (set *SandboxSet) GetSandbox(opts ...GetOption) (Sandbox, error)
func (set *SandboxSet) ReleaseSandbox(sb Sandbox, opts ...ReleaseOption) error
func (set *SandboxSet) DestroyAndRemove(sb Sandbox, reason string) error
func (set *SandboxSet) Warm(target int) error
func (set *SandboxSet) Shrink(target int) error
func (set *SandboxSet) Stats() map[string]int
func (set *SandboxSet) Metrics() *PoolMetrics

Key Features

• Options pattern for flexible configuration (timeout, health checks)
• Rich metrics and observability (hit rate, capacity utilization, error tracking)
• Event system for monitoring sandbox lifecycle
• Operational controls for warming and shrinking pools
• Health checks with timeout protection using split-lock pattern
• RWMutex for concurrent read access while protecting writes

Design Decisions

• Linear search instead of priority queues (pools are typically 5-10 sandboxes)
• RWMutex allows concurrent reads while protecting writes
• Options pattern enables future enhancements without API changes
• Event system provides hooks for custom monitoring
• All operations are atomic and thread-safe

Testing

22 unit tests using MockSandbox run in 0.172 seconds. Tests cover basic operations, reuse, pool growth, capacity, health checks, concurrent access, warm/shrink, options pattern, metrics, events, and error handling.

9 integration tests with real Docker containers run in 16.6 seconds. Tests verify container lifecycle, sandbox reuse, dynamic pool growth, capacity enforcement, destroy/remove operations, pause/unpause health checks, failed health check detection, concurrent access (12 goroutines with 12 real containers), and error handling.

Next Steps

Step 2: Replace ImportCacheNode with SandboxSet to eliminate complex reference counting logic.

Step 3: Eliminate LambdaInstance goroutines by having Lambda Function call SandboxSet directly.

Performance Impact

Estimated memory savings: 8KB per Lambda Instance reduced to approximately 500 bytes per SandboxSet pool (94% reduction in per-pool overhead).

Concurrency improvements: RWMutex enables concurrent reads, eliminates channel blocking and goroutine scheduling overhead, replaces message passing with direct function calls.

Backward Compatibility

This PR is fully backward compatible. Existing sandbox code is unchanged, new SandboxSet is opt-in and not yet integrated, and all existing tests pass.