Summary

This PR implements a flexible third-party dependency management system for paimon-cpp, allowing users to control how dependencies are resolved and built.

Relates to #103

Key Features

1. Global Dependency Source Control

• PAIMON_DEPENDENCY_SOURCE: Control all dependencies at once
  • AUTO (default): Try system libraries first, fall back to bundled
  • BUNDLED: Always build from source (current behavior)
  • SYSTEM: Use only system libraries
  • CONDA: Use conda environment libraries

2. Per-Dependency Control

• <PACKAGE>_SOURCE: Override individual dependencies
• <PACKAGE>_ROOT: Specify installation directory
• PAIMON_PACKAGE_PREFIX: Set unified path prefix

3. Library Linkage Control

• PAIMON_DEPENDENCY_USE_SHARED: Global shared/static control
• Per-dependency options: PAIMON_<PKG>_USE_SHARED

Changes

• cmake_modules/DefineOptions.cmake: Add dependency management options
• cmake_modules/ThirdpartyToolchain.cmake: Implement resolve_dependency() and build_dependency() macros
• cmake_modules/Find*Alt.cmake: Add Find modules for Snappy, zstd, lz4, glog
• docs/DependencyManagement.md: Complete documentation with examples

Benefits

• ⚡ Faster builds: Reuse pre-installed libraries
• 🔧 Flexible deployment: Support diverse environments (conda, vcpkg, system packages)
• 🚀 Better CI/CD: Cache dependencies across builds
• 🔄 Backward compatible: Defaults to AUTO mode, existing builds work unchanged
• 📦 Ecosystem friendly: Works with conda, vcpkg, brew, apt, yum, etc.

Example Usage

Use System Libraries

cmake -B build -DPAIMON_DEPENDENCY_SOURCE=SYSTEM \
  -DSnappy_ROOT=/usr/local \
  -Dzstd_ROOT=/usr/local

Mixed Approach

cmake -B build -DPAIMON_DEPENDENCY_SOURCE=AUTO \
  -DSnappy_SOURCE=SYSTEM \
  -Dzstd_SOURCE=BUNDLED

Conda Environment

cmake -B build -DPAIMON_DEPENDENCY_SOURCE=CONDA

Implementation Status

Currently supported dependencies with flexible resolution:

• ✅ Snappy (compression)
• ✅ zstd (compression)
• ✅ lz4 (compression)
• ✅ glog (logging)
• 🔄 zlib, fmt, RapidJSON, Arrow, tbb (basic support)
• 🔄 Protobuf, avro, orc (basic support)

TODO (before ready for review)

• Test builds in different modes (AUTO, BUNDLED, SYSTEM)
• Test with system-installed libraries
• Test with conda environment
• Add CI workflow to test different dependency modes
• Consider adding Find modules for more dependencies (Arrow, fmt, etc.)
• Add troubleshooting section to docs based on testing

Testing Checklist

• Build with BUNDLED mode (verify backward compatibility)
• Build with AUTO mode with no system deps (should fall back to bundled)
• Build with AUTO mode with some system deps available
• Build with SYSTEM mode with all deps available
• Build in conda environment with CONDA mode
• Verify all existing tests pass
• Test with mixed SOURCE settings

References

• Issue [Feature] Flexible Third-Party Dependency Management System #103: Feature request discussion
• Apache Arrow implementation: ThirdpartyToolchain.cmake
• Documentation: DependencyManagement.md

Note: This PR is marked as draft while testing is in progress. Will mark as ready for review once all testing is complete.