Comet Contributor Guide

Contributor Guide

• Getting Started
  • Finding issues to work on
  • Reporting issues
  • Asking for Help
  • Regular public meetings
• Comet Plugin Architecture
  • Overview
• Plugin Components
  • Comet SQL Plugin
  • CometSparkSessionExtensions
  • Query Execution
  • Shuffle
• Arrow FFI
  • Overview
  • Arrow FFI Basics
  • JVM → Native Data Flow (ScanExec)
  • Native → JVM Data Flow (CometExecIterator)
  • Memory Ownership Rules
  • Further Reading
• JVM Shuffle
  • Overview
  • When JVM Shuffle is Used
  • Input Handling
  • Architecture
  • Key Classes
  • Data Flow
  • Memory Management
  • Configuration
• Native Shuffle
  • Overview
  • When Native Shuffle is Used
  • Architecture
  • Key Classes
  • Data Flow
  • Partitioning
  • Memory Management
  • Compression
  • Configuration
  • Comparison with JVM Shuffle
• Parquet Scans
  • S3 Support
• Development Guide
  • Project Layout
  • Development Setup
  • Build & Test
  • Common Build and Test Pitfalls
  • Development Environment
  • Plan Stability Testing
  • Benchmark
  • Debugging
  • Submitting a Pull Request
  • How to format .md document
• Debugging Guide
  • Debugging for Advanced Developers
  • Verbose debug
• Benchmarking Guide
  • Current Benchmark Results
  • Benchmarking Guides
• Adding a New Operator
  • Overview
  • Implementing a Native Operator
  • Implementing a Sink Operator
  • Implementing a JVM Operator
  • Common Patterns and Helpers
  • Debugging Tips
• Adding a New Expression
  • Finding an Expression to Add
  • Implementing the Expression
  • Shimming to Support Different Spark Versions
  • Resources
• Tracing
  • Definition of Labels
• Profiling Native Code
  • Running micro benchmarks with cargo bench
  • Profiling with cargo-flamegraph
• Spark SQL Tests
  • 1. Install Comet
  • 2. Clone Spark and Apply Diff
  • 3. Run Spark SQL Tests
  • Creating a diff file for a new Spark version
  • Generating The Diff File
  • Running Tests in CI
• SQL File Tests
  • Running the tests
  • Test file location
  • File format
  • Adding a new test
  • Handling test failures
  • Architecture
• Roadmap
  • Major Initiatives
  • Ongoing Improvements
• Github and Issue Tracker