Apache Arrow Type Support in SeaORM

This document explains Apache Arrow's decimal and timestamp formats and their integration with SeaORM.

Arrow Decimal Types Overview

Apache Arrow provides fixed-point decimal types for representing precise numeric values with a specific precision and scale:

1. Decimal64

• Storage: 64-bit (8 bytes) fixed-point decimal
• Precision: Up to 18 decimal digits
• Format: DataType::Decimal64(precision, scale)
  • precision (u8): Total number of decimal digits (1-18)
  • scale (i8): Number of digits after decimal point (can be negative)
• Use case: Compact precision decimals that fit in an i64 (e.g., prices, exchange rates)
• Example: Decimal64(10, 2) represents numbers like 12345678.90

2. Decimal128

• Storage: 128-bit (16 bytes) fixed-point decimal
• Precision: Up to 38 decimal digits
• Format: DataType::Decimal128(precision, scale)
  • precision (u8): Total number of decimal digits (1-38)
  • scale (i8): Number of digits after decimal point (can be negative)
• Use case: Standard precision decimals (e.g., financial calculations requiring > 18 digits)
• Example: Decimal128(20, 4) represents numbers like 9999999999999999.9999

3. Decimal256

• Storage: 256-bit (32 bytes) fixed-point decimal
• Precision: Up to 76 decimal digits
• Format: DataType::Decimal256(precision, scale)
  • precision (u8): Total number of decimal digits (1-76)
  • scale (i8): Number of digits after decimal point (can be negative)
• Use case: High-precision scientific calculations, very large numbers
• Example: Decimal256(76, 20) for extreme precision requirements

Precision vs Scale

• Precision: Total number of significant digits (before + after decimal point)
• Scale: Number of digits after the decimal point
  • Positive scale: digits after decimal (e.g., scale=2 → 123.45)
  • Zero scale: integer (e.g., scale=0 → 12345)
  • Negative scale: multiplier (e.g., scale=-2 → 12300 stored as 123)

Examples:

Decimal64(10, 2)   → Can store: -99999999.99 to 99999999.99 (compact, i64)
Decimal64(5, 0)    → Can store: -99999 to 99999 (integers, compact)
Decimal128(20, 4)  → Can store: -9999999999999999.9999 to 9999999999999999.9999
Decimal128(10, 4)  → Can store: -999999.9999 to 999999.9999
Decimal256(38, 10) → High precision: up to 28 digits before, 10 after decimal

SeaORM Decimal Support

SeaORM supports two Rust decimal libraries:

1. rust_decimal (feature: with-rust_decimal)

• Type: rust_decimal::Decimal
• Precision: 28-29 significant digits
• Scale: 0-28
• Storage: 128-bit
• Value type: Value::Decimal
• Best for: Most business applications, financial calculations

2. bigdecimal (feature: with-bigdecimal)

• Type: bigdecimal::BigDecimal
• Precision: Arbitrary (limited by memory)
• Scale: Arbitrary
• Storage: Variable (uses BigInt internally)
• Value type: Value::BigDecimal
• Best for: Arbitrary precision requirements, scientific computing

Arrow → SeaORM Mapping

Decimal64Array → rust_decimal::Decimal

• When: Feature with-rust_decimal is enabled
• Limitation: Precision ≤ 18 (fits in i64)
• Conversion: Cast i64 to i128, then Decimal::from_i128_with_scale()
• Column Type: ColumnType::Decimal(Some((precision, scale)))

Decimal64Array → bigdecimal::BigDecimal

• When: Feature with-bigdecimal is enabled (fallback if rust_decimal not available)
• Conversion: Convert i64 via BigInt
• Column Type: ColumnType::Decimal(Some((precision, scale)))

Decimal128Array → rust_decimal::Decimal

• When: Feature with-rust_decimal is enabled
• Limitation: Precision ≤ 28, Scale ≤ 28
• Conversion: Direct mapping using i128 to Decimal::from_i128_with_scale()
• Column Type: ColumnType::Decimal(Some((precision, scale)))

Decimal128Array → bigdecimal::BigDecimal

• When: Feature with-bigdecimal is enabled (fallback if rust_decimal fails or not available)
• Limitation: None (arbitrary precision)
• Conversion: Convert via BigInt
• Column Type: ColumnType::Decimal(Some((precision, scale))) or ColumnType::Money(_)

Decimal256Array → bigdecimal::BigDecimal

• When: Feature with-bigdecimal is enabled
• Required: BigDecimal for precision > 38
• Conversion: Convert via byte array to BigInt, then apply scale
• Column Type: ColumnType::Decimal(Some((precision, scale)))

Implementation Strategy

1. Decimal64Array:

   • Try with-rust_decimal first (always fits, precision ≤ 18)
   • Fallback to with-bigdecimal if needed
   • Return type error if neither feature is enabled

2. Decimal128Array:

   • Try with-rust_decimal first (if precision/scale fit)
   • Fallback to with-bigdecimal if needed
   • Return type error if neither feature is enabled

3. Decimal256Array:

   • Requires with-bigdecimal (rust_decimal can't handle precision > 28)
   • Convert byte representation to BigInt
   • Apply scale to create BigDecimal

4. Null Handling:

   • Return Value::Decimal(None) or Value::BigDecimal(None) for null values

Apache Arrow Timestamp Types Support

Arrow provides several temporal types for representing dates, times, and timestamps with varying precision.

Arrow Temporal Types Overview

Date Types

1. Date32

• Storage: 32-bit signed integer
• Unit: Days since Unix epoch (1970-01-01)
• Format: DataType::Date32
• Range: Approximately ±5.8 million years from epoch
• Use case: Calendar dates without time component

2. Date64

• Storage: 64-bit signed integer
• Unit: Milliseconds since Unix epoch
• Format: DataType::Date64
• Range: Much larger than Date32
• Use case: Dates with millisecond precision (though time is typically zeroed)

Time Types

1. Time32

• Storage: 32-bit signed integer
• Units: Second or Millisecond
• Variants:
  • Time32(TimeUnit::Second) - seconds since midnight
  • Time32(TimeUnit::Millisecond) - milliseconds since midnight
• Range: 0 to 86,399 seconds (00:00:00 to 23:59:59)
• Use case: Time of day without date

2. Time64

• Storage: 64-bit signed integer
• Units: Microsecond or Nanosecond
• Variants:
  • Time64(TimeUnit::Microsecond) - microseconds since midnight
  • Time64(TimeUnit::Nanosecond) - nanoseconds since midnight
• Range: 0 to 86,399,999,999,999 nanoseconds
• Use case: High-precision time of day

Timestamp Types

Timestamp types represent absolute points in time with optional timezone.

• Storage: 64-bit signed integer
• Units: Second, Millisecond, Microsecond, or Nanosecond
• Timezone: Optional timezone string (e.g., "UTC", "America/New_York")
• Format: Timestamp(TimeUnit, Option<String>)

Variants:

DataType::Timestamp(TimeUnit::Second, None)        // No timezone
DataType::Timestamp(TimeUnit::Millisecond, None)   // No timezone
DataType::Timestamp(TimeUnit::Microsecond, None)   // No timezone
DataType::Timestamp(TimeUnit::Nanosecond, None)    // No timezone

DataType::Timestamp(TimeUnit::Second, Some("UTC".into()))      // With timezone
DataType::Timestamp(TimeUnit::Microsecond, Some("UTC".into())) // With timezone
DataType::Timestamp(TimeUnit::Nanosecond, Some("UTC".into()))  // With timezone

TimeUnit Precision:

• Second: 1 second precision (1,000,000,000 ns)
• Millisecond: 1 millisecond precision (1,000,000 ns)
• Microsecond: 1 microsecond precision (1,000 ns)
• Nanosecond: 1 nanosecond precision (highest)

SeaORM Temporal Type Support

SeaORM supports two Rust datetime libraries for temporal types:

1. chrono (feature: with-chrono) - Preferred

• Type Mappings:
  • chrono::NaiveDate - Date without timezone
  • chrono::NaiveTime - Time without date/timezone
  • chrono::NaiveDateTime - DateTime without timezone
  • chrono::DateTime<Utc> - DateTime with UTC timezone
• Value types:
  • Value::ChronoDate(Option<NaiveDate>)
  • Value::ChronoTime(Option<NaiveTime>)
  • Value::ChronoDateTime(Option<NaiveDateTime>)
  • Value::ChronoDateTimeUtc(Option<DateTime<Utc>>)
• Best for: Most applications needing date/time support

2. time (feature: with-time) - Alternative

• Type Mappings:
  • time::Date - Date without timezone
  • time::Time - Time without date/timezone
  • time::PrimitiveDateTime - DateTime without timezone
  • time::OffsetDateTime - DateTime with timezone offset
• Value types:
  • Value::TimeDate(Option<Date>)
  • Value::TimeTime(Option<Time>)
  • Value::TimeDateTime(Option<PrimitiveDateTime>)
  • Value::TimeDateTimeWithTimeZone(Option<OffsetDateTime>)
• Best for: Projects preferring the time crate ecosystem

Feature Priority

• Both enabled: Prefers chrono, with automatic fallback to time if type mismatch
• Only chrono: Uses chrono types exclusively
• Only time: Uses time crate types exclusively

Arrow → SeaORM Timestamp Mapping

Date32/Date64Array → Date

chrono:

Date32Array → chrono::NaiveDate
Date64Array → chrono::NaiveDate

• Conversion: Calculate days from epoch (1970-01-01) and add/subtract

time:

Date32Array → time::Date
Date64Array → time::Date

• Conversion: Julian day calculation (Unix epoch = Julian day 2,440,588)

Time32/Time64Array → Time

chrono:

Time32(Second)      → chrono::NaiveTime
Time32(Millisecond) → chrono::NaiveTime
Time64(Microsecond) → chrono::NaiveTime
Time64(Nanosecond)  → chrono::NaiveTime

• Conversion: Break down time units into (hours, minutes, seconds, nanoseconds)

time:

Time32(Second)      → time::Time
Time32(Millisecond) → time::Time
Time64(Microsecond) → time::Time
Time64(Nanosecond)  → time::Time

• Conversion: Extract hours, minutes, seconds, and nanoseconds from total time value

TimestampArray → DateTime

Without Timezone:

• Arrow: Timestamp(TimeUnit, None)
• chrono: → chrono::NaiveDateTime
• time: → time::PrimitiveDateTime
• Column Type: ColumnType::DateTime or ColumnType::Timestamp

With Timezone:

• Arrow: Timestamp(TimeUnit, Some(tz))
• chrono: → chrono::DateTime<Utc>
• time: → time::OffsetDateTime
• Column Type: ColumnType::TimestampWithTimeZone

Conversion Details by TimeUnit

References

• Arrow DataType Documentation
• Arrow Decimal Module
• Decimal256Type
• Arrow Temporal Types