Supported Spark Expressions

The following Spark expressions are currently available. Any known compatibility issues are noted in the following tables.

Literal Values

Expression

Notes

Literal values of supported data types

Unary Arithmetic

Expression

Notes

UnaryMinus (-)

Binary Arithmetic

Expression

Notes

Add (+)

Subtract (-)

Multiply (*)

Divide (/)

Remainder (%)

Conditional Expressions

Expression

Notes

CaseWhen

If

Comparison

Expression

Notes

EqualTo (=)

EqualNullSafe (<=>)

GreaterThan (>)

GreaterThanOrEqual (>=)

LessThan (<)

LessThanOrEqual (<=)

IsNull (IS NULL)

IsNotNull (IS NOT NULL)

In (IN)

String Functions

Expression

Notes

Ascii

BitLength

Chr

ConcatWs

Contains

EndsWith

InitCap

Instr

Length

Like

Lower

OctetLength

Repeat

Negative argument for number of times to repeat causes exception

Replace

Reverse

StartsWith

StringSpace

StringTrim

StringTrimBoth

StringTrimLeft

StringTrimRight

Substring

Translate

Upper

Date/Time Functions

Expression

Notes

DatePart

Only year is supported

Extract

Only year is supported

Hour

Minute

Second

TruncDate

TruncTimestamp

Year

Math Expressions

Expression

Notes

Abs

Acos

Asin

Atan

Atan2

Ceil

Cos

Exp

Floor

IsNaN

Log

Log2

Log10

Pow

Round

Signum

Signum does not differentiate between 0.0 and -0.0

Sin

Sqrt

Tan

Hashing Functions

Expression

Notes

Md5

Hash

Sha2

XxHash64

Boolean Expressions

Expression

Notes

And

Or

Not

Bitwise Expressions

Expression

Notes

ShiftLeft (<<)

ShiftRight (>>)

BitAnd (&)

BitOr (|)

BitXor (^)

BitwiseNot (~)

BoolAnd (bool_and)

BoolOr (bool_or)

Aggregate Expressions

Expression

Notes

Avg

BitAndAgg

BitOrAgg

BitXorAgg

Corr

Count

CovPopulation

CovSample

First

Last

Max

Min

StddevPop

StddevSamp

Sum

VariancePop

VarianceSamp

Complex Types

Expression

Notes

CreateNamedStruct

ElementAt

Arrays only

GetArrayItem

GetStructField

StructsToJson

Other

Expression

Notes

Cast

See compatibility guide for list of supported cast expressions and known issues

BloomFilterMightContain

ScalarSubquery

Coalesce

NormalizeNaNAndZero