Warning
This is out-of-date documentation. The latest Comet release is version 0.10.0.
Supported Spark Expressions¶
The following Spark expressions are currently available. Any known compatibility issues are noted in the following tables.
Literal Values¶
Expression |
Notes |
|---|---|
Literal values of supported data types |
Unary Arithmetic¶
Expression |
Notes |
|---|---|
UnaryMinus ( |
Binary Arithmetic¶
Expression |
Notes |
|---|---|
Add ( |
|
Subtract ( |
|
Multiply ( |
|
Divide ( |
|
IntegralDivide ( |
|
Remainder ( |
Conditional Expressions¶
Expression |
Notes |
|---|---|
CaseWhen |
|
If |
Comparison¶
Expression |
Notes |
|---|---|
EqualTo ( |
|
EqualNullSafe ( |
|
GreaterThan ( |
|
GreaterThanOrEqual ( |
|
LessThan ( |
|
LessThanOrEqual ( |
|
IsNull ( |
|
IsNotNull ( |
|
In ( |
String Functions¶
Expression |
Notes |
|---|---|
Ascii |
|
BitLength |
|
Chr |
|
ConcatWs |
|
Contains |
|
EndsWith |
|
InitCap |
|
Instr |
|
Length |
|
Like |
|
Lower |
|
OctetLength |
|
Repeat |
Negative argument for number of times to repeat causes exception |
Replace |
|
Reverse |
|
StartsWith |
|
StringRPad |
|
StringSpace |
|
StringTrim |
|
StringTrimBoth |
|
StringTrimLeft |
|
StringTrimRight |
|
Substring |
|
Translate |
|
Upper |
Date/Time Functions¶
Expression |
Notes |
|---|---|
DatePart |
Only |
Extract |
Only |
Hour |
|
Minute |
|
Second |
|
TruncDate |
|
TruncTimestamp |
|
Year |
Math Expressions¶
Expression |
Notes |
|---|---|
Abs |
|
Acos |
|
Asin |
|
Atan |
|
Atan2 |
|
Ceil |
|
Cos |
|
Exp |
|
Floor |
|
IsNaN |
|
Log |
|
Log2 |
|
Log10 |
|
Pow |
|
Round |
|
Signum |
Signum does not differentiate between |
Sin |
|
Sqrt |
|
Tan |
Hashing Functions¶
Expression |
Notes |
|---|---|
Md5 |
|
Hash |
|
Sha2 |
|
XxHash64 |
Boolean Expressions¶
Expression |
Notes |
|---|---|
And |
|
Or |
|
Not |
Bitwise Expressions¶
Expression |
Notes |
|---|---|
ShiftLeft ( |
|
ShiftRight ( |
|
BitAnd ( |
|
BitOr ( |
|
BitXor ( |
|
BitwiseNot ( |
|
BoolAnd ( |
|
BoolOr ( |
Aggregate Expressions¶
Expression |
Notes |
|---|---|
Avg |
|
BitAndAgg |
|
BitOrAgg |
|
BitXorAgg |
|
Corr |
|
Count |
|
CovPopulation |
|
CovSample |
|
First |
|
Last |
|
Max |
|
Min |
|
StddevPop |
|
StddevSamp |
|
Sum |
|
VariancePop |
|
VarianceSamp |
Arrays¶
Expression |
Notes |
|---|---|
ArrayAppend |
Experimental |
ArrayContains |
Experimental |
ArrayIntersect |
Experimental |
ArrayJoin |
Experimental |
ArrayRemove |
|
ArraysOverlap |
Experimental |
ElementAt |
Arrays only |
GetArrayItem |
Structs¶
Expression |
Notes |
|---|---|
CreateNamedStruct |
|
GetStructField |
|
StructsToJson |
Other¶
Expression |
Notes |
|---|---|
Cast |
See compatibility guide for list of supported cast expressions and known issues |
BloomFilterMightContain |
|
ScalarSubquery |
|
Coalesce |
|
NormalizeNaNAndZero |