Warning
This is out-of-date documentation. The latest Comet release is version 0.10.0.
Supported Spark Expressions¶
The following Spark expressions are currently available. Any known compatibility issues are noted in the following tables.
Literal Values¶
Expression |
Notes |
|---|---|
Literal values of supported data types |
Unary Arithmetic¶
Expression |
Notes |
|---|---|
UnaryMinus ( |
Binary Arithmetic¶
Expression |
Notes |
|---|---|
Add ( |
|
Subtract ( |
|
Multiply ( |
|
Divide ( |
|
IntegralDivide ( |
|
Remainder ( |
Conditional Expressions¶
Expression |
Notes |
|---|---|
CaseWhen |
|
If |
Comparison¶
Expression |
Notes |
|---|---|
EqualTo ( |
|
EqualNullSafe ( |
|
GreaterThan ( |
|
GreaterThanOrEqual ( |
|
LessThan ( |
|
LessThanOrEqual ( |
|
IsNull ( |
|
IsNotNull ( |
|
In ( |
String Functions¶
Expression |
Notes |
|---|---|
Ascii |
|
BitLength |
|
Chr |
|
ConcatWs |
|
Contains |
|
EndsWith |
|
InitCap |
|
Instr |
|
Length |
|
Like |
|
Lower |
|
OctetLength |
|
Repeat |
Negative argument for number of times to repeat causes exception |
Replace |
|
Reverse |
|
StartsWith |
|
StringRPad |
|
StringSpace |
|
StringTrim |
|
StringTrimBoth |
|
StringTrimLeft |
|
StringTrimRight |
|
Substring |
|
Translate |
|
Upper |
Date/Time Functions¶
Expression |
Notes |
|---|---|
DatePart |
Only |
Extract |
Only |
Hour |
|
Minute |
|
Second |
|
TruncDate |
|
TruncTimestamp |
|
Year |
Math Expressions¶
Expression |
Notes |
|---|---|
Abs |
|
Acos |
|
Asin |
|
Atan |
|
Atan2 |
|
Ceil |
|
Cos |
|
Exp |
|
Floor |
|
IsNaN |
|
Log |
|
Log2 |
|
Log10 |
|
Pow |
|
Round |
|
Signum |
|
Sin |
|
Sqrt |
|
Tan |
Hashing Functions¶
Expression |
Notes |
|---|---|
Md5 |
|
Hash |
|
Sha2 |
|
XxHash64 |
Boolean Expressions¶
Expression |
Notes |
|---|---|
And |
|
Or |
|
Not |
Bitwise Expressions¶
Expression |
Notes |
|---|---|
ShiftLeft ( |
|
ShiftRight ( |
|
BitAnd ( |
|
BitOr ( |
|
BitXor ( |
|
BitwiseNot ( |
|
BoolAnd ( |
|
BoolOr ( |
Aggregate Expressions¶
Expression |
Notes |
|---|---|
Avg |
|
BitAndAgg |
|
BitOrAgg |
|
BitXorAgg |
|
Corr |
|
Count |
|
CovPopulation |
|
CovSample |
|
First |
|
Last |
|
Max |
|
Min |
|
StddevPop |
|
StddevSamp |
|
Sum |
|
VariancePop |
|
VarianceSamp |
Arrays¶
Expression |
Notes |
|---|---|
ArrayAppend |
Experimental |
ArrayCompact |
Experimental |
ArrayContains |
Experimental |
ArrayDistinct |
Experimental: behaves differently than spark. Datafusion first sorts then removes duplicates while spark preserves the original order. |
ArrayExcept |
Experimental |
ArrayInsert |
Experimental |
ArrayIntersect |
Experimental |
ArrayJoin |
Experimental |
ArrayMax |
Experimental |
ArrayRemove |
|
ArrayRepeat |
Experimental |
ArraysOverlap |
Experimental |
ArrayUnion |
Experimental: behaves differently than spark. Datafusion sorts the input arrays before performing the union, while spark preserves the order of the first array and appends unique elements from the second. |
ElementAt |
Arrays only |
GetArrayItem |
Structs¶
Expression |
Notes |
|---|---|
CreateNamedStruct |
|
GetStructField |
|
StructsToJson |
Other¶
Expression |
Notes |
|---|---|
Cast |
See compatibility guide for list of supported cast expressions and known issues |
BloomFilterMightContain |
|
ScalarSubquery |
|
Coalesce |
|
NormalizeNaNAndZero |
|
ToPrettyString |