The Spark SQL Statement task executes a Spark SQL script on a Spark enabled connection.
As is the case with the SQL Statement task, this task does not show output and is intended for creating and modifying database tables.
Currently the Spark SQL Statement task type only supports execution on Databricks connections. For general Hadoop Spark invocation use the Spark task type.
Once the Spark SQL task type is selected, all that’s required for the Spark SQL task is a valid connection and the statement to run.
For Databricks connections you will be required to select the cluster definition or existing cluster to use for the statement.
The task type accepts any valid Spark SQL statement, and will submit the statement as part of the task. Currently Spark SQL Statement do not have the ability to be validated so Loome Integrate cannot check the syntax of your statements.
You can enable a setting for this task to run scripts in parallel. This will allow the Loome Integrate agent to execute multiple scripts in parallel, rather than in sequence. Learn more here.