Shipyard HPC Script

A Shipyard HPC Script task can execute a HPC job using Azure Batch Shipyard.

You may need to set up connections, such as your container registry, on the connections page and a cluster definition before you can create and submit this task.

For a Shipyard HPC Script task, you will need:

  • Azure Batch Connection
  • Azure Blob Connection (Azure Storage connection)
  • Cluster Definition
  • Container Type (Docker/Singularity)
  • Container Image
  • Script

Optional fields:

  • Container Registry Connection: Providing a custom container registry will allow for the usage of Container images not available from the Docker Hub or other Container Libraries.
  • Signing Key Fingerprint: You can provide this if you have selected a Singularity container
  • Shared Volume Connection: Select an Azure Storage Connection. This allows for an Azure Blob or File Share resource to be mounted to the container as a shared volume.
  • Volume File Definition
  • Task Parameters

Containers

You have the option of using either Docker or Singularity containers in a Shipyard HPC script task. You can specify which container you would like to use as part of the shipyard process in the field Container Type.

If you choose to use a Singularity container, you also have the option to provide a ‘Signing Key Fingerprint’.

Singularity container and signing key fingerprint fields

Parameters

Nvidia Driver

You can specify a different nvidia driver for a different workloads using a new parameter.

You can create a parameter with the name NvidiaSource.

The value for this parameter would be the public URL of the Nvidia driver that you want to use.

VM Size

You can also add a parameter to change the VM size used in Shipyard tasks. It will specify a different VM size, and if this parameter is present, this should override the VM size from the Azure Batch Cluster Definition.

Name your parameter task_shipyardvmsize.

The value for this parameter should be the VM size of a VM found here.

You can copy a VM size value from the above link into Loome Integrate’s parameter value field.

How to Set up a Shipyard HPC Task

In your chosen job, add a new task.

Select Shipyard HPC Script as the task type.

Shipyard HPC Task Type

Select the Azure Batch connection from the dropdown.

Select the Azure Blob Storage connection from the dropdown.

Select your Cluster Definition from the dropdown.

Choose from the dropdowns

Choose your Container Type from the dropdown. This is where you choose Docker or Singularity.

Docker or Singularity container type

Enter the name of an image in your Container Registry.

If you chose Singularity as your Container Type, you can optionally provide a Signing Key Fingerprint.

Select your Container Registry Connection from the dropdown field.

Enter the script that should be run as part of your job.

Provide a script

You can optionally add storage via other Azure Storage accounts using the fields Shared Volume Connection and Volume File Definition.

Optionally add storage via other Azure Storage accounts

You can then enter any parameters. To add a parameter, add a name and default value, and then click on the Add Parameter button on the right of the row.

Click Submit and it will save this task.

You can then run the Loome Integrate Job to run this Task.

You can monitor the logs by viewing the execution history of this Job.