Import/Export & Github Actions

Overview

This setup guide goes through how to configure a CI/CD for Loome Intergrate in Github Actions. By the end of this guide, you’ll be able to utilize GitHub Actions’ Repos and Pipelines to move changes from one Tenant to another.

Pre-Requisites

The following are required for this setup guide:

  • Two Loome Integrate tenants, both with at least one healthy agent connected.
  • Group Admin privileges in both
  • A GitHub hosted Git Repository

Terminology

Development Tenant

The development tenant is the tenant that has the jobs we will be exporting into Github.

Production Tenant

The production tenant is the tenant that we will be importing jobs into from Github Actions. By the end of this guide, the contents of the production tenant should match those in the development tenant.

Step 1. Setting Up Github Actions for Loome Integrate

In Github, CI/CD is achieved through a Workflow Action yaml file. As there is no code to “build” for Loome Integrate tenant imports, most of the work is achieved in the Workflow Action, with the workflow pipeline mainly existing to ensure all the contents of the tenant are provided in a standard format.

In Github, navigate to the Repositor that will contain your Loome Integrate Export and select “Actions” from the menu bar.

Github Actions

We will now go through the process of creating a simple Github Action Workflow file. Begin setup by selecting Simple Workflow for this workflow. This will give a basic template file to start that we will overwrite.

Select a Simple workflow

template workflow

Once on the Actions page, you will need to overwrite the template basic workflow with the following yaml script. You will need to make some modifications to the script to import into your chosen tenant, we will explain these modifications in our next step.

# This is the Loome Integrate Git Integration Workflow file
name: CI
# Controls when the action will run. Triggers the workflow on pushes to master
on:
  push:
    branches: [ master ]
jobs:
  build:
    # The type of runner that the job will run on in this case windows:
    runs-on: [windows-latest]
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2
    - name: Import Tenant
      env: 
        ClientId: YOUR_CLIENT_ID
        IdHostName: YOUR_ID_HOSTNAME
        ApiHostName: YOUR_API_HOSTNAME
        AgentHostName: YOUR_AGENTHOSTNAME
      shell: pwsh
      run: |
        try {
        Add-Type -AssemblyName System.IO.Compression.FileSystem
        $ClientId = $env:ClientId;
        $IdHostName= $env:IdHostName;
            
        $TokenRequestBody = @{
            "grant_type"    = "client_credentials";
            "client_id"     = $ClientId;
            "client_secret" = $ClientId;
            "scope"         = "webApi"
        };
        $AccessToken = ((Invoke-WebRequest -Uri "https://$IdHostName/connect/token" -Method Post -Body $TokenRequestBody).Content | ConvertFrom-Json).access_token;
            
        $ArtifactOutputDir = $ENV:GITHUB_WORKSPACE;
        $ApiHostName = $env:ApiHost;
        $BaseUrl = "https://$env:ApiHostName/api/v1/";
        $AgentHostname = $env:AgentHostName;
            
        function Upload-Zip ($ZipLocation) {
            $AccessHeaders = @{"Authorization" = "bearer $AccessToken" }
            
            $ZipUploadUrl = $BaseUrl + "Import/Zip?agentHostName=$AgentHostName";
            
            try {
                $Response = Invoke-RestMethod $ZipUploadUrl -Method 'POST' -Headers $AccessHeaders -InFile $ZipLocation -ContentType "application/zip" -SkipHttpErrorCheck;            
            }
            catch {
              throw "an unexpected error occurred";
            }

            $ResponseMessage = $Response | Out-String;
            
            if ($ResponseMessage.Contains("Import failed"))
            {
              throw $ResponseMessage;
            }
                
        }
        
        $ZipPath = "tenant.zip";
        Remove-Item $ZipPath -ErrorAction Ignore;
        Compress-Archive -Path $ArtifactOutputDir -DestinationPath $ZipPath;
        Upload-Zip -ZipLocation $ZipPath;
        }
        catch {
          Write-Output "Failed Run";
          Write-Error $_.Exception.Message;
        }
    - name: Upload Zipped Tenant
      uses: actions/upload-artifact@v1
      with:
        name: Tenant
        path: "tenant.zip"

This script will upload and compress the contents of your tenant as a zip and then upload it directly to Loome Integrate. If anything goes wrong during the import (for example, if new connections were not imported before the import) then the task will fail.

Step 2. Editing the Script for your Tenant

Select a Simple workflow

Edit the Script to include the following variables. This step varies slightly depending on the region of Integrate you are using, however there will always be four variables to set:

AgentHostName

  • This is the host name of the agent on your Production tenant.
  • This can be grabbed from the agents screen or the bar at the bottom of the application.
  • This is the agent the imported tasks will be assigned to.

ApiHostName

  • For AU Clients this will be dg-api-au.perspectiveilm.com
  • For US Clients this will be dg-api-us.perspectiveilm.com

ClientId

  • This is the Client ID you created at the start of this step.

IdHostName

  • For AU Clients this will be identity-au.perspectiveilm.com
  • For US Clients this will be identity-us.perspectiveilm.com

You will now simply need to Start Commit in the top right corner of the page. Add a simple messeage outlining the commit.

Start Commit

Step 3. Setting Up Git Sync in the Development Tenant

Open up Loome Integrate and navigate to your development tenant. To setup Git Syncing you will need to create a new “Git Connection”.

New Git Connection

The details of the Git Connection will be the following:

Field Value
Username Your Personal Username
Password Your Personal Password or Personal Access Token
Branch Leave empty (Loome Integrate will create branches for Import/Export)
Git Repository URL The URL of your Git Repository (This is the URL you use to clone the repository)

Create and validate the connection in Integrate, once the connection is validated open up the “Import/Export” settings from the top right hand dropdown.

Open up the “Export to Git” tab, you will need to provide the Git connection that you will use for exporting here. Once you click the save button you will be able to export the contents of your Tenant to your Github repository.

Setup Git Connection

Once this is enabled, you will see a button appear in the top right of the application that says “Sync Changes”. This button will appear any time there are changes available to sync to Git.

Sync Changes Button

In the rare case this button doesn’t appear on the enabling of Git sync, adding an empty job to a project will enable the button as a change will be recorded.

Clicking this button and then confirming the sync from the pop up will export the contents of the tenant to Azure Repos. Once the sync is completed, you will see a notification at the bottom right hand of the page showing the name of the created branch.

Notification Completion

Navigating to Azure Repos will show that this branch has been created and your tenant contents have synced.

Step 4. Importing Connections into the Production Tenant

Before importing jobs into another tenant, you will need to ensure that all connections used by those jobs are also available in both tenants.

Loome Integrate does not include connections in the Import/Export process, there are a few reasons for this:

  1. It is standard practice to not include database connection details and credentials in source control, abiding by this practice ensures that Integrate is correctly handling sensitive data.

  2. There may be different connections used in development and production environments (e.g., production loads may use a cloud data warehouse with more resources available or a different database entirely for some migrations).

  3. Connections that are validated on one tenant with a specific agent may require minor changes to be validated on another tenant, especially if the production tenant has an agent on a different host.

On the development tenant, open up the Import/Export page and select the Import/Export Connections tab.

Export connections

Clicking the Export Connections button will download a JSON file with all the connections from the development tenant.

On the production tenant, open up the Import/Export page and select the Import/Export Connections tab.

Import connections

Clicking the Import Connections button will prompt you to provide the connections file downloaded previously. Once you have provided the file, you will be prompted with the number of new connections that have been imported in a notification.

Loome Integrate will only add connections that do not exist in the tenant yet, if a connection with the same name is already present it will not be imported.

Once the connections have been imported, you will need to validate them before you can run jobs that use them.

As mentioned previously, you will need to reconfigure the following as it is not brought over with the import.

  • Usernames
  • Passwords
  • Domains
  • OAuth Configurations

Once you have validated the connections, you are ready to set up the CI/CD pipeline for your jobs.

Step 5. Wrapping it all up

In this setup guide, we have created a full Continuous Integration/Continuous Deployment pipeline for our Integrate tenants. At a high level we support the following flow:

  1. A user in the Development tenant makes a change in the Development tenant.
  2. The user syncs the changes to a new branch in Github Actions
  3. A project administrator in Github merges the changes from the new branch into master
  4. Once the changes have been merged, Github Action pushes the changes to the Production tenant.