How can I automate the upload Media Library files available in folder under App_Data?

Question

I build my frontend artifacts (theme files, including pre-optimized-min versions of my CSS and JavaScript) using SXA Umbrella, and in my DevOps release pipeline I make them available in the folder SitecoreFiles in the App_Data folder of my Content Management server. How can I provide a Web API end-point in Sitecore XP to upload these files into the Sitecore master database? I would like to call this Web API end-point in my DevOps release pipeline.

Answer 1

The following PowerShell script can be made available as a Web API endpoint within Sitecore. Instructions on how to do this are documented in the script:

# Deploy Media Library artifacts
# https://www.sergevandenoever.nl, https://www.linkedin.com/in/sergevandenoever/
#
# Deploy Media Library artifacts from a given folder in the App_Data folder to Sitecore.
#
# The deployment looks for all folders containing the marker file 'deployroot.txt' in the App_Data folder.
# It will at each found folder delete all files in the corresponding folders in Sitecore, and 
# copy over all files from the Sitecore Content Delivery server filesystem to Sitecore.
#
# Deployment of script as Web API endpoint:
# In Content Editor, navigate to /sitecore/system/Modules/PowerShell/Script Library
# Right-click on "Script Library" and select: Insert > Module Wizard
# - Name the module (e.g. name of customer - for example customer)
# - For integration points to create select "Web API"
# Right-click on the folder /sitecore/system/Modules/PowerShell/Script Library/<customer name>/Web API
# and select: Insert > PowerShell Script and name the script customer_DeployMediaLibraryArtifacts.ps1
#
# This will result in a Web API end-point on the Content Management server.
#
# Usage:
# https://customer_cm.com/-/script/v2/master/customer_DeployMediaLibraryArtifacts
#
# Querystring parameters:
# - app_data_folder: name of folder in App_Data web site folder that contains the Medial Library artifacts to deploy
#
# Example
# https://customer_cm.com/-/script/v2/master/customer_DeployMediaLibraryArtifacts?app_data_folder=SitecoreFiles
#
# Authentication:
# If authentication is needed, add the following parameters to the URL: user=<user>&password=<password>

$global:debug = $false
$global:log = @()
$global:stopwatch = [system.diagnostics.stopwatch]::startNew()

function Log {
    param (
        [string]$message
    )

    if ($debug) {
        Write-Output $message
    }
    $global:log += $message
}

function New-MediaItem {
    [CmdletBinding()]
    param(
        [Parameter(Position=0, Mandatory=$true, ValueFromPipeline=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$filePath,

        [Parameter(Position=1, Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$mediaPath
    )
    
    $mco = New-Object Sitecore.Resources.Media.MediaCreatorOptions
    $mco.Database = [Sitecore.Configuration.Factory]::GetDatabase("master");
    $mco.Language = [Sitecore.Globalization.Language]::Parse("en");
    $mco.Versioned = [Sitecore.Configuration.Settings+Media]::UploadAsVersionableByDefault;
    $mco.Destination = "$($mediaPath)/$([System.IO.Path]::GetFileNameWithoutExtension($filePath))";

    $mc = New-Object Sitecore.Resources.Media.MediaCreator
    $mc.CreateFromFile($filepath, $mco);
}

function Ensure-MediaFolder {
    [CmdletBinding()]
    param(
        [Parameter(Position=0, Mandatory=$true, ValueFromPipeline=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Path
    )
    
    if (!(Test-Path -Path $Path)) {
        $parentFolder = Split-Path -Path $Path -Parent
        Ensure-MediaFolder -Path $parentFolder
        New-Item -Path $Path -ItemType '{FE5DD826-48C6-436D-B87A-7C4210C7413B}' # ItemType = /sitecore/templates/System/Media/Media folder
    }
}

try {
    if ($debug) {
        [string]$AppDataFolder = 'SitecoreFiles'
    } else {
        [string]$AppDataFolder = [System.Web.HttpContext]::Current.Request.QueryString['app_data_folder']
        if ([String]::IsNullOrWhiteSpace($AppDataFolder)) {
            Throw "Url parameter app_data_folder is missing"
        }
    }

    $frontendArtifactsFolder = Join-Path -Path  $SitecoreDataFolder -ChildPath $AppDataFolder
    if (!(Test-Path -Path $frontendArtifactsFolder -PathType Container)) {
        throw "Folder with Media Library artifacts $frontendArtifactsFolder not found on server"
    }
    
    Log "========= BEGIN DEPLOY MEDIA LIBRARY ARTIFACTS ========="    
    
    $deploymentFolders = (Get-ChildItem -Path $frontendArtifactsFolder -Name 'deployroot.txt' -Recurse) | ForEach-Object { Split-Path -Parent $_ }
    $deploymentFolders | ForEach-Object {
        $sourceFolder = Join-Path -Path $frontendArtifactsFolder -ChildPath $_
        $destinationFolder = $_
        Log "Source folder: $sourceFolder"
        Log "Destination folder: $destinationFolder"
        
        # Delete al items in the destination folder
        Get-ChildItem -Path "master:\sitecore\$destinationFolder" | ForEach-Object {
            Log "Deleting $($_.Name)"
            Remove-Item -Path $_ -Force
        }
        
        # Deploy files
        $sourceFiles = Get-ChildItem -Path $sourceFolder -File -Exclude 'deployroot.txt' -Recurse
        Log "Files in folder $sourceFolder"
        $sourceFiles | Foreach-Object {
            [string]$sourceFilePath = $_
            $relativeMediaPath = $sourceFilePath.Substring($frontendArtifactsFolder.length);
            $destinationMediaPath = "/sitecore$($relativeMediaPath)"
            $destinationMediaFolder = Split-Path -Path $destinationMediaPath -Parent
            $destinationMediaFolder = $destinationMediaFolder -replace '\\', '/'
    
            Log "From   : $sourceFilePath"
            Log "To     : $destinationMediaFolder"
                   
            Ensure-MediaFolder -Path $destinationMediaFolder
            $result = New-MediaItem -FilePath $sourceFilePath -MediaPath $destinationMediaFolder
            Log "Written: $($result.Path) - Extension: $($result.Extension), MimeType: $($result.MimeType) [$($result.Size) bytes]"
        }
    }
    
    Log "========= END DEPLOY MEDIA LIBRARY ARTIFACTS ========="    

    $global:stopwatch.Stop()
    
    @{
        process = "DeployMediaLibraryArtifacts"
        app_data_folder = $AppDataFolder
        success = $true
        log = $log
        duration = $global:stopwatch.Elapsed.ToString()
    } | ConvertTo-Json
} catch {
    $global:stopwatch.Stop()

    @{
        process = "DeployMediaLibraryArtifacts"
        app_data_folder = $AppDataFolder
        success = $false
        exception = $_.ToString()
        log = $log
        duration = $global:stopwatch.Elapsed.ToString()
    } | ConvertTo-Json
    
    [System.Web.HttpContext]::Current.Response.StatusCode = 500
    [System.Web.HttpContext]::Current.Response.TrySkipIisCustomErrors = $true
}

The Web API call returns a JSON object with status and log information. In case of success the HTTP status code is 200, in case of error the HTTP status code is 500.

To call this in your DevOps release pipeline, you can create a step in your task group of type Azure PowerShell with an inline script like below:

$result = Invoke-WebRequest -Uri "https://customer_cm.azurewebsites.net/-/script/v2/master/customer_DeployMediaLibraryArtifacts?app_data_folder=SitecoreFiles&user=admin&password=$(sitecoreAdminPassword)"
if ($result.StatusCode -ne 200) {
  Throw "Request to url '$Url' failed with status code " + $result.StatusCode
  [Environment]::Exit(1)
}

Note the $(sitecoreAdminPassword) variable used in the Url.

Note: the function New-MediaItem is borrowed from the great SSE post Bulk create Media Items.