How To Reference Azure Managed Service Identity (MSI) During ARM Template Deployment

Despite the long title, sharing this information out to the broader community as I had this specific need for a customer scenario and found it in a reply on this StackOverflow thread.  I developed a full ARM template and tweaked the initial solution to better suit my customer’s needs.  You can either download the reference ARM template (no warranties, provided as-is) or implement the pieces that you need.

Scenario

As of the time of writing this, Azure has released into preview the Managed Service Identity (MSI) functionality into preview.  In essence this allows specific Azure resources (ex. app service, VM, etc.) to be granted a service principal in Azure AD which can then be granted permissions in role based access control (RBAC) type fashion.  For a customer they needed to deploy an Azure Function and associated Key Vault.  The MSI for the Azure Function needed to have read (get) access to the secrets within the key vault.  While this could be configured post-ARM template deployment it is easier and more reliable to do so at deployment time.

Solution

The reference ARM template can be downloaded in full from the following location.

https://github.com/BrianTJackett/Blog-Samples/blob/master/MSI-ARM-Template/azure-function-with-MSI-to-key-vault-template.json

In the “Microsoft.Web/sites” resource be sure to enabled the MSI by including the following element at the root of the resource:

"identity": {
         "type": "SystemAssigned"
       }

In order to add the access policy to the key vault, add the following elements as children of the “properties” element:

note: sites_name is the name of a parameter for my given ARM template.  Either hardcode this value or supply a value to this parameter.

"tenantId": "[subscription().tenantid]",

"accessPolicies": [
   {
     "tenantId": "[subscription().tenantid]",
     "objectId": "[reference(concat(resourceId('Microsoft.Web/sites', parameters('sites_name')), '/providers/Microsoft.ManagedIdentity/Identities/default'), '2015-08-31-PREVIEW').principalId]",
     "permissions": {
       "keys": [],
       "secrets": [
         "get"
       ],
       "certificates": []
     }
   }

]

The highlighted portion references the MSI (principalId) of the resource that is being looked up (the Azure Function).

Lastly be sure to establish a dependsOn relationship from the key vault to the Azure Function with the following:

"dependsOn": [
         "[resourceId('Microsoft.Web/sites', parameters('sites_name'))]"
       ]

Conclusion

This post was more of a mental reminder to myself about how to reference an Azure MSI within an ARM template, but if you have need of this same solution hopefully you found it useful.  Feel free to share and questions or feedback in the comments.

-Frog Out

Automate Creation of Azure AD Application with OAuth Permissions

<Update 2018-02-06>Updated with snippet to list out GUIDs for app roles that can be assigned.</Update>

In this post I will show how to automate the creation of an Azure AD Application and assign OAuth permissions to that application.  The latter part is tricky as there is not currently a PowerShell commandlet or Azure CLI command to assign OAuth permissions.  Instead we will leverage an authenticated call to the Microsoft Graph to assign the permissions.  For more in depth information about Azure AD apps, verifying the results, and more please see the following post which I am borrowing heavily from.  I had difficulty finding this information so this post is my attempt to spread the word and also add a few clarifications on the ADAL libraries used.

(Read this first!) Automating the creation of Azure AD Applications by Christer Ljung

http://www.redbaronofazure.com/?p=7197

Problem

Creating Azure AD apps typically involves logging into the Azure Portal (classic or “new” / Ibiza version) and manually clicking through multiple screens.  When developing a solution that needs to leverage Office 365 services (as is my case with a current project) it is helpful to automate the process of creating the Azure AD app and assigning the permissions.  If you happen to be assigning Admin permissions then additional steps will be required by an Azure AD domain administrator (see following screenshot).

AzureADApp1

Solution

Creating an Azure AD application can be accomplished in 2 lines of PowerShell.  Login to Azure then create the app.


Login-AzureRmAccount

$aadapp = New-AzureRmADApplication -DisplayName "Some amazing app" -HomePage https://localhost:8081/ -IdentifierUris https://localhost:8081/

***BONUS***

If you want to create an app that uses certificate based authentication you can use the following PowerShell commandlets.

Note: The commandlets for creating and exporting a certificate require Windows 8 or higher.  There are workarounds for Windows 7 or similar OS.  Feel free to reach out if you are in that scenario.


$pwd = Read-Host -AsSecureString -Prompt "Enter certificate password"

# process for Windows 8+ type OS
$ssc = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -Provider "Microsoft Enhanced RSA and AES Cryptographic Provider" `
                           -Subject "cn=MySuperSpecialCert" -KeyDescription "Used to access Azure Resources" `
                           -NotBefore (Get-Date).AddDays(-1) -NotAfter (Get-Date).AddYears(1)

# Export cert to PFX - uploaded to Azure App Service
Export-PfxCertificate -cert cert:\localMachine\my\$($ssc.Thumbprint) -FilePath ExportedSpecialCertFile.pfx -Password $pwd –Force
$KeyStorageFlags = [System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable, [System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::MachineKeySet, [System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::PersistKeySet
$certFile = Get-ChildItem –Path <path to certificate file>
$x509 = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$x509.Import($certFile.FullName, $pwd, $KeyStorageFlags)
$certValue = [System.Convert]::ToBase64String($x509.GetRawCertData())

# should match our certificate entries above.
$validFrom = [System.DateTime]::Now.AddDays(-1)
$validTo = [System.DateTime]::Now.AddYears(1)

$aadapp = New-AzureRmADApplication –DisplayName "Some amazing app" -HomePage "https://localhost:8080/" `
                                   -IdentifierUris "https://localhost:8080/" -CertValue $certValue `
                                   -StartDate $validFrom -EndDate $validTo

The next step involves granting OAuth permissions to the recently created Azure AD app.  As of the writing of this blog (Feb 2, 2018) there is not a PowerShell commandlet nor Azure CLI command to assign those permissions.  There is however a way to use the Microsoft Graph to assign permissions.  This is an adapted version of Christer’s example that I referenced earlier and uses a local version of the Active Directory Authentication Library (ADAL) DLLs.  Currently these are at version 3.19.1.

ADAL NuGet package

https://www.nuget.org/packages/Microsoft.IdentityModel.Clients.ActiveDirectory/

Extract the following DLLs into the folder where you are executing other PowerShell commands:

  • Microsoft.IdentityModel.Clients.ActiveDirectory.dll
  • Microsoft.IdentityModel.Clients.ActiveDirectory.Platform.dll

$Tenant = "<Office 365 tenant name, ex. Contoso>"
$aadTenant = "$Tenant.onmicrosoft.com"
$adminUser = "<admin account with access to authenticate against MS Graph>"

# load ADAL DLLs
$adal = ".\Microsoft.IdentityModel.Clients.ActiveDirectory.dll"
$adalforms = ".\Microsoft.IdentityModel.Clients.ActiveDirectory.Platform.dll"

[System.Reflection.Assembly]::LoadFrom($adal) | Out-Null
[System.Reflection.Assembly]::LoadFrom($adalforms) | Out-Null

 $clientId = "1950a258-227b-4e31-a9cf-717495945fc2"  # Set well-known client ID for AzurePowerShell
  $redirectUri = "urn:ietf:wg:oauth:2.0:oob" # Set redirect URI for Azure PowerShell
  $resourceAppIdURI = "https://graph.windows.net/" # resource we want to use
  $adminUserId = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.UserIdentifier" -ArgumentList ($adminUser, "OptionalDisplayableId")

 # Create Authentication Context tied to Azure AD Tenant
  $authContext = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext" -ArgumentList $authority

  # Acquire token
  $authResult = $authContext.AcquireToken($resourceAppIdURI, $clientId, [Uri]$redirectUri, [Microsoft.IdentityModel.Clients.ActiveDirectory.PromptBehavior]::Always, $adminUserId)

 $authHeader = $authResult.CreateAuthorizationHeader()
  $headers = @{"Authorization" = $authHeader; "Content-Type"="application/json"}   

# make call against MS Graph to apply OAuth permissions
$url = "https://graph.windows.net/$aadTenant/applications/$($aadapp.ObjectID)?api-version=1.6"
$postData = "{'requiredResourceAccess':[
     {'resourceAppId':'00000003-0000-0ff1-ce00-000000000000','resourceAccess':[{'id':'fbcd29d2-fcca-4405-aded-518d457caae4','type':'Role'}]},
     {'resourceAppId':'00000002-0000-0000-c000-000000000000','resourceAccess':[{'id':'311a71cc-e848-46a1-bdf8-97ff7156d8e6','type':'Scope'}]}
     ]}";
$result = Invoke-RestMethod -Uri $url -Method "PATCH" -Headers $headers -Body $postData  

Note the use of specific resoureAppId and resourceAccess values above.  These two examples grant the “read and write all items in SharePoint Online” admin consent permission and the default “read user profile data” delegated permission respectively.  In order to find out the GUIDs you may need you’ll need to add the permissions through the Azure portal UI, check the manifest file, and extract the GUIDs.  See Christer’s post for more details.

<Update 2018-02-06>  I recently found out it is possible to list out the Application role permissions and GUIDs needed above by running the following PowerShell against the Azure AD module (I’m using Azure AD “V2” Preview module, haven’t verified against the existing V1 module).

Connect-AzureAD
# 00000003-0000-0ff1-ce00-000000000000 is the AppId for SharePoint Online, call Get-AzureADServicePrincipal by itself to find other AppIds
$SPOApi = Get-AzureADServicePrincipal -Filter "AppId eq '00000003-0000-0ff1-ce00-000000000000'"
$SPOApi.AppRoles

</Update>

If you happen to assign an admin consent permissions (such as the “read and write all items in SharePoint Online” permission) an Azure AD domain administrator will still need to consent to that permission by clicking “Grant permission” inside the Azure portal.  I’m not aware of a way to automate that process but if you do know please share in the comments below.

Conclusion

Originally I had hoped automating creation of an Azure AD app would be a simple process.  Creation of the Azure AD app is easy, but adding certificate authentication and / or assigning OAuth permissions adds extra work to be done.  As seen in this post though much of that can be automated.  Hopefully this post saves you time and effort.  Feel free to leave any feedback or questions in the comments below.

-Frog Out

Slides and Demo Files from SPTechCon DC 2017

   Big thanks to Stacy and her crew of organizers, all of the attendees, and fellow speakers at SPTechCon DC 2017.  This was one of the best ones I’ve attended from an engagement and networking perspective.  Below are my slides and source code.  Feel free to let me know about any follow up questions or comments.

PowerApps and Microsoft Flow for Developers

GitHub link to demo project files

https://github.com/BrianTJackett/BTJ.PowerApps.AzureDBSample

Slides

Intro to Power BI for Office 365 Developers

Slides

Sample Financial Data file

http://go.microsoft.com/fwlink/?LinkID=521962

Blog posts on SMAT data report

https://aka.ms/SMAT2013BTJpart1

https://aka.ms/SMAT2013BTJpart2

      -Frog Out

Using Microsoft Flow to Start and Stop a Set of Azure VMs

   In this blog post I’ll walk through creating a Microsoft Flow flow for starting (and another for stopping) a set of Azure Resource Manager (ARM) VMs.  Note that this is not my own original work.  I implemented this based on the work of someone else I found online but can no longer find the original owner’s reference.  If you do find this elsewhere please feel free to let me know in the comments.

Background

   While it is possible to start and stop Azure VMs from the newly released Azure mobile app, most time I need to start up a set (3-5) VMs at a time for a SharePoint farm / app dev environment / etc.  I was able to find a sample someone wrote in Microsoft Flow to trigger the start / stop from the Flow mobile app.  The flow calls Azure AD to get an access token using an Azure AD app that has permissions to start / stop VMs.  The access token is then passed into a series of REST calls to start up VMs in order (usually domain controller, database server, app server, web front end, etc.)  Finally the flow will send a mobile push notification letting me know that the VMs have started.

Word of caution

   This solution embeds the client ID and client secret (essentially user name and password) for the Azure AD app which has permissions to the Azure VM.   This could be a security risk and as such should be cautioned from doing this.  Treat this sample as a proof of capability for development purposes only.  I’m continuing to explore alternatives (ex. Managed Service Identity, Azure connector in Microsoft Flow) which would increase security for this solution.  If anyone has any suggestions please feel free to let me know in the comments.

Solution – Start Azure VMs

   I won’t go into detail on each and every step as some of these are self explanatory or a repeat of others (ex. 2nd and 3rd VM to be started.)  Before going into the flow to be created, ensure you have an Azure AD app registered with permissions on the desired VMs to be started / stopped.

Register Azure AD App

   Log into the “new” Azure portal (portal.azure.com) and go into the Azure AD screen.  First click on Properties to view the directory ID.  Make note of this for future use.

image

   Click App registrations and create a new app of type “Web app / API”.

image

   Make note of the application ID (also known as client ID).

image

   Go into the Required Permissions setting for the app.  Add a permission for the “Windows Azure Service Management API”.  Choose the permission “Access Azure Service Management as organization users” which is currently in preview.

   Create a key for the Azure AD app and write this down.  You will only get to see this key once and cannot retrieve it at a later time.  If you lose the key value you will need to create a new one.

Assign access control to resource group

   Now that the Azure AD App has been registered it will need access control to the resource group (or individual Azure VMs, more administration if this option) so that the app can start / stop the desired VMs.  I granted Virtual Machine Contributor role to the Azure AD App but more fine grained controls might be possible if security concerns are a factor.

image

Microsoft Flow sample

  1. Manually trigger a flow
  2. Get access token for Azure
  3. Parse JSON to extract access token
  4. Start VMs (in series)
  5. Push notification if successful

image

Manually trigger a flow

   This is self explanatory.  This will let you initiate the flow from Flow web portal or the Flow mobile app.

Get access token for Azure

   This step will use an HTTP POST action to the Azure AD directory where the Azure AD app is registered.  Ideally you should send a request to this URI using Postman or a similar REST endpoint testing tool to get a sample of the JSON response to be used in the following step.

image

  • Method: POST
  • Uri: https://login.microsoftonline.com/<directoryID from previous step>/oauth2/token
  • Headers
    • Content-Type: application/x-www-form-urlencoded
  • Body: resource=https://management.azure.com/&client_id=<client ID from previous step>&grant_type=client_credentials&client_secret=<client secret from previous step>

Example JSON response using Postman:

{
   “token_type”: “Bearer”,
   “expires_in”: “3599”,
   “ext_expires_in”: “0”,
   “expires_on”: “1508115492”,
   “not_before”: “1508111592”,
   “resource”: “https://management.azure.com/”,
   “access_token”: “<removed value>”
}

Parse JSON

   Either using the sample JSON response above or your own you can define the schema of the JSON to be parsed.  Specify the “Body” of the JSON response from the prior HTTP POST action.  The important element to parse out is “access_token”.

image

{

    “type”: “object”,

    “properties”: {

        …<other properties here>…,

        “access_token”: {

            “type”: “string”

        }

    }

}

Start VM REST call

   Add another HTTP POST action this time specifying the following configuration.

image

  • Method: POST
  • Uri: https://management.azure.com/subscriptions/<Azure subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.Compute/virtualMachines/<Azure VM name>/start?api-version=2016-04-30-preview
  • Headers
    • Authorization: Bearer <insert the bearer token “input” from prior Parse JSON step>

   Note that I used an older version for the “api-version=” portion of query string (highlighted in green).  A newer version might also be available and compatible but I haven’t tested anything newer.

   Create as many additional HTTP POST actions that call off to additional VMs as needed.  I hand coded the Uri for each as Microsoft Flow didn’t yet support expressions and other dynamic variables when this solution was first created.  You may want to investigate those to reduce repeated syntax if possible.

Notify when VMs started

   Straight forward action with a simple notification to let me know when flow has completed.

image

Solution – Stop Azure VMs

   The steps for stopping a set of Azure VMs will be identical to the “start” flow except that stopping VMs can be done in parallel as the order is not as important.  In your own scenario the order may be important so consider that when creating your own solution.

  1. Manually trigger a flow
  2. Get Access Token for Azure
  3. Parse JSON to extract access token
  4. Stop VMs (in parallel)
  5. Push notification if successful

image

   The other important difference will be to call to “deallocate” (highlighted in red) the VM rather than “start” using the Azure Service Management API.  See example below for the HTTP POST to a VM.

Sample Execution

   As you can see from the below sample executions of both flows the start and stop of each VM can take some time (2-3 minutes) but is still an easier process of clicking one button rather than multiple clicks within the Azure Portal or mobile app.

image

image

Conclusion

   Hopefully this walkthrough will help others who are interested in automating Azure VMs to start and stop (or any other authenticated actions against Azure resources).  I’m hoping to try out additional options to remove the need to store client ID and secret within the flow.  For the time being try out this process and let me know if you have any issues.

      -Frog Out

Slides and Demo Files from Dogfood Con 2017

   A big thanks to Cassandra, Trey, Danillo, and all of the other organizers of Dogfood Con 2017.  Thanks also to all of the folks who attended my two sessions.  Below are my slides and source code.  Feel free to let me know about any follow up questions or comments.

PowerApps and Microsoft Flow for Developers

GitHub link to demo project files

https://github.com/BrianTJackett/BTJ.PowerApps.AzureDBSample

Slides

Intro to Power BI for Office 365 Developers

Slides

Sample Financial Data file

http://go.microsoft.com/fwlink/?LinkID=521962

Blog posts on SMAT data report

https://aka.ms/SMAT2013BTJpart1

https://aka.ms/SMAT2013BTJpart2

-Frog Out

SharePoint CSOM to Traverse All Sites in SharePoint Online

[Update 2018-07-31] Special thanks for my peer John Ferringer for pointing out an issue with my prior CSOM snippet. Updating CSOM code to handle large number of sites as previous sample would truncate results if large number of sites were traversed.[/Update]

In the past I’ve written posts for “PowerShell to Enumerate SharePoint 2010 or SharePoint 2013 Permissions” or “PowerShell Script To Traverse All Sites In SharePoint 2010 (or 2007) Farm” to assist with traversing through all sites within a SharePoint on-prem farm.  In this post I’ll share a snippet I recently used for traversing through all site collections and subsites within a SharePoint Online tenant.

 

Background

If you’ve worked with the SharePoint Online Management Shell you may know that originally it was not able to retrieve Personal Sites (also known as My Sites / OneDrive for Business sites) in SharePoint Online.  As far as I’m aware this was primarily a limitation of the underlying client side libraries (Microsoft.SharePoint.Client.*).  Fast forward to a recent release (I don’t have the specific one but I can confirm it is in the 16.1.6621.1200 release of the Microsoft.SharePointOnline.CSOM NuGet package) and now it is supported to retrieve Personal Sites.  In the management shell this is accomplished by calling the following:

 

Note: If you do not see the below Gist please refer to code at this location: PS-Get_All_Sites_SPO.ps1

 

The problem is that if you want to traverse all site collections in SharePoint Online in client side object model (CSOM) you would need to know how the SharePoint Online Management Shell implements that inclusion of Personal Sites with the rest of site collections.  In order to find this out I used a disassembler (ILSPY in my case, but there are many alternatives available as well) on the underlying libraries to recreate the process in my own code.

 

Solution

The resulting CSOM that I came up with is below.  Feel free to borrow this and use in your own code but note that it is provided as-is with no warranty.

Note: If you do not see the below Gist please refer to code at this location: CSOM_Traverse_All_Sites_SPO.txt

 

Conclusion

In this post I shared a snippet for traversing all site collections in SharePoint Online with C# CSOM code.  In my daily job I’ve been using this in combination with Azure Functions for a number of interesting interactions with a SharePoint Online tenant.  More to come on those scenarios in future weeks.  For now let me know in the comments if you have any questions or issues implementing the above snippet in your own code.  Happy coding!

 

-Frog Out

Slides and Demo Files from CincyDevelop() Conference

   A big thanks to Phil and all of the other organizers of Cincy.Develop() / Day of Agile Conference.  Thanks also to all of the folks who attended my Integrate All the Things: PowerApps and Flow for Developers session.  Below are my slides and source code.  Feel free to let me know about any follow up questions or comments.

PowerApps and Microsoft Flow for Developers

GitHub link to demo project files

https://github.com/BrianTJackett/BTJ.PowerApps.AzureDBSample

Slides

-Frog Out