In my engineering role within our Azure Stack product team at Dell Technologies, I have the opportunity to talk about what value Microsoft and Dell Technologies bring to our customers’ hybrid strategy for our internal sales teams and our important partners. In addition, I also get to discuss these same values directly with our customers. In these discussions, I get to hear about their cloud goals, requirements and provide deep technical guidance to ensure that the targeted solutions will meet their objectives. There are many strategies that come out of these discussions that customers are looking to achieve; they could be looking to reduce their OnPrem IT spend by moving their core infrastructure to a public cloud or they may be looking at how they can target workloads that should live in the public cloud do so while still supporting infrastructure OnPrem in a more modernized way, whether it be application modernization or IT modernized management. It’s those conversations where the customers are seeing a real business value by taking a hybrid approach across their organizations. This ability to harness the power to embrace both public cloud and local datacenter, and may provide edge capabilities, is what is enabling customers to improve their IT agility and efficiencies. At Microsoft’s core, it’s always been about the Enterprise and running workloads within the customer’s datacenter. With the advent of Azure, Microsoft continued to see the importance of the OnPrem infrastructures that customers have traditionally invested in while still seeing the importance that public cloud infrastructures bring. This mindset has allowed Microsoft to invest and build their assets that are uniquely built for hybrid. Many of these hybrid strategies evolve into delivering new application patterns using intelligent cloud and intelligent edge.
Over the past couple of years, Microsoft has been honing their hybrid strategy to enable their partners, like Dell Technologies, to deliver modernized infrastructure built on Enterprise-proven assets integrated with connectivity to Azure Services. With the Azure Stack portfolio, customers can find the solution that meets their hybrid goals. Each of these have different delivery models, control planes, services, and targeted workloads. These variations across the Azure Stack ecosystem help to support complex customer requirements, many through Microsoft validated or Integrated systems via their hardware partners.
Before we jump into what those solutions look like from Dell Technologies, I’m going to circle back to our customer discussions and what we look to glean from these discussions to identify the appropriate solutions to meet their requirements. During these informative meetings, we want to understand:
- Use-Case Scenarios
Current OnPrem investments
- Are they heavily invested in one virtualization Platform – i.e. Microsoft, VMWare, nothing, or something else?
What are their long term OnPrem and Cloud goals
- Are they going to shift their OnPrem investments to the Public cloud?
- Investments in one Public Cloud ecosystem or heavily skewing toward one over another
- Have they identified workloads that have business or technical requirements that would prohibit them from living in public cloud
Investing toward application modernizing to take advantage of cloud-native design principles or simply a lift-and-shift to the cloud of their current workloads?
- Container Workloads – Kubernetes/AKS/GKE/etc, OpenShift, AWS ECS,
- Moving to streamline development pipeline operations w/ DevOps
- Standardizing on a cloud platform control plane; Azure Resource Manager (ARM), Amazon EKS, Cloudflare or VMware Cloud Foundation (VCF)
This list is not exhaustive but you get the idea that there are lot of things to consider as a customer traverses their path to the cloud and considerations they must be aware of, as well as their decisions, for both their OnPrem and Cloud investments. This includes things they may not have considered as well, such as: learning new tools or platforms, costs associated with unforeseen expenses in the cloud, application migrations or re-architecture requirements. As we discuss all these things with the customer, we want to ensure they are thinking about these as we move through this engagement. They will not all be understood, though sometimes they are, but it’s a case of planning and re-visiting the plan often.
Breaking down this list to bucketize them will help arrive at the best solution. Depending on the outcome of these discussions, you may find that the customer’s, or your, initial thoughts around a specific solution may not meet their overall requirements and you need to shift entire solution sets. That’s ok. At Dell Technologies, we have a broad playbook that allows us to shift to something else that is a better fit for their needs. We don’t want to shoehorn a solution just because that is the only thing we can provide. That path will not end in a satisfactory way and lead to customer issues and may abandon the course. Not good.
This is going to be focused around the Azure Stack portfolio so assuming our conversations and outcomes from the above list are keeping on this track, we will discuss some of these considerations.
If we look at the image of the Azure Stack portfolio, there are three distinct product offers. Each one has services and values that provide the ability for our customers to deliver their application needs directly to their customers:
Starting with Azure Stack Hub, which was the first solution delivered as part of the Azure Stack family, provides customers the ability to deliver a cloud-native experience OnPrem, behind their firewall, either connected for fully disconnected to Azure Public. These allows customers to deliver applications that take advantage of Platform-as-a-Service (PaaS) services, such as Azure Functions, Web and API applications, Event Hub, and IoT Hub that can be consumed from a local Hub instance but also has the same control plane as Azure Public resources. In fact, Azure Stack Hub is just another region that can be targeted like other Azure regions (i.e. US East, US West, etc) using the same tools you are familiar with; Azure CLI, PowerShell, and even Visual Studio, for example. Just keep in mind that Hub is a subset of Public Azure, so you need to ensure that the services you want to deploy to Hub are available. Do not assume that it is available for your application use since it does not have the scale factor that Azure Public has. Scale for Hub is from 4 – 16 compute nodes per region/scale-unit. However, if a customer is considering a full investment into the Azure ecosystem and they are trying to unify their control plan between OnPrem and Public cloud, using ARM, this could be the best path for them. Think of Azure Stack Hub as a development platform which provides:
- A way for customers to implement a standardized DevOps practice between Azure Public and OnPrem
- Modernize OnPrem applications from traditional monolithic application models to lightweight microservices while still close by to the OnPrem data source instead, keeping latency low and performance high while allowing time for the application to be migrated over time.
Delivering cloud-native applications in a completely disconnected pattern for edge or far-edge deployments. Think in terms of remote locations: Mines, Combat zones, Oil and Gas industries, etc.
- Example: Process IoT sensor data from a mining operation that measures critical mine shaft data (air quality, oxygen levels, gas levels, seismic activity) that are processed locally in real-time for the safety of the miners.
- Support of containers using Azure Kubernetes Engine (AKSe) and, soon, Azure Kubernetes Service (AKS).
Moving to our right in the image, to the center solution, we are to Azure Stack HCI.
Out of the portfolio, Azure Stack HCI has evolved the most over the last few years. Initially referenced as Windows Server Software-Defined Datacenter (WSSD), more of reference architecture, it was built upon a strong virtualization platform in Hyper-V, storage built upon Storage Spaces Direct (S2D) and, the optional, Software-Defined-Networking (SDN) to deliver a modernized, scalable virtualization platform. Dell Technologies was an early provider with WSSD solutions through our PowerEdge-based S2D Ready Nodes product line that were the building blocks to support virtual machines for those customers who were strong Microsoft datacenters. Over the last year, Microsoft has invested heavily in maturing the platform and is now a 1st-party citizen with Azure Public services; Azure Monitor, Azure Backup, etc. This provides a native integration layer directly with Azure that other solution providers just cannot deliver today. What this translates to is that customers running Azure Stack HCI can have a highly scalable infrastructure built on a proven and reliable Hyperconverged Infrastructure hardware delivered as an integrated experience. Dell Technologies has taken the years of experience delivering Hyperconverged Infrastructure solutions across different platforms, like VxRail using VMware to engineer and deliver these solutions. Scalability is key to the platform and Azure Stack HCI will scale from 2 nodes to a maximum of 16 nodes. Over the past 6-8 months, Microsoft and Dell Technologies have gone further and improved the Azure Stack HCI platform by introducing:
- Dell EMC Integrated System for Microsoft Azure HCI, improving the original Ready Node concept that we launched with a few years ago by adding a new AX product line specifically designed and targeted for Azure Stack HCI. This new platform provides new full stack Lifecycle Management features that are simply not available in Ready Nodes or PowerEdge servers.
- Microsoft introduced HCI-OS, a pared down OS focusing only on HCI services optimized to deliver best in-class virtualization. This OS uses an Azure subscription-based billing for the HCI Operating system and will provide faster updates, aligning more to the Azure release model. Also, it will target HCI features, such as stretch-clustering, that will not be available on Windows Server Datacenter.
All of this allows for a performance-driven platform optimized for delivering the best experience for virtualized workloads. Circling back to our original list above, use-cases for HCI will be targeting:
- Windows Server or Hyper-V Refresh from Server 2008/2012/2016/2019 to Azure Stack HCI: If a customer is Microsoft-centric today and has a Hyper-V presence, it’s a natural progression to the more modernized HCI platform.
- Virtual Desktop Infrastructure (VDI) – As you can imagine, with the pandemic, many customers have looked at investing in VDI and Azure Stack HCI is a great consideration for this use-case. Powerful and performance driven, Dell Technologies has configurations that can support the demand.
- Remote Office / Branch Office (ROBO) or Edge Infrastructure – Cluster sizing down to a minimum of 2 nodes makes using HCI at the edge or branch office.
- High Performance, latency sensitive applications – Any applications that are highly performance driven or require low latency, such as Microsoft SQL server.
- Azure Kubernetes Service (AKS) – Microsoft released AKS recently for Azure Stack HCI making container support running OnPrem a reality giving you the ability to support containers using a proven and scalable solution like AKS.
For management, HCI is done with Windows Admin Center. Unlike Hub, which uses ARM, using Windows Admin Center’s native integration of HCI OS Cluster management operations and Dell Technologies Lifecycle Management integration, allows customers to manage many HCI cluster instances from a single instance.
The net here is that if the customer is a strong Microsoft customer today, looking to invest more in Microsoft or shift their support to Microsoft, Azure Stack HCI is a strong factor aligning to their use cases.
Finally, let’s talk about the final solution in the image, Azure Stack Edge. Unlike the other two in the Azure Stack portfolio, Azure Stack Edge is delivered directly by Microsoft through the Azure portal. This solution is also very specific in the target workloads. Since it isn’t scalable like the other two platforms, this is looking at those workloads which are targeting lighter duties for IoT and/or Machine learning (ML) capabilities. Also, keep in mind that Edge always comes with a GPU or FPGA for those ML-focused workloads so that cost is always factored into the price. If you aren’t looking to support workloads of this type or don’t need GPU/FPGA support, it may not be the best or cost effective for your particular needs. Since this is directly sold by Microsoft, Dell Technologies typically does not speak much about it except that it is available and what the targeted workloads typically are. For more information for Azure Stack Edge, I urge you to read more about it at Azure Stack Edge | Microsoft Azure.
One thing that we have not discussed today, Azure Arc. Azure Arc’s goal is to offer simplified management, faster application development, and consistent Azure Services. You can easily organize, govern and secure OS’s, Windows or Linux, SQL Server and Kubernetes clusters across datacenters, the edge and even multicloud environments directly from the Azure Portal.
We have discussed quite a bit today but this is only the tip of the spear. There are a lot of considerations for any of these solutions and within those solutions, different configuration options to support the targeted workloads. Below you will find reference documentation to align to the information I’ve reviewed today. Also, if you are considering a move to the Azure Stack Portfolio, I urge you to reach out us at Dell Technology or directly with Microsoft to get a conversation started. We can partner with you on your journey, answer any questions and help size any solution that is required.
Hello…I’m back again but I’m having to change direction a little bit from my planned blog for this month. Initially, my follow-up to my previous blog was to be about going through a walkthrough of deploying workloads to Azure Stack HCI that has AKS deployed to it. Basically, putting it all together after you have AKS up and running. However, I’ve hit a snag with the latest AKS Preview and have been unable to get it deployed to my 2-node Azure Stack HCI instance.
So, with that, I’m going to change it up a little and shift my focus to Deploying the AKS Engine to Azure Stack Hub followed by deploying a Kubernetes cluster to it. I’m a bit surprised, really though, as my Azure Stack Hub ASDK instance – a single node instance of Hub that allows you do some functionality testing and get familiar with how hub works – is still running as I’ve not done much with it as of late and it has been unable to start in my nested virtualization environment. Anyway, spent a few minutes this morning working through the startup issues and got it to boot up and here we are…Recovering from that may be a story for another day.
Before I jump into the deployment process here for the AKS Engine, I did want to note what is different here versus what I did in my last blog with deploying AKS to Azure Stack HCI. The AKS Engine is a command-line tool that allows you to deploy and manage a Kubernetes cluster in both Azure Public and Azure Stack Hub. You use this to create, upgrade, and scale Azure Resource Manager (ARM) native clusters. On Hub, this can be used to deploy a cluster in both connected or disconnected environments. Understand there are difference between running the AKS Engine in Azure Public and onPrem with Azure Stack Hub. Be sure to check out those difference to have the right expectations on what you can and cannot do with the AKS Engine on Hub.
With that out of the way, so let’s do this…
We need to first get our prerequisites inline:
Azure Stack Hub Prerequisites that will need to be carried out by a Cloud Operator. This may be you if you are using an ASDK for this exercise.
|Azure Stack Hub 1910+||Yes||AKS Engine requires Azure Stack Hub 1910+. Currently the latest version is 2008.|
|Linux Custom Script Extension 2.0.6+||Required||This be available in your subscription. If it isn’t, reach out to your Cloud Operator to have them add it|
|AKS Base images||Required||Like the custom extension scripts, this needs to be available to your subscription so have your Operator add it. If you are the Cloud Operator, refer here to get it added to the Marketplace.|
|Service Principle Identity (SPN)||Required||Note that if you are using an AAD SPN, Internet access is required from the VMs in the Kubernetes cluster so that the SPN can be authenticated with Azure AD. If Internet is not available, the Kubernetes cluster will not be functional.|
|Assign SPN to the Contributor Role||Required||You Azure Stack Operator will need to complete this action before you get started. Refer to Assign a role for more instructions|
The next Prerequisites will be done by you:
|Azure Stack Hub Subcription||Yes||To deploy any workloads in Hub, you need a subscription from the tenant portal. Go to Azure Stack Hub Subscription for more info|
|Resource Group||Optional||I’ll be using MySub-rg for this exercise. You can create a new one or use an existing one from your tenant subscription.|
|Private Public Key||Required||Refer to SSH Key Generation for instructions to create a public/private key pair.|
This blog assumes you have already reviewed and implemented the above Prerequisites so will not be walking through those aspects here.
We’ve got all the prerequisites secured and we are ready to go. First, we will need either a Linux or Windows VM that we’ll use to host the AKS engine in order to deploy and manage a Kubernetes cluster. For this exercise, I’ll be using a Linux VM based on Ubuntu 18.04 LTS. This is available in my Hub marketplace. So, I’ll create a new resource for this VM using the portal as you can see below. If you are using Windows, the steps will be the same here.
For the creation of the VM, you will need to specify a few parameters that are specific to your environment. Note, that you will need to provide your SSH public key here that was created as part of the requirements at the start of this. You can use an SSH password but that is not as desirable as it offers less security as a certificate authentication practice.
Choose the Azure VM type/size that you need to support your workloads. For my case in this exercise, I’m taking the default (denoted by the star)
On the next screen, configure to your needs but I chose the default except for the exposed public inbound ports since I need to connect to this VM by SSH. I allowed for the SSH port to be opened.
Once everything is ready to go and you’ve reviewed it for accuracy, click OK to begin the Linux VM deployment.
This will take a bit of time but once complete, you will see it listed in the Virtual Machines blade (found on left menu pane) and this new VM should reflect that it is running.
I now have the VM up and running. You will connect to it and will need to determine what version we will use based on the AKS Base Image that you pulled down as part of the prerequisites. Again, in my case, the AKS Base Image is using the Ubuntu 18.04 LTS version so I need to ensure the AKS Engine version I use aligns to that. This can be determined using the table found here. Since I’m using the latest versions available for both Hub (2008) and Ubuntu 18.04 LTS, my AKS Engine version will be V0.60.1.
Having this info, we will now issue the following command from my Linux console:
|curl -o get-akase.sh https://raw.githubusercontent.com/Azure/aks-engine/master/scripts/get-akse.sh chmod 700 get-akse.sh ./get-akse.sh –version v0.60.1|
Once we have completed the above command, let’s verify the installation is OK.
- Connect to your client VM
- Run the following Command:
- Next, if you are using the ASDK and, thus, a self-signed certificate, you need to explicitly add the root certificate to the trusted certificate store of the machine. You can find the root certificate in the VM at this directory: /var/lib/waagent/Certificates.pem.
Copy the certificate file with the following command:
|sudo cp /var/lib/waagent/Certificates.pem /usr/local/share/ca-certificates/azurestacka.crt sudo update-ca-certificates|
OK…we’ve got our AKS-engine installed on our host and our self-signed certificate trusted; we are now ready to deploy a Kubernetes cluster with the AKS engine on Hub.
To do this, we will first need to define our cluster specification that uses a JSON file for our API model. This API model is what is used by the AKS Engine to create the cluster specification.
- Let’s grab the API model file from:
|curl -o kubernetes-azurestack.json https://raw.githubusercontent.com/Azure/aks-engine/patch-release-v0.60.1/examples/azure-stack/kubernetes-azurestack.json|
- Open the downloaded JSON file, kubernetes-azurestack.json
- Find the appropriate orchestratorRelease and orchestratorVersions in this file for the Kubernetes version that your AKS-Engine supports. For mine, it is version 1.17 and 1.17.17, respectively.
- From there, find the customCloudProfile in that section and provide the URL to the tenant portal. In my case, since this the ASDK, it is https://portal.local.azurestack.external. If you are using ADFS in your Hub deployment, you will need to also modify the identitySystem as well to reflect this but otherwise, you can leave it blank.
- Next, move down to the masterProfile to set the following fields in the snapshot:
A couple of notes on these fields…
- dnsPrefix – This needs to be a unique string that will identify the hostname of VMs. I’ve used the name of my resource group.
- Count – This is the number of masters you want for your deployment. The minimum is 3 for any HA deployment but for non-HA/Test environments, 1 is sufficient.
- vmSize – Enter a supported Azure Stack Hub VM size
- distro – Define what Linux distro you are using that maps to the correct AKS-Engine and Kubernetes version you are using. In my case, this is Ubuntu 18.04
- Then we want to modify the section right below this for agentPoolProfiles
More info on these specific fields:
- Count – Enter the number of agents you want for your deployment. The maximum of nodes to use per subscription is 50.
- vmSize – Enter a supported Azure Stack Hub VM size
- distro – like the previous entry, provide the linux distro you will be using.
- Finally, in the linuxProfile section:
The public key is what was created as part of the prerequisites at the start of this blog. Note: This private key must have no spaces and must be pre-faced with ssh-rsa like in the output above.
Now, we’ve prepared our API model specific via the downloaded JSON file, mapping it to the requirements for our environment. Next up, let’s get that cluster created.
Before you start the actual deployment, it’s always good to ensure that Hub is running correctly so you can do this yourself if you are the Cloud Operator (or running the ASDK) or you will need to ask your Cloud Operator to do this if you don’t have access to Hub directly.
Assuming you do have the required Cloud Operator status, you run Test-AzureStack Powershell command from the Privileged Endpoint. For more information on connecting to the Privileged Endpoint, please refer to “Use the privileged endpoint in Azure Stack Hub”
Let’s issue the following command to deploy our Kubernetes Cluster:
|aks-engine deploy \ |
–azure-env AzureStackCloud \
–location local \ If using ASDK, this should reflect local, otherwise should be location for your multi-node stack
–resource-group MySub-rg \ Modify to reflect your Resource Group
–api-model ./kubernetes-azurestack.json \
–output-directory MySub-rg \ This is going to be in my Resource group that was specified above
–client-id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \ This is the SPN application id you created as part of the Prerequisites
–client-secret xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \ SPN client-secret that was created as part of the Prerequisites –subscription-id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \ Sub ID to your tenant
–identity-system adfs Optional: Only required if using ADFS, otherwise, you can omit.
Once that completes, we should have an active Kubernetes cluster so let’s verify it is working as expected.
At completion, you should see that the deployment succeeded like below:
To verify it’s working as expected, we need to grab the public IP address of any one of the master nodes VMs found in the Hub portal:
Then, from a machine that has access to your Hub instance, connect via SSH into this master node using an SSH client, like PuTTy. For the SSH username, use the default ‘azureuser’ or the one that you specified in the API model configuration file. I used sysadmin for my case. Also, use the private key that you provided for the deployment of the cluster.
If successful, you should now be at the Master node’s prompt where you can issue the following commands to create a sample deployment of a Guestbook application called mongo-deployment. Note: This only works for connected Hub instances.
|kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/application/guestbook/mongo-deployment.yaml|
Query the list of pods:
|kubectl get pods|
Then you can also view the deployment logs using:
|kubectl logs -f <pod name>|
You can also issue the command:
As well as review the node states:
|kubectl get nodes|
That’s it! You have successfully deployed the AKS-Engine and a Kubernetes Cluster to your Azure Stack Hub instance. You’ve also deployed a sample application to the cluster. You can clean up the sample application using kubectl as well:
|kubectl delete deployment -l app=mongo|
A couple of things to consider now that you have it running. Since this is not AKS in the sense that it is a managed service like you would expect to see in Azure Public. Some general housekeeping is up to you, the owner, to do. One of those chores is to rotate your service principal secret regularly. For more of a complete reference to the AKS Engine, please refer to the online documentation at GitHub.
I hope you found this information useful. Check back soon for a follow-up here where I’ll configure the Kubernetes Dashboard which is included with Kubernetes that provides a clean way to view your cluster status. Also, I’ll be walking through workload deployments against both Azure Stack Hub and Azure Stack HCI.