Pillars of Azure: 4 trends to watch in your cloud career

In this post, (based on my session from the recent ACG Community Summit) I’m going to lay out what I view as the four pillars of Azure, trends we're seeing around these, where I think they’re heading, and how you might plan your cloud career around these areas.

Before we get too far down the path, we have to talk about the elephant in the room: Azure Pillars.

I can see you Googling (or, sure, maybe you're Bing-ing) it now: “What are the Azure pillars?” These aren’t official or based on any kind of super-secret Microsoft insider intel, but rather my impressions of the industry and where it’s headed. So, I’ve made the executive decision to appoint four pillars of Azure. 

The pillars of Azure (according to me) are:

• DevOps
• Security
• Data Science
• Data Engineering

For each of these four areas, we’ll look at the technologies and trends around each, the market and job opportunities, and the related certifications. Let’s dive in!

Are you looking to begin your Azure career or take your skills to the next level? Our Azure Cloud Architect learning path offers customized paths to excel your cloud journey!

Out of all the pillars, I’d say this is the most mature pillar of Microsoft Azure. In this pillar, I would place a focus on hybrid and multi-cloud integration, two areas where Microsoft has put a tremendous focus over the last 12–18 months.

Cloud is here to stay. Almost every company is either in cloud, moving to cloud, or late to the party and beginning planning.

Of the two roads diverged in a cloudy wood, hybrid is the road most taken.

Hybrid can offer an easier path to cloud adoption. For companies not yet in the cloud, hybrid technology is absolutely the interim path to success. 

The growth in cloud has accelerated even faster due to COVID. (You didn't think I'd make it through this without talking about COVID, did you?)

Because of COVID, it seemed everyone everywhere was working remote. Even as policies begin to relax, the pressure to have a remote workforce is not. The logical step for many companies is on-premises to hybrid and then hybrid to cloud.

Microsoft has been busy implementing new hybrid options and discussing how to work in a hybrid environment, letting companies take advantage of a slightly easier setup time, reduce costs, and lets them get their toe in the water — with a good step toward the future.

You see a lot of companies moving from on-premises completely to hybrid or multi-cloud environments. And then from there transitioning into full cloud over time (once they get through regulations and get a chance to explore a little further).

Post-COVID DevOps: Accelerating the Future
How has COVID affected — or even accelerated — DevOps best practices? Watch this free, on-demand webinar with DevOps leaders as we explore DevOps in a post-COVID world.

Along with hybrid, we also have multi-cloud integration using AWS and Azure or others.

Many companies, especially the larger ones, operate in a siloed environment. This leads to departments, whether by product, geographic location, function — adopting multi-cloud solutions based upon need or personal preference.

Vendors have seen this coming, and no one wants to get caught up in the "you can only use my software on my platform" mentality. This has led to the adoption of a cloud-agnostic focus with topics such as infrastructure.

Yes, you can use Azure Resource Manager. Or you can use third-party tools, such as Ansible, Chef, or Puppet. It's a platform-agnostic solution. Want to run in Linux, Mac OS, Windows? No problem. How about writing in Python, Java, PHP, C#, Ruby? Of course, you can!

This trend is one that will continue as we march towards 2025 and beyond. If nothing else, it provides comfort to companies that are concerned about getting cornered in with one specific cloud provider.

Having multi-cloud integration also means direct integration with other providers, such as AWS, Oracle, or Google Cloud. Want to monitor an AWS machine in Azure? No problem. You can leverage Log Analytics.

Familiar with AWS but new to Azure? Check out the AWS user’s guide to Azure.

As we look at the marketplace, job opportunities around DevOps are plentiful.

DevOps engineers and other related roles are top-paying jobs. Expect average based salaries of above $100,000 USD and a red-hot market that is expected to get even hotter over the next few years.

If you're starting out in Azure DevOps, look at the AZ-900 Azure Fundamentals Certification. Then follow with the AZ-104 Microsoft Azure Administrator Certification. This is going to help you to build a foundation on a solid path to higher pay and better opportunities. 

The AZ-400 is not likely to be required in a job, but it's certainly going to help differentiate you over other candidates — or help you to land that interview you've been hoping for.

(Sidenote: New to Azure? Don’t be turned away by the confusing exam and certification names. We can help you plot an Azure DevOps Certification Path from cloud newcomer to Azure guru.)

As we look at Azure security, the initial observations of hybrid and multi-cloud . . . well, they remain true. Let’s start by taking a look at Azure Defender and Azure Sentinel.

Azure Defender is a built-in tool that provides advanced threat protection for workloads.

It protects hybrid data hosted in Azure, on-premises, or in other clouds, and it detects unusual attempts to access Azure storage. You can then further augment that security toolset by introducing Azure Sentinel.

When we talk about adding Azure Sentinel onto Azure Defender, keep in mind that when we talk about security, (like lasagna) it's all about the layers. We're going to layer multiple levels of security so that we have different ways of protecting from intrusions.

And you can see that in the image below. This is an example of how Defender works from Microsoft.

Azure Defender works with one-click enablement. (Simplicity is a trend we’ll see more as we move forward.)

If an attacker tries to break into storage, Azure Defender helps protect against that. It gives you a security alert. From there you can create responses — either through investigations with Azure Sentinel or through an automatic response or an admin looking at it. And then that leads to some sort of threat remediation to get rid of the problem.

At its core, this is how Azure Defender functions as part of the holistic Azure security landscape.

Azure Sentinel is a SIEM, or Security Information and Event Management solution.

A good security plan involves collecting information about the environment, detecting anomalies or abnormalities, investigating those abnormalities (like any good detective), and then responding appropriately. Sentinel does all these things.

It collects information from sources inside Azure or other cloud providers (i.e., multi-cloud, right?), on-prem, and, of course, hybrid solutions.

Sentinel helps correct prioritization issues by combining artificial intelligence and traditional collect and analyze systems to help evaluate real threats and eliminate noise. This allows organizations to maintain and scale more easily without putting additional stress on overworked security operations teams.

What Azure Sentinel does:

• Collecting information about the environment
• Detecting anomalies using machine learning
• Investigating anomalies
• And — if there are abnormalities or action that needs to take place — we'll introduce a mitigation strategy.

Now, as I discussed above, any topic involving security should involve a discussion about layered protection. And as you can imagine, as we introduce layered protection, we're introducing more and more data coming from all those various layers.

Machine learning is coming more into play to help solve these problems. In security, I see more interaction with machine learning, and (as we discussed earlier) virtually any new security service is likely to include multi-cloud and hybrid.

Taking a look at the marketplace, job opportunities around Azure security and for security engineers are extremely plentiful. The security market is going to continue to grow as everything from cars to fridges involves the internet of things (IoT).

Security like DevOps seems like it will be reasonably stable over the years to come.

If you're starting out, you need the AZ-900. Then follow that by the AZ-104. But then you want to branch into security and look at the AZ-500. It's important that you look at the AZ-104 and the AZ-500: you want to understand not just security, but also DevOps and how the different systems play together. A Cloud Guru's Azure Security Certification Path is a great place to start too!

Get the Cloud Dictionary of Pain
Speaking cloud doesn’t have to be hard. We analyzed millions of responses to ID the top concepts that trip people up. Grab this cloud guide for succinct definitions of some of the most painful cloud terms.

Now, let’s take a look at data engineering. 

No cloud review would be complete without talking about Big Data. Big Data has been developing for well over a decade now.

When discussing data engineering, the phrase that you need to know is “data democratization.”

Data democratization is just a fancy way of saying that everyone that should have access to the data can have access to the data.

In the past, if you needed anything done, it had to pass through IT. The only people who had access and the knowledge to use it were in information technology silos.

The goal now as Microsoft and other companies are pushing toward, is to allow, for instance, a business analyst, to be able to access data, or an executive to be able to access a report, or create a report, or make light changes to a report without requiring weeks or months of development from an IT team.

Azure Synapse Analytics is one of the ways that Microsoft uses to encourage data democratization.

Custom Learning Experiences at Scale

Does your team need an IT career development plan? There’s no one-size-fits-all solution when it comes to cloud learning. No matter your needs, there’s a path for that with ACG Custom Learning Paths.

Azure Synapse Analytics helps to simplify data engineering in cloud, by combining ingest, explore, store, orchestrate, prep and train, model, and serve.

We can explore this with this graphic from Microsoft.

By combining multiple services into one interface, data engineering becomes more accessible with simplified processes. It's less training, it's less setup time.

Azure Synapse Analytics also helps customers to support their democratization efforts with workload isolation.

And, taking a look at the image above, you can see that really what Synapse is — a combination of multiple different services into one platform. Why?

The benefits of Azure Synapse Analytics:

• It simplifies things.
• It enables easier access to the data.
• It enables less complexity while working with the data.
• All of that means more data democratization, more access and less cost. And that's the focus.

I think that this trend continues. I think that Microsoft is going to push to simplify services, and they're going to look to continue to evolve as we look toward the future of data engineering.

The other less-settled trend is the continuing growth of data storage, combined with the need for decreased processing times.

There’s a massive amount of data that needs to go through cloud architecture, especially as we introduce concepts like machine learning and as security gets more complex. There's more data that needs to be processed, and this all runs through data engineering.

Data collection is growing at an exponential rate, and that's going to continue to happen. And so it turns into data mining. How do we get that data into the system, figure out what's important and pull just those bits out? Because that's going to lead to fewer costs and decreased processing times.

You could imagine with all of these challenges, the demand for data engineers and the job market for data engineering is probably the hottest of any of the cloud sectors. (Which keep in mind, as I've said before, anything in cloud right now, it's at all-time highs for opportunity.)

That trend is going to continue to grow as more companies move toward cloud, internet of things, increased security, and data science.

From a certification standpoint, data engineering is much newer than security or DevOps, and there's much more turbulence in the certifications.

The most recent changes to this would be the introduction of the DP-900 Azure Data Fundamentals, which is an intro data engineering certification. And the combination of the DP-200 and 201 into a much larger DP-203 Data Engineer Associate certification. This certification is not only a combination of two exams, but it reflects the changing thought process on what is required to be a data engineer.

The new certification requires more fundamental knowledge of data engineering and basic programming, as well as an understanding of data engineering tools available in Azure and how to apply them to solve engineering problems.

I foresee this trend continuing as well for the next couple of years, as data engineering in Azure continues to mature. The good news is the opportunity is plentiful, and well worth the effort required to achieve a data engineering certification.

If you're interested in data engineering, jump in, take the DP-900, take the DP-203, and start moving forward in that journey. Those two certifications, especially the DP-203 are going to be very critical in your success.

Finally, we round out our journey with a brief, and I mean, very brief look at data science. This is a huge field. And as we look here, we're peaking just a little bit into the future.

Don’t get me wrong. Data science is already here, but we are still looking at the very early stages of what's possible. Data science will continue to grow and become more mainstream over at least the next 5–10 years.

When we talk about data science, what are we talking about?

• We’re talking about AI. This means self-driving cars, voice assistance, intelligent ads, predictive analytics, and tons more.

• Machine learning is then a subset of that focusing on algorithms that help to aid machines in learning to solve problems. 

• Data science is all of that combined together plus some more stuff that we just don't have time to even talk about.

From an Azure perspective, Microsoft is in a race to accomplish the vision we’ve touched on in the previous pillars: accessibility, democratization, and time to insight.

How do we enable portability of data, integration of multi-cloud, hybrid, accessibility of third-party platforms, and the programming language of your choice?

Microsoft has multiple platforms to access and process third-party data.

When we're talking about machine learning, the flagship for Azure is going to be Azure Machine Learning Studio. You can also process machine learning in tools like Databricks.

Both Machine Learning Studio and Databricks allow you to program in multiple languages. It allows you to use a wide variety of compute options, portability of your data through notebooks and containers. And again, it provides avenues to introduce hybrid capabilities through complex pipelines.

How do we grant access to data science? In this application, the focus is on combining multiple systems, integrating those graphic user interfaces, and removing the need for programming and advanced statistical knowledge wherever possible.

Please, don't misunderstand. I am not suggesting that your doctorate in data science and advanced statistical theory, isn't helpful. What I am suggesting though, is that Microsoft has a vision that will allow a solutions architect or a business analyst with minimal data science knowledge to begin playing around with and building simple systems to solve business problems — i.e., data democratization.

As you can imagine, ever complicating systems require more data. More data means more processing. And that means more money and more time.

This trend is not unique to machine learning or cloud, or even technology for that matter: everyone is looking to do everything faster and for less money.

In machine learning, this turns into processing the data faster, streamlining pre-processing so that the data that is processed is more relevant, and parallel processing. And of course, better tools to access and search and store the data that is needed.

The challenges here aren't that different than what we just talked about in data engineering.

• Give me access to data
• Make it fast and affordable
• Don’t lock me into a provider
• Use machine learning in solutions

Data science, much like data engineering is a very hot field. There are tons of opportunities in data science. And data science is still in its infancy. It's going to continue to grow and get larger.

So there are a lot of opportunities, and tons of time to get in and establish a career in data science. And it’s a career that's likely going to be highly profitable (and fun) as you move forward.

From a certification perspective then, what you really need to focus on is the DP-100 for the Azure Data Scientist Associate certification.

Of course, we've talked about the AZ-900, and it also wouldn't hurt you to look at the DP-900 Azure Data Fundamentals as well to get kind of a standing foundation in both Azure and data engineering.

But if you have those concepts down, the DP-100 is the certification for you. That's going to help establish your relevancy in Azure as a data scientist.

Here's what I hope you to get from all of this.

It's not data democratization.

It's not hybrid or data efficiency or processing.

What I really want you to get from this talk, and the state of the Azure pillars is, the future is bright.

There is a wealth of opportunity in all of the Azure pillars. All of them are growing, all of them have new challenges, new things to explore, excitement. It's going to be an awesome next 5–10 years in cloud.

Take this time to look through your Azure certification path, figure out what you want to do in the direction that you want to head, and then start to build some goals.

Start to work on those certifications. Start to work on your plan for how you're going to grow and change your career path, whether that's in machine learning or data engineering or DevOps or quantum computing (whatever that looks like).

Level up your career. Learn more about the most in-demand tech skills with A Cloud Guru. Check out this month's free courses or get a free 7-day trial.