Episode #
1

Unlocking Leadership and Cost Optimization in Tech with Mark Sass

Episode Description

Discover how seasoned tech leader Mark Sass navigates the challenges of leadership, cost efficiency, and scaling platform engineering.

This episode delves into practical strategies, leadership evolution, and the importance of understanding business context, all from a leader with a rich background across industries.

Main Topics Covered:

- Mark Sass’s career journey and leadership style evolution

- The impact of an MBA on technical decision-making and cross-functional communication

- Practical approaches to cost efficiency in cloud infrastructure

- Strategies for gaining buy-in for infrastructure modernization

- Building high-performing teams through leadership principles and mentorship

- The role of data and metrics in driving cloud cost savings

- Overrated trends in platform engineering, emphasizing value over hype

Links & Resources

Connect with Stephen Koza: LinkedIn

Connect with Mark Sass: LinkedIn

Connect with EverOps on LinkedIn

Books mentioned by Mark:

Transcript

00:00:09 - 00:00:34 Stephen Koza
Hello and welcome to Tech Pod Talks. Here we do candid conversations with leaders that are building what's next. I'm Stephen Koza. I'm the CEO here at EverOps and every episode we bring in somebody that's a true practitioner, somebody actually in the arena to tell us what's happening behind the scenes and platform engineering in cloud and DevOps and more importantly, technical leadership.

00:00:35 - 00:00:59 Stephen Koza
So really excited to have our guest on today. He is exactly that kind of person. His name is Mark Sass. And Mark, I got to tell you, you've got one of the most interesting career arcs that I've seen in platform engineering. So Mark is build shred teams at Home Depot, managing payment infrastructure for thousands of stores and hundreds of billions of dollars in transactions.

00:00:59 - 00:01:30 Stephen Koza
He's led engineering teams at Meta. More recently senior director of online engineering at 2K games, which is one of the biggest names in gaming, building the infrastructure behind titles that some of you've probably played. Also holds an MBA from Texas A&M. I won't hold that against you. Mark knows that I went to the rival University of Texas, and today Mark is senior director of platform engineering at SolarWinds, which is a company in the middle of a pretty major transformation.

00:01:30 - 00:01:35 Stephen Koza
And I'll let him tell you more about that. So, Mark, welcome to Tech Pod Talks.

00:01:35 - 00:01:37 Mark Sass
Thanks, Stephen. I appreciate you having me on.

00:01:37 - 00:01:54 Stephen Koza
Yeah, it's my pleasure. It's good to, good to catch up. Since we've known each other for a little while. So, Mark, I want to start with your career. You've told me about it, and we've certainly done some research to fill in some of the blanks. You've you've done some pretty interesting work across some household names.

00:01:54 - 00:02:03 Stephen Koza
Home Depot, Meta, Indeed. 2K Games. Now SolarWinds, those are all pretty different companies. What's the thread that connects all of them for you?

00:02:03 - 00:02:29 Mark Sass
It's interesting question. So, you have to go back a little bit farther than what you have listed there. So, years and years ago, I was a technical trainer. I got tired of doing a lot of the teaching and wanted to work on practice. So I started working for a startup and during the dotcom bubble, and, you started working on the practical experience of that, and then I started my own consulting practice, and then I did some more time in startups.

00:02:29 - 00:02:54 Mark Sass
And so, you know, and what I've done is I've kind of bob, between early in my career, between startups where I absolutely love the agility and the problem solving and doing stuff with a shoestring budget. I get frustrated with that. And I move to a bigger company, like a Dell or a Visa that are that are, you know, way back in the time machine for me, where we had all this budget and just really slow and bureaucratic processes.

00:02:54 - 00:03:11 Mark Sass
And so, I've kind of bounced between those companies. And so recently I've kind of settled into the mid-sized companies, although I can't really tell with the time it met in Home Depot. Those are really large companies that had great opportunities, but indeed was a relatively small company. 2K was a relatively small company, and and SolarWinds is not a huge company as well.

00:03:11 - 00:03:24 Mark Sass
So I really like the opportunities there to really flex the technology, but also flex leadership skills where there's opportunities for us to grow the organization and grow the grow the careers of people work for me.

00:03:24 - 00:03:46 Stephen Koza
Yeah, it makes a lot of sense. Thanks for sharing. I want to ask about the MBA because I think that's pretty unique. I don't see that a ton. In, technical leadership, certainly there's some folks like you, but they're the minority and not the majority. So tell me, how does having that business lens change the way you think about technical decisions?

00:03:46 - 00:03:50 Stephen Koza
And then also what was the motivator? Why go do that?

00:03:50 - 00:04:08 Mark Sass
Oh, so that's even a better story. So I actually went to Texas A&M out of out of high school. I was a terrible student. You know, if if they had the diagnosis at that time, I probably wouldn't, ADHD or or on the, on the spectrum there, couldn't focus. And so I had to go through the school of hard knocks.

00:04:08 - 00:04:34 Mark Sass
And when I started working at del back in 2013, they offered free tuition or helpful with the tuition. And so I was like, well, now's a chance. So I went back and I didn't get my degree in computer science like I studied in college. I went to a management because the idea was like, what's the quickest way for me to get my degree in the, the, the, the irony of that is, the reason why I decided to get my degree was because I had applied for a, for a job at EA, a competitor took and they left me.

00:04:34 - 00:05:02 Mark Sass
But they said, you don't have a degree, we can't hire you. And so I started seeing that as a glass ceiling for me. So I got my degree at a at an online university. And then I was like, the problems that I was facing, Adele, because this was during my del time, was there was a lot of conversations about CapEx and opex and amortization, and there was, you know, I was having conversations with product people and marketing people and all these other things, and I understood what they were saying, but I didn't really understand what it meant for me.

00:05:02 - 00:05:32 Mark Sass
And so the lens that it changed, for me, at least from that perspective, is up to that point, I've been in it in other places where I just kept asking for money, and I had no way to really articulate why that was valuable, where the value came from spending money. And so the MBA really kind of opened that lens up to me, started making me think about things like a return on investment for what we wanted to do, and allowed me to understand how product marketing works and how to to exercise that muscle, to say I'm not just building a service that people take advantage of.

00:05:32 - 00:05:51 Mark Sass
I'm building a product and make and understand my customers so that when I built that product, I had, you know, a feeling for what they wanted. I can go deliver it. And it became a partnership, collaborative event that I did as opposed to what I did normally in the IT world, which is like, I'm going to do this stuff and it's going to make my life easier.

00:05:51 - 00:06:06 Mark Sass
I try to make it. How can I make this easier for the customer? So that was really what a gain for me was just that insight to be able to understand how to talk to the finance people, how to talk to product people, how to talk to executives that I lacked before then.

00:06:06 - 00:06:23 Stephen Koza
I was also not a great student, at least in my early years. I don't know if I should out myself, but, University of Texas told me if I don't get a better grades after my first semester, that they weren't going to let me stay in the engineering school in my second semester. So I had to figure that out, too.

00:06:23 - 00:06:44 Stephen Koza
I never did the MBA kind of wish I did. I was always I don't know if you call it a regret, but, recognize a lot of the value. Well, what advice would you have for people earlier in their career that want to do that? But maybe it's not in the cards. Maybe they can't afford it. Maybe they don't want to carve the time out, but still want the kind of benefits that you mentioned.

00:06:44 - 00:06:47 Stephen Koza
The ability to understand the other perspectives.

00:06:47 - 00:07:04 Mark Sass
One of the things that I advise a lot of people that work for me is, you know, map out your next ten years, and try to figure out where you want to be in ten years from now. And that kind of helps drive the situation. So if I have people who are engineers and they're like, I want to be a principal engineer, I want to stay in the technology field and just be an engineer.

00:07:04 - 00:07:24 Mark Sass
That's a far different path than someone that says, you know, I'm looking into management. That's the place I want to go. And unfortunately, a lot of companies today actually kind of top out engineers where the only way they can continue to grow their career is to move to management. And I think that's a mistake. So I would say for most people is figure out, like, what do you want to do in ten years where you want to take your career?

00:07:24 - 00:07:40 Mark Sass
And then and of course, that is like, you know, super high level. You could be if you're a software engineer may say, I want to be a director of software engineering ten years. And the idea there is and map that out and say, okay, if I want to be there in ten years where I need to be in five, and if I want to be there in five, what do I have to accomplish in three?

00:07:40 - 00:07:57 Mark Sass
And if I want to be there in three, what I have to do in the next year and the idea around that is then it gives you, you know, more realistic goals of how you achieve that and kind of stair step up. And if you're looking to move into leadership, that's when you start thinking about how do I build the skill set that's needed for leadership?

00:07:57 - 00:08:23 Mark Sass
A technical person might be saying, I need to understand bigger architecture. I need to, you know, understand scaling. But someone wants to move into leadership, might start reading books like the Five Dysfunctions teams or, five, five levels of leadership or extreme ownership or all these. There's literally you can't throw a rock without hitting a book about leadership, but they might try to define like a leadership ethos that they want to follow, and then they want to start understanding the skill sets that engineering backgrounds don't give you.

00:08:23 - 00:08:37 Mark Sass
Like how to have a hard conversation, how to motivate people, all these type of things that come into play, and then the business stuff will actually flow beyond that. It's like, okay, now I can lead. Now I need to understand how to talk to my peers and how to talk to the business side of the other folks.

00:08:37 - 00:09:01 Stephen Koza
Yeah. Love the book, Rex. By the way. I didn't prep you for that, but, a couple good ones in there. Mark, let's pivot a little bit more into your career and some of the places you've been. And, the way I want to do that is focus on a couple specific topics that maybe there was an arc or some common threads across your different roles.

00:09:01 - 00:09:29 Stephen Koza
And that topic I want to talk about is cost efficiency. So based on what I know about you and how we've got to know each other, that's been an area you've driven some pretty significant impact around. And back to the MBA conversation we just had. I'm not surprised why that is. And so I know that in at least one of your roles, you were able to shave a pretty significant amount off your cloud bill.

00:09:29 - 00:09:57 Stephen Koza
In fact, maybe a couple companies. And I don't want to put words in your mouth or misrepresent. And so I'd love if you could just walk us through your mental model when you come into a new company or a new role, what are you looking for and how do you wrap your arms around the not just current state and where a company is or where their infrastructure costs are, but how do you actually start looking for the opportunities?

00:09:57 - 00:10:05 Stephen Koza
Well, what are the questions you ask? What's the data? What are the things you're poking at to form an opinion before you get into action mode?

00:10:05 - 00:10:23 Mark Sass
Sure. So, one of the first questions I try to understand, or one of the first topics to understand, is this how is the Cogs versus R&D conversation happen? What's considered Cogs? What's considered R&D? What are the expected values out of there from the finance group. So understanding the budget, understanding how they do forecasting at that point.

00:10:23 - 00:10:54 Mark Sass
And then from a technical perspective I start diving into how they manage that process. Now how are they managing cost controls. And so one of the things I've noticed at the last few companies I have is that, you know, companies really jump into the cloud and how and all the great things it gives you, which is that, you know, instant gratification, for lack of a better way to describe it compared to what it was in the data center days, where if you ask for an a VM, it might take weeks, and now you can get a virtual machine or an EC2 instance in, in, in minutes.

00:10:54 - 00:11:14 Mark Sass
And you can stand up large clusters and you can do all this work in seconds. But generally what people forget about is, that's all opex number one, which is, you know, more costly on on a bottom line generally than data centers. But, the other thing is that if there's not the proper governance, you lose track of where who's put spun that up, what its purpose was, what it's being used for.

00:11:14 - 00:11:36 Mark Sass
And so the first thing I talked about like A2K and even I've had that problem here, my new company is what's the tagging structure that we have. What are the requirements that are improved place to to to to look at this. How do we establish ownership of infrastructure. And that's kind of the big thing is that if you have good tagging strategy then you can start aligning your cost better to what the finance teams want to see and the business owners.

00:11:36 - 00:11:56 Mark Sass
When you see if you don't do that, then it becomes really difficult to cost optimize. And then the other thing you need to figure out is you're looking at, how we're spending money in terms of is there a significant amount of on demand spend if we're talking about the clouds and versus compute savings plans, if we're talking about AWS, what's that value at?

00:11:56 - 00:12:18 Mark Sass
You know, how are they doing that that those calculations, if they are doing it and where we can get some easy, quick savings, quick wins by doing stuff like savings plans. And that's something what we did at that at 2K was, you know, we did have a tagging strategy when I started. We started tagging stuff, and the problem is, the older your company gets, the harder it is to track down how things should be tagged.

00:12:18 - 00:12:34 Mark Sass
And so you start. You know, I wasn't taking shortcuts, but you start tagging things into a bucket that's like I'll go looking at later almost. And those are the parts where you might save, you know, 5 or 10% of your bill. But then I said going to look at, you know, how are you spending what, you know, understanding your demand curve of your services.

00:12:34 - 00:12:51 Mark Sass
So, so like in the 2K world, you know, there was a very unicameral kind of view of it. It went up during the day, went down at night. And so you're trying to and then understand how are we auto scaling to match that, that that demand curve, that demand curve of the services that you have. So you can get your supply curve above that.

00:12:51 - 00:13:05 Mark Sass
So those kind of things I'm looking at is trying to understand how the business functions. You know, how the business recognizes the costs and then where some quick, easy wins that we can get to. And then setting ourselves up for a longer term success you through like a tagging strategy to help us out that way.

00:13:05 - 00:13:33 Stephen Koza
I'm curious you you and I have talked about tagging strategies in the past, so I'm not surprised to hear you bring that one up. When you look at a cost savings opportunity and you try and think like you've got the data, now it's tag, you understand where the maybe the over vision resources are, is it usually a people problem or a process problem or a technology problem, or is it all of them.

00:13:33 - 00:14:01 Mark Sass
It's kind of all of them in that sense, because it's probably more related to time. If you think about, the way the cycles work, the dev cycles work with developers or even engineers, they're focused on, I need to deliver product, I need to deliver services to, to my customers, whether those are internal customers or external customers. And so the things that get dropped are the things that are about cost optimization, because those take time to research and go into it from that perspective.

00:14:01 - 00:14:16 Mark Sass
You know, there are tools out there that make great recommendations to you. But even though you look at those recommendations where says, oh, you can save $50,000 a month by turning on intelligent-tiering, you know, inside your S3 buckets, that's an easy win until you start looking at how you're using that and you know, you got to go.

00:14:16 - 00:14:38 Mark Sass
Then understand the data lifecycle of that. And so I think, that's the biggest problem is time to get people to focus on. So one of the things I've been successful at at two K and that I'm implementing here is getting that ability to push that, that ownership to the teams that can have control over that.

00:14:38 - 00:14:51 Mark Sass
So, you know, in a lot of cases like two K my my journey was really interesting. I was given the control of cloud. And they say you own the cloud. So I started looking at expand and I was like, oh my gosh, we're spending so much money. And I would start saying, we need to stop doing this.

00:14:51 - 00:15:07 Mark Sass
We need to start doing this, we need to stop doing this. And we we saved a lot of money. But then six months later, we were in the same situation. So I was like, okay, let me go look around and see how I can save money. And I realized that I was doing that wrong to a certain extent, because the people that were spending the money, you know, weren't my teams, they were other teams.

00:15:07 - 00:15:27 Mark Sass
And so one of the things that I've implemented it at A2K and I'm implementing that my current can be SolarWinds, is the idea of pushing budgets on to these teams for the services they run. So they now have a budget that's been established and it's now their responsibility to understand and giving them visibility, what they're spending. So they now control that, of making sure that they're meeting those budgets.

00:15:27 - 00:15:43 Mark Sass
So if if finance comes and says you have $1 million a month or you have $1 million, divide it amongst all the different services, give them a budget, let them focus on. And then if they're out of bounds, I don't I don't have to be the bad guy. They get a roll up, their manager gets a roll up, the director gets a roll up.

00:15:43 - 00:16:15 Mark Sass
A senior director's Robert says, you know, if it all washes at the higher level, nobody's going to care. But if it starts looking over, people are going to start looking at that, and then that's when they can come back to my teams, the S3 teams generally, and try to understand how can we cost optimize. And so what you're really trying to do is put some guardrails on them so that, you know, they don't spend too much and they don't they don't cut corners too much because that's that's burned you as well, where, you know, some of the services within AWS specifically, you know, if they if you if you move them down too small, they

00:16:15 - 00:16:24 Mark Sass
could be knocked off the network if they're using too much network traffic and unexpectedly. And so that causes outages. You don't want to have. So you're trying to, you know, like I said, put those guardrails out there.

00:16:24 - 00:16:57 Stephen Koza
You know, that that makes me kind of wonder once, let's say you're successful, the each consumer owns the budget. There's some built in accountability there. But I imagine that alone probably doesn't just magic wand saw the cost problem. So I'm curious when you find something that's inefficient and there's an opportunity to modernize or cost optimize, say something's overprovision or a architecture doesn't make sense.

00:16:57 - 00:17:11 Stephen Koza
How do you actually get the buy in to make a major change, even once you've done the budgeting thing where the departments have that pressure, I imagine there's more required. Tell me about that.

00:17:11 - 00:17:33 Mark Sass
Yeah. So it's kind of there's got to two pieces to that. The way to help drive cost optimization is really giving some data driven metrics that help you help the, the those customers understand what they're spending, where they're at. So one of the things we talked about it at two K was driving what I refer to as utilization metric, which is what is the percentage of utilization we have our Kubernetes cluster and what's our goal.

00:17:33 - 00:17:56 Mark Sass
So if our goal is 80% and we're sitting at 60%, we're effectively losing, you know, wasting 20% of our of our spend because we're not where we need to be. On the other side of the equation, you have the application saturation metric, which is how much capacity of our provisioned and how much am I actively using. So if I've got a thousand TPS out there and I'm using 300, I'm at 30%.

00:17:56 - 00:18:14 Mark Sass
Now. Once again, what I when I look at when I surface these metrics, it's not my job to say this is good or bad. It's the it's the product team's decision on where they want to be. A 30% is where they think they need to be. Have at it. That's that's their point. But if they want to be aiming for 80%, then they need to understand that they need to take make changes to their provision.

00:18:14 - 00:18:42 Mark Sass
So that's that's one side of the equation is giving them really good metrics that help them drive the conversation of what they're spending currently. And then when we start talking about like architectural discussions, those become very interesting because then you're trying then you need to really understand the application structure, how they've designed it. And unfortunately you get into a situation where decisions were made long before you got there that have just been built upon for, you know, years and years and years and years.

00:18:42 - 00:19:08 Mark Sass
And so then it becomes, okay, what do we do to fix that problem? Is it a complete, you know, almost like a rewriting of the entire architecture, or are there some simple changes? And those are where you get into conversations like, I just had this conversation. I won't go into details where I'm setting up a bunch of of architects slash principal engineers to talk about how we can build more resiliency than some of our services, because we have some single points of failure that are causing us problems here at my company.

00:19:08 - 00:19:25 Mark Sass
And so then it becomes a collaborative effort to say, okay, and this is what I tell my team all the time is like, if this is where we're at and this is what good looks like, what's our plan to get from place to place? In a lot of cases, teams don't know what good looks like. They really don't know where they should be.

00:19:25 - 00:19:46 Mark Sass
They just know where they're at, and they're kind of moving in the direction of what they think it is. So really kind of formulating that big, big picture idea of like, this is what I think good looks like. And then making sure everybody buys into it and go in that direction, it gives them the path to say, okay, now we can we can look at what good looks like and it helps them understand that path forward to getting to that final architecture.

00:19:46 - 00:19:53 Mark Sass
And sometimes, like I said, that's a complete wipe. Sometimes that is, you know, tweaks that we can do, some tactical changes we can do to make that happen. Super.

00:19:53 - 00:20:35 Stephen Koza
We've known each other for a little while, and I think that's one of the things we as a company pride ourselves on, is we get to work with lots of different clients and lots of different orgs, and as a result, know what good looks like. And I know you've led teams at a bunch of different companies at probably pretty different scales, and so keen off that if you look across gaming and social media and retail and all these different industries, I'd love to start talking about leadership and first question is, tell me a little bit about your leadership style and how it's changed over the years, and not necessarily what you might have done

00:20:35 - 00:20:44 Stephen Koza
differently, but how have you changed Org to org, and what have you learned and and how has that evolved as is your career has grown.

00:20:44 - 00:21:02 Mark Sass
Yeah. So great question. So earlier in my career a lot of a lot of a lot of Marc lore here, backstory lore here for me. But, my wife stayed at home to take care of our four children. And so I was the only person there. So I kind of drove my career off of what gets me more money and what gets me, you know, up the ladder more.

00:21:02 - 00:21:19 Mark Sass
And so, like, I moved into management positions naturally, because I generally would step into a leadership role within the team. And then it just moved into management. And I really didn't have what I would consider in a leadership ethos. What I was doing was like, I didn't like this when I was an engineer, so I'm not going to do that.

00:21:19 - 00:21:44 Mark Sass
And so not the best way to pick a leadership. But then, you know, probably 15, you know, almost 20 years ago, it came across that I needed to figure out, like, I need to pick a path. And there are, you know, an immense amount of different types of authoritarian relationships, like authoritarian or to democracies. I can't say that word, but democracy related ones or, the one that that really kind of stood out to me is the servant leadership.

00:21:44 - 00:22:10 Mark Sass
And the reason why that was stood out for me is because in reality, I find far more interest in helping people maneuver their careers than necessarily the technology itself. I love solving complex problems, but so for me, I gravitated towards how do I help my teams and enable my teams to be better so that I look good by by their their improve productivity, their improved capabilities and do it that way?

00:22:10 - 00:22:29 Mark Sass
But I take a lot of pieces from other places. You know, in terms of like sometimes you have to be authoritarian, right? I mean, I've been in lots of meetings where there's been, you know, discussions on how we solve a problem and there are boils into 2 or 3 camps. And I have to make a decision, say, we're going this way, and that's the way we're going to do it.

00:22:29 - 00:22:52 Mark Sass
And then there's other places where I've had to use more democratic principles, where it's like, you know, what do we want to do? What's the best case? And so you have to go through kind of this, this mentality, what works best with the team you have. So if you have a team that is, already, going through the motions have already been high performing, then it's a different vibe than if you're trying to build a team up.

00:22:52 - 00:23:10 Mark Sass
And I, I think the best example I have is like, if you there's a, there's a book out there called Extreme Ownership, which I'm actually doing has a book club with almost every company I've worked for over the last, you know, five companies. And one of the great things they talk about inside there is in the first section, there's, about they have these boat crews, it's about Navy Seals, about boat crews.

00:23:10 - 00:23:30 Mark Sass
And there's one guy that one team that's always winning and one team that's always in last of the people in last. They're not treated really nicely. So you really don't want to be in last. And they took the leaders and flattened and switched them. So the person that was leading the winning team was now at the back. And the team that was there was winning, still had the team that had the leader that had been at the back all the time.

00:23:30 - 00:23:49 Mark Sass
And what they saw was that leader, that was you know, at the back, he took over a team, a high performance. He was able to keep it going. They were consistently finishing first or second in all the competitions. But then over that same period of time, that leader who had built that team in that first place was able to come in and really drive that team up so that where they were competing with that first place team.

00:23:49 - 00:24:05 Mark Sass
And so you got to understand what kind of leader are you, you know, can you do both? A lot of people can't. A lot of people. It's really struggle to how you build, you know, camaraderie, build a mission, build a vision into a team that delivers and drives those results from a dysfunctional team to a high performing team.

00:24:05 - 00:24:08 Mark Sass
That's that is a far different problem from an illusion perspective.

00:24:08 - 00:24:28 Stephen Koza
Yeah, 100%. I, I was chuckling when you gave the book example because, you know, and I promise we didn't actually prep for this, but I'm literally reading that book right now. And I literally just finished the chapter of the example you gave Book Crew to Boat Crew six. So, you know, I'm not making that up. Yeah.

00:24:28 - 00:24:55 Stephen Koza
How funny. Yeah. Good. Good book so far. So since you mentioned, any you shared some of the initial motivations for getting into leadership and then how that's changed. What advice would you have for early in career engineering managers, and specifically, what do you think the best, or the biggest misconception is that folks have about what the job actually is?

00:24:55 - 00:25:14 Mark Sass
Generally, what you see happen for someone who is an engineer and moves into an engineer in a leadership position, is that they were the best on that team. And what people don't understand a lot of times is what what got you promoted, what got you into that energy engineering leadership role is not what makes you successful there. And that's the big misconception that a lot of people have.

00:25:14 - 00:25:31 Mark Sass
So a lot of people, you know, they step into a leadership role. And, you know, the worst case scenario is they step into a leadership role of the team they used to lead, because the dynamic completely changed as you go from a buddy that you can share all, you know, your gripes and complaints to to someone's like, oh, they're on the other side of the equation now.

00:25:31 - 00:25:51 Mark Sass
They're, they're, they're, they're the bad guys in this case. And so that's one good thing. But for leaders who are stepping into a new leadership role on a team they they haven't been a part of, I think the biggest thing as an engineering is learning to let go, right. So first off, you need to establish you need to figure out what your leadership ethos is going to be, how what kind of a leader do I want to be?

00:25:51 - 00:26:08 Mark Sass
And then from there, you also need to turn around and say, okay, looking at my team, looking at who's on their team, who are the people that have the most, you know, influence across their team, and that's the person you need to figure out how you can work. I work side by side with because that's the person that makes or breaks.

00:26:08 - 00:26:27 Mark Sass
Your success on that team is if you get someone who's the most influential person on the team and they're at odds with you, it's going to make for bad leadership. If they become your partner and you're in your and your friend and your ability to leverage them, then they can leverage the whole team across the board. And so as an end, new engineering leader, that's really what it is.

00:26:27 - 00:26:45 Mark Sass
I think the other thing that the last thing that I would say is two minutes engineering leaders don't use enough context when they're describing things. So like what's the business outcome of why we're doing this work? So a lot of people say, go do this work. And, you know, some director will come down to a manager and say, we need this done.

00:26:45 - 00:27:07 Mark Sass
You that engineer leader may not get context of what they're trying to do, what the purpose of this work is. And so they just pass it on and people do it, but then they're shocked that the results don't match what the expectations were. And so understanding like why I'm doing this work is critically important to explain the why to the team as well, so that they can then build, you know, based off of that rather than just building to it.

00:27:07 - 00:27:24 Mark Sass
And there's that there's that funny meme you see out there where it's like, this is what the customer wanted, and it shows a nice swing under a tree. And then there's, there's all these different things that are come about because of the, the lack of translation. And so trying to make sure you translate that mission, that vision, the business context and outcomes that you're looking for.

00:27:24 - 00:27:26 Mark Sass
Critically important for an engineer leader.

00:27:26 - 00:27:55 Stephen Koza
Yeah. Couldn't agree more. That you got me thinking about my my background a little different. Started at engineering, kind of pivoted to business and sales early on. So I led sales teams for a long time, and I always just took it for granted. Those types of folks, they have to get the business problem and be thinking about the business value and that sort of thing, because that's the only that's going to make you successful or unsuccessful.

00:27:55 - 00:28:16 Stephen Koza
And then joining up, spending more time with engineering teams and technical leaders, not so much the leaders, but I, I took it for granted the value and understanding that perspective in that lens. And so I always encourage our team to ask the why questions. You're doing something. Why what's the benefit? What's the value? What are you hoping to achieve.

00:28:16 - 00:28:41 Stephen Koza
Why is that important? And I think we're getting there. And, you know, hopefully I'm sure you've gotten your teams there. So let me kind of pivot to, some of the unsexy parts of leadership. So you gave me a good, maybe segue on this. You talked about, you know, getting to know the individuals and their motivations and having to get them on board.

00:28:41 - 00:29:05 Stephen Koza
So, do you have an example of a moment where maybe there was a challenge team morale was not in a great place. The the book example you you gave a minute ago. And, like, tell me about that. What was happening? How did you know it was an issue? Was it a crisis? Did you prevent a crisis?

00:29:05 - 00:29:10 Stephen Koza
I'd. I'd love to hear a little bit about, something real world that you've navigated. Yeah.

00:29:10 - 00:29:26 Mark Sass
So I think what I'll say is, in general, you see this a lot on teams that are behind the scenes. And what I mean behind the scenes is like, if I look at a team that's building a product and they're interacting with the customer, and they're dealing with solving customer problems and adding features to do that, that's great.

00:29:26 - 00:29:49 Mark Sass
But like generally, my experience has been on the side of the infrastructure side. So we're one layer at least removed from those, those those customers. And so a lot of times that can feel like you're a second class citizen, for lack of a better describe it. And, an example that we have here at SolarWinds is, on call is dreadful.

00:29:49 - 00:30:04 Mark Sass
Right. And what I mean by that is just a lot of incidents going on. And they're not serious incidents, but there's just a lot of noise in the system. And it causes a morale problem because when people are on call, it's like, you know, I'm getting my phone's going to go off, you know, several hundred times over my 12 or 24 hour shift what it is.

00:30:04 - 00:30:25 Mark Sass
And they're tethered to their computer and everything else like that. And so really what you gotta do is like, you know, understand. It's almost like a blameless postmortem. You have to go through. It's like, okay, what is leading to this poor morale? And then you got to keep asking yourself, why is that the case? So, like, you know, when I started here was like, why are the engineers, you know, experiencing so much dread about being on call?

00:30:25 - 00:30:43 Mark Sass
It's because there's so many answers. Why are there so much incidents coming on? And so you start going down the path of trying to figure out why. And so that's a big, big play there. And then I think another thing that most engineers or most engineering leaders, you know, need to understand as well is that in general, it's not what you demand from your team.

00:30:43 - 00:31:04 Mark Sass
It's more of what you accept. And so if you think about, you know, you have an engineer who's underperforming, you can't you can't miss that that potential performance impact that has on the rest of the team. Because if someone's putting in their 40 or 50 hours and they're doing the best they can, and then someone's, you know, you know, seems to be coasting, you know, that that's going to impact the team as well.

00:31:04 - 00:31:22 Mark Sass
So I always boil things down. There's like a will and a skill problem. So do they. Do they have the will to do the job. Do they have the skill to do the job and to answer those questions on an individual motivation basis, but then understand the impact on the team of what's happening to the team because of those issues that they're having and how we can get the team to build them up.

00:31:22 - 00:31:45 Mark Sass
So it goes back to that, you know, that extreme ownership book. There are several places inside the where they talk about, a situation where one team is complaining about another team, not doing their best. And so they just don't understand the what that other team needs. And so it's a matter of reach across the aisle and try to figure out how do I how do I help my other engineers do better, how to help my teams do better, and intro really trying to understand this problem spaces.

00:31:45 - 00:32:01 Stephen Koza
Yeah. Great perspective. I imagine you've probably had a bunch of great leaders and mentors over the years. Anybody specifically that stands out and you can name them or not. But more importantly what do they teach you or would you learn from them?

00:32:01 - 00:32:21 Mark Sass
I won't name his name, but he was my executive director. Adel. Great guy. He was the one who actually, you know, brought me in and said, what do you want to do with your life? Essentially. Right. Because at that point, you know, I was a senior manager at Dell. I was running the backup and infrastructure environment over there, and I was I was relatively successful.

00:32:21 - 00:32:36 Mark Sass
But he was like, what's your next steps? What do you want to go? The life and I never thought about at that point, and I was in my 40s, to be honest. That was probably early 40s. And so that's really kind of a sharp eye opening moment when you turn around and say, well, you know, I don't I don't know what the end game is.

00:32:36 - 00:32:51 Mark Sass
And I think that was extremely valuable to me to help out, you know, that that that ten year plan I talked about when I, what I sat in front of my engineer, everybody reports to I, I asked them politely to fill out that ten year plan of where they want to be in ten years, and I share mine.

00:32:51 - 00:33:05 Mark Sass
The good news mine. The ten year plan is now retirement. So I get to you know, I get to, you know, ease off on on on career goals and be like that. But you know, that's really what kind of drove me to go back to school. What drove me to get my MBA is that I started thinking, this is where I want to be.

00:33:06 - 00:33:20 Mark Sass
And it's funny because if you would talk to me during my MBA time out, it's like, I want to be a C-suite executive. That's what I want to be able to be a CEO or CTO or something like that. And then as I got farther along my career, I was like, no, that's okay. I was like, you know, this right now, and I'm at right now.

00:33:20 - 00:33:38 Mark Sass
I would consider somewhat a terminal level, maybe get to VP someday. But, you know, I'm I'm perfectly happy doing what I'm doing right now. And I have no desire to move up into more political offices that that have the sea in front of you. Right? So you change, you're kind of you got to make sure you're constantly looking at, you know, is that still what I want to do?

00:33:38 - 00:33:43 Mark Sass
And, and revisit it relatively often to see if you're still on track with what you want to do.

00:33:43 - 00:34:06 Stephen Koza
Yeah, a little bit. Yeah, definitely. Interesting how your motivations and what you value and how you weight those things changes throughout your life, especially once you have a family and kids. And I know you got a bunch of them, so not surprised. Mark. Thank you. This is been genuinely pretty interesting. And I really, really appreciate you carving out the time.

00:34:06 - 00:34:29 Stephen Koza
Thanks for being so candid and or vulnerable and sharing some great insights. I love the book recommendations. I love the earlier in career mentorship story. So listeners everything Mark mentioned today will be in the show notes. And Mark, I think you mentioned, your LinkedIn is probably the best place to find you. So we'll include that as well.

00:34:29 - 00:34:47 Stephen Koza
And there we go. That's it for this episode of Tech Pod Talks. If you enjoyed it and you got something valuable out of it, please share it with somebody else. It's, building platforms at scale and leading teams. That's who we're making this for. I'm Stephen Koza with EverOps. And see you on the next episode.

00:00:09 - 00:00:34 Stephen Koza
Hello and welcome to Tech Pod Talks. Here we do candid conversations with leaders that are building what's next. I'm Stephen Koza. I'm the CEO here at EverOps and every episode we bring in somebody that's a true practitioner, somebody actually in the arena to tell us what's happening behind the scenes and platform engineering in cloud and DevOps and more importantly, technical leadership.

00:00:35 - 00:00:59 Stephen Koza
So really excited to have our guest on today. He is exactly that kind of person. His name is Mark Sass. And Mark, I got to tell you, you've got one of the most interesting career arcs that I've seen in platform engineering. So Mark is build shred teams at Home Depot, managing payment infrastructure for thousands of stores and hundreds of billions of dollars in transactions.

00:00:59 - 00:01:30 Stephen Koza
He's led engineering teams at Meta. More recently senior director of online engineering at 2K games, which is one of the biggest names in gaming, building the infrastructure behind titles that some of you've probably played. Also holds an MBA from Texas A&M. I won't hold that against you. Mark knows that I went to the rival University of Texas, and today Mark is senior director of platform engineering at SolarWinds, which is a company in the middle of a pretty major transformation.

00:01:30 - 00:01:35 Stephen Koza
And I'll let him tell you more about that. So, Mark, welcome to Tech Pod Talks.

00:01:35 - 00:01:37 Mark Sass
Thanks, Stephen. I appreciate you having me on.

00:01:37 - 00:01:54 Stephen Koza
Yeah, it's my pleasure. It's good to, good to catch up. Since we've known each other for a little while. So, Mark, I want to start with your career. You've told me about it, and we've certainly done some research to fill in some of the blanks. You've you've done some pretty interesting work across some household names.

00:01:54 - 00:02:03 Stephen Koza
Home Depot, Meta, Indeed. 2K Games. Now SolarWinds, those are all pretty different companies. What's the thread that connects all of them for you?

00:02:03 - 00:02:29 Mark Sass
It's interesting question. So, you have to go back a little bit farther than what you have listed there. So, years and years ago, I was a technical trainer. I got tired of doing a lot of the teaching and wanted to work on practice. So I started working for a startup and during the dotcom bubble, and, you started working on the practical experience of that, and then I started my own consulting practice, and then I did some more time in startups.

00:02:29 - 00:02:54 Mark Sass
And so, you know, and what I've done is I've kind of bob, between early in my career, between startups where I absolutely love the agility and the problem solving and doing stuff with a shoestring budget. I get frustrated with that. And I move to a bigger company, like a Dell or a Visa that are that are, you know, way back in the time machine for me, where we had all this budget and just really slow and bureaucratic processes.

00:02:54 - 00:03:11 Mark Sass
And so, I've kind of bounced between those companies. And so recently I've kind of settled into the mid-sized companies, although I can't really tell with the time it met in Home Depot. Those are really large companies that had great opportunities, but indeed was a relatively small company. 2K was a relatively small company, and and SolarWinds is not a huge company as well.

00:03:11 - 00:03:24 Mark Sass
So I really like the opportunities there to really flex the technology, but also flex leadership skills where there's opportunities for us to grow the organization and grow the grow the careers of people work for me.

00:03:24 - 00:03:46 Stephen Koza
Yeah, it makes a lot of sense. Thanks for sharing. I want to ask about the MBA because I think that's pretty unique. I don't see that a ton. In, technical leadership, certainly there's some folks like you, but they're the minority and not the majority. So tell me, how does having that business lens change the way you think about technical decisions?

00:03:46 - 00:03:50 Stephen Koza
And then also what was the motivator? Why go do that?

00:03:50 - 00:04:08 Mark Sass
Oh, so that's even a better story. So I actually went to Texas A&M out of out of high school. I was a terrible student. You know, if if they had the diagnosis at that time, I probably wouldn't, ADHD or or on the, on the spectrum there, couldn't focus. And so I had to go through the school of hard knocks.

00:04:08 - 00:04:34 Mark Sass
And when I started working at del back in 2013, they offered free tuition or helpful with the tuition. And so I was like, well, now's a chance. So I went back and I didn't get my degree in computer science like I studied in college. I went to a management because the idea was like, what's the quickest way for me to get my degree in the, the, the, the irony of that is, the reason why I decided to get my degree was because I had applied for a, for a job at EA, a competitor took and they left me.

00:04:34 - 00:05:02 Mark Sass
But they said, you don't have a degree, we can't hire you. And so I started seeing that as a glass ceiling for me. So I got my degree at a at an online university. And then I was like, the problems that I was facing, Adele, because this was during my del time, was there was a lot of conversations about CapEx and opex and amortization, and there was, you know, I was having conversations with product people and marketing people and all these other things, and I understood what they were saying, but I didn't really understand what it meant for me.

00:05:02 - 00:05:32 Mark Sass
And so the lens that it changed, for me, at least from that perspective, is up to that point, I've been in it in other places where I just kept asking for money, and I had no way to really articulate why that was valuable, where the value came from spending money. And so the MBA really kind of opened that lens up to me, started making me think about things like a return on investment for what we wanted to do, and allowed me to understand how product marketing works and how to to exercise that muscle, to say I'm not just building a service that people take advantage of.

00:05:32 - 00:05:51 Mark Sass
I'm building a product and make and understand my customers so that when I built that product, I had, you know, a feeling for what they wanted. I can go deliver it. And it became a partnership, collaborative event that I did as opposed to what I did normally in the IT world, which is like, I'm going to do this stuff and it's going to make my life easier.

00:05:51 - 00:06:06 Mark Sass
I try to make it. How can I make this easier for the customer? So that was really what a gain for me was just that insight to be able to understand how to talk to the finance people, how to talk to product people, how to talk to executives that I lacked before then.

00:06:06 - 00:06:23 Stephen Koza
I was also not a great student, at least in my early years. I don't know if I should out myself, but, University of Texas told me if I don't get a better grades after my first semester, that they weren't going to let me stay in the engineering school in my second semester. So I had to figure that out, too.

00:06:23 - 00:06:44 Stephen Koza
I never did the MBA kind of wish I did. I was always I don't know if you call it a regret, but, recognize a lot of the value. Well, what advice would you have for people earlier in their career that want to do that? But maybe it's not in the cards. Maybe they can't afford it. Maybe they don't want to carve the time out, but still want the kind of benefits that you mentioned.

00:06:44 - 00:06:47 Stephen Koza
The ability to understand the other perspectives.

00:06:47 - 00:07:04 Mark Sass
One of the things that I advise a lot of people that work for me is, you know, map out your next ten years, and try to figure out where you want to be in ten years from now. And that kind of helps drive the situation. So if I have people who are engineers and they're like, I want to be a principal engineer, I want to stay in the technology field and just be an engineer.

00:07:04 - 00:07:24 Mark Sass
That's a far different path than someone that says, you know, I'm looking into management. That's the place I want to go. And unfortunately, a lot of companies today actually kind of top out engineers where the only way they can continue to grow their career is to move to management. And I think that's a mistake. So I would say for most people is figure out, like, what do you want to do in ten years where you want to take your career?

00:07:24 - 00:07:40 Mark Sass
And then and of course, that is like, you know, super high level. You could be if you're a software engineer may say, I want to be a director of software engineering ten years. And the idea there is and map that out and say, okay, if I want to be there in ten years where I need to be in five, and if I want to be there in five, what do I have to accomplish in three?

00:07:40 - 00:07:57 Mark Sass
And if I want to be there in three, what I have to do in the next year and the idea around that is then it gives you, you know, more realistic goals of how you achieve that and kind of stair step up. And if you're looking to move into leadership, that's when you start thinking about how do I build the skill set that's needed for leadership?

00:07:57 - 00:08:23 Mark Sass
A technical person might be saying, I need to understand bigger architecture. I need to, you know, understand scaling. But someone wants to move into leadership, might start reading books like the Five Dysfunctions teams or, five, five levels of leadership or extreme ownership or all these. There's literally you can't throw a rock without hitting a book about leadership, but they might try to define like a leadership ethos that they want to follow, and then they want to start understanding the skill sets that engineering backgrounds don't give you.

00:08:23 - 00:08:37 Mark Sass
Like how to have a hard conversation, how to motivate people, all these type of things that come into play, and then the business stuff will actually flow beyond that. It's like, okay, now I can lead. Now I need to understand how to talk to my peers and how to talk to the business side of the other folks.

00:08:37 - 00:09:01 Stephen Koza
Yeah. Love the book, Rex. By the way. I didn't prep you for that, but, a couple good ones in there. Mark, let's pivot a little bit more into your career and some of the places you've been. And, the way I want to do that is focus on a couple specific topics that maybe there was an arc or some common threads across your different roles.

00:09:01 - 00:09:29 Stephen Koza
And that topic I want to talk about is cost efficiency. So based on what I know about you and how we've got to know each other, that's been an area you've driven some pretty significant impact around. And back to the MBA conversation we just had. I'm not surprised why that is. And so I know that in at least one of your roles, you were able to shave a pretty significant amount off your cloud bill.

00:09:29 - 00:09:57 Stephen Koza
In fact, maybe a couple companies. And I don't want to put words in your mouth or misrepresent. And so I'd love if you could just walk us through your mental model when you come into a new company or a new role, what are you looking for and how do you wrap your arms around the not just current state and where a company is or where their infrastructure costs are, but how do you actually start looking for the opportunities?

00:09:57 - 00:10:05 Stephen Koza
Well, what are the questions you ask? What's the data? What are the things you're poking at to form an opinion before you get into action mode?

00:10:05 - 00:10:23 Mark Sass
Sure. So, one of the first questions I try to understand, or one of the first topics to understand, is this how is the Cogs versus R&D conversation happen? What's considered Cogs? What's considered R&D? What are the expected values out of there from the finance group. So understanding the budget, understanding how they do forecasting at that point.

00:10:23 - 00:10:54 Mark Sass
And then from a technical perspective I start diving into how they manage that process. Now how are they managing cost controls. And so one of the things I've noticed at the last few companies I have is that, you know, companies really jump into the cloud and how and all the great things it gives you, which is that, you know, instant gratification, for lack of a better way to describe it compared to what it was in the data center days, where if you ask for an a VM, it might take weeks, and now you can get a virtual machine or an EC2 instance in, in, in minutes.

00:10:54 - 00:11:14 Mark Sass
And you can stand up large clusters and you can do all this work in seconds. But generally what people forget about is, that's all opex number one, which is, you know, more costly on on a bottom line generally than data centers. But, the other thing is that if there's not the proper governance, you lose track of where who's put spun that up, what its purpose was, what it's being used for.

00:11:14 - 00:11:36 Mark Sass
And so the first thing I talked about like A2K and even I've had that problem here, my new company is what's the tagging structure that we have. What are the requirements that are improved place to to to to look at this. How do we establish ownership of infrastructure. And that's kind of the big thing is that if you have good tagging strategy then you can start aligning your cost better to what the finance teams want to see and the business owners.

00:11:36 - 00:11:56 Mark Sass
When you see if you don't do that, then it becomes really difficult to cost optimize. And then the other thing you need to figure out is you're looking at, how we're spending money in terms of is there a significant amount of on demand spend if we're talking about the clouds and versus compute savings plans, if we're talking about AWS, what's that value at?

00:11:56 - 00:12:18 Mark Sass
You know, how are they doing that that those calculations, if they are doing it and where we can get some easy, quick savings, quick wins by doing stuff like savings plans. And that's something what we did at that at 2K was, you know, we did have a tagging strategy when I started. We started tagging stuff, and the problem is, the older your company gets, the harder it is to track down how things should be tagged.

00:12:18 - 00:12:34 Mark Sass
And so you start. You know, I wasn't taking shortcuts, but you start tagging things into a bucket that's like I'll go looking at later almost. And those are the parts where you might save, you know, 5 or 10% of your bill. But then I said going to look at, you know, how are you spending what, you know, understanding your demand curve of your services.

00:12:34 - 00:12:51 Mark Sass
So, so like in the 2K world, you know, there was a very unicameral kind of view of it. It went up during the day, went down at night. And so you're trying to and then understand how are we auto scaling to match that, that that demand curve, that demand curve of the services that you have. So you can get your supply curve above that.

00:12:51 - 00:13:05 Mark Sass
So those kind of things I'm looking at is trying to understand how the business functions. You know, how the business recognizes the costs and then where some quick, easy wins that we can get to. And then setting ourselves up for a longer term success you through like a tagging strategy to help us out that way.

00:13:05 - 00:13:33 Stephen Koza
I'm curious you you and I have talked about tagging strategies in the past, so I'm not surprised to hear you bring that one up. When you look at a cost savings opportunity and you try and think like you've got the data, now it's tag, you understand where the maybe the over vision resources are, is it usually a people problem or a process problem or a technology problem, or is it all of them.

00:13:33 - 00:14:01 Mark Sass
It's kind of all of them in that sense, because it's probably more related to time. If you think about, the way the cycles work, the dev cycles work with developers or even engineers, they're focused on, I need to deliver product, I need to deliver services to, to my customers, whether those are internal customers or external customers. And so the things that get dropped are the things that are about cost optimization, because those take time to research and go into it from that perspective.

00:14:01 - 00:14:16 Mark Sass
You know, there are tools out there that make great recommendations to you. But even though you look at those recommendations where says, oh, you can save $50,000 a month by turning on intelligent-tiering, you know, inside your S3 buckets, that's an easy win until you start looking at how you're using that and you know, you got to go.

00:14:16 - 00:14:38 Mark Sass
Then understand the data lifecycle of that. And so I think, that's the biggest problem is time to get people to focus on. So one of the things I've been successful at at two K and that I'm implementing here is getting that ability to push that, that ownership to the teams that can have control over that.

00:14:38 - 00:14:51 Mark Sass
So, you know, in a lot of cases like two K my my journey was really interesting. I was given the control of cloud. And they say you own the cloud. So I started looking at expand and I was like, oh my gosh, we're spending so much money. And I would start saying, we need to stop doing this.

00:14:51 - 00:15:07 Mark Sass
We need to start doing this, we need to stop doing this. And we we saved a lot of money. But then six months later, we were in the same situation. So I was like, okay, let me go look around and see how I can save money. And I realized that I was doing that wrong to a certain extent, because the people that were spending the money, you know, weren't my teams, they were other teams.

00:15:07 - 00:15:27 Mark Sass
And so one of the things that I've implemented it at A2K and I'm implementing that my current can be SolarWinds, is the idea of pushing budgets on to these teams for the services they run. So they now have a budget that's been established and it's now their responsibility to understand and giving them visibility, what they're spending. So they now control that, of making sure that they're meeting those budgets.

00:15:27 - 00:15:43 Mark Sass
So if if finance comes and says you have $1 million a month or you have $1 million, divide it amongst all the different services, give them a budget, let them focus on. And then if they're out of bounds, I don't I don't have to be the bad guy. They get a roll up, their manager gets a roll up, the director gets a roll up.

00:15:43 - 00:16:15 Mark Sass
A senior director's Robert says, you know, if it all washes at the higher level, nobody's going to care. But if it starts looking over, people are going to start looking at that, and then that's when they can come back to my teams, the S3 teams generally, and try to understand how can we cost optimize. And so what you're really trying to do is put some guardrails on them so that, you know, they don't spend too much and they don't they don't cut corners too much because that's that's burned you as well, where, you know, some of the services within AWS specifically, you know, if they if you if you move them down too small, they

00:16:15 - 00:16:24 Mark Sass
could be knocked off the network if they're using too much network traffic and unexpectedly. And so that causes outages. You don't want to have. So you're trying to, you know, like I said, put those guardrails out there.

00:16:24 - 00:16:57 Stephen Koza
You know, that that makes me kind of wonder once, let's say you're successful, the each consumer owns the budget. There's some built in accountability there. But I imagine that alone probably doesn't just magic wand saw the cost problem. So I'm curious when you find something that's inefficient and there's an opportunity to modernize or cost optimize, say something's overprovision or a architecture doesn't make sense.

00:16:57 - 00:17:11 Stephen Koza
How do you actually get the buy in to make a major change, even once you've done the budgeting thing where the departments have that pressure, I imagine there's more required. Tell me about that.

00:17:11 - 00:17:33 Mark Sass
Yeah. So it's kind of there's got to two pieces to that. The way to help drive cost optimization is really giving some data driven metrics that help you help the, the those customers understand what they're spending, where they're at. So one of the things we talked about it at two K was driving what I refer to as utilization metric, which is what is the percentage of utilization we have our Kubernetes cluster and what's our goal.

00:17:33 - 00:17:56 Mark Sass
So if our goal is 80% and we're sitting at 60%, we're effectively losing, you know, wasting 20% of our of our spend because we're not where we need to be. On the other side of the equation, you have the application saturation metric, which is how much capacity of our provisioned and how much am I actively using. So if I've got a thousand TPS out there and I'm using 300, I'm at 30%.

00:17:56 - 00:18:14 Mark Sass
Now. Once again, what I when I look at when I surface these metrics, it's not my job to say this is good or bad. It's the it's the product team's decision on where they want to be. A 30% is where they think they need to be. Have at it. That's that's their point. But if they want to be aiming for 80%, then they need to understand that they need to take make changes to their provision.

00:18:14 - 00:18:42 Mark Sass
So that's that's one side of the equation is giving them really good metrics that help them drive the conversation of what they're spending currently. And then when we start talking about like architectural discussions, those become very interesting because then you're trying then you need to really understand the application structure, how they've designed it. And unfortunately you get into a situation where decisions were made long before you got there that have just been built upon for, you know, years and years and years and years.

00:18:42 - 00:19:08 Mark Sass
And so then it becomes, okay, what do we do to fix that problem? Is it a complete, you know, almost like a rewriting of the entire architecture, or are there some simple changes? And those are where you get into conversations like, I just had this conversation. I won't go into details where I'm setting up a bunch of of architects slash principal engineers to talk about how we can build more resiliency than some of our services, because we have some single points of failure that are causing us problems here at my company.

00:19:08 - 00:19:25 Mark Sass
And so then it becomes a collaborative effort to say, okay, and this is what I tell my team all the time is like, if this is where we're at and this is what good looks like, what's our plan to get from place to place? In a lot of cases, teams don't know what good looks like. They really don't know where they should be.

00:19:25 - 00:19:46 Mark Sass
They just know where they're at, and they're kind of moving in the direction of what they think it is. So really kind of formulating that big, big picture idea of like, this is what I think good looks like. And then making sure everybody buys into it and go in that direction, it gives them the path to say, okay, now we can we can look at what good looks like and it helps them understand that path forward to getting to that final architecture.

00:19:46 - 00:19:53 Mark Sass
And sometimes, like I said, that's a complete wipe. Sometimes that is, you know, tweaks that we can do, some tactical changes we can do to make that happen. Super.

00:19:53 - 00:20:35 Stephen Koza
We've known each other for a little while, and I think that's one of the things we as a company pride ourselves on, is we get to work with lots of different clients and lots of different orgs, and as a result, know what good looks like. And I know you've led teams at a bunch of different companies at probably pretty different scales, and so keen off that if you look across gaming and social media and retail and all these different industries, I'd love to start talking about leadership and first question is, tell me a little bit about your leadership style and how it's changed over the years, and not necessarily what you might have done

00:20:35 - 00:20:44 Stephen Koza
differently, but how have you changed Org to org, and what have you learned and and how has that evolved as is your career has grown.

00:20:44 - 00:21:02 Mark Sass
Yeah. So great question. So earlier in my career a lot of a lot of a lot of Marc lore here, backstory lore here for me. But, my wife stayed at home to take care of our four children. And so I was the only person there. So I kind of drove my career off of what gets me more money and what gets me, you know, up the ladder more.

00:21:02 - 00:21:19 Mark Sass
And so, like, I moved into management positions naturally, because I generally would step into a leadership role within the team. And then it just moved into management. And I really didn't have what I would consider in a leadership ethos. What I was doing was like, I didn't like this when I was an engineer, so I'm not going to do that.

00:21:19 - 00:21:44 Mark Sass
And so not the best way to pick a leadership. But then, you know, probably 15, you know, almost 20 years ago, it came across that I needed to figure out, like, I need to pick a path. And there are, you know, an immense amount of different types of authoritarian relationships, like authoritarian or to democracies. I can't say that word, but democracy related ones or, the one that that really kind of stood out to me is the servant leadership.

00:21:44 - 00:22:10 Mark Sass
And the reason why that was stood out for me is because in reality, I find far more interest in helping people maneuver their careers than necessarily the technology itself. I love solving complex problems, but so for me, I gravitated towards how do I help my teams and enable my teams to be better so that I look good by by their their improve productivity, their improved capabilities and do it that way?

00:22:10 - 00:22:29 Mark Sass
But I take a lot of pieces from other places. You know, in terms of like sometimes you have to be authoritarian, right? I mean, I've been in lots of meetings where there's been, you know, discussions on how we solve a problem and there are boils into 2 or 3 camps. And I have to make a decision, say, we're going this way, and that's the way we're going to do it.

00:22:29 - 00:22:52 Mark Sass
And then there's other places where I've had to use more democratic principles, where it's like, you know, what do we want to do? What's the best case? And so you have to go through kind of this, this mentality, what works best with the team you have. So if you have a team that is, already, going through the motions have already been high performing, then it's a different vibe than if you're trying to build a team up.

00:22:52 - 00:23:10 Mark Sass
And I, I think the best example I have is like, if you there's a, there's a book out there called Extreme Ownership, which I'm actually doing has a book club with almost every company I've worked for over the last, you know, five companies. And one of the great things they talk about inside there is in the first section, there's, about they have these boat crews, it's about Navy Seals, about boat crews.

00:23:10 - 00:23:30 Mark Sass
And there's one guy that one team that's always winning and one team that's always in last of the people in last. They're not treated really nicely. So you really don't want to be in last. And they took the leaders and flattened and switched them. So the person that was leading the winning team was now at the back. And the team that was there was winning, still had the team that had the leader that had been at the back all the time.

00:23:30 - 00:23:49 Mark Sass
And what they saw was that leader, that was you know, at the back, he took over a team, a high performance. He was able to keep it going. They were consistently finishing first or second in all the competitions. But then over that same period of time, that leader who had built that team in that first place was able to come in and really drive that team up so that where they were competing with that first place team.

00:23:49 - 00:24:05 Mark Sass
And so you got to understand what kind of leader are you, you know, can you do both? A lot of people can't. A lot of people. It's really struggle to how you build, you know, camaraderie, build a mission, build a vision into a team that delivers and drives those results from a dysfunctional team to a high performing team.

00:24:05 - 00:24:08 Mark Sass
That's that is a far different problem from an illusion perspective.

00:24:08 - 00:24:28 Stephen Koza
Yeah, 100%. I, I was chuckling when you gave the book example because, you know, and I promise we didn't actually prep for this, but I'm literally reading that book right now. And I literally just finished the chapter of the example you gave Book Crew to Boat Crew six. So, you know, I'm not making that up. Yeah.

00:24:28 - 00:24:55 Stephen Koza
How funny. Yeah. Good. Good book so far. So since you mentioned, any you shared some of the initial motivations for getting into leadership and then how that's changed. What advice would you have for early in career engineering managers, and specifically, what do you think the best, or the biggest misconception is that folks have about what the job actually is?

00:24:55 - 00:25:14 Mark Sass
Generally, what you see happen for someone who is an engineer and moves into an engineer in a leadership position, is that they were the best on that team. And what people don't understand a lot of times is what what got you promoted, what got you into that energy engineering leadership role is not what makes you successful there. And that's the big misconception that a lot of people have.

00:25:14 - 00:25:31 Mark Sass
So a lot of people, you know, they step into a leadership role. And, you know, the worst case scenario is they step into a leadership role of the team they used to lead, because the dynamic completely changed as you go from a buddy that you can share all, you know, your gripes and complaints to to someone's like, oh, they're on the other side of the equation now.

00:25:31 - 00:25:51 Mark Sass
They're, they're, they're, they're the bad guys in this case. And so that's one good thing. But for leaders who are stepping into a new leadership role on a team they they haven't been a part of, I think the biggest thing as an engineering is learning to let go, right. So first off, you need to establish you need to figure out what your leadership ethos is going to be, how what kind of a leader do I want to be?

00:25:51 - 00:26:08 Mark Sass
And then from there, you also need to turn around and say, okay, looking at my team, looking at who's on their team, who are the people that have the most, you know, influence across their team, and that's the person you need to figure out how you can work. I work side by side with because that's the person that makes or breaks.

00:26:08 - 00:26:27 Mark Sass
Your success on that team is if you get someone who's the most influential person on the team and they're at odds with you, it's going to make for bad leadership. If they become your partner and you're in your and your friend and your ability to leverage them, then they can leverage the whole team across the board. And so as an end, new engineering leader, that's really what it is.

00:26:27 - 00:26:45 Mark Sass
I think the other thing that the last thing that I would say is two minutes engineering leaders don't use enough context when they're describing things. So like what's the business outcome of why we're doing this work? So a lot of people say, go do this work. And, you know, some director will come down to a manager and say, we need this done.

00:26:45 - 00:27:07 Mark Sass
You that engineer leader may not get context of what they're trying to do, what the purpose of this work is. And so they just pass it on and people do it, but then they're shocked that the results don't match what the expectations were. And so understanding like why I'm doing this work is critically important to explain the why to the team as well, so that they can then build, you know, based off of that rather than just building to it.

00:27:07 - 00:27:24 Mark Sass
And there's that there's that funny meme you see out there where it's like, this is what the customer wanted, and it shows a nice swing under a tree. And then there's, there's all these different things that are come about because of the, the lack of translation. And so trying to make sure you translate that mission, that vision, the business context and outcomes that you're looking for.

00:27:24 - 00:27:26 Mark Sass
Critically important for an engineer leader.

00:27:26 - 00:27:55 Stephen Koza
Yeah. Couldn't agree more. That you got me thinking about my my background a little different. Started at engineering, kind of pivoted to business and sales early on. So I led sales teams for a long time, and I always just took it for granted. Those types of folks, they have to get the business problem and be thinking about the business value and that sort of thing, because that's the only that's going to make you successful or unsuccessful.

00:27:55 - 00:28:16 Stephen Koza
And then joining up, spending more time with engineering teams and technical leaders, not so much the leaders, but I, I took it for granted the value and understanding that perspective in that lens. And so I always encourage our team to ask the why questions. You're doing something. Why what's the benefit? What's the value? What are you hoping to achieve.

00:28:16 - 00:28:41 Stephen Koza
Why is that important? And I think we're getting there. And, you know, hopefully I'm sure you've gotten your teams there. So let me kind of pivot to, some of the unsexy parts of leadership. So you gave me a good, maybe segue on this. You talked about, you know, getting to know the individuals and their motivations and having to get them on board.

00:28:41 - 00:29:05 Stephen Koza
So, do you have an example of a moment where maybe there was a challenge team morale was not in a great place. The the book example you you gave a minute ago. And, like, tell me about that. What was happening? How did you know it was an issue? Was it a crisis? Did you prevent a crisis?

00:29:05 - 00:29:10 Stephen Koza
I'd. I'd love to hear a little bit about, something real world that you've navigated. Yeah.

00:29:10 - 00:29:26 Mark Sass
So I think what I'll say is, in general, you see this a lot on teams that are behind the scenes. And what I mean behind the scenes is like, if I look at a team that's building a product and they're interacting with the customer, and they're dealing with solving customer problems and adding features to do that, that's great.

00:29:26 - 00:29:49 Mark Sass
But like generally, my experience has been on the side of the infrastructure side. So we're one layer at least removed from those, those those customers. And so a lot of times that can feel like you're a second class citizen, for lack of a better describe it. And, an example that we have here at SolarWinds is, on call is dreadful.

00:29:49 - 00:30:04 Mark Sass
Right. And what I mean by that is just a lot of incidents going on. And they're not serious incidents, but there's just a lot of noise in the system. And it causes a morale problem because when people are on call, it's like, you know, I'm getting my phone's going to go off, you know, several hundred times over my 12 or 24 hour shift what it is.

00:30:04 - 00:30:25 Mark Sass
And they're tethered to their computer and everything else like that. And so really what you gotta do is like, you know, understand. It's almost like a blameless postmortem. You have to go through. It's like, okay, what is leading to this poor morale? And then you got to keep asking yourself, why is that the case? So, like, you know, when I started here was like, why are the engineers, you know, experiencing so much dread about being on call?

00:30:25 - 00:30:43 Mark Sass
It's because there's so many answers. Why are there so much incidents coming on? And so you start going down the path of trying to figure out why. And so that's a big, big play there. And then I think another thing that most engineers or most engineering leaders, you know, need to understand as well is that in general, it's not what you demand from your team.

00:30:43 - 00:31:04 Mark Sass
It's more of what you accept. And so if you think about, you know, you have an engineer who's underperforming, you can't you can't miss that that potential performance impact that has on the rest of the team. Because if someone's putting in their 40 or 50 hours and they're doing the best they can, and then someone's, you know, you know, seems to be coasting, you know, that that's going to impact the team as well.

00:31:04 - 00:31:22 Mark Sass
So I always boil things down. There's like a will and a skill problem. So do they. Do they have the will to do the job. Do they have the skill to do the job and to answer those questions on an individual motivation basis, but then understand the impact on the team of what's happening to the team because of those issues that they're having and how we can get the team to build them up.

00:31:22 - 00:31:45 Mark Sass
So it goes back to that, you know, that extreme ownership book. There are several places inside the where they talk about, a situation where one team is complaining about another team, not doing their best. And so they just don't understand the what that other team needs. And so it's a matter of reach across the aisle and try to figure out how do I how do I help my other engineers do better, how to help my teams do better, and intro really trying to understand this problem spaces.

00:31:45 - 00:32:01 Stephen Koza
Yeah. Great perspective. I imagine you've probably had a bunch of great leaders and mentors over the years. Anybody specifically that stands out and you can name them or not. But more importantly what do they teach you or would you learn from them?

00:32:01 - 00:32:21 Mark Sass
I won't name his name, but he was my executive director. Adel. Great guy. He was the one who actually, you know, brought me in and said, what do you want to do with your life? Essentially. Right. Because at that point, you know, I was a senior manager at Dell. I was running the backup and infrastructure environment over there, and I was I was relatively successful.

00:32:21 - 00:32:36 Mark Sass
But he was like, what's your next steps? What do you want to go? The life and I never thought about at that point, and I was in my 40s, to be honest. That was probably early 40s. And so that's really kind of a sharp eye opening moment when you turn around and say, well, you know, I don't I don't know what the end game is.

00:32:36 - 00:32:51 Mark Sass
And I think that was extremely valuable to me to help out, you know, that that that ten year plan I talked about when I, what I sat in front of my engineer, everybody reports to I, I asked them politely to fill out that ten year plan of where they want to be in ten years, and I share mine.

00:32:51 - 00:33:05 Mark Sass
The good news mine. The ten year plan is now retirement. So I get to you know, I get to, you know, ease off on on on career goals and be like that. But you know, that's really what kind of drove me to go back to school. What drove me to get my MBA is that I started thinking, this is where I want to be.

00:33:06 - 00:33:20 Mark Sass
And it's funny because if you would talk to me during my MBA time out, it's like, I want to be a C-suite executive. That's what I want to be able to be a CEO or CTO or something like that. And then as I got farther along my career, I was like, no, that's okay. I was like, you know, this right now, and I'm at right now.

00:33:20 - 00:33:38 Mark Sass
I would consider somewhat a terminal level, maybe get to VP someday. But, you know, I'm I'm perfectly happy doing what I'm doing right now. And I have no desire to move up into more political offices that that have the sea in front of you. Right? So you change, you're kind of you got to make sure you're constantly looking at, you know, is that still what I want to do?

00:33:38 - 00:33:43 Mark Sass
And, and revisit it relatively often to see if you're still on track with what you want to do.

00:33:43 - 00:34:06 Stephen Koza
Yeah, a little bit. Yeah, definitely. Interesting how your motivations and what you value and how you weight those things changes throughout your life, especially once you have a family and kids. And I know you got a bunch of them, so not surprised. Mark. Thank you. This is been genuinely pretty interesting. And I really, really appreciate you carving out the time.

00:34:06 - 00:34:29 Stephen Koza
Thanks for being so candid and or vulnerable and sharing some great insights. I love the book recommendations. I love the earlier in career mentorship story. So listeners everything Mark mentioned today will be in the show notes. And Mark, I think you mentioned, your LinkedIn is probably the best place to find you. So we'll include that as well.

00:34:29 - 00:34:47 Stephen Koza
And there we go. That's it for this episode of Tech Pod Talks. If you enjoyed it and you got something valuable out of it, please share it with somebody else. It's, building platforms at scale and leading teams. That's who we're making this for. I'm Stephen Koza with EverOps. And see you on the next episode.