Episode #
13

Coding Stopped Being the Bottleneck. So What Replaced It?

Episode Description

Back in April, Stephen Koza opened episode two with a claim: coding is not the bottleneck, it is everything else. Over the next ten episodes he put a version of that question to ten engineering leaders, and they did not agree with each other. This episode runs their answers side by side and lands on the one thing they had in common, which is that almost none of the answers were technical.

Main Topics Covered:

  • Where the bottleneck actually went once code generation stopped being the constraint, from code review to deployment to go to market
  • Why story points are collapsing to a factor of one, and which agile roles can be handed to an agent
  • Request volume jumping 25x in a month, and why cost optimization is always the first work to get dropped
  • Token costs passing human costs, and the case for hiring people back
  • Why AI pilots stall on org structure rather than technology, and the argument for thinning vertically before flattening horizontally
  • The guests who reject the premise, from the limits of vibe coding to the junior engineer pipeline

Links & Resources

Guests featured in this episode:

Connect with Stephen Koza: LinkedIn
Connect with EverOps: Website | LinkedIn

References:

Transcript

[00:00:08] Stephen Koza

Every engineering team in the world just got the ability to write ten or even 100 times more code, but almost none of them are shipping ten times the value. Why? Well, coding is not the bottleneck. It's everything else: the reviews, the testing, the rework, the waiting. Now that AI has flooded the system with more code than people can possibly review. You get a fork in the road. Either you figure out how to remove humans from the loop, or watch somebody else do it first.

[00:00:37] Meir Wasserman (Ep 5)

I'll tell you, like any engineer that's experienced will say, like, more code's not better. It gives me like no hope to say we're going to generate a ton of code. I would actually feel a lot better to say, like, we've actually found a way to generate less code and have a less surface area for attacks.

[00:00:51] Stephen Koza

So that was me in April, episode two, and that's Meir Wasserman a few episodes later telling me I got it wrong. I'm Stephen Koza. I run a cloud and AI services firm called EverOps, and this is TechPod Talks. We're ten episodes in. Ten people who actually run engineering orgs. I asked them all basically the same question: if writing code stopped being the hard part. What about now? And I thought I would get more agreement than I got. Google's DORA team asked 5,000 engineers that same thing and same split. AI has pushed up throughput, but stability has gone down. And at the same time, things are faster, shakier, and nobody's solved for that yet. So what you're about to hear doesn't quite line up. A couple of these guys flat out contradicted each other, but the part they do agree on is the part I didn't see coming. Let's get into it.

[00:01:49] Robert Gonzalez (Ep 6)

When it comes to engineering, I look at what we do within engineering as a series of technical things, one of which happens to be coding. And there's a lot of talk in professional circles, and you don't have to go very far, even on LinkedIn, to see this, where people talk about AI and how it writes code and how much slop there is and how terrible the code is. And all of these challenges to why we shouldn't use AI to do this one thing in an engineering space. My view is a little bit more broad than that. Reducing an engineer to code takes away easily 50% of the work that an engineer does. The value that an engineer brings to the business is so much beyond code. We research, we prototype, we use math in a lot of our calculations. We diagram, we collaborate. We actually offer solutions and all kinds of other services outside of the code rewrite. And I think any engineer that's been in the business more than a year knows that code is one of the tools in our toolbox, but it's not our toolbox. So when you look at what AI can do, it can accelerate the engineer in a number of ways, not just code. Now, it will accelerate with code, but it's going to accelerate everything you do. It's going to accelerate architecture. It's going to accelerate documentation. It's going to accelerate testing. It's going to accelerate design. And when you can apply that acceleration to the role, you get a lot more output as a person. One of the struggles that I've seen, and I've talked about this a little bit, is that particularly in the AI space, accelerating one piece of a puzzle just moves what happens in a bottleneck situation to a different part of that puzzle. In an AI driven workflow, if you're saying, hey, I want engineers to be driven by AI, it's awesome. What does your deployment look like? Is your deployment covered with AI? If it's not, you're going to have a bottleneck. Is your go to market covered by AI? If not, you're going to have a bottleneck there. So when I look at AI right now, even as an engineering leader, I look at it more as a business leader, which is how are we using AI to accelerate all facets of the business. So that way we make that fast lane for everybody, not just for engineering. Engineering can go really fast, but if you're not ready to receive that speed and that pace, you're just going to get overwhelmed with the things that engineering builds. And then the value of that acceleration starts to drop off pretty quick.

SEGMENT 1: The bottleneck moved downstream

[00:04:15] Stephen Koza (Ep 6)

Yeah. You know what this reminds me? We had another engineering lead from another SaaS company on. We talked about the same thing. It was like, okay, now you can generate, you know, unlimited code. Code is not the bottleneck. And it turns out, well, you know, something else is now the bottleneck. It's pipeline or testing, and I like how you included go to market and even other parts of the business. So couldn't agree with you more for sure.

[00:04:40] Francisco Trindade (Ep 2)

You know, I think maybe this is a poor analogy, but like the farmer, you know, when the Industrial Revolution came about and all of a sudden they had better, more automated ways to do their job, their job was never to plow the field. Their job was to produce food. And that's what a developer's job is. It's to create a product. Now the coding part has been reduced, which I think is creating the question, like, where are the bottlenecks going to be? And I think we're seeing different things happening in terms of like, you know, product design, maybe sometimes being a bottleneck, but also there is this kind of sometimes expansion of coding, of people trying to do a lot because they can do a lot now, but that's becoming distracting. Through the ultimate goal they're trying to deliver and how to adapt to that. The thing is, the big question, I think what companies need to do. But my kind of perspective is I like what software engineers were doing before, what I was doing now hasn't changed, or shouldn't have changed. And that was deliver value through software. And that's something that we're still going to continue doing.

[00:05:52] Stephen Koza (Ep 2)

Yeah, that's a good take. I like that. So if coding isn't the bottleneck, super easy to generate tons of code now. I know you wrote a piece recently where you talked about this. What is the bottleneck? What's broken that needs to change?

[00:06:06] Francisco Trindade (Ep 2)

Yeah, I think, like, I mean, the [UNCLEAR] is happening is like, you know, now everyone's like, what are we going to do with code reviews, right? Which is, I think, other questions. Can we get rid of code reviews at some point because we, you know, we trust agents to write code consistently well, and we have systems to prevent things from going wrong, which I think is very likely and probably what most startups are doing right now. If you're starting a company now, you're probably not even reviewing code, because you're trying to move fast.

[00:06:39] Chris Harden (Ep 7)

There are a lot of efficiencies that engineering can gain from using agents to do all this work with and for them, and even building new features, and also teaching you about areas that you don't know. You can learn a lot working with the agents. The challenge there, though, is you have to really be technically savvy enough to understand what the agent is making and ensuring that it's not giving you slop, that you then have to deal with as tech debt for the next 5 or 6 years. So there is a risk there. And that's sort of especially true when you're dealing with mature products, mature games, mature tools, and just putting agents in there. And they can wreak all sorts of havoc that you can't necessarily see if you're not careful. When you're doing greenfield brand new development, those teams move super fast because they're just coding away and bringing something to market and then they shape it.

[00:07:28] Robert Gonzalez (Ep 6)

One of the measurements that I've changed in the way that I'm looking at when it comes to metrics is I'm moving away from story points. A lot of agile shops have used story points instead of just pure time based estimations to measure time and complexity, right? But what I'm realizing is as code becomes less and less costly, and as agents are building more and more of our product, what used to be a measure of time and complexity is now becoming a factor of one. So one of the things I think that is becoming clear to me, it might not be clear to others yet, but it's clear to me and my staff, is that a lot of what is baked into agile was built around people and roles within that system, and a lot of those roles can actually be automated. One of the things I just got done doing this week is collaborating with one of my team members to build a Scrum Master agent, and that Scrum Master agent looks at actual true capacity for throughput, looks at actual calendars of people that are going to be at work, who aren't going to be at work. It looks at requests that might be coming in from product, or maybe it's a PRD, maybe it's an epic, whatever. It looks at the volume of what is being asked to do and can actually measure what is on the team against what can be done within a period of time. Even if we stay with a two week sprint, we still need to be able to set out a plan for that sprint. That's something that a Scrum Master historically has done, and that's something that AI agents are actually very good at doing: taking in a lot of data, analyzing it and planning. Right? So the role of the Scrum Master, in my opinion, can become [UNCLEAR]. And it's one of the first things that we're working on. But there's some more pieces to that, which is there's the planning bit and then there's the part that kind of keeps track of what everybody's doing on a given point within the sprint.

SEGMENT 2: It costs something, and someone has to own it

[00:09:11] Tom Kershaw (Ep 9)

That's a judgment call on how you handle the same as, as I mentioned, the Rubicon situation, where all of a sudden the volume of requests and pages we got went up by about 25x, like in a matter of a month. So your choices are don't process them all and figure out which ones to drop and which ones to process. Or you drive your marginal cost to zero. Those are your two options, right? And so my preferred option was driving my marginal cost to zero. Didn't work out super well. But we found ways to filter out the technologies or filter out the requests that were likely to monetize and those that weren't. And that is exactly what we have to do in the music world. There's an interesting study that Deezer has been throwing out there. I'm not going to get the numbers exactly right, but they put out some statement that 60% of their songs that were uploaded to Deezer were AI. And I'm like, whoa, that's like, so essentially you've had a tripling of volumes of songs, and a big chunk of those are AI. But here's the thing. 0.03% of their plays were AI. So what that is, is there's a ton of music being created, but if it's never consumed, if you generated 8 billion songs and nobody ever heard any of them, did they even really exist?

[00:10:31] Mark Sass (Ep 1)

It's kind of all of them in that sense, because it's probably more related to time. If you think about the way the dev cycles work with developers or even engineers, they're focused on, I need to deliver product, I need to deliver services to my customers, whether those are internal customers or external customers. And so the things that get dropped are the things that are about cost optimization, because those take time to research and go into it from that perspective. You know, there are tools out there that will make great recommendations to you. But even though you look at those recommendations where it says, oh, you can save $50,000 a month by turning on intelligent tiering inside your S3 buckets, that's an easy win until you start looking at how you're using that. And you know, you got to go understand the data lifecycle of that. You know, in a lot of cases, like 2K, my journey was really interesting. I was given the control of cloud. And they say you own the cloud. So I started looking at our spend and I was like, oh my gosh, we're spending so much money. And I would start saying, we need to stop doing this. We need to stop doing this. We need to stop doing this. And we saved a lot of money. But then six months later, we were in the same situation. And so I was like, okay, let me go look around and see how I can save money. And I realized that I was doing that wrong to a certain extent, because the people that were spending the money weren't my teams, they were other teams. And so one of the things that I've implemented at 2K and in the implementation at my current company, SolarWinds, is the idea of pushing budgets onto these teams for the services they run. So they now have a budget that's been established, and it's now their responsibility to understand, and giving them visibility into what they're spending. So they now control making sure that they're meeting those budgets. So if finance comes and says you have $1 million a month, or you have $1 million divided amongst all the different services, give them a budget, let them focus on it. Then if they're out of bounds, I don't have to be the bad guy. They get a roll up, their manager gets a roll up, the director gets a roll up, a senior director gets a roll up and says, you know, if it all washes at the higher level, nobody's going to care. But if it starts looking over, people are going to start looking at that, and then that's when they can come back to my teams, the teams generally, and try to understand how can we cost optimize. And so what you're really trying to do is put some guardrails on them so that they don't spend too much and they don't cut corners too much, because that's burned you as well. Some of the services within AWS specifically, if you move them down too small, they could be knocked off the network if they're using too much network traffic unexpectedly. And so that causes outages that you don't want to have. So you're trying to, like I said, put those guardrails out there.

[00:13:05] Patrick McKinney (Ep 10)

Well, I think too, when you combine a lot of, I mean, I'm sure people have seen the articles that now with token costs, AI is becoming more expensive than humans. So it's like now we got to hire some humans back because it's cost effective, and I get that side of it. But I also, you know, AI can't, I mean it can model but it can't really see the future, can't predict the same way the human brain does because it's being trained on data that's here and before. So it can't accurately forecast. And I think humans can't accurately forecast for the most part. I mean, look at most weather channels. But at the same time, I think the human element that needs to be in security teams, engineering teams is, you know, because with security, it's obviously vulnerabilities. With engineering teams it's resiliency. It's making sure that things are up and running. That forecasting, that foresight is something that humans are still doing better than AI for the most part.

[00:13:59] Chris Robertson (Ep 4)

And I think that's where that extra 40% really goes. Like, what's the next thing? What's the next thing? What's the next thing? You're not trying to pretzel people into bad architectural decisions to get a cost target to hit or these other ratios or anything like that. You're understanding your actual customer metrics, you're understanding your unit economics, and you are mapping your back end across a couple of those dimensions. And that is where it gets really powerful. Because if you can [UNCLEAR] your product team and you can say, you know that feature you wanted to roll out to 20%, what if I can get that price down by 50% for you, and now you can roll it out to 50% of the user base? What's that going to do for our attach rate? What's that going to do for our growth rates? You know, Life360 is all relatively public. But you know, we completely changed the unit economics there, and that allowed so much flexibility for the organization because tens of millions of dollars in costs were not hanging over the company. Or early on that journey, you know, here at Arlo, but already some pretty significant wins in that direction as well.

[00:15:22] Tom Kershaw (Ep 9)

We've got parts of our business that are very, very complicated and structured and some that are simpler but have high degrees of scale. And I think the structure problem is the big one. I mean, first of all I'd say is that billions of performances a day by internet terms is tiny. And most of what I deal with here compared to at a Rubicon or an internet company, not even in the same league where we would deal with a billion transactions a second in some cases. Here, a billion an hour is a big number. But I think about it in two ways. So first, there's the database of the song. So here's all the songs that BMI represents. And that's, as you mentioned, relatively small. 25 million in internet terms is tiny. So that's a classic structured data problem. And you've got the song, you've got whoever wrote it. And if it was written by one person, that's easy. But what if it was three people who wrote it and they didn't split it equally? One gets 50% and the other two get 25%, and they're both represented by different publishers and different managers. Okay, great. I can count and I can pay them out percentage wise. But what happens when one of them dies and leaves it to their three kids and one of those kids divorced, and the ex-wife gets half of his share? This is where you get into really complex problems, because copyright in the United States is 100 years. So right now we're collecting and paying on songs that were written in 2020 and, sorry, 1927. And so imagine the ownership changes. And you know, we represent many estates, like Michael Jackson's estate, for example. And so there's a lot of, it's a small database with a massively high degree of complexity. And the thing that adds to that complexity right now is, and I'm sure your listeners have all read this, everyone seems to be packaging their portfolios up and selling them to investors. So the rights holder may not even be the songwriter. Could be some investment company that runs a portfolio of songs and catalogs that they've bought over the years, and they have different motivations than the actual singers and songwriters. Singers and songwriters have a specific way of doing things, which we all appreciate. Investors, maybe a little bit different way of doing things. So running a business where I'm paying everything from scaled investors to individual songwriters that are just starting out at age 12 or 13 or 14 or whatever, like Taylor Swift was when she started songwriting. It's very complicated.

SEGMENT 3: Nobody agreed on the goal

[00:17:55] Chris Robertson (Ep 4)

There's a couple of key things. So if you want to do a journey from some small number of nines to some larger number of nines in terms of reliability, there's a couple of really key things you have to start with before you can figure out if that journey is possible. So one, do you have organizational agreement on the goal? That sounds maybe silly, maybe obvious, but it's actually not. Do you have your marketing or product teams willing to wait for features so that you can fix the reliability issue? Do you have agreement on your exact [UNCLEAR] on if you're a two nine, three nine or a seven nines environment? Do you have an understanding of your customer SLAs? Like all of these types of things take that which you think would be a simple answer and make it actually pretty nuanced. And that's actually been typically the biggest hurdle. Do you have the agreement? Do you have people actually willing to change behaviors in order to do that, to not just give lip service, but actually make meaningful changes? So that's probably 40 to 50% of the problem. Once you have that, the next big thing you need is a way to describe the reliability or whatever the improvement is that you're trying to make. And what I mean by describe is, do you have a vocabulary? Do you have a set of metrics? Do you have a shared understanding of where you are currently and what it is that needs to actually specifically change?

[00:19:44] Janet Sherlock (Ep 3)

That's one of the biggest complaints today about AI, is things getting stuck in, you know, proof of concepts or experiments. And I think that most companies get stuck in experimentation because they're not organized to convert that into impact. And of course, I'm a little biased because, you know, but I believe that literally org structure can be attributed to most of AI failure, lack of adoption or ability to scale it. Literally. If you end up peeling back the onion, you'd see that org structure ends up being behind it. So, you know, when you see lots of pilots and lots of activity, but everyone's on their own and following their own direction, whether it's data, technology, creating models, third party strategies, you know, sometimes you'll still see excitement. But where it falls off is there's no mechanism to scale. So there's not alignment, no true ownership and definitive roles and responsibilities. So you end up not able to create repeatable results. These intermediate vertical layers are often the residue of organizational growth that was never rationalized. So they don't typically create a lot of value. Unfortunately they end up absorbing value. So we're seeing that a lot. Another example you're seeing a lot today, Stephen, is AI teams. So you'll have an AI team that's being created that sits between, you know, a technology team and business teams. It's the same thing. So many times that will literally slow down the speed of implementing AI capabilities as opposed to speeding it up. So, you know, removing vertical layers, it does more than just reduce payroll. It can shorten decision paths, it accelerates response times. And it's even more important today in the era of AI, agentic AI. And it's interesting because, you know, you talk about flattening the organization. So I have a slightly different perspective on simplifying structure than many others, especially today. Right now, you know, the tech companies and some other companies are just aggressively laying off middle management to flatten the organization. And it's a trend driven by cost cutting and AI integration. And the term right now, I think that they're using is it's the great flattening. So major firms like Meta and Amazon, Microsoft, Intel, Expedia, I mean, they've reduced mid-level roles, and their attempt is to streamline bureaucracy. So the instinct is to flatten horizontally. But first I recommend looking for opportunities to flatten or to thin vertically first. And that's, you know, you look for those teams that exist between teams, those coordination layers that were created to bridge two departments that probably should have been talking together all along. It's called the CFD model, and it's a center of enablement, federated data science, and democratized data and insights. So that's center of enablement, which is typically within the technology function. They own the platform, the data, all the infrastructure governance. And that's what allows companies to be able to build and scale. And then you have federated data science where, and we talked about it just a little bit earlier, where the domain expertise sits in the business, and they can focus on their high value use cases, and they can be directly tied to the outcomes that are needed for the business to succeed. So, you know, and they use the platforms and the services provided by that center of enablement. And then the last is democratized data and insights. So this is where you extend access to data and personal AI tools more broadly across the organization. This is where you have your access to the company's GPT and tools for their own personal efficiency and effectiveness. When you have a structure like this, this creates a flywheel, because now you have people who are empowered within their own areas to actually do what's needed for their particular domain. So the center enables platforms and standards. But, you know, business teams deliver business outcomes. That creates your credibility, that creates the demand and that drives the adoption. So you have a compounding value behind it. So then the big trick then, Stephen, becomes making sure you have a good governance process, and then making sure that you don't have that center of enablement be a bottleneck. So you need to have a really good prioritization process. But if you already have a good IT prioritization process, intake, portfolio management, before, if you layer that in, you'll probably be in really good shape.

[00:25:16] Meir Wasserman (Ep 5)

Most problems we solve as technical leaders certainly are people problems. It's not what's the technology choice we should make, or you know what works best in this moment. It's how do we get a team or an organization to align that this is a problem that we need to solve and then go ahead and execute that. Which, that's what I mean when I say people problems, that's what I mean. It's not like a person behaving badly. It's just dealing with the kind of inertia of human nature and the desire to stay the course more than to make drastic change. So, you know, first step is like identifying problems. Yes. But then you have to form relationships and you have to socialize the problems and you have to make them understandable to non-technical crowds. Generally that's like not always a skill set that everyone has, is like take this deep technical problem and level it up to a point where it makes sense but doesn't lose its impact and understanding and accuracy. All those things have to be true before you can even begin to start to institute change. It's not just like, where's AI headed? It's also like, how do we incorporate it? A big problem space I think about a lot with AI is like, how do you take what is capable now and create production line code out of it, or production line outputs, proper scale, secure, all that stuff. We're not like crazy far off from that, but there does need to be some structure around it. You can't just, people say vibe coding is kind of shorthand for like conversational coding. That's not generally sufficient to create production code. I'm sure some people would disagree and say they did it. That's fine. But I think systematically it's not quite right. And so I think a lot about like, how do we bridge the gap between vibe coding and production, like enterprise level code? I think those answers are out there. We're kind of trying some out now, and I'm eager to see where that heads.

SEGMENT 4: The ones who reject the question

[00:27:20] Chris Robertson (Ep 4)

I think that's the most powerful thing about it. I mean, obviously there's like, hey, do you need more hands or some other stuff like that? But that's not the high leverage thing in my head. You know, the high leverage is being able to come in and have a highly skilled team that has opinions and be like, I can give you three options. You know, here's different ways to do it. Here's why I've seen this one work or that one, and being able to debate that with the team because they have that external viewpoint. And I think that's why I tend to go to outside teams. You know, the outside team is never going to have the domain knowledge your internal team does, but they're going to have a breadth of experience that your internal team's never going to have. I think that's the big one. Maybe AI related, maybe not AI related. I think it helps to iterate that faster with the AI stuff. But I think fundamentally, AI is not going to really give you the nuanced questions. And if you can't ask a good question of the tools, the tools will tell you exactly what you want to hear and they will drive you straight off a cliff.

[00:28:29] Meir Wasserman (Ep 5)

But there are also some technical concerns. I had like, oh, does the, I've since gotten over these, by the way, so I'll go there. But it's like, does this kill our junior pipeline? Like, although AI is very powerful in the hands of an experienced senior, what about five years from now when that person, ten years from now, the person retires, and then the junior like never had the chance to learn those lessons. And that was like one concern I had. And then another concern I initially had was like, well, does it kind of make us dimmer? Like if we don't have to think about the code interface to machines, and that's obfuscated from us, then when things go right, not a problem. It's great. But when things go wrong, like are we as a population capable of reasoning through why it's gone wrong and then actually solving it? That one, I'm not so sure yet. I think the jury is out if that's going to be long term harmful. But I think the junior pipeline thing is going to be just fine because what I've seen happen, especially as the models have improved, is like people are using AI. There's kind of two ways this is happening. And, you know, some people are like incorporating it into their current workflows and that's like the slow way of going about it. And then there's people who are just saying, like, what if I never had a current workflow? How would I use this tool and how would I then make whatever I want to make, whatever outcome I want to have happen, happen?

[00:29:54] Francisco Trindade (Ep 2)

Exactly. I think, like, I don't know if it's, maybe it's a hot take, and I don't know how much of a hot take it is, but there's a lot of this kind of talk about like, oh, the junior engineers, they are doomed. And I'm thinking like, when I was a junior engineer, the technology was. So they were like, the amount of revolutions I have gone through my career. Of course, this is a big one. But it's just like, you know, in the beginning of my career, we would spend like a week of six people trying to build a build server so we could commit code, right? Like that was like hundreds of thousands of dollars investment, like that is how much was involved then. And we had to learn. Why people had to learn in this period, like I think generally, years ago I learned and they're going to be fine doing whatever they need to do in like five, ten years, because that's what we all do, right?

[00:30:46] José Gonzalez (Ep 8)

There's a lot of great candidates for just about anyone. And you can shape the role to be whatever you need to do. And oftentimes, like after you hire someone, right? It's like just you have that. And so it's just trying not to be overly perfect when I'm in that space, and that like, okay, what's like the rough sets of traits that we're looking for for this person, right? And can this person come in and redefine this space? Because like, otherwise we're all actually doing this role, right? It's a critical role. Whether it's an engineering hire, [UNCLEAR]. And so it's like, okay, I just need someone to run with it and honestly just redefine this space for us. I just think much more about like, what are the experiences that this person is going on, and like the kind of thinking they're going to have to have and can do that. And there's exceptions to that. If you're hiring for like an AI senior leader, then you probably want to look at AI senior leader, right?

[00:31:48] Patrick McKinney (Ep 10)

Yeah. So some of the things that we went with early on were like we wanted to build the team lean, organically. I wasn't able to hire, you know, 15 people at once or 20 people at once. So as people would come in for different functions, you know, infosec, apps, cloud [UNCLEAR], all that sort of stuff, detection, response, fraud, I would have those people work with me and we would basically build what the function should look like. What are the processes that are repeatable? What are the ones that we could trust to AI if the AI was quality, and which ones would always need human oversight? And where does that human oversight look? A quick example is, you know, you got a lot of these tools out now for apps that will find the vulnerability and they'll cut a PR to fix the vulnerability. Well, we still want human oversight of that PR to make sure it doesn't break something just outside of fixing the vulnerability. It's kind of how it works.

[00:32:40] Chris Harden (Ep 7)

It's also true with developing an application, game dev or whatever. You are the person orchestrating the agents. And yes, there are arrangements where an agent will orchestrate other agents for you and all this kind of stuff. But the most common use, once you get past just talking into the chat, is you tell it what to do, and you verify it over and over again, and that's you. You're the orchestrator and the verifier.

[00:33:04] Stephen Koza (Ep 9)

And yeah, you just, you said the two key words in the same sentence.

[00:33:08] Tom Kershaw (Ep 9)

Focus is no. That's what focus means. It literally means the word no.

[00:33:12] Stephen Koza (Ep 9)

Yeah, yeah. You know, I love when there's a list of ten priorities. And, you know, it's easy to say, well, nothing's really a priority then.

[00:33:19] Tom Kershaw (Ep 9)

Yeah. I say which one is more important, and the business team says they're all important. And what are you going to...

[00:33:27] Stephen Koza (Ep 9)

Yeah, we were in a meeting the other day and there was a list of metrics underneath the heading North Star. And my CTO astutely pointed out that there's only one North Star in the universe.

[00:33:41] Stephen Koza

Nobody talked about the code, and it's intuitive, but that's kind of the part I didn't see coming. Push on any of these people, and the problem just moves somewhere else. Who owns the outcome? Did people even agree on the goal in the first place? Did the downstream team believe they could absorb what the upstream team just sped up? Well, Chris Robertson put a number on it. 40 to 50% of the work is just getting to agreement before you actually try and fix anything. When DORA ran the numbers, 5,000 engineers, they all landed in the same spot. AI doesn't actually fix a team, it amplifies what's already there. Ten conversations, same conclusion, and none of us were trying to get there. We wrote up the longer version of this on our website, checklist included. We'll put the link in the show notes. So thanks to all the guests so far. It's been super fun. We got a great lineup around the corner for season two. See you on the next one, and thanks for listening.

[00:00:08] Stephen Koza

Every engineering team in the world just got the ability to write ten or even 100 times more code, but almost none of them are shipping ten times the value. Why? Well, coding is not the bottleneck. It's everything else: the reviews, the testing, the rework, the waiting. Now that AI has flooded the system with more code than people can possibly review. You get a fork in the road. Either you figure out how to remove humans from the loop, or watch somebody else do it first.

[00:00:37] Meir Wasserman (Ep 5)

I'll tell you, like any engineer that's experienced will say, like, more code's not better. It gives me like no hope to say we're going to generate a ton of code. I would actually feel a lot better to say, like, we've actually found a way to generate less code and have a less surface area for attacks.

[00:00:51] Stephen Koza

So that was me in April, episode two, and that's Meir Wasserman a few episodes later telling me I got it wrong. I'm Stephen Koza. I run a cloud and AI services firm called EverOps, and this is TechPod Talks. We're ten episodes in. Ten people who actually run engineering orgs. I asked them all basically the same question: if writing code stopped being the hard part. What about now? And I thought I would get more agreement than I got. Google's DORA team asked 5,000 engineers that same thing and same split. AI has pushed up throughput, but stability has gone down. And at the same time, things are faster, shakier, and nobody's solved for that yet. So what you're about to hear doesn't quite line up. A couple of these guys flat out contradicted each other, but the part they do agree on is the part I didn't see coming. Let's get into it.

[00:01:49] Robert Gonzalez (Ep 6)

When it comes to engineering, I look at what we do within engineering as a series of technical things, one of which happens to be coding. And there's a lot of talk in professional circles, and you don't have to go very far, even on LinkedIn, to see this, where people talk about AI and how it writes code and how much slop there is and how terrible the code is. And all of these challenges to why we shouldn't use AI to do this one thing in an engineering space. My view is a little bit more broad than that. Reducing an engineer to code takes away easily 50% of the work that an engineer does. The value that an engineer brings to the business is so much beyond code. We research, we prototype, we use math in a lot of our calculations. We diagram, we collaborate. We actually offer solutions and all kinds of other services outside of the code rewrite. And I think any engineer that's been in the business more than a year knows that code is one of the tools in our toolbox, but it's not our toolbox. So when you look at what AI can do, it can accelerate the engineer in a number of ways, not just code. Now, it will accelerate with code, but it's going to accelerate everything you do. It's going to accelerate architecture. It's going to accelerate documentation. It's going to accelerate testing. It's going to accelerate design. And when you can apply that acceleration to the role, you get a lot more output as a person. One of the struggles that I've seen, and I've talked about this a little bit, is that particularly in the AI space, accelerating one piece of a puzzle just moves what happens in a bottleneck situation to a different part of that puzzle. In an AI driven workflow, if you're saying, hey, I want engineers to be driven by AI, it's awesome. What does your deployment look like? Is your deployment covered with AI? If it's not, you're going to have a bottleneck. Is your go to market covered by AI? If not, you're going to have a bottleneck there. So when I look at AI right now, even as an engineering leader, I look at it more as a business leader, which is how are we using AI to accelerate all facets of the business. So that way we make that fast lane for everybody, not just for engineering. Engineering can go really fast, but if you're not ready to receive that speed and that pace, you're just going to get overwhelmed with the things that engineering builds. And then the value of that acceleration starts to drop off pretty quick.

SEGMENT 1: The bottleneck moved downstream

[00:04:15] Stephen Koza (Ep 6)

Yeah. You know what this reminds me? We had another engineering lead from another SaaS company on. We talked about the same thing. It was like, okay, now you can generate, you know, unlimited code. Code is not the bottleneck. And it turns out, well, you know, something else is now the bottleneck. It's pipeline or testing, and I like how you included go to market and even other parts of the business. So couldn't agree with you more for sure.

[00:04:40] Francisco Trindade (Ep 2)

You know, I think maybe this is a poor analogy, but like the farmer, you know, when the Industrial Revolution came about and all of a sudden they had better, more automated ways to do their job, their job was never to plow the field. Their job was to produce food. And that's what a developer's job is. It's to create a product. Now the coding part has been reduced, which I think is creating the question, like, where are the bottlenecks going to be? And I think we're seeing different things happening in terms of like, you know, product design, maybe sometimes being a bottleneck, but also there is this kind of sometimes expansion of coding, of people trying to do a lot because they can do a lot now, but that's becoming distracting. Through the ultimate goal they're trying to deliver and how to adapt to that. The thing is, the big question, I think what companies need to do. But my kind of perspective is I like what software engineers were doing before, what I was doing now hasn't changed, or shouldn't have changed. And that was deliver value through software. And that's something that we're still going to continue doing.

[00:05:52] Stephen Koza (Ep 2)

Yeah, that's a good take. I like that. So if coding isn't the bottleneck, super easy to generate tons of code now. I know you wrote a piece recently where you talked about this. What is the bottleneck? What's broken that needs to change?

[00:06:06] Francisco Trindade (Ep 2)

Yeah, I think, like, I mean, the [UNCLEAR] is happening is like, you know, now everyone's like, what are we going to do with code reviews, right? Which is, I think, other questions. Can we get rid of code reviews at some point because we, you know, we trust agents to write code consistently well, and we have systems to prevent things from going wrong, which I think is very likely and probably what most startups are doing right now. If you're starting a company now, you're probably not even reviewing code, because you're trying to move fast.

[00:06:39] Chris Harden (Ep 7)

There are a lot of efficiencies that engineering can gain from using agents to do all this work with and for them, and even building new features, and also teaching you about areas that you don't know. You can learn a lot working with the agents. The challenge there, though, is you have to really be technically savvy enough to understand what the agent is making and ensuring that it's not giving you slop, that you then have to deal with as tech debt for the next 5 or 6 years. So there is a risk there. And that's sort of especially true when you're dealing with mature products, mature games, mature tools, and just putting agents in there. And they can wreak all sorts of havoc that you can't necessarily see if you're not careful. When you're doing greenfield brand new development, those teams move super fast because they're just coding away and bringing something to market and then they shape it.

[00:07:28] Robert Gonzalez (Ep 6)

One of the measurements that I've changed in the way that I'm looking at when it comes to metrics is I'm moving away from story points. A lot of agile shops have used story points instead of just pure time based estimations to measure time and complexity, right? But what I'm realizing is as code becomes less and less costly, and as agents are building more and more of our product, what used to be a measure of time and complexity is now becoming a factor of one. So one of the things I think that is becoming clear to me, it might not be clear to others yet, but it's clear to me and my staff, is that a lot of what is baked into agile was built around people and roles within that system, and a lot of those roles can actually be automated. One of the things I just got done doing this week is collaborating with one of my team members to build a Scrum Master agent, and that Scrum Master agent looks at actual true capacity for throughput, looks at actual calendars of people that are going to be at work, who aren't going to be at work. It looks at requests that might be coming in from product, or maybe it's a PRD, maybe it's an epic, whatever. It looks at the volume of what is being asked to do and can actually measure what is on the team against what can be done within a period of time. Even if we stay with a two week sprint, we still need to be able to set out a plan for that sprint. That's something that a Scrum Master historically has done, and that's something that AI agents are actually very good at doing: taking in a lot of data, analyzing it and planning. Right? So the role of the Scrum Master, in my opinion, can become [UNCLEAR]. And it's one of the first things that we're working on. But there's some more pieces to that, which is there's the planning bit and then there's the part that kind of keeps track of what everybody's doing on a given point within the sprint.

SEGMENT 2: It costs something, and someone has to own it

[00:09:11] Tom Kershaw (Ep 9)

That's a judgment call on how you handle the same as, as I mentioned, the Rubicon situation, where all of a sudden the volume of requests and pages we got went up by about 25x, like in a matter of a month. So your choices are don't process them all and figure out which ones to drop and which ones to process. Or you drive your marginal cost to zero. Those are your two options, right? And so my preferred option was driving my marginal cost to zero. Didn't work out super well. But we found ways to filter out the technologies or filter out the requests that were likely to monetize and those that weren't. And that is exactly what we have to do in the music world. There's an interesting study that Deezer has been throwing out there. I'm not going to get the numbers exactly right, but they put out some statement that 60% of their songs that were uploaded to Deezer were AI. And I'm like, whoa, that's like, so essentially you've had a tripling of volumes of songs, and a big chunk of those are AI. But here's the thing. 0.03% of their plays were AI. So what that is, is there's a ton of music being created, but if it's never consumed, if you generated 8 billion songs and nobody ever heard any of them, did they even really exist?

[00:10:31] Mark Sass (Ep 1)

It's kind of all of them in that sense, because it's probably more related to time. If you think about the way the dev cycles work with developers or even engineers, they're focused on, I need to deliver product, I need to deliver services to my customers, whether those are internal customers or external customers. And so the things that get dropped are the things that are about cost optimization, because those take time to research and go into it from that perspective. You know, there are tools out there that will make great recommendations to you. But even though you look at those recommendations where it says, oh, you can save $50,000 a month by turning on intelligent tiering inside your S3 buckets, that's an easy win until you start looking at how you're using that. And you know, you got to go understand the data lifecycle of that. You know, in a lot of cases, like 2K, my journey was really interesting. I was given the control of cloud. And they say you own the cloud. So I started looking at our spend and I was like, oh my gosh, we're spending so much money. And I would start saying, we need to stop doing this. We need to stop doing this. We need to stop doing this. And we saved a lot of money. But then six months later, we were in the same situation. And so I was like, okay, let me go look around and see how I can save money. And I realized that I was doing that wrong to a certain extent, because the people that were spending the money weren't my teams, they were other teams. And so one of the things that I've implemented at 2K and in the implementation at my current company, SolarWinds, is the idea of pushing budgets onto these teams for the services they run. So they now have a budget that's been established, and it's now their responsibility to understand, and giving them visibility into what they're spending. So they now control making sure that they're meeting those budgets. So if finance comes and says you have $1 million a month, or you have $1 million divided amongst all the different services, give them a budget, let them focus on it. Then if they're out of bounds, I don't have to be the bad guy. They get a roll up, their manager gets a roll up, the director gets a roll up, a senior director gets a roll up and says, you know, if it all washes at the higher level, nobody's going to care. But if it starts looking over, people are going to start looking at that, and then that's when they can come back to my teams, the teams generally, and try to understand how can we cost optimize. And so what you're really trying to do is put some guardrails on them so that they don't spend too much and they don't cut corners too much, because that's burned you as well. Some of the services within AWS specifically, if you move them down too small, they could be knocked off the network if they're using too much network traffic unexpectedly. And so that causes outages that you don't want to have. So you're trying to, like I said, put those guardrails out there.

[00:13:05] Patrick McKinney (Ep 10)

Well, I think too, when you combine a lot of, I mean, I'm sure people have seen the articles that now with token costs, AI is becoming more expensive than humans. So it's like now we got to hire some humans back because it's cost effective, and I get that side of it. But I also, you know, AI can't, I mean it can model but it can't really see the future, can't predict the same way the human brain does because it's being trained on data that's here and before. So it can't accurately forecast. And I think humans can't accurately forecast for the most part. I mean, look at most weather channels. But at the same time, I think the human element that needs to be in security teams, engineering teams is, you know, because with security, it's obviously vulnerabilities. With engineering teams it's resiliency. It's making sure that things are up and running. That forecasting, that foresight is something that humans are still doing better than AI for the most part.

[00:13:59] Chris Robertson (Ep 4)

And I think that's where that extra 40% really goes. Like, what's the next thing? What's the next thing? What's the next thing? You're not trying to pretzel people into bad architectural decisions to get a cost target to hit or these other ratios or anything like that. You're understanding your actual customer metrics, you're understanding your unit economics, and you are mapping your back end across a couple of those dimensions. And that is where it gets really powerful. Because if you can [UNCLEAR] your product team and you can say, you know that feature you wanted to roll out to 20%, what if I can get that price down by 50% for you, and now you can roll it out to 50% of the user base? What's that going to do for our attach rate? What's that going to do for our growth rates? You know, Life360 is all relatively public. But you know, we completely changed the unit economics there, and that allowed so much flexibility for the organization because tens of millions of dollars in costs were not hanging over the company. Or early on that journey, you know, here at Arlo, but already some pretty significant wins in that direction as well.

[00:15:22] Tom Kershaw (Ep 9)

We've got parts of our business that are very, very complicated and structured and some that are simpler but have high degrees of scale. And I think the structure problem is the big one. I mean, first of all I'd say is that billions of performances a day by internet terms is tiny. And most of what I deal with here compared to at a Rubicon or an internet company, not even in the same league where we would deal with a billion transactions a second in some cases. Here, a billion an hour is a big number. But I think about it in two ways. So first, there's the database of the song. So here's all the songs that BMI represents. And that's, as you mentioned, relatively small. 25 million in internet terms is tiny. So that's a classic structured data problem. And you've got the song, you've got whoever wrote it. And if it was written by one person, that's easy. But what if it was three people who wrote it and they didn't split it equally? One gets 50% and the other two get 25%, and they're both represented by different publishers and different managers. Okay, great. I can count and I can pay them out percentage wise. But what happens when one of them dies and leaves it to their three kids and one of those kids divorced, and the ex-wife gets half of his share? This is where you get into really complex problems, because copyright in the United States is 100 years. So right now we're collecting and paying on songs that were written in 2020 and, sorry, 1927. And so imagine the ownership changes. And you know, we represent many estates, like Michael Jackson's estate, for example. And so there's a lot of, it's a small database with a massively high degree of complexity. And the thing that adds to that complexity right now is, and I'm sure your listeners have all read this, everyone seems to be packaging their portfolios up and selling them to investors. So the rights holder may not even be the songwriter. Could be some investment company that runs a portfolio of songs and catalogs that they've bought over the years, and they have different motivations than the actual singers and songwriters. Singers and songwriters have a specific way of doing things, which we all appreciate. Investors, maybe a little bit different way of doing things. So running a business where I'm paying everything from scaled investors to individual songwriters that are just starting out at age 12 or 13 or 14 or whatever, like Taylor Swift was when she started songwriting. It's very complicated.

SEGMENT 3: Nobody agreed on the goal

[00:17:55] Chris Robertson (Ep 4)

There's a couple of key things. So if you want to do a journey from some small number of nines to some larger number of nines in terms of reliability, there's a couple of really key things you have to start with before you can figure out if that journey is possible. So one, do you have organizational agreement on the goal? That sounds maybe silly, maybe obvious, but it's actually not. Do you have your marketing or product teams willing to wait for features so that you can fix the reliability issue? Do you have agreement on your exact [UNCLEAR] on if you're a two nine, three nine or a seven nines environment? Do you have an understanding of your customer SLAs? Like all of these types of things take that which you think would be a simple answer and make it actually pretty nuanced. And that's actually been typically the biggest hurdle. Do you have the agreement? Do you have people actually willing to change behaviors in order to do that, to not just give lip service, but actually make meaningful changes? So that's probably 40 to 50% of the problem. Once you have that, the next big thing you need is a way to describe the reliability or whatever the improvement is that you're trying to make. And what I mean by describe is, do you have a vocabulary? Do you have a set of metrics? Do you have a shared understanding of where you are currently and what it is that needs to actually specifically change?

[00:19:44] Janet Sherlock (Ep 3)

That's one of the biggest complaints today about AI, is things getting stuck in, you know, proof of concepts or experiments. And I think that most companies get stuck in experimentation because they're not organized to convert that into impact. And of course, I'm a little biased because, you know, but I believe that literally org structure can be attributed to most of AI failure, lack of adoption or ability to scale it. Literally. If you end up peeling back the onion, you'd see that org structure ends up being behind it. So, you know, when you see lots of pilots and lots of activity, but everyone's on their own and following their own direction, whether it's data, technology, creating models, third party strategies, you know, sometimes you'll still see excitement. But where it falls off is there's no mechanism to scale. So there's not alignment, no true ownership and definitive roles and responsibilities. So you end up not able to create repeatable results. These intermediate vertical layers are often the residue of organizational growth that was never rationalized. So they don't typically create a lot of value. Unfortunately they end up absorbing value. So we're seeing that a lot. Another example you're seeing a lot today, Stephen, is AI teams. So you'll have an AI team that's being created that sits between, you know, a technology team and business teams. It's the same thing. So many times that will literally slow down the speed of implementing AI capabilities as opposed to speeding it up. So, you know, removing vertical layers, it does more than just reduce payroll. It can shorten decision paths, it accelerates response times. And it's even more important today in the era of AI, agentic AI. And it's interesting because, you know, you talk about flattening the organization. So I have a slightly different perspective on simplifying structure than many others, especially today. Right now, you know, the tech companies and some other companies are just aggressively laying off middle management to flatten the organization. And it's a trend driven by cost cutting and AI integration. And the term right now, I think that they're using is it's the great flattening. So major firms like Meta and Amazon, Microsoft, Intel, Expedia, I mean, they've reduced mid-level roles, and their attempt is to streamline bureaucracy. So the instinct is to flatten horizontally. But first I recommend looking for opportunities to flatten or to thin vertically first. And that's, you know, you look for those teams that exist between teams, those coordination layers that were created to bridge two departments that probably should have been talking together all along. It's called the CFD model, and it's a center of enablement, federated data science, and democratized data and insights. So that's center of enablement, which is typically within the technology function. They own the platform, the data, all the infrastructure governance. And that's what allows companies to be able to build and scale. And then you have federated data science where, and we talked about it just a little bit earlier, where the domain expertise sits in the business, and they can focus on their high value use cases, and they can be directly tied to the outcomes that are needed for the business to succeed. So, you know, and they use the platforms and the services provided by that center of enablement. And then the last is democratized data and insights. So this is where you extend access to data and personal AI tools more broadly across the organization. This is where you have your access to the company's GPT and tools for their own personal efficiency and effectiveness. When you have a structure like this, this creates a flywheel, because now you have people who are empowered within their own areas to actually do what's needed for their particular domain. So the center enables platforms and standards. But, you know, business teams deliver business outcomes. That creates your credibility, that creates the demand and that drives the adoption. So you have a compounding value behind it. So then the big trick then, Stephen, becomes making sure you have a good governance process, and then making sure that you don't have that center of enablement be a bottleneck. So you need to have a really good prioritization process. But if you already have a good IT prioritization process, intake, portfolio management, before, if you layer that in, you'll probably be in really good shape.

[00:25:16] Meir Wasserman (Ep 5)

Most problems we solve as technical leaders certainly are people problems. It's not what's the technology choice we should make, or you know what works best in this moment. It's how do we get a team or an organization to align that this is a problem that we need to solve and then go ahead and execute that. Which, that's what I mean when I say people problems, that's what I mean. It's not like a person behaving badly. It's just dealing with the kind of inertia of human nature and the desire to stay the course more than to make drastic change. So, you know, first step is like identifying problems. Yes. But then you have to form relationships and you have to socialize the problems and you have to make them understandable to non-technical crowds. Generally that's like not always a skill set that everyone has, is like take this deep technical problem and level it up to a point where it makes sense but doesn't lose its impact and understanding and accuracy. All those things have to be true before you can even begin to start to institute change. It's not just like, where's AI headed? It's also like, how do we incorporate it? A big problem space I think about a lot with AI is like, how do you take what is capable now and create production line code out of it, or production line outputs, proper scale, secure, all that stuff. We're not like crazy far off from that, but there does need to be some structure around it. You can't just, people say vibe coding is kind of shorthand for like conversational coding. That's not generally sufficient to create production code. I'm sure some people would disagree and say they did it. That's fine. But I think systematically it's not quite right. And so I think a lot about like, how do we bridge the gap between vibe coding and production, like enterprise level code? I think those answers are out there. We're kind of trying some out now, and I'm eager to see where that heads.

SEGMENT 4: The ones who reject the question

[00:27:20] Chris Robertson (Ep 4)

I think that's the most powerful thing about it. I mean, obviously there's like, hey, do you need more hands or some other stuff like that? But that's not the high leverage thing in my head. You know, the high leverage is being able to come in and have a highly skilled team that has opinions and be like, I can give you three options. You know, here's different ways to do it. Here's why I've seen this one work or that one, and being able to debate that with the team because they have that external viewpoint. And I think that's why I tend to go to outside teams. You know, the outside team is never going to have the domain knowledge your internal team does, but they're going to have a breadth of experience that your internal team's never going to have. I think that's the big one. Maybe AI related, maybe not AI related. I think it helps to iterate that faster with the AI stuff. But I think fundamentally, AI is not going to really give you the nuanced questions. And if you can't ask a good question of the tools, the tools will tell you exactly what you want to hear and they will drive you straight off a cliff.

[00:28:29] Meir Wasserman (Ep 5)

But there are also some technical concerns. I had like, oh, does the, I've since gotten over these, by the way, so I'll go there. But it's like, does this kill our junior pipeline? Like, although AI is very powerful in the hands of an experienced senior, what about five years from now when that person, ten years from now, the person retires, and then the junior like never had the chance to learn those lessons. And that was like one concern I had. And then another concern I initially had was like, well, does it kind of make us dimmer? Like if we don't have to think about the code interface to machines, and that's obfuscated from us, then when things go right, not a problem. It's great. But when things go wrong, like are we as a population capable of reasoning through why it's gone wrong and then actually solving it? That one, I'm not so sure yet. I think the jury is out if that's going to be long term harmful. But I think the junior pipeline thing is going to be just fine because what I've seen happen, especially as the models have improved, is like people are using AI. There's kind of two ways this is happening. And, you know, some people are like incorporating it into their current workflows and that's like the slow way of going about it. And then there's people who are just saying, like, what if I never had a current workflow? How would I use this tool and how would I then make whatever I want to make, whatever outcome I want to have happen, happen?

[00:29:54] Francisco Trindade (Ep 2)

Exactly. I think, like, I don't know if it's, maybe it's a hot take, and I don't know how much of a hot take it is, but there's a lot of this kind of talk about like, oh, the junior engineers, they are doomed. And I'm thinking like, when I was a junior engineer, the technology was. So they were like, the amount of revolutions I have gone through my career. Of course, this is a big one. But it's just like, you know, in the beginning of my career, we would spend like a week of six people trying to build a build server so we could commit code, right? Like that was like hundreds of thousands of dollars investment, like that is how much was involved then. And we had to learn. Why people had to learn in this period, like I think generally, years ago I learned and they're going to be fine doing whatever they need to do in like five, ten years, because that's what we all do, right?

[00:30:46] José Gonzalez (Ep 8)

There's a lot of great candidates for just about anyone. And you can shape the role to be whatever you need to do. And oftentimes, like after you hire someone, right? It's like just you have that. And so it's just trying not to be overly perfect when I'm in that space, and that like, okay, what's like the rough sets of traits that we're looking for for this person, right? And can this person come in and redefine this space? Because like, otherwise we're all actually doing this role, right? It's a critical role. Whether it's an engineering hire, [UNCLEAR]. And so it's like, okay, I just need someone to run with it and honestly just redefine this space for us. I just think much more about like, what are the experiences that this person is going on, and like the kind of thinking they're going to have to have and can do that. And there's exceptions to that. If you're hiring for like an AI senior leader, then you probably want to look at AI senior leader, right?

[00:31:48] Patrick McKinney (Ep 10)

Yeah. So some of the things that we went with early on were like we wanted to build the team lean, organically. I wasn't able to hire, you know, 15 people at once or 20 people at once. So as people would come in for different functions, you know, infosec, apps, cloud [UNCLEAR], all that sort of stuff, detection, response, fraud, I would have those people work with me and we would basically build what the function should look like. What are the processes that are repeatable? What are the ones that we could trust to AI if the AI was quality, and which ones would always need human oversight? And where does that human oversight look? A quick example is, you know, you got a lot of these tools out now for apps that will find the vulnerability and they'll cut a PR to fix the vulnerability. Well, we still want human oversight of that PR to make sure it doesn't break something just outside of fixing the vulnerability. It's kind of how it works.

[00:32:40] Chris Harden (Ep 7)

It's also true with developing an application, game dev or whatever. You are the person orchestrating the agents. And yes, there are arrangements where an agent will orchestrate other agents for you and all this kind of stuff. But the most common use, once you get past just talking into the chat, is you tell it what to do, and you verify it over and over again, and that's you. You're the orchestrator and the verifier.

[00:33:04] Stephen Koza (Ep 9)

And yeah, you just, you said the two key words in the same sentence.

[00:33:08] Tom Kershaw (Ep 9)

Focus is no. That's what focus means. It literally means the word no.

[00:33:12] Stephen Koza (Ep 9)

Yeah, yeah. You know, I love when there's a list of ten priorities. And, you know, it's easy to say, well, nothing's really a priority then.

[00:33:19] Tom Kershaw (Ep 9)

Yeah. I say which one is more important, and the business team says they're all important. And what are you going to...

[00:33:27] Stephen Koza (Ep 9)

Yeah, we were in a meeting the other day and there was a list of metrics underneath the heading North Star. And my CTO astutely pointed out that there's only one North Star in the universe.

[00:33:41] Stephen Koza

Nobody talked about the code, and it's intuitive, but that's kind of the part I didn't see coming. Push on any of these people, and the problem just moves somewhere else. Who owns the outcome? Did people even agree on the goal in the first place? Did the downstream team believe they could absorb what the upstream team just sped up? Well, Chris Robertson put a number on it. 40 to 50% of the work is just getting to agreement before you actually try and fix anything. When DORA ran the numbers, 5,000 engineers, they all landed in the same spot. AI doesn't actually fix a team, it amplifies what's already there. Ten conversations, same conclusion, and none of us were trying to get there. We wrote up the longer version of this on our website, checklist included. We'll put the link in the show notes. So thanks to all the guests so far. It's been super fun. We got a great lineup around the corner for season two. See you on the next one, and thanks for listening.

Become a Guest
ABOUT THE PODCAST
Honest Conversations. Hard-Won Lessons.

TechPod Talks is a podcast from EverOps featuring candid conversations with the leaders behind the platforms. Each episode dives into topics like leadership, AI, cost efficiency, and what it actually takes to build and scale in tech. No scripts. No fluff. Just real conversations with people who've been in the trenches.

Have a story worth sharing? We're always looking for tech leaders, founders, and operators with real-world experience to join the show. Tell us about yourself and we'll be in touch.