S12 E28: The AI Throughput Illusion: Why Splurging on Expensive Models Fails to Ship Code and How to Measure Real Engineering Output with Emilie Schario, Co-Founder & Head of Product & Engineering at Kilo Code

0:01 [SPEAKER_01]: One of the things that I think we all take for granted in this world of working with AI coding assistance is that there is always going to be some AI coding assistant available. [SPEAKER_01]: And so we've built our on-call process very much around the fact that you can leverage Kilo to do things for you. [SPEAKER_01]: We've had that process in place for a while. [SPEAKER_01]: It's been working really well. [SPEAKER_01]: And then a couple of weeks ago, we had an incident where Kilo was down, and so our whole on-call process is predicated on Kilo being up.

0:39 [SPEAKER_01]: My name's Emily Sherio, I'm co-founder and I lead

0:49 [SPEAKER_03]: a podcast bringing you interviews with tech visionaries. [SPEAKER_03]: Six, six months moonlighting goes. [SPEAKER_03]: It's the last and all of the backgrounds who share what it takes to change an industry. [SPEAKER_00]: I don't exactly know what to do. [SPEAKER_03]: It doesn't go as to get right. [SPEAKER_02]: who built the teams that have their bad company is its team's help each other, which is proud of our team. [SPEAKER_03]: Keeping scalability top of mind, all that infrastructure was up there.

1:13 [SPEAKER_03]: Yes, we've been fighting it as we grew up. [SPEAKER_03]: Total waste of time. [SPEAKER_03]: The stories you don't read in the headlines. [SPEAKER_02]: It's not an easy thing to achieve. [SPEAKER_03]: To get yourself a deficit of off, try to begin to ride the ups and downs of the start-up line. [SPEAKER_02]: Need to really want it. [SPEAKER_03]: Not just about technology. [SPEAKER_03]: All this and more on code story. [SPEAKER_03]: I'm your host, Noah Labpart. [SPEAKER_03]: And today, how Emily Sherio has built the best way to code with AI.

1:39 [SPEAKER_03]: Fully open source and open model, sans breaking the bank.

Guest introduction: Emilie Schario's background and career journey

1:49 [SPEAKER_03]: Emily Sherio grew up in New Jersey outside of Newark, and attended college in state. [SPEAKER_03]: Currently she lives in Columbus, Georgia, outside of Atlanta. [SPEAKER_03]: She mentioned she got into technology so she could easily follow her husband's career geographically, and has had much success in the industry. [SPEAKER_03]: That's how to text she's married with three boys, all five and under. [SPEAKER_03]: So there's a lot of wrestling in her household. [SPEAKER_03]: She admits she is often quoted saying she does three things in her life.

2:15 [SPEAKER_03]: Work, parenting, and if she's lucky, attends CrossFit three times a week. [SPEAKER_03]: In fact, she finds a great sense of community in that world and brings her kids with her to cheer her on.

2:27 [SPEAKER_03]: A year and a half ago, Emily's current venture was started to build the open source orchestrator or harness for AI coding agents. [SPEAKER_03]: Through some shuffle in the early team, Emily joined and started in building the fastest AI coding app on the market.

2:44 [SPEAKER_03]: This is the creation story of Keylob.

Overview of Kilo Code as an open source agentic engineering harness

2:51 [SPEAKER_01]: Kilo is the agentic engineering platform, so think of it like your favorite coding assistant. [SPEAKER_01]: People are probably familiar with Codex or Cloud Code. [SPEAKER_01]: Kilo is very similar. [SPEAKER_01]: It's the coding harness that you use to interact with AI models. [SPEAKER_01]: The thing that really differentiates Kilo is what we call model freedom.

Differentiating through model freedom and supporting 500 plus AI models

3:14 [SPEAKER_01]: So when you use a tool like Codex or Cloud Code, you're using a software that's tied to a particular lab. [SPEAKER_01]: So if you're using Cloud Code, you can only use anthropic models. [SPEAKER_01]: If you're using Codex, you are using OpenAI models. [SPEAKER_01]: When you're using Kilo, you have over 500 models available to you. [SPEAKER_01]: So you can use models from OpenAI or Anthropic, but you can also use GLM or DeepSeek or MiniMax or RC or Grock or any other models that are interesting to you.

3:46 [SPEAKER_01]: Think one of the things that we hear from our users over and over is that they want to be able to choose the right model for the task. [SPEAKER_01]: And Kilo is one of the few ways that they can do that.

Kilo Code founding story with Sid Sijbrandij and team history

3:58 [SPEAKER_01]: Healow was founded about a year and a half ago. [SPEAKER_01]: Sid's a brandy who was then, he was founder and for a long time, he was CEO at GitLab. [SPEAKER_01]: Now he's executive chairman. [SPEAKER_01]: He saw the opportunity to develop something in this space. [SPEAKER_01]: The project got started with it's then co-founder, J.P. P. P. P. P. P. P. P. P. P. P. P. P. [SPEAKER_01]: GP had to step away and then myself and Scott right in other came in to really assure the project forward.

4:26 [SPEAKER_01]: So about a year and a half whole old globally distributed team split across North America, South America, Europe and Africa and really just focus on building the best coding harness that we can.

Defining the evolving MVP for AI coding tools in a fast-moving market

4:42 [SPEAKER_03]: So tell me about what you would consider the MVP for Kilo then the first version of the product. [SPEAKER_03]: Perhaps you came into push the project forward. [SPEAKER_03]: Tell me about that version or maybe even the version before. [SPEAKER_03]: And what sort of tools were used to bring it to life and how long did it take to get it out? [SPEAKER_01]: Every day at Kilo is about figuring out what the MVP for coding harnesses because this is an area where the measuring stick is definitely moving.

5:12 [SPEAKER_01]: What people expect from their coding tools in this AI era is changing. [SPEAKER_01]: And I'll give you a really specific example is that maybe two months ago, someone had a tweet Goviral Pete Steinberger that found Deriv open claw had a tweet Goviral

The rapid shift from manual prompt engineering to autonomous loops

5:31 [SPEAKER_01]: If you're writing prompts, you're doing it wrong. [SPEAKER_01]: We're all about loops now. [SPEAKER_01]: And most tools did not have loops, I don't know, like a week after that. [SPEAKER_01]: And now suddenly everyone has to support loops. [SPEAKER_01]: And so it's constantly a moving target in this space that is truly [SPEAKER_01]: that is truly evolving. [SPEAKER_01]: The way I think about it, the first version is can you give a prompt and can that prompt lead to code changing? [SPEAKER_01]: And now we're constantly working to not just build the best, but to make sure we have the things that make it possible for people to use Kilo every single day and in-day out to do most of their work.

6:11 [SPEAKER_03]: I'm interested in a decision or trade off you had to make in the early days of joining and starting Kilo of how the product was going to be shaped.

Trade-offs and resource allocation: Deprecating the Kilo App Builder

6:19 [SPEAKER_03]: Right now here that every day is about those MVP decisions and I totally relate to that. [SPEAKER_03]: I'm curious about maybe a Kilo when you had to make that's become a pillar in how you've approached creating Kilo. [SPEAKER_01]: For a lot of the, let's say, second half of last year, journey, we were really focused on broadening our ICP, so not just serving developers really well, but how do we serve all knowledge workers really well? [SPEAKER_01]: And I don't think we've decided against that, but I do think we had to confront our limited resources against that goal.

6:55 [SPEAKER_01]: And realize that we only have so many engineers, we can only tackle so many projects if we're gonna do them well. [SPEAKER_01]: So an example of a trade-off we had to make was around our Kilo App Builder. [SPEAKER_01]: Kilo App Builder is something akin to lovable or bolt if you're familiar with those. [SPEAKER_01]: Really that prompt to hosted website or hosted app experience. [SPEAKER_01]: And kilo app builder was live. [SPEAKER_01]: People built absolutely incredible things with it.

7:24 [SPEAKER_01]: But at some point, we had to decide that we could not invest in app builder the way we wanted to. [SPEAKER_01]: So either we were going to continue to provide this subpar experience or we were going to proactively decide that we were not going to do more on app builder right now. [SPEAKER_01]: And so a couple of weeks ago, [SPEAKER_01]: Maybe a couple of months ago, time and AI is a very weird thing. [SPEAKER_01]: But we decided we were going to make it no longer available to users who were unfamiliar with it.

7:59 [SPEAKER_01]: So kind of hit it from some of the marketing pages and then we hit it in the app itself. [SPEAKER_01]: So you couldn't find it if you weren't already a user and that was our way of communicating that people who already have this as part of their workflow. [SPEAKER_01]: You're going to be able to continue to use it, but people who don't yet have it as part of their workflow. [SPEAKER_01]: We're not going to let you embed it into your workflow. [SPEAKER_01]: And now the next part of that is what do we do for these users who are using it?

8:28 [SPEAKER_01]: And I think we're making that decision now, so for the time this airs will have broader communication. [SPEAKER_01]: But I think it's likely that we work with them to sunset it, so we'll [SPEAKER_01]: Share information, communicate, help migrate these users over to something else. [SPEAKER_01]: Make sure everything is exportable, but when you're building a company, you only have so many resources, so much energy to go around. [SPEAKER_01]: You have to be able to take lots of bets, and I'm really proud of the bets that we've taken.

8:55 [SPEAKER_01]: I think broadly speaking, we've made the right bets, and they've paid off. [SPEAKER_01]: But sometimes you also have to decide when you're going to stop investing in something. [SPEAKER_01]: And that's this is an example of where we did that.

Modern AI product management: Why multi-year roadmaps no longer work

9:09 [SPEAKER_03]: Okay, let's move forward then, this will be in your bread and butter, I'm curious about how you've progressed and matured the product from even a year and a half ago, even not that long ago, and I'm curious about how you build your roadmap, how you go about deciding that, okay, this is the next most important thing to build or to address with Kilo. [SPEAKER_01]: It's a tricky balance, because traditional, I'm doing air quotes right now, but traditional product management is one where you have these long, what's a six month plan, what's a year long plan, what's a five year plan, and we don't really have any of that.

9:47 [SPEAKER_01]: When I'm thinking about what is the thing we're going to tackle, let's say next month. [SPEAKER_01]: Right? [SPEAKER_01]: I'm factoring in what are the things that our enterprise customers are asking for, what are the things that will unblock a deal that's in the pipeline, also what are we hearing from our users, and then what is our team feel? [SPEAKER_01]: And I think these are really good stakeholders to consider, because for when I think about the next phase in product management, especially in this AI,

Shifting PM responsibilities from tracking engineers to setting context

10:19 [SPEAKER_01]: It's not as much about project managing engineers. [SPEAKER_01]: That part is going away. [SPEAKER_01]: Engineers can do more than they've ever done before. [SPEAKER_01]: The funnel is different now, and engineering, software engineering, is no longer the bottleneck. [SPEAKER_01]: That means engineers' job is going to change, and that means product managers' job is also going to change. [SPEAKER_01]: So, what a product manager needs to do, or whoever's setting that product to vision, [SPEAKER_01]: Their number one priority is understanding, what are the features that will build a business?

10:53 [SPEAKER_01]: What are the things that people are going to pay us for? [SPEAKER_01]: So it's really important that we understand what our customers want and need, and what they need in order to buy and pay for a kilo. [SPEAKER_01]: On the other side, we're also an open source and an open core product. [SPEAKER_01]: Kilo Codebase itself is open source and then the Kilo Cloud Codebase is source available. [SPEAKER_01]: That means that anyone can contribute to it. [SPEAKER_01]: We are a developer-led community.

11:19 [SPEAKER_01]: We've got lots of folks in Discord every day who are active and sharing feedback and we want to make sure their voices are heard. [SPEAKER_01]: We are regularly considering their needs as well and a lot of the developer features that we build come both from feedback from the community and pull requests from the community. [SPEAKER_01]: And then we're also keeping an eye on what our competitors are doing. [SPEAKER_01]: What are the other tools in the space shipping? [SPEAKER_01]: Do we need to think about what our version of that feature might look like?

11:50 [SPEAKER_01]: Do we think it's a feature that's not going to take off and so we can sit tight and weigh a little bit? [SPEAKER_01]: There's a combination of things that go into it, but we're also very much deciding what our own bets are and leaning into where we think the right strategic bets for the org are.

12:08 [SPEAKER_03]: So here you say, we tell me about how you built your team and what you look for in those people to indicate that they are the winning horses to join you. [SPEAKER_01]: I will say, I'm so proud of the team we've built at Kilo. [SPEAKER_01]: We're about 35 people. [SPEAKER_01]: I mentioned earlier that we're globally distributed, just really, so talented and I'm really grateful for the team. [SPEAKER_01]: We've really high bar for hiring and first-hanging at Kilo, and I'm glad we do because it's [SPEAKER_01]: makes it such a joy to be here.

12:39 [SPEAKER_01]: There's not politics, there's not. [SPEAKER_01]: None of the things that create you down like the things that create you down in a company with B players. [SPEAKER_01]: When I think about the team, we're broadly split into two parts. [SPEAKER_01]: Products and engineering, which is led by me, and then go to market, which is led by my co-founder Scott. [SPEAKER_01]: Both sides of the org operate very similarly.

The AI throughput illusion: Why expensive models don't equal shipped code

13:01 [SPEAKER_01]: We all have a daily stand-up for setting ambitious goals for tomorrow. [SPEAKER_01]: We've got one product manager in the Kilo Core side of the business who works across many an accordion for structure, [SPEAKER_01]: Court of Kilo, platform-like responsibilities, but all of our engineers are what I would call product engineers to really focus on making sure that [SPEAKER_01]: Everyone has autonomy to make decisions and go quickly. [SPEAKER_01]: And that works really well in our space because the number one power users of Kilo are the people on the team who are using Kilo to build Kilo every day.

13:43 [SPEAKER_01]: On the other side of the house, we've got the go-to-market team where we're structured as we've got one salesperson and then one customer experience lead who lead sales and support. [SPEAKER_01]: We've also got a really talented bunch of devrails who are just truly absolutely incredible firing on all cylinders all the time, staying on top of model launches and partnerships with the labs, just incredible systems, and a really strong partnership's lead and Ari, who is regularly understanding what's coming, what models are coming, what do we need to do to best support the different model partners that we're working with.

14:26 [SPEAKER_01]: We're a small but mighty team, and I, one of my favorite pieces of feedback that I get is when people say something along the lines of, oh, I just assumed Kilo was a 150 person org, and then I get to tell them we're just a small fraction of that.

14:45 [SPEAKER_03]: Let's move into scalability, and this will be interesting to hear where it comes about, or where it was considered kind of an early day. [SPEAKER_03]: How did you approach it in the early days? [SPEAKER_03]: I think what's most interesting to me is that there have been interesting areas where you've had to fight technological scale or team business scale, all the above, as you've thrown. [SPEAKER_01]: One of the really nice things about working at Kilo is the seniority on our team.

15:12 [SPEAKER_01]: So the average engineer at Kilo has 15 years of professional experience, but the least experienced engineer at Kilo has 10 years of experience. [SPEAKER_01]: Which means that everyone on the team is just [SPEAKER_01]: incredibly senior. [SPEAKER_01]: They've all seen it before. [SPEAKER_01]: So a lot of the problems around scale or data structures that companies have as they grow. [SPEAKER_01]: We're lucky enough to have folks who kind of saw some of these things coming, which is really great.

15:47 [SPEAKER_01]: And nothing in particular comes to mind. [SPEAKER_01]: But I know that they're [SPEAKER_01]: Even with this very senior team, there's going to be, there's going to be goals and goblins just around the corner.

16:02 [SPEAKER_03]: So as you step out on the balcony, you look across all that you've built thus far with Kilo. [SPEAKER_03]: What do you most proud of? [SPEAKER_01]: there are lots of things I'm proud of. [SPEAKER_01]: I'm proud of the product. [SPEAKER_01]: I'm proud of the users who send us really incredible messages about how much, how much the product makes a difference to their lives and to the work they're doing. [SPEAKER_01]: But truly, the thing that I am most proud of,

16:29 [SPEAKER_01]: and that's selfish because I've hired a lot of them but this is this team is a joy to work with. [SPEAKER_01]: I enjoy going to work every day. [SPEAKER_01]: I think the thoughts and the insights we have and being able to have Woody Banner with folks around what kind of vacation is best. [SPEAKER_01]: Is it an active one or a beach one? [SPEAKER_01]: These are the things that I think make a real big difference when you get to go to work every day. [SPEAKER_01]: When I talk to people, especially in the closing process of making someone unoffer, I always say compensation is, but your cash, it's your bonus, it's your equity.

Measuring true engineering output and productivity in the AI era

17:06 [SPEAKER_01]: It's also how much do you want to come to work every day. [SPEAKER_01]: And the thing that we've really nailed at Kilo is that these are the people that you want to be working with. [SPEAKER_01]: And as much as I'm proud of what we're building, I'm super proud of this team. [SPEAKER_03]: Okay, let's flip the script a little bit. [SPEAKER_03]: Tell me about a mistake you made and how you and your team responded to it.

17:32 [SPEAKER_01]: One of the things that I think we all take for granted in this world of working with AI coding assistance is that there is always going to be some AI coding assistant available. [SPEAKER_01]: And so we've built our on-call process very much around the fact that you can leverage kilo to do things for you. [SPEAKER_01]: have Kilo look at logs or depending on the kind of incident you can have Kilo spin up subagents to go identify some patterns or help identify what the root cause of a problem might be.

18:06 [SPEAKER_01]: We've had that process in place for a while, it's been working really well. [SPEAKER_01]: And then a couple of weeks ago, we had an incident where Kilo was down. [SPEAKER_01]: And so our whole on-call process is predicated on Kilo being up. [SPEAKER_01]: And so ultimately it was fine, we were able to figure it out. [SPEAKER_01]: We did the same things we always do. [SPEAKER_01]: Get people into a war room, hop on a call. [SPEAKER_01]: There's an incident commander called the incident.

18:35 [SPEAKER_01]: But we did realize we had to reflect on the actual incident process afterwards and say, how do we want to handle it in a world where Kilo itself is down? [SPEAKER_03]: And there's not a coding agent available. [SPEAKER_01]: It's a little bit like, how do you handle it if Slack goes down? [SPEAKER_01]: Like, you take some of this infrastructure. [SPEAKER_01]: That's not like your product infrastructure, but you're tooling for getting things done. [SPEAKER_01]: You take it for granted.

19:06 [SPEAKER_03]: Okay, Emily, let's move forward then. [SPEAKER_03]: This will be really exciting. [SPEAKER_03]: What does the future look like? [SPEAKER_03]: Or kilo for what you're building for where the industry is going and moving so fast for all the things, what does the future look like? [SPEAKER_01]: My crystal ball over here is a little bit cloudy, so I don't have a ton of great answers for you. [SPEAKER_01]: But what I can say is that I think we're seeing people experiment with models more frequently and that the future is definitely multi-modal.

19:41 [SPEAKER_01]: The idea that you're going to only use one model for all your tasks doesn't make any sense to me. [SPEAKER_01]: In a world where we've seen specialization be effective across people, why wouldn't we also expect our models to specialize? [SPEAKER_01]: So the future is definitively multi-modal, but what does being multi-modal look like and how companies adjust to it? [SPEAKER_01]: I'm still trying to un... add some windex to my crystal ball, but I think that Kilo will be a part of that story.

20:17 [SPEAKER_03]: Let's switch to you. [SPEAKER_03]: Who influences the way that you work? [SPEAKER_03]: Name a person or many persons or something. [SPEAKER_03]: You look up to him. [SPEAKER_03]: Why? [SPEAKER_01]: I'm really lucky to have incredible co-founders in both Scott, Brighton, and other N-Sid, Subrandy. [SPEAKER_01]: I worked with Sid at GitLab many years ago and many of the kind of core working principles of mine came from watching him work. [SPEAKER_01]: Formulating my own opinions on what worked and what didn't, and so just truly grateful to have learned and worked with him and gotten the chance to work with him again.

20:55 [SPEAKER_01]: And then Scott very similarly, Scott and I had known each other for a long time when the kilo opportunity came up by tapped him on the shoulder and said, the text message I sent him was like,

Building resilient engineering cultures around AI coding platforms

21:08 [SPEAKER_01]: He can I pitch you on a crazy idea? [SPEAKER_01]: He was like, yeah, I want to hop on a call in a little bit And I'm so glad we did and it's one of those serendipitous things where right place right time right opportunity And I really feel like we've gotten the chance to make magic over the last year [SPEAKER_01]: There's still a lot that I have to learn for sure, but I think I have the right people working with me all the time. [SPEAKER_01]: I think there's lots of other people I really enjoy the writing of Laura Vandercam as I think about how I manage my time.

21:44 [SPEAKER_01]: I was a number of writing from Barbara Palmer that I really appreciate as I think about motherhood and working motherhood, especially. [SPEAKER_01]: So there's lots of people, but the two that I think drive the biggest difference day in and day out are definitely my co-founders, because they're probably who I spend the most time talking to, who I'm not also married to. [SPEAKER_03]: Okay, last question Emily, so you're getting on a plane and you're sitting next to a young entrepreneur who's built the next big thing.

22:11 [SPEAKER_03]: They're jazzed about it that can't be showed off to the world and can't be showed off to you right there on the plane. [SPEAKER_03]: What advice do you give that person having gone down this road a bit several times? [SPEAKER_01]: My advice every time is a version of don't do it. [SPEAKER_01]: Don't do it. [SPEAKER_01]: You don't want to do this. [SPEAKER_01]: It's way harder than you think because if they're crazy enough to do it anyway, then they should probably do it. [SPEAKER_01]: I think starting a company is there's a lot out there that makes it look really quote unquote sexy and attractive.

22:43 [SPEAKER_01]: Like it's a credential you collect along the way. [SPEAKER_01]: But it is stressful and heartbreaking and hard in a way most people will never understand. [SPEAKER_01]: Lots of company executives out there, but they do not carry the stress of a CEO worried around about money running out or how you're going to run payroll that week or any number of things. [SPEAKER_01]: And so I'm always going to advise people don't do it. [SPEAKER_01]: and the crazy ones will do it anyway. [SPEAKER_01]: And if they're not crazy enough to do it anyway, based on some advice a stranger on a plane gave them, they're probably not crazy enough to do it.

23:23 [SPEAKER_03]: I really like that, couldn't agree more. [SPEAKER_03]: Limley, thank you for being on the show today, and thank you for telling the creation story of Kelo.

Closing thoughts and where to learn more about Kilo Code

23:30 [SPEAKER_01]: Thanks for having me. [SPEAKER_03]: And this concludes another chapter of Code Story.

23:43 [SPEAKER_03]: code story is hosted and produced by Noah Labhart. [SPEAKER_03]: Be sure to subscribe on Apple Podcasts, Spotify or the podcasting app at your choice. [SPEAKER_03]: And when you get a chance, leave us a review, both things help us out tremendously.

23:59 [SPEAKER_03]: And thanks again for listening.