S13 Bonus: Why Local On-Device AI Compute is the Future of Privacy with Behnam Bastani, Co-Founder & CEO of OpenInfer

0:00 [SPEAKER_01]: And everyone was using AI as a time as chatbot, messages, and we said no, AI is going to be long lasting living with us. [SPEAKER_01]: That means some pieces need to respond fast, some pieces respond slow, they become heavily dependent on each other which today we refer to them as agents. [SPEAKER_01]: So we stepped out with that mission in mind that compute is going to be heavily consumed by inference. [SPEAKER_01]: And inference behavior is going to be completely different than what it was in 24.

0:33 [SPEAKER_01]: The infrastructure, the software ecosystem, will not be able to let us scale. [SPEAKER_01]: On Benham, Bastani, CEO and co-founder of OpenInfer.

0:47 [SPEAKER_02]: This is Code Story. [SPEAKER_02]: a podcast bringing you interviews with tech visionaries. [SPEAKER_02]: Six, six months moonlighting goes. [SPEAKER_02]: It's the last and all of the backgrounds who share what it takes to change an industry. [SPEAKER_00]: I don't exactly know what to do. [SPEAKER_02]: It doesn't go as to get right. [SPEAKER_03]: who built the teams that have their bad company is its team's help each other, which is proud of our team. [SPEAKER_02]: Keeping scalability top of mind, all that infrastructure was up there.

1:13 [SPEAKER_02]: Yes, we've been fighting it as we grew up. [SPEAKER_02]: Total waste of time. [SPEAKER_02]: The stories you don't read in the headlines. [SPEAKER_03]: It's not an easy thing to achieve. [SPEAKER_02]: To get yourself, we've tested it off and tried it again. [SPEAKER_02]: To ride the ups and downs of the start-up line. [SPEAKER_02]: Need to really want it, not just about technology. [SPEAKER_02]: All this and more, on code story. [SPEAKER_02]: I'm your host, Noah Labpart. [SPEAKER_02]: And today, help Benham Bustani, has built the inference OS for the agentic era to close the gap between AI's promise and what in-fruf can deliver.

1:49 [SPEAKER_02]: Benham Bustani is originally from Tehran, Iran. [SPEAKER_02]: He left the country the age of 17 setting out for Canada where he got his master's and PhD before moving to California. [SPEAKER_02]: He's always looked at things with the big picture in mind, whether that is artwork, woodwork, engineering, et cetera. [SPEAKER_02]: He's worked for Hewlett Packard, Mehta, and Roblox before starting his current venture. [SPEAKER_02]: But outside of technology, he loves to exercise and participate in water activities.

2:15 [SPEAKER_02]: For peace of mind, he gardens, experimenting with growing trees, replanting, and seeding.

2:23 [SPEAKER_02]: Previously, Benham went to meta to pursue the future of wearables. [SPEAKER_02]: He immediately realized that these wearables required a connected system. [SPEAKER_02]: He started to ask the question, why can't we use computer round us to fuel wearables? [SPEAKER_02]: Eventually, through stance at meta and roblox, he and his co-founder knew that they wanted this type of system and started with the infrastructure to enable it.

2:48 [SPEAKER_02]: This is the creation story of OpenInfer.

2:54 [SPEAKER_01]: I mentioned system is the thinking that I had system building, looking at the entire stack as a system. [SPEAKER_01]: It was at the meta, Facebook, where I was hired, I left Google to meta to think about future of wearables, virtual reality augmented reality. [SPEAKER_01]: It was in that moment that it became very clear to me how the systems were built, virtual reality and augmented reality, was peace, meal of pieces together, a software, a lens, an optics system, computing, and we were getting limited, limited experience.

3:34 [SPEAKER_01]: We were getting either virtual reality headset that had to get connected to a PC,

3:45 [SPEAKER_01]: Why can't we have a great experience on the head at super low cost? [SPEAKER_01]: Why can't we use ecosystem around us, compute around us? [SPEAKER_01]: Maybe it's a PC, maybe it's cloud, why can't we use that for VR? [SPEAKER_01]: and everyone said because of latency and it caused people to say, so on and so forth. [SPEAKER_01]: That was the time that means I said, you know what? [SPEAKER_01]: If we look at the entire system, how compression, how streaming, how rendering, how AI works together, we can come up with a solution that perfectly brings best experience of VR into mobile headset.

4:24 [SPEAKER_01]: That's where I actually met my co-founder, the Razer, Razer and I built Oculus Link and I NK, which really changed the way virtual reality was being done not just at meta across the industry. [SPEAKER_01]: Link was a platform compute platform where we broke down pieces of compute from rendering to graphics to audio to haptics in a way that heavy portion can be offloaded and we could use AI to mitigate latency. [SPEAKER_01]: So it became a pipeline of pieces that some happens on your head, some happens on a PC,

5:10 [SPEAKER_01]: That changed the game, that let us introduce the first head-sets that was the cheapest in the world, best experience, and we shut down competition by far. [SPEAKER_01]: Zuckerberg announced in 2019, I went on stage with my co-founder, we talked about it, and then we scaled it to millions, and then we three took on a bigger challenge, leaving meta together to Roblox, the gaming company, and we repeated the same game there. [SPEAKER_01]: In that time, it was bringing voice moderation, all the kids talking multilingual, away from AWS and video GPUs to run on Roblox's infrastructure, which was only CPU based.

5:53 [SPEAKER_01]: No one thought that you can do all AI inference on CPU this is 2023 and doing it at such a hard large scale. [SPEAKER_01]: And that was possible because we look at the entire stack again as a system. [SPEAKER_01]: moderation is not just model, it's not just inference, it's not deployment, it's everything. [SPEAKER_01]: And when we do you do everything together, you look at the system, then you think you make things that are completely thought being impossible in the past. [SPEAKER_01]: So we saw the opportunity, we saw the world is moving heavily into AI inference, and usage of AI.

6:31 [SPEAKER_01]: You said, the world's biggest compute need is going to be around inference usage of AI. [SPEAKER_01]: And that was in 2024 where everyone was saying modeling training is the big topic. [SPEAKER_01]: You said, no, usage is becoming the topic. [SPEAKER_01]: And everyone was using AI as a time as chatbot, messages, and said no, AI is going to be long lasting living with us. [SPEAKER_01]: That means some pieces need to respond fast, some pieces respond slow, they become heavily dependent on each other which today we refer to them as agent.

7:06 [SPEAKER_01]: So we stepped out with that mission in mind that compute is going to be heavily consumed by inference and inference behavior is going to be completely different than what it was in 24. [SPEAKER_01]: The infrastructure, the software ecosystem, that is built around it, will not be able to let us scale. [SPEAKER_01]: We need to do something fundamentally different. [SPEAKER_01]: We need to look at the entire system in terms of how the hardware, how low-level kernels, how the management of resources, how the processing of AI to even how routing between different servers are done, everything built with that mission in mind of long-lasting collaborative AI inferences, aka agent, the AI.

7:52 [SPEAKER_01]: So we stepped out in 24, 2024, [SPEAKER_01]: We started the company and my co-founder and I now, we're in this journey, we started the opening fair with the mission in mind to bring AI and friends anywhere, any hardware, any infrastructure by making AI ubiquitous.

8:13 [SPEAKER_02]: Let's dive into what you would consider the MVP then so that first version when you stepped out how long it takes you build and what sort of tools are you using to bring it to life? [SPEAKER_01]: MVP is the hardest, and especially for startups, particularly these days, that the market also changes very quickly. [SPEAKER_01]: We went after portable ecosystem devices, where inference was very underserved. [SPEAKER_01]: Soon we realized the market pulled us into the cloud Neo Clouds, which is the new topic, actually.

8:45 [SPEAKER_01]: These are all smaller cloud providers, computer providers. [SPEAKER_01]: where they wanted to serve AI, but they don't have the infrastructure and capability to bring up inference. [SPEAKER_01]: They can only rent out GPUs. [SPEAKER_01]: We started with devices, but then soon we got pulled into this direction. [SPEAKER_01]: We had to adjust. [SPEAKER_01]: New cloud became a topic early 2026 and demand was very big. [SPEAKER_01]: We had to repivate ourselves to make sure we can hit that demand.

9:16 [SPEAKER_01]: Agility was critical. [SPEAKER_01]: And we built today our MVP serving Neo Clouds, bringing inference AI to their infrastructure, so they can start selling tokens, they don't just rent out GPUs, they can rent out GPUs with a software stack on top of it that does inference, typically their profit margin goes up by as much as 50%.

9:42 [SPEAKER_01]: Huge value prop for them, they can now sell a lot more, higher value prop under pure hardware and a huge gain for their clients because now they don't have to bring software pieces from here and there, they get one stack everything runs on them. [SPEAKER_01]: It reminds me of VMware, where they say, you know what, I don't care about your hardware underneath it, I'm gonna visualize the work for you. [SPEAKER_01]: We are now virtualizing inference for new clouds. [SPEAKER_01]: Don't care if it's a type of GPU you have, you have older newer ones, you have a mixture of CPUs or you're actually seeing a lot newer compute coming in, we virtualize it.

10:23 [SPEAKER_01]: Meaning, be abstracted out, you bring your agent, you bring your workload, it works on that new cloud. [SPEAKER_01]: paying reduction for the user, paying the reduction for Neocloud, the ones that are offering in friends, and us, they're taking the portion of that to win. [SPEAKER_02]: I want to stay on that for a minute. [SPEAKER_02]: I'm curious about a decision or trade-off you had to make in building that first MVP in. [SPEAKER_02]: Get probably cherry pick a couple of things from what you said, but curious about one that really shaped the product and really shaped your approach and maybe it's like acceptance of debt or how you go to marketing like that.

10:57 [SPEAKER_02]: Tell me about one of those you had to make and how you coped with it. [SPEAKER_01]: I've been on the other side where startups were pitching to me all the time and I'm like, you're solving their own problem. [SPEAKER_01]: I'm not going to buy it. [SPEAKER_01]: So, rather than us jumping into building, we spend quite a bit of time to really understanding the problem and [SPEAKER_01]: And we made sure we have major design partners through that journey. [SPEAKER_01]: These design partners are clients, our customers, that are willing to start early on, call prototype with us, and they're big enough that they can shape the industry.

11:39 [SPEAKER_01]: So obviously it's really hard early on to find these these partners are willing to take a risk to work with us start up. [SPEAKER_01]: And it requires a lot of work, a lot of prep, a lot of prototyping together. [SPEAKER_01]: But that was the biggest item. [SPEAKER_01]: We are lucky now that we have five major Fortune 500s that work with us with the past five years and they're sticking along and we're going to major launchers together. [SPEAKER_01]: They shaped us. [SPEAKER_01]: they guided us.

12:06 [SPEAKER_01]: Of course, we're not taking every single direction, but it's informing what the industry is actually looking. [SPEAKER_01]: That was the biggest, I think, biggest lesson I took as a win to make sure the company stays in the right direction and we're serving the problem that's worth significant dollar. [SPEAKER_02]: So then, you're year and a half in, let's say after that MVP point. [SPEAKER_02]: How have you progressed and matured the product or how are you progressing and maturing it?

12:39 [SPEAKER_02]: And I think to wrap that in a box a little bit, what I'm looking for is, how do you build your roadmap? [SPEAKER_02]: How do you go about deciding that, okay, this is the next most important thing, the builder to address with open infrastructure. [SPEAKER_01]: I again use our customers. [SPEAKER_01]: Once you pick these clients, when they have massive market share, continuously learn about what's the next problem, what's the next problem, what's the next problem? [SPEAKER_01]: Today, as an example, Neoclouds, huge, and warrior design partners, those that make servers, those that make silicons, those that make AI models, [SPEAKER_01]: Those are your design partners we go in.

13:18 [SPEAKER_01]: And those design partners are saying new clouds how we use them. [SPEAKER_01]: But now a trend of sovereign clouder coming on, the idea of AI where the data lives. [SPEAKER_01]: So now you start thinking about feature of AI, maybe a portion lives in the cloud, maybe a portion lives on the neocloud, maybe a portion lives on your own infrastructure, maybe a portion of that AI runs on your laptop. [SPEAKER_01]: how would now that ecosystem work? [SPEAKER_01]: So that's basically shaping a roadmap and we're seeing the roadmap as actually going in that direction.

13:56 [SPEAKER_01]: The direction that AI needs to live where the data is. [SPEAKER_01]: Banks, robotics, drones, they're ingesting generating tons of data. [SPEAKER_01]: Sometimes they need the responses fast, sometimes they can shove the data to the cloud, it costs huge, dollar a month, sometimes it means loss of sovereignty, how do you make it happen? [SPEAKER_01]: It's good to say you bring AI where the data is, but then how do you bring the compute? [SPEAKER_01]: When you may not want to bring the whole compute, how do you split the compute?

14:29 [SPEAKER_01]: How do you make sure the compute's work together? [SPEAKER_01]: So this is the beginning of the MVP, solving a hard problem, generating revenue, but I look at it as, I guess Tesla is a good example, right? [SPEAKER_01]: They built the Roadster, Brotster wasn't to generate money, it was such a niche market. [SPEAKER_01]: But taught them a lot, then they built Model S. Model S was about scaling. [SPEAKER_01]: Can you actually scale now that Roadster 2 mass adoption? [SPEAKER_01]: And after that it was cost reduction.

15:01 [SPEAKER_01]: Brot Model 3. [SPEAKER_01]: Not even it teaches you about the market, but also it brings you revenue and lets you fund bigger and bigger ambitions. [SPEAKER_01]: Same thing with it for us. [SPEAKER_01]: We're building our own roadster in new clouds. [SPEAKER_01]: Very niche market. [SPEAKER_01]: It's going to form our own scale operation and then after that the cost reduction.

15:25 [SPEAKER_02]: I'm curious about team, this is a challenging problem you're solving, and I'm curious about how you win about bringing your team on what you look for in those people to indicate that they were the winning horses to join you. [SPEAKER_01]: Being in the valley for 15-20 years, working in industries for close to 30 years, I learned a lesson, sometimes in the hard way that people is the biggest asset of the company. [SPEAKER_01]: Talent, harmony of that talent is the biggest one. [SPEAKER_01]: Once you bring that right team, you structure it, innovation, progress, comes in, inefficiency goes away.

16:05 [SPEAKER_01]: So we put a lot of effort to hand-pick and select, filter out, [SPEAKER_01]: and sometimes if you had to say goodbye to partners, so the team is structured well. [SPEAKER_01]: A lot of people say, hey, how big is your team? [SPEAKER_01]: It's just like 50-100, no, these days with AI, with a genetic with code gen and all those tools we have around us. [SPEAKER_01]: It's not about a number of people we have. [SPEAKER_01]: It's about the design, it's about structure, it's about harmony and selecting the people building the team is the most critical one.

16:43 [SPEAKER_01]: So we made a hard choice. [SPEAKER_01]: Everyone has to be local, we have to meet in person all the time, that's where creativity comes in. [SPEAKER_01]: Of course there's a huge cost that goes with it, being in the Bay Area where I could have hired in other areas,

17:03 [SPEAKER_01]: when it makes sense for specific areas of the operation and then making sure through that journey the culture we put a lot of thoughts in terms of culture fit collaboration and even when we were super small we always put a trial period does work or not [SPEAKER_01]: Let's both sides need to figure out if this is the right fit. [SPEAKER_01]: And you know what? [SPEAKER_01]: I had situations that I really wanted that candidate. [SPEAKER_01]: Candidate came in and you're like, this is not a fit for me.

17:35 [SPEAKER_01]: That's better than if they came in and they're then they walk out. [SPEAKER_00]: So, all right. [SPEAKER_01]: That's okay. [SPEAKER_01]: It's not selling and getting the people, it's getting the right people in the right group. [SPEAKER_01]: So we're small team, we're under 20, we thrive, we're the top in the world we believe that knows how to work together, make things happen that no one thought it's possible in the past. [SPEAKER_01]: And quite a few of us are, we work together in the past.

18:05 [SPEAKER_01]: Yeah, most of us, good portion of us are X meta folks that we went to different routes and now coming back because we built things together a couple of times and we like repeating that.

18:20 [SPEAKER_02]: Let's move into scalability, and this is baked into your bread and butter here, but I'm curious about how scale was approached from the early days, and also if there have been interesting areas where you've had to fight scale as you've grown. [SPEAKER_01]: They're two angles that we severely think about. [SPEAKER_01]: One is the direction that we need to scale and be how we approach the scale. [SPEAKER_01]: As an example, when does it make sense to bring scale, build up, or go to markets, sales operation, marketing?

18:53 [SPEAKER_01]: And then how do you see that scale? [SPEAKER_01]: Do you bring head of sales or do we bring a junior person? [SPEAKER_01]: Do you bring them full-time or you contract them out? [SPEAKER_01]: Do you bring sales or you bring revenue person? [SPEAKER_01]: You bring product person or you bring XYZ. [SPEAKER_01]: And I'm not saying I've done the best and I got the best recipe. [SPEAKER_01]: But it's when things are a bit dark and you're trying to scale and you're trying to figure out directions you want to scale and how you scale it there is always an art to evaluate directions especially there is a quite a bit of ambiguity.

19:28 [SPEAKER_01]: the evaluation becomes more clear when you look at how others are approaching it, what have they learned, what they but things haven't gone well, but also coming up with creative ways to give it a try, give it a shot and not be afraid to lose and repeat in a different directions. [SPEAKER_01]: Yes, scale is a big one. [SPEAKER_01]: If you scale incorrectly, it can make you go bankrupt, go crazy, not be productive at all. [SPEAKER_01]: And it's an art and it's an art that we should be okay to back out and adjust and repeat again.

20:05 [SPEAKER_02]: And that's a approach I've been taking.

20:09 [SPEAKER_02]: As you step out on the balcony, you'll across all that you've built as far with open-in for what you most proud of. [SPEAKER_02]: Three things. [SPEAKER_01]: I said the team, the architecture, if that's the system we built, and see the vision that they picked. [SPEAKER_01]: The vision, what we're truly unlocking, like two years ago, no one believed in us, and now it's like the hottest thing. [SPEAKER_01]: The team, like, this is the top end of a role that has done this multiple times, things that people thought it's impossible.

20:43 [SPEAKER_01]: And architecture, system designs, the modular, so creative that it can literally today can run on big cloud environment, and we have examples in our research lab, it can squeeze and run on a tiny Raspberry Pi, single core CPU. [SPEAKER_01]: how the scale can adjust itself, how can it now become swarm of operations? [SPEAKER_01]: So I'm very proud of how architecturally we thought about it, we designed it, but also we took one sliver through that design and we make sure MVP gets launched within time and generates revenue.

21:26 [SPEAKER_01]: So those are the three steps I'm proud of. [SPEAKER_02]: Let's flip the script a little bit, tell me about a mistake you made and how you and your team responded to it.

21:37 [SPEAKER_01]: We started the company and we wanted to go after a market to be heavily differentiated. [SPEAKER_01]: When you become a, when you want to be heavily differentiated, it's you typically go to a market that is fairly new. [SPEAKER_01]: But maybe that wasn't this smart choice. [SPEAKER_01]: We went after it very new market. [SPEAKER_01]: Collaborative robotics, physical AI, collaborative physical AI. [SPEAKER_01]: And we said future is about that. [SPEAKER_01]: The challenges we're seeing and maybe now it's obvious.

22:06 [SPEAKER_01]: In 2024, we were expecting by 2026, the robotics is generating revenues, right? [SPEAKER_01]: They're selling. [SPEAKER_01]: Reanity is, I think there's going to be another 3, 4, 5, maybe, something years, that you actually see people are selling robots, it's getting more and more messed, messed, messed, adopted. [SPEAKER_01]: We could have gone at much simpler market, markets that are already there, and just go serve there, and then when others come in, we go after those. [SPEAKER_01]: So maybe that was a mistake, not always being differentiated or in a market is the right thing.

22:40 [SPEAKER_01]: It's just solving the problem and if others are making money, it means it generates money, so let's just go after that. [SPEAKER_01]: But the good thing, the success is we are not afraid of adjusting, we are not afraid of learning and quickly adjusting. [SPEAKER_01]: And we took the learnings from that because it helps us to really architect the problem with the vision in mind that we start from a new cloud, but inference is done around everywhere, AI is done around everywhere. [SPEAKER_01]: So, it was a mistake.

23:11 [SPEAKER_01]: It took it as an learning opportunity, and it took pieces to really position us better in the next trial.

23:24 [SPEAKER_02]: Okay, let's move into the future then. [SPEAKER_02]: Tell me what the future looks like for open and for for where the industry is going for all the things that are happening for what you're building. [SPEAKER_02]: Tell me what the future looks like. [SPEAKER_01]: There's two angles. [SPEAKER_01]: One running AI is a portion of that. [SPEAKER_01]: AI should be solving a problem. [SPEAKER_01]: A problem means there needs to be a application. [SPEAKER_01]: There needs to be a place to run it.

23:49 [SPEAKER_01]: There needs to be type of data progresses. [SPEAKER_01]: So we need to think more and more vertically. [SPEAKER_01]: how we get closer and closer to the end customers. [SPEAKER_01]: And then, technologically, think about how AI inference is changing. [SPEAKER_01]: It's started from hyperscalers, we're seeing now in Neil Clouds. [SPEAKER_01]: You're going to see more and more maturity that these things become more affordable to run in your own infrastructure. [SPEAKER_01]: So there is heterogeneity of the hardware of the compute of the stacks that you're going to see.

24:31 [SPEAKER_01]: And you're going to see more and more players that are solving hard problems, but problems that have to be solved with the best experience, where these agents to live with you, buy you, and more importantly, grow with you, learn from you. [SPEAKER_01]: And these two dimensions are, we're going to walk steps between the two axes. [SPEAKER_01]: In the next couple of years to make sure every, we are taking a set of stairs and we have products and each step. [SPEAKER_02]: What's which to you, Benham, who influences the way that you work?

25:07 [SPEAKER_02]: Am a person or many persons or something, you look up to him why. [SPEAKER_01]: I listen to a lot of successful leaders quite often, but also spiritually I listen to a lot of folks that have gone through spiritual path, whether it's meditation, whether it's calmness, leave it or not, both have huge impact in shaping and leadership style. [SPEAKER_01]: But also innovation, I constantly listen to podcasts, I constantly listen to YouTube's interviews by a large, by successful leaders in industries, past and new, and the ones that are live now, and also spatially how people have shared their experience and things have gone well.

25:52 [SPEAKER_01]: And those two, I think, become a good mix of directions that I can absorb, and I can aid their aid on. [SPEAKER_01]: But also, it keeps me focused and calm through that journey.

26:07 [SPEAKER_02]: OK, Ben, I'm last question. [SPEAKER_02]: So you're getting on a plane. [SPEAKER_02]: And you're sitting next to a young entrepreneur who's built the next big thing. [SPEAKER_02]: They're jazzed about it. [SPEAKER_02]: I can't wait to show it off to the world. [SPEAKER_02]: And can we show it off to you right down the plane? [SPEAKER_02]: What advice do you give that person? [SPEAKER_02]: Having gone down this road a bit? [SPEAKER_01]: Can you succinctly tell me the problem? [SPEAKER_01]: And then tell me why you are you ready for this journey?

26:28 [SPEAKER_01]: Do you know what it takes? [SPEAKER_01]: Knowing the problem, knowing why you? [SPEAKER_01]: And then after that, I would see the excitement in their eyes. [SPEAKER_01]: Do you know how you want to approach this? [SPEAKER_01]: But the first two are the most critical ones. [SPEAKER_01]: What's the problem? [SPEAKER_01]: Do you really think there is a problem? [SPEAKER_01]: Can you see the future of the problem? [SPEAKER_01]: And are you off for it? [SPEAKER_01]: Why you? [SPEAKER_01]: You should be able to answer that, right, and then they should have at least a story and I love it if the story is short One minute, 30 seconds How they're gonna how they're gonna attack all this so like elevator pitch if you will that's the most critical one if you're not able to succinctly say that that means you haven't understood the problem [SPEAKER_02]: I think all that is fantastic.

27:12 [SPEAKER_02]: It's fantastic way to approach it and fantastic advice. [SPEAKER_02]: Then I'm thank you for being on the show today and thank you for telling the creation story of OpenInfer. [SPEAKER_02]: Thank you.

27:23 [SPEAKER_02]: And this concludes another chapter of Code Story.

27:33 [SPEAKER_02]: code story is hosted and produced by Noah Labhart. [SPEAKER_02]: Be sure to subscribe on Apple podcast, Spotify, or the podcasting app at your choice. [SPEAKER_02]: And when you get a chance, leave us a review. [SPEAKER_02]: Both things help us out tremendously.

27:49 [SPEAKER_02]: And thanks again for listening.