S13 Bonus: Building Generative AI Tools & Multiplayer Engines for Game Developers with Anupam Singh, VP of Eng at Roblox

0:01 [SPEAKER_01]: Hello listeners. [SPEAKER_01]: Today we have a special guest on the podcast, a NuPum scene, VP of Engineering, AI platform and infrastructure at Roblox. [SPEAKER_01]: As you all may know, Roblox is a gaming platform, with millions of different games within it, allowing players to immerse themselves in gameplay, watching interactive clips, chat with their friends, and even build worlds of their own. [SPEAKER_01]: In today's episode, a new poem is going to illuminate his career journey, which has landed him at Roblox, along with the incredible tech he and his team are building in Foundation AI and Infrastructure.

0:36 [SPEAKER_01]: Open up them. [SPEAKER_01]: Thank you for being on the show today. [SPEAKER_01]: Thanks for being on code story. [SPEAKER_01]: I'm going to see you in a lot. [SPEAKER_01]: Absolutely. [SPEAKER_01]: Excited to dive into all of our topics today, which includes your journey to Roblox, a lot around foundation AI and infrastructure, the group that you're leading at Roblox, building games, and then some reflections on your career and all the things that you've done. [SPEAKER_01]: So let's dive right into it with your journey to Roblox.

1:01 [SPEAKER_01]: So [SPEAKER_01]: your career spans, Oracle, Sybase, Infirmax, some startups, as a co-founder, and then cloud dara. [SPEAKER_01]: And now Roblox, tell me about the throughline. [SPEAKER_01]: Tell me about what connects all of these companies from your experience and what that throughline is. [SPEAKER_00]: I've always been attracted to hard data problems at massive scale. [SPEAKER_00]: So early in my career, it was about making databases faster because that was the era where every company on the planet was adopting databases to automate workflows.

1:37 [SPEAKER_00]: The next maybe decade was about open source big data.

1:46 [SPEAKER_00]: on the internet is powered by some open source big data platform. [SPEAKER_00]: So I spent time there and then it seemed many of these problems were solved. [SPEAKER_00]: And so I took a retirement, early retirement, also to take care of a family member. [SPEAKER_00]: And so it paused and was looking for another massive data challenge.

2:16 [SPEAKER_01]: After you took that time off to take care of your family member, right? [SPEAKER_01]: I was about a year time frame that you took off as that correct? [SPEAKER_01]: It's during that time frame or after that time frame, you met the CEO of Roblox. [SPEAKER_01]: And were you planning to found a company at that point or did you always intend to join a company after this break? [SPEAKER_01]: And how did you end up joining as SVP of Engineering? [SPEAKER_00]: So great question, or brings back memories.

2:41 [SPEAKER_00]: I was at home and this new paper at Paramaharan Google on Transformers. [SPEAKER_00]: It was called Adention is All You Need, if I didn't believe it. [SPEAKER_00]: as you hone in on that was very tempting. [SPEAKER_00]: So even though I retired I was at home, I started buying a few GPUs and started training a model just on my, or we started becoming a starter by India. [SPEAKER_00]: But then I had breakfast with the Pazuki, the founder and CEO of Rockbox, and we started talking about what we took a lot of interaction data, 3T data, and what could we do with?

3:24 [SPEAKER_00]: And just trying to answer that question, we started jamming almost a whiteboard session, and a couple of meetings later, [SPEAKER_00]: I didn't actually join in as we'd be engineering. [SPEAKER_00]: I joined just as somebody responsible for the data and AI strategy. [SPEAKER_00]: And at that time, in 2021, I'll bring you back to the pre GPT days, pre chat GPT days. [SPEAKER_00]: AI strategy itself was our clear. [SPEAKER_00]: Remember there was no coding model, there was no large model, so to speak.

4:02 [SPEAKER_00]: more for impact than for Dido, and that's where my roadblocks journey started. [SPEAKER_01]: Awesome. [SPEAKER_01]: I bet that was an interesting lunch there that Dave swade you out of the out of the start-up world into Roblox. [SPEAKER_01]: That's really interesting. [SPEAKER_01]: What surprised you most about Roblox, once you got inside? [SPEAKER_00]: One was the scale. [SPEAKER_00]: As somebody who doesn't really play that much Roblox, now of course I play a lot of Roblox's part of my job, but I had a teenager at home and they played minecraft and Roblox.

4:36 [SPEAKER_00]: And I remember joining Roblox and within three weeks we had this massive event and Roblox went down for three days. [SPEAKER_00]: So for three days we were out, the surprising thing was, every time we tried to bring back the platform, we would get a thundering hoods of players trying to log into Roblox.com. [SPEAKER_00]: So we would never get breathing space. [SPEAKER_00]: And that became for Dave and me, the first idea that I worked on, which was how to solve for thundering hoods.

5:10 [SPEAKER_00]: It was surprising that every Saturday almost

5:17 [SPEAKER_00]: are creators. [SPEAKER_00]: would launch these events, whether it is destoing dress, whether it is grow of garden or 99 nights, they could just add 3 million 10 million and then last year in a single event, they got 45.7 million players to play Roblox at the same time. [SPEAKER_00]: That's been surprising. [SPEAKER_00]: How do you get 45 million people to use your software? [SPEAKER_00]: I could have never seen that in enterprise software [SPEAKER_01]: Yeah, that's an incredible scale.

5:51 [SPEAKER_01]: And then incredible, I'm sure engineering problem to stare at. [SPEAKER_01]: And kind of be wild by and then also how do I set my company up for success in that sort of scale there? [SPEAKER_00]: Even planning for infrastructure and all. [SPEAKER_00]: If you're, if you are for Gauss S-22 million players, but 45 million sure, you're not going to go by GPUs. [SPEAKER_00]: the morning of the Saturday, right? [SPEAKER_00]: And so just figuring out our capacity, working with our partners, working with our vendors, and even our vendors get very excited because they have family members that play Roblox.

6:30 [SPEAKER_00]: So I get texts from very senior executives saying, hey, please make sure Roblox doesn't go down to horror because my, [SPEAKER_00]: my giant is excited about some of it. [SPEAKER_00]: That's been the surprise that even the senior most people in worldwide have family members that are active robots creators or peers. [SPEAKER_01]: Yeah, no pressure from those text messages to keep the platform up. [SPEAKER_01]: Okay, let's move into foundation AI and infrastructure, the group that you're leading now and building out.

7:01 [SPEAKER_01]: What does foundation AI and infrastructure actually do? [SPEAKER_01]: And what was it when you started? [SPEAKER_00]: The origin story a distinctly remember was all of us large machine learning through things like recommendations. [SPEAKER_00]: So when you load a phone page, the platform will recommend certain games for you. [SPEAKER_00]: And initially, my responsibilities included our homepage recommendation system and that was the biggest tenant on our platform. [SPEAKER_00]: So built the platform really to power recommendation.

7:37 [SPEAKER_00]: Now, as the platform grew, safety became the most imported AI project for us. [SPEAKER_00]: So every character that you

7:51 [SPEAKER_00]: Every [SPEAKER_00]: game that you upload goes through a very sophisticated AI pipeline to assure safety of our players. [SPEAKER_00]: So that became the second day. [SPEAKER_00]: The first day it is recommendations, the second is safety, and then we decided to build our own foundation models. [SPEAKER_00]: In voting, in video, 3D, and NPCs.

8:22 [SPEAKER_00]: is very different from our starting mind because we focus on four foundation models. [SPEAKER_01]: Makes total sense. [SPEAKER_01]: I'm curious about the process of building those models. [SPEAKER_01]: What's it? [SPEAKER_01]: What was it like creating those pillars there in the models? [SPEAKER_01]: And what was the process you went through to make it to what it is today? [SPEAKER_01]: Obviously not sharing anything proprietary, but really the experience. [SPEAKER_00]: As a leader and I know you have a founder and leader, we all know great technology starts with great talent.

8:54 [SPEAKER_00]: So a lot of the initial years was just bringing more and more founder type talent into the company. [SPEAKER_00]: As a founder, I do have [SPEAKER_00]: a bias towards hiring founders because they are able to take the biggest challenges. [SPEAKER_00]: without worrying about sort of laws of physics, right? [SPEAKER_00]: And as a founder, I'm sure you have that experience yourself. [SPEAKER_00]: That's an important part. [SPEAKER_00]: That's the first step. [SPEAKER_00]: The second step was, of course, building out our own infrastructure because if you're serious about AI, you do want to build a really infrastructure.

9:40 [SPEAKER_00]: And then the third part was convincing our creators

9:46 [SPEAKER_01]: Interesting. [SPEAKER_01]: Okay, that's a that's a really good segue into my next kind of area of questions around building games and getting serious about building infrastructure and then again convincing the creators of the games those are all pretty interesting hurdles you had to go over or difficult things you have to head to work through what was the biggest technical barrier that's that's been there for creators and and how did you go about it? [SPEAKER_00]: So the biggest technical barrier is understanding 3D, so all of these foundation models are really good in understanding text and image and maybe even video, but 3D objects, so think about a car in a game.

10:31 [SPEAKER_00]: you should be able to open the door, you should be able to get in, turn the steering wheel and accelerate. [SPEAKER_00]: That means an object that is functional, not just pretty to look at if you, right? [SPEAKER_00]: Whereas in a video you can't really get inside the car as a player. [SPEAKER_00]: So, that was the main technical insight we started working on three years ago, which became the Q Foundation model and then we have been building out from that Q Foundation. [SPEAKER_01]: That makes so much sense, especially in the platform that you are.

11:06 [SPEAKER_01]: The three-dimensional aspect of it and anything you see out there with AI right now, it is text space. [SPEAKER_01]: There's generative parts there's, but even the generative parts that you see are typically image video and then two-dimensional types of stuff. [SPEAKER_01]: So that's interesting. [SPEAKER_01]: They've got my mind rolling a bit there. [SPEAKER_01]: What is the future of creation on Roblox look like? [SPEAKER_01]: So you've got these models you've created, the abstraction you've created, [SPEAKER_01]: or the game builders on board and you're able to build with AI, with that 3D, the 3D space in mind, what role does your team play in making the future of creation on Roblox possible?

11:48 [SPEAKER_00]: So the future of creation is making more and more players, being able to just create games that are entertaining and interesting. [SPEAKER_00]: So, around [SPEAKER_00]: three years ago, 11 San Francisco was watching these fireworks go off in the fog. [SPEAKER_00]: It's pretty beautiful when you see it on 4th of July. [SPEAKER_00]: And I was thinking, I just want to make a game where you can fly through San Francisco while fireworks are on. [SPEAKER_00]: Because remember we are video game, right?

12:21 [SPEAKER_00]: So we should be able to do this. [SPEAKER_00]: But the amount of work that takes is monumental. [SPEAKER_00]: Roblox is founding principal walls that anybody can fill the game and we have delivered on that promise we have studios with just two people who are millions of players. [SPEAKER_00]: Even then we only have a single digit percentage of our players that are creators. [SPEAKER_00]: So our big goal would be, what if anybody can create a game? [SPEAKER_00]: What if no one locks into robots, that says, I have an idea.

13:00 [SPEAKER_00]: So no, if you had an idea, what would you build today? [SPEAKER_01]: This is a good question. [SPEAKER_01]: What popped in my head was like an old, tiny, two-tier airplane flight simulator. [SPEAKER_01]: That's just what popped in my head. [SPEAKER_00]: There you go, super popular, flight simulators, and you should check out some of the shipping simulators on roadblocks. [SPEAKER_00]: It's just beautiful. [SPEAKER_00]: And so now know as an idea, you could open roadblocks. [SPEAKER_00]: You should be able to take this idea and then build an entire game out of it.

13:30 [SPEAKER_00]: We are delivering on that promise this year, we have a product called Bill where you can write a text from exactly what you just said. [SPEAKER_00]: I want a two-tier airplane simulator. [SPEAKER_00]: It will ask you a series of questions. [SPEAKER_00]: Do you want the controls to be obvious? [SPEAKER_00]: Do you want the plane to look futuristic? [SPEAKER_00]: Or do you want to look at some ways prehistoric?

14:00 [SPEAKER_00]: and build a game for you, and then you can do another 200, 300 prompts and build a digital mechanics. [SPEAKER_00]: So what kind of mechanic do you want to build in that game? [SPEAKER_01]: It would be interesting to have irregular gravity at certain spots, maybe less gravity or more gravity and certain simulation, it would make it interesting. [SPEAKER_00]: There you go. [SPEAKER_00]: So that's game playing mechanics. [SPEAKER_00]: So now you have a 3D object called the plane and you just added some gameplay mechanics into it, building that gameplay mechanics till we introduced build in the market would require you to write a bunch of code, right?

14:39 [SPEAKER_00]: You and I are fairly technical, you could open an interactivity, development environment to write a bunch of code saying this is how gravity should behave, etc. [SPEAKER_00]: but in this new world you should be able to just express irregular gravity and it will generate all the code to build that irregular gravity in the head. [SPEAKER_00]: So we have the plane [SPEAKER_00]: we have the game mechanic of irregular gravity and like any good engineer we want to test it. [SPEAKER_00]: Does it work?

15:12 [SPEAKER_00]: Now the classical game development would mean you would have to invite some friends and family to test you again. [SPEAKER_00]: But what if you could get some non-plane characters and they could test you again for it? [SPEAKER_00]: So that's the NPC model that we had heard. [SPEAKER_00]: So we went from 3D to coding to NPC. [SPEAKER_00]: And now the aim might be ready. [SPEAKER_00]: And then we didn't ask each other this question. [SPEAKER_00]: When is this plane flying? [SPEAKER_00]: Dude.

15:39 [SPEAKER_01]: That's a good question. [SPEAKER_01]: In my head, I am picturing mountains around it. [SPEAKER_01]: So maybe we've got some snow cover. [SPEAKER_01]: Think around Alaska. [SPEAKER_01]: We'll say that. [SPEAKER_00]: What you just described is our Roblox reality foundation one. [SPEAKER_00]: You should be able to say, okay, my plane is going to fly across no cap mountains. [SPEAKER_00]: And as soon as you said, Alaska also started thinking about crossing the Pacific Ocean, right? [SPEAKER_00]: If we are flying out from San Francisco, and so you need some beautiful water, and you need some beautiful clouds, and you need some mountains.

16:12 [SPEAKER_00]: Doing this, a conventional game engine, even a modern one. [SPEAKER_00]: would take you a lot of effort, building the mountains, building the ripples in the water, and do you want fluffy clouds or something else? [SPEAKER_00]: The roadblocks reality model that we demonstrated as a research preview in RDC would let you describe the mountains. [SPEAKER_00]: but let you describe the water and then it would go off and upscale all of this for you. [SPEAKER_00]: So, coding 3D NPC and video.

16:51 [SPEAKER_00]: Those are our four foundation models. [SPEAKER_00]: That's how I generally answer it. [SPEAKER_00]: People ask me, hey what does foundation AI do? [SPEAKER_00]: We build these four foundational technologies so that you can build the game literally from your thought all the video are actual [SPEAKER_01]: That's pretty fascinating. [SPEAKER_01]: It's a really cool approach to it as well. [SPEAKER_01]: I can think I'm not a game developer, but I do enjoy playing games and I can imagine stepping out into something like this and it's leveling the playing field of being able to create these games based on imagination, not just, I'm not imagination and required coding skills.

17:29 [SPEAKER_00]: Yeah, so reduce the coding skills, let the imagination take more, and make Noah a game developer without having to fit an interactive independent one, right? [SPEAKER_00]: That would be the sort of the mission statement of AI trope locks. [SPEAKER_01]: Excellent. [SPEAKER_01]: That's super cool. [SPEAKER_01]: Okay, let's move into some more maybe personal reflections. [SPEAKER_01]: Tell me about you outside of being involved in technology. [SPEAKER_01]: So this could be your past at this point.

17:57 [SPEAKER_01]: We talked a little bit about that and the influences, hobbies, family, things like that, but whatever you want to tell me. [SPEAKER_00]: A lot of travel because [SPEAKER_00]: seeing other cultures is very interesting. [SPEAKER_00]: So I was in Kyoto last week for a vacation after our developer's conference and just seeing different types of art. [SPEAKER_00]: Very passionate about art and I'm very involved in SF Jazz and the SF San Francisco American Conservatory Theatre. [SPEAKER_00]: So that

18:31 [SPEAKER_00]: and of course being a bit in, with sometimes seems like full-dined jobs, sometimes seems like an easy job. [SPEAKER_01]: Certainly understand that. [SPEAKER_01]: How many kids you have? [SPEAKER_00]: I want kid who is in college in New York. [SPEAKER_01]: Very cool. [SPEAKER_01]: That travel all sounds really fun, and as well as your family there. [SPEAKER_01]: So moving into some more reflection on what you're proud of. [SPEAKER_01]: And you have a fantastic career outside of Roblox, and now you're inside of Roblox.

18:57 [SPEAKER_01]: I'm curious about what you're most proud of as you look across all that you've built and that could be, I'm curious about two answers one, what you've built prior to Roblox and then what you most proud of in what you've built inside of Roblox. [SPEAKER_01]: Tell me about that. [SPEAKER_00]: Yeah, for me, the proudest moment is when somebody's using your software to do something not knowing [SPEAKER_00]: that across the table or across the counter, it's the author of this software. [SPEAKER_00]: Open source, Hadoob, today is embedded everything that we do.

19:29 [SPEAKER_00]: And sometimes I see it, I'm sitting in an airline seat, and sometimes the audio or video system is rebooting, and you see all the log lines, and I'm sitting there, oh, we built this, we built that. [SPEAKER_00]: Oh, this is [SPEAKER_00]: This plane actually, this flight is going to use software that started as an open source project maybe 10 years ago. [SPEAKER_00]: So you bigger this software and then roadblocks is very easy. [SPEAKER_00]: I was in a small, I was hiking through a small town slash village in India.

20:02 [SPEAKER_00]: And I happened to be wearing my Roblox T-shirt and I can't do you know there were 10 or 20 people who sounded me just giving me feedback about the app and how the game might be lagging or their favorite aim. [SPEAKER_00]: So that was just an amazing moment to be in a place which had an intermittent network and the phones were not as powerful as the ones that we carry and yet people just love playing [SPEAKER_01]: Okay, let's flip the script a little bit. [SPEAKER_01]: Tell me about a mistake you've made and how you and your team were teams responded to it.

20:36 [SPEAKER_01]: That could be prior to Roblox or during Roblox. [SPEAKER_01]: Tell me about that. [SPEAKER_00]: It depends on how much time you have, lots of mistakes. [SPEAKER_00]: One of the mistakes would be underestimating how fast something could be adopted. [SPEAKER_00]: Right? [SPEAKER_00]: So I'll give you a recent example. [SPEAKER_00]: A year ago, I didn't feel that coding models would be this far. [SPEAKER_00]: If I would go back to years, I would invest way more in coding models. [SPEAKER_00]: Because AI was so text-based, right?

21:06 [SPEAKER_00]: You and I and all that, it was chat-based. [SPEAKER_00]: And I always thought, hey, it cannot generate sophisticated code. [SPEAKER_00]: Because code has a particular structure. [SPEAKER_00]: And so, I personally underestimate how fast coding models will proliferate in the industry. [SPEAKER_00]: even though some of my colleagues passionately advocated for that. [SPEAKER_00]: So we're still in a good place, but it's something that I think about. [SPEAKER_01]: All right, appreciate you sharing that.

21:32 [SPEAKER_01]: Okay, who influences the way that you work, and name a person or many persons or something you look up to and why? [SPEAKER_00]: This will sound a little self-serving, but I spend a lot of time with the founders CEO, Dave Bazooki, and I learned so much from him about how to focus on our creators and our players and dig big principal decisions. [SPEAKER_00]: So that's one person and then a lot of my heroes and technology are open source folks because they start communities of engineers and customers without really a commercial

22:17 [SPEAKER_00]: Kafka, all of these open source projects have created massive human benefit, but the person who started it did not start it for commercial reasons. [SPEAKER_00]: And that inspires me every day. [SPEAKER_00]: It's almost like art and art is very close to my heart. [SPEAKER_01]: I love that, Anupam. [SPEAKER_01]: Okay, I cut one more question. [SPEAKER_01]: You're getting on a plane and you're sitting next to a young entrepreneur who's built the next big thing. [SPEAKER_01]: They're jazzed about it.

22:46 [SPEAKER_01]: They can't much show off the world and can we show off to you right there on the plane. [SPEAKER_01]: What advice do you give that person having gone down this road a bit as a founder as a builder inside of roadblocks really from a lot of different perspectives what advice do you give that person. [SPEAKER_00]: My advice will be more human or it would be be less anxious. [SPEAKER_00]: A lot of founders have anxiety, will their idea make it and if their idea makes it, will there be customers?

23:11 [SPEAKER_00]: If there are customers, will they pay all these questions can overwhelm you? [SPEAKER_00]: And my advice for the founder would be take the next step, found the found, just go look for one customer who's passionate. [SPEAKER_00]: about your idea and make them really successful. [SPEAKER_00]: Worst is worrying about how big will my company get, how much money will I raise, the mechanics of building a company are less important than making one customer view of the app. [SPEAKER_01]: Well, I think that's fantastic advice.

23:45 [SPEAKER_01]: But, Nupam, thanks for being on the show today. [SPEAKER_01]: It's really interesting to hear about your career and the throughline of your career and would guide you to Roblox and I'm especially geeking out about how you're going about building these foundational models and infrastructure at Roblox to enable the game creators to build based on their imagination. [SPEAKER_01]: It's really interesting. [SPEAKER_01]: Those pillars that you created there, that you and the team have created, [SPEAKER_01]: I'm really excited for the future for what Roblox has is bringing to the market.

24:14 [SPEAKER_01]: What Roblox is bringing to the market for those creators and for the players and hoping you continue to solve those massive scaling problems you learned from the very beginning. [SPEAKER_01]: So, really interesting conversation, Nupon. [SPEAKER_01]: Thanks for being on the ship. [SPEAKER_00]: Thank you, Nupon. [SPEAKER_00]: This was very interesting, made me think, which is beautiful.

24:32 [SPEAKER_01]: Quite the story from a new poem with the solid throughline in his experience of enterprise, startups, and large-scale systems, leading him into his new role of Roblox. [SPEAKER_01]: As we've heard, he and his team are pushing the boundaries of what is available, not only the players, but to the creators themselves, giving them faster and more intelligent ways to build and deploy games, founded on AI. [SPEAKER_01]: If you'd like to learn more about Roblox, or jump in and play, go to Roblox.com.