Developer Chats - Oleksandr Piekhota - Code Story: Insights from Startup Tech Leaders

[SPEAKER_00]: Hello listeners.

[SPEAKER_00]: Today we are dropping another episode in our series Developer Chats, hearing from the large-scale system builders themselves.

[SPEAKER_00]: In this episode, we are talking with Alexander Picoto, Principal Software Engineer at Teaching Strategies LLC.

[SPEAKER_00]: Alexander helps to show us, at what point of scale, platform approaches are required.

[SPEAKER_00]: When to run experiments, when to stop, and perhaps more importantly, engineering ownership beyond the code.

[SPEAKER_00]: Alexander, thank you for being on the show today.

[SPEAKER_00]: Thank you for being on code story.

[SPEAKER_00]: Thanks for having me.

[SPEAKER_00]: Absolutely, really excited to dive into all of your experience today as moving into more technical leadership as a principal engineer and platform versus feature thinking and all the things we're going to dive into.

[SPEAKER_00]: Before we do, tell me in my audience a little bit about you.

[SPEAKER_01]: So basically, I am currently working

[SPEAKER_01]: I can count like 14 years in production engineering, so I think I have at least of someone to stand like what I'm talking about.

[SPEAKER_01]: My background is mostly focused about PHP and GoLank.

[SPEAKER_01]: I do have some...

[SPEAKER_01]: extra experience in different languages, but that's not that reach to say that I'm strong in it.

[SPEAKER_01]: Have two masters degrees, one computer science, and to the second one in high performance computing.

[SPEAKER_01]: The last one I've got a few years ago in University of Edinburgh, really kept about it.

[SPEAKER_01]: And last year, I also found it a community called Detox Edinburgh, and we're just trying to make some meetups and to bands for people to share their technical experience and thoughts and give some richness to the audience.

[SPEAKER_00]: Very cool, congrats on all your success and your recent degree acquisition and that's a lot of work put into.

[SPEAKER_00]: To get those advanced degrees, I'd love to dive into my first question, which you've moved from hands-on engineering into principle and technical leadership roles.

[SPEAKER_00]: Kind of more working on architecture and platforms.

[SPEAKER_00]: At what point did you realize your work was no longer about individual features, but about the system as a whole?

[SPEAKER_01]: Oh, good start.

[SPEAKER_01]: I think it's responsibility shift, so basically when you realize that your decisions influence performance, cost, reliability, maybe like team velocity at the same time and when you understand that you're responsible for the system at the hall, like not in the video features, then when you understand that like your manager can call you,

[SPEAKER_01]: during the night because like of the client issue or something happened to the production instance and they need your help no matter how long it takes and like how late it is then this shift starts.

[SPEAKER_01]: I remember like a case from like my video-fueler's public IPI experience when my manager called me about clients issue and I tried to help him and debug it so even though it was late

[SPEAKER_01]: but the problem was that we haven't stored all the necessary traces to figure out what exact problem was.

[SPEAKER_01]: It's was a flaky issue when one time it repeats and the second one it doesn't.

[SPEAKER_01]: And because we didn't have all of those necessary traces to reproduce exact particular problem, at that moment I understood that I need to think a little bit out of the box.

[SPEAKER_01]: and like you need to design those kind of like things because the next time like when the module calls you, you need to be prepared like to answer those questions and be ready to handle like various situations.

[SPEAKER_01]: Yeah, like that's kind of like shift, I think is important.

[SPEAKER_01]: And when you get all of those responsibilities like this kind of understanding,

[SPEAKER_01]: comes to you like naturally and it's very hard like even though it sounds simple, but many developers don't get it unless they get into like similar scenario.

[SPEAKER_00]: I appreciate you walking through that.

[SPEAKER_00]: Okay, across several projects.

[SPEAKER_00]: Growth doesn't break functionality, right?

[SPEAKER_00]: Across projects, growth doesn't break functionality.

[SPEAKER_00]: It can expose architectural limits, and I'm sure you've experienced this many times.

[SPEAKER_00]: Can you recall a moment when it became clear that shipping more features wouldn't solve a problem, and a platform approach was required?

[SPEAKER_01]: I have one example in my mind, so when we work in my previous company on a B2B product, we pay a situation where we got like rapid team growth, like we tried to hire like many people from within the company, like we tried to hire like some people from like outside the company.

[SPEAKER_01]: And I guess we managed to get from maybe 10, 15 people as a core to about 80 like 90 people.

[SPEAKER_01]: And it's a huge growth, like we got that, like, from, like, for, like, during the period of half a year.

[SPEAKER_01]: And lack of standards, like, cross-tim development, microservice approach, like, pushed development, like, in such a fast-paced, that when we eventually get the point that we have some features, we face a situation that local development is terrible.

[SPEAKER_01]: it's nearly impossible to run like the whole environment.

[SPEAKER_01]: Integration between services like it's just to complete mess.

[SPEAKER_01]: We even tried to assign delivery managers from different teams just to make a release possible.

[SPEAKER_01]: So delivery manager like goes to each team and ask them basically, have you guys deployed everything?

[SPEAKER_01]: Did you did this?

[SPEAKER_01]: Did you did that?

[SPEAKER_01]: And yeah, like it made those features very complicated.

[SPEAKER_01]: And finally, when we were ready to release like production flow and our first production flow basically was our internal test and when we asked how our internal people to go and test like some flow, we figured out that the product can't handle like the load we created like within the company.

[SPEAKER_01]: I'm not even saying like we published it to some other external customers.

[SPEAKER_01]: And yes, so this gives me, like, the understanding that, like with such a pace, like with such an approach, going further makes no any good progress at all.

[SPEAKER_01]: Like, things has to be with a good thought, to rethink in some way that the approach is going to be different.

[SPEAKER_01]: And yeah, we started thinking about it and like we tried to make some microservices, try to put them together, we tried to introduce tracing, we tried to invest in like more mature local development flow, hired people that were responsible for like automation tests and regression tests.

[SPEAKER_01]: And yeah, all of these together gave a lot of experience to understand that

[SPEAKER_00]: Okay, cool.

[SPEAKER_00]: Let's move into my next question here, and with PDF filler, started as a B2C product.

[SPEAKER_00]: But entering into B2B required public APIs and much higher reliability standards.

[SPEAKER_00]: For business to business side of things, what was the hardest part of that transition?

[SPEAKER_00]: From an engineering perspective, not necessarily a business one.

[SPEAKER_01]: Yeah, it's actually funny because the hardest part wasn't even like skill in infrastructure, but it was more about contracts and like formalizing interface, which you develop.

[SPEAKER_01]: So basically when they developed like some kind of like core functionality,

[SPEAKER_01]: for the B2C part of the product so when you that we can develop these we can develop that you can go and talk to front-end engineers or you can talk with mobile devs.

[SPEAKER_01]: You can always introduce something else for them.

[SPEAKER_01]: If like previous part of the code like doesn't work then they can just use another endpoint.

[SPEAKER_01]: They can work in any way like you grew with them during the call all like during the lunch or something like that.

[SPEAKER_01]: and yeah this internal behavior like it makes things so much easier and simpler to work with so basically like you focus on completely different things.

[SPEAKER_01]: When you go like into B2B and you're trying to develop a public API that's going to be used to by other companies, other teams,

[SPEAKER_01]: becomes more like about the public contract itself.

[SPEAKER_01]: Inslike, backward compatibility, like conversion in strategy, like rate limiting, so things like, where I like much more important now.

[SPEAKER_01]: And the design that you put into every endpoint, it's actually interesting because I was working with my manager a lot, spending a huge amount of time just discussing,

[SPEAKER_01]: a single-epian point, and how we should name, like, and point itself, how we should name, like, the request params, response params, IDs, like errors, like names of those errors, maybe, like, some kind of messages and stuff like that.

[SPEAKER_01]: This would make public API much more available for other people.

[SPEAKER_01]: And at some point, how easy it is to read it, how easy it is to work

[SPEAKER_01]: how easy like it would it be to develop SDK for such an API.

[SPEAKER_01]: This kind of like predictability and trust that this is what makes like just an API like a great API I would say.

[SPEAKER_00]: Very cool.

[SPEAKER_00]: Okay.

[SPEAKER_00]: So moving into round internal systems, examples like invoicing and HR workflows, they begin as automation, but eventually they evolve into real products.

[SPEAKER_00]: What tells you that an internal tool is worth developing seriously?

[SPEAKER_00]: Rather than treating it as a temporary workaround, when I think about that, I think about when you're using vaping process together for an internal system versus

[SPEAKER_01]: I think that it really depends and it's hard like to reflect on this question from my experience.

[SPEAKER_01]: I can say that we try to develop internal tools that we've seen like a problem with because the company I was working in was about document and workflows automation.

[SPEAKER_01]: Basically like we've seen that for example in voice flow wasn't optimized properly

[SPEAKER_01]: And on the other hand, like when we start to develop in public API, we've noticed that before we go into real production, this real client, we still need to have some feedback about our API usage.

[SPEAKER_01]: And the best thing is to develop like a client for ourselves, and see how it goes on a scale of some workflow like this.

[SPEAKER_01]: And what you figure out is like when develop in such a tool,

[SPEAKER_01]: We were able to see all the kind of design issues they made during prototype in this IPI.

[SPEAKER_01]: And this helped us a lot to get all of those feedback.

[SPEAKER_01]: And on the other hand, we managed to create like an app that's really simplified and optimized in turn-of-law.

[SPEAKER_01]: So all of those internal clients that gonna use the tool that you've built, they gonna give you like a very valuable and important feedback that you probably won't be able to get from real customers because they see it outside of your office and maybe like you need to analyze some kind of like logs and traces which is completely different story.

[SPEAKER_01]: That's like solution that an approach that

[SPEAKER_01]: I'm not sure that it's probably going to work for everybody and like everywhere but again if you really look into a good client for your system, maybe it makes sense to try to use your system itself by your particular needs and this will help you maybe find in some issues and maybe solve

[SPEAKER_00]: very good.

[SPEAKER_00]: I appreciate you all can do that.

[SPEAKER_00]: Okay.

[SPEAKER_00]: So in R&D, you explored CICD automation, serverless infrastructure experiments, the whole nine yards, not all reached production.

[SPEAKER_00]: How do you decide when an experiment should continue?

[SPEAKER_00]: And I don't even add in there when it starts.

[SPEAKER_00]: And when it's no longer worth the engineering costs.

[SPEAKER_01]: Basically, from what I've noticed, like from my personal experience, is when you try to do some R&D work, it becomes very emotional.

[SPEAKER_01]: When you spend a week or two or even more, like trying to understand something technology, then you spend some time trying to develop a proof of concept.

[SPEAKER_01]: And then like maybe prepare some presentation about it.

[SPEAKER_01]: You're trying to be like, not exactly trying, you're going to be looking like from the perspective when you want to sell this tool, because you'll already invest it like so many, you personal time in it.

[SPEAKER_01]: And this is the problem because sometimes it really makes no real sense or sometimes like perspective, maybe like managers perspective or like teams perspective, could bring you a very valuable information like feedback.

[SPEAKER_01]: So, answering the question I would say that it's very nice and good to have team working with you together.

[SPEAKER_01]: So, you can share some thoughts and hear some feedback.

[SPEAKER_01]: And it's really cool if you're going to have a skilled manager that can lead you towards the right direction, or maybe slow you down if you're trying to go too deep or like to,

[SPEAKER_00]: Very good.

[SPEAKER_00]: Okay, so you've hired teams at standards and shape long-term technical direction.

[SPEAKER_00]: And at what point does an engineer stop being a contributor and start owning business-level outcomes?

[SPEAKER_00]: I'm really excited to hear what you have to say here, so I think it applies across the board.

[SPEAKER_01]: Hard to say when exactly this happens and I assume it also depends on the environment you are in.

[SPEAKER_01]: and the type of company you're working with.

[SPEAKER_01]: So for example, many outsourcing companies, people they are working like just from project to project.

[SPEAKER_01]: And like when you talk to such people that their focus mostly is shifted towards delivering the project like doing what requirements are and just working towards these kind of like cadences like where

[SPEAKER_01]: And even if there are some people that have some kind of like opinion about their architecture, about like the things, how it should be developed, they try to like get knowledge about what products going to do.

[SPEAKER_01]: The problem is that they opinion usually doesn't matter in terms of like business.

[SPEAKER_01]: And I can say even more that people who make decisions won't even able to hear them

[SPEAKER_01]: They may be sit-like in different country, in different office, and there are a whole bunch of operation managers that just proxy the information, and because of such a structure, this information would never like reach proper years.

[SPEAKER_01]: And because of that, people who choose to stay in such environments probably won't become owners of business level outcomes.

[SPEAKER_01]: And on the other hand, there is small product oriented team,

[SPEAKER_01]: and there are like very few people there like maybe some found an engineer like few folks that works with him and they see what they are doing and the impact they do like the code they deliver like reaches the clients instantly just after they deploy they see like how it changes like on the real product

[SPEAKER_01]: And maybe like when they see that their work is appreciated, when they try to have some conversation and they hear like real, like discussion and basically like when the moment of appreciation happens, I think this is where like people start in, like this business level outcomes.

[SPEAKER_00]: very cool and that makes sense.

[SPEAKER_00]: So you've contributed to open source tools that later became part of your company's infrastructure.

[SPEAKER_00]: Why do you see open source contributions as part of serious engineering work, rather than a side activity, because I think some people can see it as side projects and scratching itches that maybe you don't do during the day-to-day job, but really can contribute to the whole thing, the whole picture of what you're trying to build.

[SPEAKER_01]: Basically, when you start any new project, you're basically trying to include some framework or library, like in PSP World, for example, you can use like composer, write to install something that are like many frameworks and majority of those tools like frameworks are open sourced.

[SPEAKER_01]: And many people don't think about it, but the fact is people invest a huge amount of effort to make those tools, which later become part of your code.

[SPEAKER_01]: It's funny, but we trust many of those libraries blindly, just because we try the community.

[SPEAKER_01]: If you take a look at some popular libraries, for example, in GoLank, there is PGX library for Postgres,

[SPEAKER_01]: It has a huge amount of likes and community.

[SPEAKER_01]: In PSP world, the Riz Lake El Rabel framework, like everyone knows about it.

[SPEAKER_01]: In JavaScript, Riz Lake React, and this is also like open source solution.

[SPEAKER_01]: And I think like we need to pay our debt back.

[SPEAKER_01]: in one form or another.

[SPEAKER_01]: And I can say that even like in the company like I'm working the moment and like in all of those like I didn't do work in the past and so on, companies do a lot of research like proof of concepts, experiments.

[SPEAKER_01]: very often you already have the code that is possible can be open sourced you just need to maybe add a read me and publish it and it's very simple to do but like the call of those small steps all of those small publications will create a whole bunch of like very interesting open sourstools and from a practical perspective I can say that if you can publish your pc code and in the

[SPEAKER_01]: They can make a production-ready solution, which means that you can get benefits of such a solution for free, if your original problem is still relevant.

[SPEAKER_01]: What I've noticed recently is like developers, for example, in my team, understand that we are going to open source like some component.

[SPEAKER_01]: They just start to think differently.

[SPEAKER_01]: They try to put much more effort into developing the library because they know that this work will become their public activity and maybe like part of their resume.

[SPEAKER_01]: And just saying let's open-sourced, it helps people to produce much better, much more

[SPEAKER_01]: and maybe like last but not least like sometimes you need some experience working with something but you don't have any project that you can work with.

[SPEAKER_01]: Or maybe like you work in the company and company doesn't have the stack, you're interested to work with to work into.

[SPEAKER_01]: And I guess like the easiest way here is simply to come an open source contributor, and you will get all of that knowledge.

[SPEAKER_01]: And just doing so, you probably can simply jump from one position to another in terms of singularity.

[SPEAKER_01]: By simply expanding like maybe an hour of time after work or something like on the weekend.

[SPEAKER_01]: And this is so simple to do.

[SPEAKER_00]: absolutely super simple and it doesn't take that long and it really access to these types of things is just astronomically different than it used to be.

[SPEAKER_00]: If you look across your projects, and curious how you recognize a truly mature engineering system, is it based on code quality, is it based on process, or how teams respond when things go wrong?

[SPEAKER_00]: How do you measure that?

[SPEAKER_01]: To be honest, like my thinking about it is going to be maybe funny in some way, and maybe like some people want to agree with my like vision, but I think like a mature system is like the system that makes

[SPEAKER_01]: and where customers have their use cases covered.

[SPEAKER_01]: This is like the base, so this is like the foundation.

[SPEAKER_01]: If you have these, everything else is secondary.

[SPEAKER_01]: So, for example, it can be like something with a terrible code base, very bad design practices, no architecture at all, but unless it makes money and solve the problem,

[SPEAKER_01]: Who cares?

[SPEAKER_01]: So, on the other hand, we can have a strong team.

[SPEAKER_01]: They can do everything solid, dry, Kubernetes, microservices, cloud, and they are called as written like in Kotlin, Golem, Scala.

[SPEAKER_01]: It's great from the engineering perspective, like it just some kind of like a ruby.

[SPEAKER_01]: And at some point that,

[SPEAKER_01]: The project has no funding or like it's difficult to change product perspective, make a quick shift like simple cost to pay for the infrastructure and the team like actually cost a lot of money itself.

[SPEAKER_01]: And the funniest part like when somebody is living the product,

[SPEAKER_01]: You can't find replacement due to the fact how complex or how mature your particular architecture is.

[SPEAKER_01]: So basically, summarizing all of these together, I can say that mature system is like a balance of everything.

[SPEAKER_01]: And the highest priority, I would still say that it's going to be like making money.

[SPEAKER_01]: if you make money and like you have nice friendly responsible team that can do like trade-offs and keep it working and still keep it yet simple and probably like architecture that can be like not that strong maybe like somewhere like simple in in such in some terms but yet flexible for like business changes this is where like it became like really mature.

[SPEAKER_01]: So, yeah, long story short, Hawaii's the trade-offs, and maybe some money income.

[SPEAKER_01]: That's pretty much it.

[SPEAKER_00]: There you go.

[SPEAKER_00]: Okay.

[SPEAKER_00]: I'll examine our last question.

[SPEAKER_00]: So, if we look five to seven years into the future, which architectural assumptions we treat as standard today are most likely to turn out to be naive or limiting.

[SPEAKER_00]: This is this will be interesting, because this is a little bit of forecasting here.

[SPEAKER_00]: What do you think?

[SPEAKER_01]: Yeah, I'm not very good in terms of forecasting things.

[SPEAKER_01]: Let me try to think about it.

[SPEAKER_01]: Basically, everything is changing now like with the AI.

[SPEAKER_01]: And I can say that AI will quietly, or maybe like loudly, affect many of our current architectures, or maybe like architecture habits.

[SPEAKER_01]: I think this happens like in one or like another way, not because those are architectures,

[SPEAKER_01]: But because they were built like for the world where engineers are like a primary operators or like integrators, right?

[SPEAKER_01]: And very soon like places where like those engineers would become bottlenecks like the first place is which is going to be affected by this AI boom.

[SPEAKER_01]: I have a good example, so there is a book called Clean Code by Robert SchMartin, and in this book, and overall, this is what we should think about, is like the way how we see how we treat a clean code, like as a primary quality magic, right?

[SPEAKER_01]: And in they are like full word, draw like like phrases, something like honesty is like in small things, is not like a small thing.

[SPEAKER_01]: or like maybe God is in the details or maybe something like because small thin smithers.

[SPEAKER_01]: And the fun part is if like now most of our code bases are a degenerated then all of those kind of like small things like about code elegance, it makes no sense anymore.

[SPEAKER_01]: And now what shift is going to be is like to focus more on like tests and contracts.

[SPEAKER_01]: If such an adoption happens, we probably would require, like, to make our CI-CD pipelines much more mature, and maybe put some much more advanced, like, gateways that will check our codebase that will validate that everything works, that, for example, like test and test coverage would be like much more important here.

[SPEAKER_01]: or maybe like deployment pipeline, supposed to be like exactly something like kind of deploy, where like changes goes to 1% of traffic than 5, then or something else.

[SPEAKER_01]: And maybe like we need to like add some kind of like security checks that will validate all the compliances, vulnerabilities, secrets, etc., etc.

[SPEAKER_01]: Maybe like human focus in this case has to be shifted into defining some kind of like good policies and rollback strategies.

[SPEAKER_01]: so the production won't fail at some point.

[SPEAKER_01]: I can say also like that and I really like the way habit goes so about model context protocol mcp the way how popular it begins now and how it grows.

[SPEAKER_01]: I can say that maybe like in some close future it can replace all of the usual APIs we used to have.

[SPEAKER_01]: and things like open-claw or like non-about, they prove that like you can get 25,000 likes in two weeks, which is just crazy.

[SPEAKER_01]: Maybe like, monolithic architecture would be much more popular against microservices because, again, like, all of those agents are much more productive when they work with a single codebase rather than just when you try to

[SPEAKER_01]: Like micreservices and like then trying to glue them together like with some kind of like a contract.

[SPEAKER_01]: Long story short, right?

[SPEAKER_01]: Yeah, I will affect a lot of practices.

[SPEAKER_01]: I tried to maybe bring some examples, but who knows?

[SPEAKER_01]: I guess we will see soon.

[SPEAKER_00]: Yeah, for sure we will see it and no one can predict the future, but I appreciate you sharing your forecast there and I really appreciate you being on the show Alexander.

[SPEAKER_00]: It's clear you've built and led teams to build platform architecture.

[SPEAKER_00]: You clearly indicate that a mature system is one that strikes a balance between all the right things and I really connected with your point on experiments and wind to call them done.

[SPEAKER_00]: So I really appreciate you being on the show today and sharing your industry experience.

[SPEAKER_01]: Sense for having me and really happy to give such a

[SPEAKER_00]: Oh, Extander highlights some really interesting points around moving from hands on engineering into principle and technical leadership.

[SPEAKER_00]: You got to think a little bit differently into end architecture, infrastructure, platform thinking, and the like, and how to hire teams and explore experiments that support bringing features to market.

[SPEAKER_00]: If you'd like to connect with Alexander check the show notes for all the appropriate links.

[SPEAKER_00]: And thanks again for listening.