Not in a passive, step-back-and-hope-for-the-best kind of way. But in the active, daily, uncomfortable way where you watch problems get solved differently than you would solve them and you have to be okay with that. Sometimes you’re genuinely impressed by the approach. Sometimes things go off track and need correction. Both of those are fine. Both are learning opportunities. The hard part is resisting the urge to jump in and steer.

Two Directions, Same Struggle

This happens in two directions that I deal with almost daily.

The first is upward. Will Larson wrote about this in his Staying Aligned with Authority post.

At the staff+ level, you’re expected to have strong opinions. You’ve built judgment over years, and you see patterns, anticipate risks, and have a vision for how things should evolve. But you’re not the final decision maker for everything. Leadership will sometimes go a different direction than what you advocated for. Maybe they have context you don’t see. Maybe they’re optimizing for something different. Or maybe you just disagree.

The challenge isn’t understanding this intellectually. Everyone gets that in theory. The challenge is not letting it turn into resentment. Or an us-against-them feeling. It’s learning to genuinely commit to the direction that was chosen, even when it wasn’t yours, without becoming passive or bitter about it. You still push back when it matters. You still raise concerns. But once the call is made, your job is to make that direction succeed. Not because you gave up on your opinion, but because that’s what the role actually requires.

Larson frames this as authority being “loaned, not owned.” And that framing helped me a lot. Your effectiveness depends on staying tightly aligned with the people who loaned you that authority. The moment you start working against the grain because you think you know better, you’re burning the trust that makes the role work in the first place.

I won’t say I’ve figured this out. There are still days when it’s frustrating. But I’ve been learning to sit with that frustration instead of acting on it. That’s a work in progress.

Downward: The Harder One

The second direction is downward, and this is the one I struggle with more. When you’ve spent your career building strong judgment about how to approach problems, watching someone take a different path triggers something. It’s ego. It’s competitiveness. It’s the instinct that says you could do this faster, or that you already know where this is heading.

It’s fighting the urge to play the hero role

But the job at this level isn’t about your solution anymore. It’s about their growth. And growth doesn’t happen when someone follows your exact blueprint. It happens when they wrestle with the problem, make their own calls, and learn from what comes out of it.

I’ve had cases where I let go and was genuinely impressed by what someone built. Approaches that I wouldn’t have considered, solutions that fit the context better than what I had in mind. And I’ve had cases where things went completely off the rails and needed course correction. Both were good outcomes. The first reminded me that my way is not the only way. The second was a learning moment for everyone, myself included.

The real tension is that you can’t fully disappear. You still need to be present, still need to catch the things that genuinely matter, still have to create conditions for people to succeed. The line between giving space and staying involved is thin, and I get it wrong more often than I’d like.

It’s like learning to become a football coach after being a player your entire career.

The Ego Part

I want to name this directly because I think a lot of engineers at this level feel it but don’t talk about it.

You want to make an impact. You’re competitive. You want to be the one solving the interesting problem. That’s probably part of what got you here. And when you step back to let someone else lead something you could have done yourself, there’s a voice that asks whether you’re still relevant. Whether you’re still contributing enough.

It’s something that every staffplus engineer knows. How to be a multiplier instead of an execution force. The same principle applies here, but at a higher altitude. You’re not just enabling your immediate team, you’re shaping how problems get approached across a wider surface area. And that only works if you learn to let go of being the one who solves them.

How I’m Approaching It

I don’t have a clean framework for this. But two things have been helping me.

The first is encouraging people to seek feedback early. When someone shares their thinking at the rough draft stage, I can give lightweight input without taking over. I can ask questions, flag risks, and then step back. The earlier the feedback happens, the less it feels like me imposing my view and the more it feels like a normal part of how we work together.

The second is being more hands-on. And this is a big shift that’s happening across the industry right now. Over the last couple of years, my productivity has been doubling almost monthly. Using Claude Code, Copilot, and other AI coding assistants has made this possible in a way that wasn’t realistic before. I’m shipping code more than ever, even at the principal level.

This matters because it changes the dynamic of letting go entirely. When you’re contributing tangibly to the codebase, you don’t feel like you’re missing out. You’re not the theoretical architect who reviews from a distance and gives abstract feedback. You’re in the code, you understand the current state of things firsthand, and that gives your input real weight when you do engage with what others are building. People take your feedback differently when they know you’re in the trenches with them.

But it also solves a personal problem. A big part of why letting go is hard is the feeling that you’re losing your identity as a builder. You became a principal because you’re good at solving problems, and suddenly the job asks you to stop doing the thing that got you here.

The taste here, and it’s a real one, is knowing how to do this without slipping back into hero mode. Being hyper-productive as a principal is great until you start absorbing problems that should belong to someone else. The line between “I’m contributing” and “I’m taking over” gets blurry fast when you can move quickly. I’m still learning where that line is.

Together, these two things make letting go feel less like losing control and more like a deliberate way of working where everyone stays connected without anyone being the single point of authority.

Still Figuring It Out

I catch myself wanting to jump in when I should step back. I still feel the pull of wanting to solve the interesting problem myself. And there are days where I tip too far in one direction, either too hands-off or too involved.

But I think that’s the nature of this role. The technical problems at this level are solvable. The human ones, the ones that require managing your own ego and trusting other people’s judgment, those take much longer to get right. I’m not there yet. But I’m working on it, and writing about it is part of how I process it.

One of the hardest lessons I learned in my career was learning to be okay with putting something imperfect out there. I used to spend a lot of time trying to perfect things before showing them to anyone. Whether it was a design, a proposal, or even a Slack message, I wanted it polished before it left my hands. And I thought that was diligence. I thought that was what good engineers do.

What I eventually learned is that this is a common, hard problem. And not just for me, but for everyone. Artists have a name for this; they call it blank canvas syndrome.

The pressure to create something perfect from nothing causes paralysis. The emptiness feels overwhelming. And the advice that artists give each other is surprisingly practical: Kill the White.

You need to start somewhere, right?

The Perfection Trap

As engineers, we love to perfect things. Covering every edge case, every input, and every possible outcome. It’s the nature of the job. And nobody wants to ship broken software or present half-baked ideas.

But that instinct becomes a trap when it stops you from putting things out early enough to get meaningful feedback.

What happens is you spend days or weeks working on something in isolation, building confidence in your own solution, and by the time you share it, you're emotionally invested. You're not really asking for feedback anymore. You're asking for validation. And if someone challenges your approach, it doesn't feel like collaboration. It feels like an attack.

I've been on both sides of this. I've been the person who spent way too long perfecting a design document before sharing it, only to realize in the first review that I missed something fundamental that someone could have pointed out on day one. And I've been in rooms where someone presented a fully baked solution and got defensive the moment anyone pushed back. In both cases, the root cause was the same: the fear of being seen as wrong.

The best advice I got on this was, “Be comfortable being an idiot.”

Your Title Is Not Your Identity

This gets worse as you get more senior. When you’re earlier in your career, being wrong is expected. Nobody blinks when a junior engineer asks a question that seems obvious or proposes something that doesn’t quite work. That’s part of learning, and everyone knows it.

But something shifts when you reach senior and beyond. There’s an unspoken expectation, sometimes from others but mostly from yourself, that you should have the answers. You start tying your value to your ideas. Your title becomes tangled with your identity, and being wrong starts to feel like it threatens both.

And the irony is that this is exactly what slows your growth down. The people who grow the fastest at senior and staff+ levels are not the ones who are always right. They’re the ones who are comfortable being wrong in front of others, because that’s how they learn the fastest. They put out the rough sketch, invite the criticism, adjust, and move on. They treat being wrong as information, not as failure.

Feedback Needs Something to Land On

This is where the empty canvas idea becomes practical. If you want useful feedback from your team, from your peers, or from your manager, you have to give them something to work with. A half-formed idea shared early is infinitely more valuable than a perfect idea shared too late. Because feedback has a shelf life. The earlier you get it, the cheaper it is to act on. The later you get it, the more painful it becomes, both practically and emotionally.

I’ve learned this the hard way more times than I’d like to admit. There were moments where I held back an idea because I wasn’t confident enough in it, and by the time I shared it, the window had passed.

And there were other moments where I forced myself to share something rough and early, felt exposed and a bit stupid, and then watched as the feedback from the team turned it into something far better than what I would have built alone.

Learn to Bring People A long

Seeking feedback early means you are bringing people along. It shows respect and collaboration. My point isn’t about speaking up every idea that crosses your mind.

When you seek out early feedback, people feel the ownership, and they are more likely to support you in more ways. It puts you in a position where you can upskill your peers and yourself. And on the contrary, late feedback puts you in a position where you it feels like you are trying to be a hero.

Real accountability is messy. It's sharing the ugly first draft. It's saying "I think this might work, but I'm not sure" in front of people who you want to respect you. It's trusting that your value as an engineer comes from how you approach problems, not from whether your first answer was correct.

Seeking Feedback as a Practice

The shift I had to make was turning feedback from something that happens to me into something I actively seek out. Instead of waiting for a review or a meeting where someone might poke holes in my work, I started going to people earlier. “Here’s what I’m thinking; what am I missing?” became a regular part of how I work.

And the thing is, once you start doing this consistently, something interesting happens. People trust you more, not less. They see that you’re not trying to protect your ego. You’re trying to get to the best answer. And that creates a dynamic where others feel safe to do the same. They start sharing their rough ideas too. They start asking for feedback earlier. The whole team gets better at iterating and learning together, and that compounds over time into something much more powerful than any individual’s polished solution.

This is what I think real engineering growth looks like at the senior and staffplus level. It’s not about knowing more or being right more often. It’s about being comfortable enough with yourself to put the imperfect thing out there, invite people to challenge it, and use that friction to build something better. Your identity isn’t your title or your track record of being right. It’s how you show up when you don’t have the answer yet.

On the surface this looks like initiative. It looks like exactly what a senior or staff engineer should do. Take the hard problem, own it, and deliver the solution. And honestly, the person doing it genuinely believes they are helping. In their mind, the best thing they can do is dive in headfirst, figure it out alone, and come back with a complete answer so the team doesn’t have to struggle with it.

But here’s the thing. The team learned nothing. The context of that problem now lives in one person’s head. Nobody else got closer to understanding the system, the failure mode, or the reasoning behind the fix. And that person just became the default for every scary, ambiguous problem. Not because they built something sustainable, but because they trained everyone around them to wait.

The Hero Fallacy

This is a fallacy that a lot of engineers fall into, especially when they are trying to level up from senior to staff. I fell into this myself. I remember times when I thought the fastest path to demonstrating impact was to take the hardest thing on the table, disappear, and come back with the answer. It felt productive. It felt like accountability.

The fact is, it wasn’t.

Accountability means you own the outcome. It means you care about the problem being solved well and that the team is better for it. Hero mode forces you to focus on the spotlight. You care about being the one who solved it. These two things feel identical from the inside, which is why the fallacy is so common. You can genuinely believe you are being accountable while actually operating in hero mode. The difference only becomes visible when you look at the impact on the people around you.

Being Proactive vs. Being a Hero

Let me be clear. I’m not saying in any way that you should stop taking on hard problems or being proactive.

My favorite quote of all time is by Thomas Jefferson

“I'm a great believer in luck, and I find the harder I work the more I have of it."

Those are great qualities, and every team needs people who see problems early and act on them. The distinction is in how you act.

Being proactive means you identify a problem before it becomes critical, you raise it with the team, and you help create clarity around what’s happening and what needs to happen. You might lead the effort to solve it, but you bring people along. You share context as you go. You make the problem visible and solvable by the team, not just by you. In fact, I wrote about this before:

But hero mode is different. It means you see the problem and you absorb it entirely. You don’t share context because you’re still figuring it out. You don’t bring people in because it feels faster to do it alone. And when you come back with the solution, the team is grateful but also a little more dependent on you and a little less capable of handling the next one.

The proactive engineer makes the team stronger. The hero makes the team more reliant. Both look like leadership from the outside, but only one of them actually is.

Being a Multiplier, Not a Solo Force

The shift from senior to staff (and beyond) is fundamentally about becoming a multiplier. Your value is no longer measured by how much you can personally deliver. It’s measured by how much you enable the people and systems around you to deliver.

Hero mode is the opposite of being a multiplier. It’s being a solo execution force with extra drama. You might ship the fix, but you didn’t debug the org. You didn’t uplift anyone. You didn’t make the system more understandable or the team more capable. You just solved one problem and set yourself up to be the only one who can solve the next one.

The engineers who I’ve seen grow the most at senior and staff plus levels are the ones who resist the urge to disappear and come back with the answer. Instead, they do the harder, slower, less glamorous work of pulling people into the problem, creating shared understanding, and letting the solution emerge from the team. Sometimes that solution is messier. Sometimes it takes longer. But the team is better for it every single time.

There are exceptions, though!

I don’t want to pretend that there’s never a time for it. Production is down, customers are affected, and you need someone to jump in and fix it now. In those moments, hero mode is the right call. You don’t need a collaborative learning experience at 2 AM during an outage.

But those moments should be the exception, not the default. If hero mode is your default operating mode, that’s a sign that something deeper is broken. Either the team doesn’t have the context to engage with hard problems, or you haven’t invested in building that context, or you’re optimizing for personal visibility without realizing it.

The best engineers I know treat those emergency moments as signals. After the fire is out, they go back and ask, Why was I the only one who could fix this? What context was missing? How do we make sure more people can handle this next time? That’s the difference between a hero and a leader.

In Ted Lasso, Coach Beard (the best character, fight me 😜) drops this wisdom:

“You know, we used to believe that trees competed with each other for light. Suzanne Simard’s field work challenged that perception, and we now realize that the forest is a socialist community. Trees work in harmony to share the sunlight.”

I can’t stop thinking about this. Trees don’t fight over sunlight. They share it. They help each other grow. Even when there is minimal resources.

I’ve been a mentor for most of my career. Even during times when I probably should have been “focusing on myself.”

My philosophy is simple:

When you help someone, really help them, they feel it. They feel valued. And people who feel valued want to give back. Sometimes that’s friendship. Sometimes it’s having your back when things get tough.

It also changes how people see you. They’re not just happy to work with you—they actively support your ideas. They fight for the same things you fight for. Not because you asked them to, but because that’s what happens when you invest in people.

I’ve watched others do the opposite. They guard their knowledge like it’s gold. They gatekeep. And you know what happens? People resent them. Everyone’s on edge. Nobody trusts anyone.

Knowledge hoarding is a losing game. It creates small kingdoms ruled by fear, where innovation dies and people leave. The hoarders think they’re protecting their value, but they’re actually challenging it. Your value isn’t tied to exclusive knowledge! 

Knowledge grows through exchange, through challenge, through teaching someone else and realizing you understand it better yourself.

If trees are helping growing each other. Human should do better..

Understanding the Learning Process

The human brain isn't a database. They're pattern-matching machines that thrive on connections and context.

When we find new information, our minds don't simply store it in a file. Instead, they wire it into existing neural networks, strengthening some connections while pruning some others.

This is why passive consumption rarely leads to understanding. Reading or watching tutorials might feel productive, but without active engagement, that information remains meaningless.

Real learning happens when we force our brains to struggle, to build bridges between what we know and what we're trying to understand. It's that process of transforming unstructured information to structured knowledge.

Identifying The Gaps

In his amazing blog post, Tackling Complexity, Dr. Doug Belshaw, explained the known/unknown spectrum in this figure.

Applying the same paradigm in the learning process. The first step is understanding what you don't know.

• Known knowns: Skills you've mastered and use confidently.

• Known unknowns: Areas where you recognize your limitations.

• Unknown knowns: Areas where you've mastered/understand but don't apply regularly.

• Unknown unknowns: Concepts you don't even know exist yet.

The real growth happens when you systematically convert all unknowns into known areas. This might seem a tiring and overwhelming process in the beginning, but once you build the learning loop. It becomes much easier to grasp.

The Information Overload Problem

Search for any topic online, and you'll find millions of resources—blog posts, videos, courses, certificates. Take Python as an example: the sheer volume of content is overwhelming. You could spend years reading every Python book ever written, but that's not learning, that's hoarding information.

The challenge isn't finding resources, it's filtering signal from noise. You need a system that transforms unstructured information into structured knowledge that your brain can actually use.

The Learning Loop

💡

This is my personal learning process and doesn't apply to everyone. My advise is to build a system that works for you.

My learning loop looks like this

1. Identify gaps: Either stuff that I encounter in my job. Or read/hear about it while interacting with people.

2. Gather focused resources: With all the available AI tools, you can easily find good quality content that you will like (or have AI build it for you).

3. Transform information into knowledge: This is the most interesting step, you go through the resources that you find thoroughly, and you start identifying practical examples that you can apply either on a silly side project or in your job.

4. Write about it: Once you build the habit of writing about this knowledge either in private notes, blog posts or even README files. You will force your mind to process this knowledge and build a deeper understanding.

5. Discover new gaps: As you are writing about what you learned or building a practical example, you start finding areas where you still need to learn about. Doesn't mean you have to go through everything, but it means those have moved from unknown unknowns to known unknowns.

6. Repeat.

This creates an infinite loop of improvement. Initially, the unknown territory feels vast and intimidating. But as you build foundations, you develop intuition for what to learn next and what you don't need to learn, but you know is out of your depth.

Practical Techniques That Work

1. Build Broken Side Projects

When learning new topics, don't spend a lot of time building a complete project. That's not the goal, Instead, create something that captures 20-30% of the core concept that you fully understand. My GitHub is littered with incomplete projects, and that's by design.

The key is setting boundaries, making Assumptions and ignoring edge cases. Focus on understanding the core problem domain. You can always add those complexities to your learning backlog.

2. Write to Teach Your Future Self (And others)

Writing is programming for humans. When you force your brain to build structured knowledge out of the information it has. The gaps in your understanding become very obvious.

This doesn't need to be a complete or good writing. You are writing what you actually understand and can reason about. It's meant to be private (though it doesn't have to). And similar to my graveyard of half-complete projects in Github, My writing tab in Bear is the same. It's full of incomplete notes of subjects that I learn about.

I usually start with the table of content, I try to reason a lot about it and then quickly try to write (or draw, I'm a visual person) about that section. And immediately I'll know what I fully understood and can reason about, and what I need to learn about more. I use AI to get reviews on those notes, and in some cases, friends or colleagues that are far more experts in those areas.

This helped me to build a habit of writing. Most of it never sees the light of day. But every time I think about a topic or read about it or even answer a question about it, I try to dedicate 10/15 mins to turn it into a mini blog post. This creates a personal knowledge base I can reference and share. And force my mind to re-process this knowledge.

This extends to talking about those topics as well. Mentoring is a big part of my day, and I can argue that it benefits me more. Teaching or discussing topics that you know (known-knowns), Usually makes you identify what you don't know and strengthens the knowledge you already have.

Transform Your Mind into a Learning Machine

The core insight is this: treat your mind as software that transforms unstructured information into structured knowledge. Every blog post you read, every video you watch, every conversation you have. These are inputs to your learning system.

With the right mental models and practical techniques, you can dramatically accelerate your growth as an engineer.

But in reality, the best software systems are the ones that feel a bit... predictable and boring.

They provide a stable foundation that allows teams to move quickly without unnecessary friction.

The Myth of Exciting Software

A lot assumes that when you work on a great product it means that every thing you build will be using latest tech, solving a very challenging problem or push you to do something you never did before.

And once you work on a system that survived the test of time you realize that a well-designed software will be the exact opposite of that almost all the time. Those systems optimize for standardization. It can be deeply complex, handling massive amounts of data at scale, server a very rough SLA but still predictable.

Predictability is what allows teams to move fast without breaking things making sure that a feature written today will still make sense months or years down the line.

The Cost of Excitement

On the contrary. Systems that optimize for “excitement” always introduce Inconsistencies, hard debugging flow and High Cognitive Load.

Unpredictable system slows everything down. Engineers waste time making choices they shouldn’t need to make. Debugging is painful because no two parts of the system behave the same way. The cognitive load compounds over time until even the smallest changes feel risky and overwhelming.

And there’s a reason this cost is often not considered a risk by most teams. It accumulates over the long term, and most people don’t stay at the same company or on the same project long enough to suffer the consequences. After a few major version releases each with its own fancy new design and architecture things become so complicated that many systems today are labeled "legacy" only a few years after going into production.

Designing for Predictability (and Boredom)

Predictability doesn’t happen by accident. It requires investment in tooling, documentation, and cultural habits that reinforce stability and help the team to achieve the most value with the least:

Tooling to eliminate unnecessary.

Automated quality checks, mandatory code reviews and automated deployment pipelines automate testing and deployment so that shipping a feature follows the same process every time. The consistency help the team to build "muscle memory" for the system and make introducing changes less overwhelming because you are only focusing on the problem at hand.

Documentation that explain ‘why,’ not just ‘how.’

Keeping project documentation up-to-date is usually a forgotten task. And over the years I found that this usually happens because documentation focus on the "how". Which change a lot with design changes, new features, bug fixes, etc. And this lands you in a situation where the documentation explain how work does a very old version of that software that doesn't exist anymore used to work.

On the other hand, focusing on the reasoning for the design will always be useful, even if the documentation is behind, the reasoning that led to the design is still relevant. Which gives people the context and requirements for the software.

• Testing to increase predictability, not just coverage.

Good testing suites doesn't necessary have high coverage rates. In fact, it's was proven that using the coverage as the main metrics for tests quality isn’t productive and negatively affects the design. Meanwhile, testing for predictability and consistency ensure that the system is always self explanatory and easy to work with. It decrease the margin of human error not just on "functional" implementation but also on design.

Achieving a high quality testing quality that ensures the system predictability is always a first-class citizen that requires a solid foundation of different testing strategies.

• Observability and alerting make both easy days and hard days manageable. A well-monitored system doesn’t just generate logs it makes it obvious what’s happening at any given moment. When an alert fires, engineers shouldn’t have to dig through five dashboards to understand the issue. Good observability turns failure into a manageable event. This require investing in writing great logging messages, monitoring what matters and investing in quality observability stack.

The Tough Tradeoffs of Boring Software

Building predictable systems requires tradeoffs. It means resisting the urge to introduce shiny new technologies just because they’re exciting. It means saying no to unnecessary abstractions that add flexibility at the cost of clarity. It also means fighting the urge of introducing new tools left and right.

The best teams are disciplined about what they introduce into their stack. They understand that every additional tool or framework adds long-term maintenance costs, learning curve and mental burden. Their goal is that the system remains maintainable for years. That their system can survive the test of time.

And those systems feel boring because they don’t surprise you. They don’t require constant context switching. They don’t demand that engineers re-learn the rules every time they touch a new part of the codebase.

A system that feels boring because it's predictable enables rapid development. Engineers can move quickly, not because they’re constantly solving new problems, but because the hard decisions have already been made.

This builds a great platform for innovation. If you are not spending your time firefighting, trying to make sense of complicated pieces or guessing how the system is working, you will find the time to solve more "real" problem with innovative solution. The comfort and protected clarity of the system will allow the engineers to think about clean solution and innovative ideas.

The best engineering teams aren’t chasing excitement. They’re building systems where the next step is always clear.

Boring software makes everything easier. It’s a hard lesson that takes years to learn.

When everything is running smoothly, log messages often go unnoticed. But the moment stuff breaks, logs are usually the lifeline. In those rough moments, a great log message can be the difference between a quick fix and hours of guesswork.

I spend a huge chunk of my work debugging and maintaining codebases that I've never touched before. And I can tell you that writing good log messages isn’t just about documenting code. It’s about helping the next engineer—or your future self—quickly understand what's happening and how it's happening, especially during tense times.

Why Good Logs Matter

Logs are the extension of documentation. They explain the flow of your system, the decisions made at each step, and the points where things went off track. If a log message is too vague like “We did something here,” it offers little to no value and sometimes can cause confusion. Logs should give enough detail to explain what happened and hint at how it happened, especially when someone can’t look at the underlying code. This usually becomes very clear when you are trying to fix a production system that you are not familiar with.

Moreover, when everyone can read and understand the messages, it’s easier to divide tasks and assign fixes. Without good logs, people might rely on trial and error, which wastes time and introduces more confusion.

Choosing the Right Log Level

Debug logs are for deep dives. They reveal internal state and detailed activity, but they should only be enabled when you need that level of insight. If debug logs are always on, they can bury important info under irrelevant details. So, the “why” behind using debug logs is to clarify complex processes when necessary—but not to spam your system with noise.

Debug logs should act as an online production "debugger," allowing you to follow the execution step, understand the call stack, data flow, and the application state. But it's not always on. You only use a debugger when you need to.

On the contrary, info logs are always on—they sit at the heart of normal operation. They mark meaningful events you’d want to see in day-to-day usage. The reason you have info logs is to confirm expected behavior.

Error logs exist to highlight failures. They deserve extra context—like what was being attempted, which process was affected, and what might have caused the error. The point of error logs is to guide you toward a fix as soon as you spot them.

How to Write Them Well

As a general rule of thumb, logs should be written for the next engineer.

This means every log message should be self-explanatory, provide value, be packed with context, and consistent with other messages. Here are some guidelines I follow to write good log messages:

Descriptive and Direct

Every log line should explain the event in a short, direct way. This helps anyone scanning your logs see the bigger picture right away.

Include Relevant Data

If an error happens, provide the relevant values. This extra context can save hours when tracing issues. Logs—especially error and debug logs—should allow you to "reproduce" the issue.

Avoid Filler

More log lines aren’t automatically better. Messages that don't add value or provide direct explanation for an operation should be deleted.

Stay Consistent

Pick a format for your messages and stick to it. Consistency makes logs easier to skim and parse. Human brains are great at pattern recognition. When scanning through thousands of log messages, even a tiny change in the format can catch your eye. This can't happen if you are not consistent.

Final Thoughts

Writing efficient log messages is a great practice that directly impacts how quickly issues get resolved. It’s a mandatory extension of your documentation and a critical component of building predictable and maintainable software. By investing the time to write clear, context-rich, and purposeful logs, you're saving your team (and often yourself) valuable hours in debugging and keeping your systems understandable, maintainable, and reliable.

One of the most common pitfalls that I saw people fall into when they start working with K8s is having a misconfigured health probe that prevents the container from stabilising.

It often looks like this:

• The container works fine locally.

• Keeps restarting and failing on K8s with no error messages in the logs.

Understanding what's happening

When you deploy your container to K8s, the K8s engine (kubelet) needs a way to verify it’s actually doing what you’ve defined. When you define a health probe for your pod, K8s uses that probe to ensure the container is running as intended.

Why Health Checks Matter

Beside that, When you deploy an application in Kubernetes, you want to be sure it’s ready to serve requests and stays alive. You might need to do some pre-start work before serving requests like loading a ML model or opening a database connection.

How K8s use health probe

When you deploy a container on K8s. It will run through a state of events that looks something similar to this:

Startup Probe

If your pod takes a long time to start, you’ll need a Startup Probe. This allows Kubernetes to wait until your pod is fully operational. For example, if you are running a JVM-based container or needs time to load an ML model, the startup probe ensures your pod has enough time without being prematurely restarted. At the same time, it tells the kubelet clearly when and if your pod failed to start.

Liveness Probe

The Liveness Probe checks if your pod is still alive. If it fails, Kubernetes restarts the container. This is the endpoint that k8s will keep hitting it to make sure your application is still healthy and keeps your application fresh, eliminating stuck or faulty states.

Readiness Probe

Even if your container is up, it might not be ready to serve requests. The Readiness Probe detects when your application is ready for traffic. If it’s not ready, Kubernetes removes the Pod from the load balancer until it passes the check.

So in the case where your container read some state, load a model or runs a db migration. You need to setup a proper startup and readiness probes that allows your pod the time it needs other wise, k8s will interrupt it and restarts it.

Anatomy of a health probe

Each health probe consist of a few attribute that allow k8s to properly check how, when and how many times it should execute the check.

Attribute Description initialDelaySeconds Seconds to wait before performing the first check. periodSeconds Frequency (in seconds) between checks. timeoutSeconds Maximum time to wait for the probe response before considering it a failure. failureThreshold Number of consecutive failures before marking the pod unhealthy/unready. successThreshold Number of successful checks required to mark a failing pod as healthy again. exec Command executed inside the container to verify health. httpGet HTTP GET request to a specified path and port to check health. tcpSocket TCP socket check on a specified port to verify container availability.

Finally

Misconfigured health probes usually lead to a very frustrating endless restarts and headaches when deploying to Kubernetes. Choosing the correct probe types and setting the correct thresholds ensures your pods have the time they need to start and be ready.

Listening to people talking about their career is such a joy. Understanding the perspective, challenges and journey for other people is a great way to learn about them. This is one of my favorite "small-talks" tricks that helps me build personal connection with people I interact with.

Most recently, I had some discussions with friends, colleagues and mentees to learn more about how they view their career so far and what are they looking for. And I noticed that folks who are a bit earlier in their career have very unrealistic view of what it means to be on a "senior" level (or beyond).

Looking back, I realize how much clarity or understanding I lacked about what being a senior engineer actually involves and how different it is from what I initially thought.

The challenge with moving to a senior role isn’t just about getting better at coding or working more hours. It’s about stepping up in ways that go beyond the technical bit of the job. Most people who gets promoted to a senior position they often assume they’ll work exclusively on the exciting tasks.

While in reality the expectations from your teams gets shifted from relying on you as an execution force to more of a multiplier effect.

This is probably the main point that is usually missed out, being a multiplier effect is net better than being execution force. (In most cases!).

That's because when you’re focused on individual impact, your value lies in how much you can get done on your own. You’re the person who delivers, fixes, and solves problems directly. And there’s nothing wrong with that—every team needs reliable contributors. But the more senior you get the bigger you should think.

On the other hand, if your focus shifts. You’re no longer the one doing all the heavy lifting; instead, you’re championing ideas, mentoring teammates, and helping shape the bigger picture. Your impact goes from the work you do yourself to the ripple effect you create through others. This means setting others up for success, advocating for improvements, and sometimes even stepping back so others can step up.

It’s a mindset shift, and it’s not always easy. But here’s the thing: the multiplier effect isn’t just better for the team—it’s better for your own growth. It allows you to grow on multiple verticals beside the pure technical abilities. It also helps you maximize your impact and influence.

Being a senior isn’t about writing less code or working less. It’s about viewing yourself as a leader who scale the team’s impact while continuing to grow.

If you liked this post. Subscribe to the newsletter for more.

In 2019, I was part of the Statsbomb team and the company was really taking off with it's football collection and analytics platform. We were asked to re-design the data collection platform to support realtime events. This was a big task that got everyone across the tech team (in 2 countries) to have deep discussions about every single UI component, service, database and tool. We had to design the system from the ground up.
Our goal was clear, but we had no idea how to achieve it, there were a lot of options and tons of tooling that could "theoretically" get us there. Therefore, lots of different opinions.
My first idea was to stream changing events from our core DB into a new Kafka cluster and have them delivered to our processing services. Well, looking at this now, it's not such a terrible idea, and if I remember correctly, after all the heated discussions we got this implemented somehow.

The Journey Begins

What I strongly remember are the long and deep discussions I had with team about how we should think about the problem itself.
This was my first real interaction with topics like FP, Domain Driven Design, EDA, Streaming Systems and most of the main areas that I have worked on ever since.

About This Series

In this series of blogs, I'll try to go over one topic at a time and write a comprehensive guide about it and share some resources that I find invaluable.
I'll cover topics such as:

• ✅ Domain Driven Design

• 🔄 Event Driven Architecture

• 🔄 Functional Programming

• 🔄 Reactive Programming

Why Start With DDD?

Specifically, in this blog, I've decided to start with Domain Driven Design. If you look at any SWE bookshelf, you will probably notice a large blue book. This is the DDD book by Eric Evans.

I decided to start with DDD since it builds the foundation for how we can think about building systems that solve problems. DDD sets the scene for how we can translate business domains into software components. Later in this series we will cover different topics that are adjacent and complementary for DDD like event driven architecture, event storming, reactive and streaming systems.

What's Domain-Driven Design

Domain-Driven Design (and the book by Eric Evans) is a set of software design principles and rules that allows you to design and build system based on a solid model of domain understanding.

This is especially important when you are building software for a complex domain that often requires deep experience to understand and is usually messy.

The Problem

When building software, the toughest part is usually getting communication right. You have the business side, product side and engineering side. And depending on your organization size the communication channel between these sides can get really messy.

But the one thing that's common in most cases regardless of the organization size is that each side of these has a different understanding and view of the problem and the solution.

Often, there are a wide range of reasons why software projects fail. I tried to capture it in this visualization:

Domain Driven Design tries to solve some of the problems in this Venn diagram. It sets a set of rules that helps the engineers build on a modeled understanding of the domain provided by the business and product sides.

The Approach

Looking at the Venn diagram, you can see that a lot of problems come from misalignment and communication gaps. DDD tries to tackle this by establishing a common ground between all sides.

Instead of having engineers build what they think is right, product people designing what they think users want, and business folks trying to explain complex domain knowledge - DDD introduces a systematic approach to get everyone on the same page.

Domain

This word gets thrown around a lot in tech (like many other words), but what does it actually mean? Think about it this way - if you're building a system like Airbnb, your domain isn't just about APIs or databases. It's about understanding how hospitality works, what makes a great guest experience, how hosts manage their properties, and what makes people trust staying in a stranger's home. Your domain is basically your problem space - it's what your business actually does, not the technical implementation of it. This is why engineers who deeply understand their business domain are usually more valuable than those who just write code (even good code).

Ubiquitous Language

This is probably my favorite part of DDD, and I wish I knew about it earlier in my career. The idea is simple - everyone involved in the project should speak the same language.

No more meetings where business calls it "user engagement" while engineers call it "activity_metric" and product calls it "interaction index". When teams use different terms for the same concept, it creates confusion and wastes time in translations. I've seen this happen countless times especially in data projects where data scientists, engineers, and business analysts all use different terms for the same metrics.

Implementing this requires a larger effort than just aligning on names. This means that your system should be designed using the common language. Each service, repo, function, etc. should have a business meaning.

A Must Read if you haven't already.

Model-Driven Design

This is where things get interesting. Once you have your domain understanding and common language, you start modeling your software around it. But unlike traditional approaches where we often model around technical constraints, in DDD you model around domain concepts. For instance, when building a real-time analytics system, instead of thinking about database tables and message queues first, you focus on modeling what an "event" actually means in your domain. How does it change? What rules govern it? This helps make better technical decisions later because your foundation is solid.

Design Principles

After understanding the core ideas behind DDD, we can dive into how to actually implement these ideas. Since DDD pushes us to cover the 2 major areas for any technical project (design and implementation), We can split domain-driven design principles into 3 layers.

Strategic vs Tactical Principles

(DDD) consists of two main design approaches: Strategic and Tactical. Here's the key difference:

Aspect Strategic Design Tactical Design Focus Big picture system organization Implementation details Components Bounded Contexts, Context Maps, Subdomains Aggregates, Entities, Value Objects, Events Scale System architecture Within single bounded contexts Who Uses It Architects and business stakeholders Technical teams Goal Define system boundaries and interactions Implement domain logic and rules

Strategic Design

Strategic design is about understanding the big picture of your system. It's about defining the boundaries of your system and how different parts of your system work together.

This is the solution space and how you can design your services to work together. This is not a technical debate of which tools you should use, but rather it's a modeling approach of how you can design your system to actually reflect the domain.

Here's the key concepts:

1) Bounded Contexts

I like to think of bounded contexts as the building blocks of the system. Take Uber for example - you have the ride-hailing context, the payment context, and the driver management context. Each of these contexts has its own rules, its own data model, and even its own language. The beauty of bounded contexts is that they help you handle complexity. Instead of building one massive system where everything is connected to everything else (we've all been there), you build smaller, focused systems that communicate through well-defined interfaces.

And don't get me wrong, this doesn't mean you can only implement this in a microservices architecture. The idea is about designing the solution space, it's a higher level than the actual system design. You can take this solution space and implement it in any software setup. Monolithic, Microservices, Event-Driven, etc. Even a local desktop application can follow this approach.

2) Context Mapping

Now we've split the system into bounded contexts, you need to figure out how they talk to each other. Context mapping is basically documenting these relationships. This is where you define the boundaries of your system and how different parts of your system work together. In general, we are trying to define the business relationships between the bounded contexts. You might have contexts that share data, contexts that translate between different models, or contexts that need to stay in sync. Understanding these relationships helps you build better integration points and avoid the "everything is connected to everything" mess.

3) Subdomains

Subdomains are the natural divisions in your business domain. It's what differentiates your business from the competition. You usually have:

• Core subdomains (the stuff that makes your business unique)

• Supporting subdomains (important but not unique)

• Generic subdomains (stuff everyone needs)

For instance, Netflix's core subdomain would be their content streaming and recommendation system. This is what gives them edge over their competitors. Their billing system? That's probably a supporting subdomain - important but not what makes them special.

Tactical Design

After laying-out the strategic design (High Level), we usually dive into the tactical design (Implementation Details). This is where we actually start building the system.

Like Strategic Design, Tactical have its own key concepts:

Entities & Value Objects

Entities and Value Objects are similar to classes in OOP, functions in FP, Pods in Kubernetes, etc. You get the idea.

These are the foundational building blocks in DDD. The key distinction lies in how we treat their identity and lifecycle. Entities maintain their identity throughout their lifecycle. No matter what changes happen to their attributes, they're still the same entity. This is not just a technical concept - it's deeply rooted in how we naturally think about things in our domain. When domain experts talk about something that maintains its identity despite changes, that's probably an entity.

Value Objects are a bit more interesting. They have no identity of their own - they're defined entirely by their attributes. When you modify a Value Object's attributes, you're effectively creating a new Value Object. This immutability isn't just a technical choice - it reflects how these concepts work in the real world. Value Objects are perfect for representing domain concepts where the values themselves matter more than any sense of identity.

The real power of Value Objects shows up when you start using them to encapsulate domain rules. Instead of having business logic scattered throughout your codebase, you can bundle it right where it belongs. This makes your code not just cleaner, but also more aligned with the business and product side.

Aggregates

When you study DDD, aggregates are one of the most confusing concepts. But when you think about it from the business side, aggregates are just any group of entities that are tightly coupled. Martin Fowler puts it this way:

Aggregate is a pattern in Domain-Driven Design. A DDD aggregate is a cluster of domain objects that can be treated as a single unit. An example may be an order and its line-items, these will be separate objects, but it's useful to treat the order (together with its line items) as a single aggregate. An aggregate will have one of its component objects be the aggregate root. Any references from outside the aggregate should only go to the aggregate root. The root can thus ensure the integrity of the aggregate as a whole. Aggregates are the basic element of transfer of data storage - you request to load or save whole aggregates. Transactions should not cross aggregate boundaries. DDD Aggregates are sometimes confused with collection classes (lists, maps, etc). DDD aggregates are domain concepts (order, clinic visit, playlist), while collections are generic. An aggregate will often contain multiple collections, together with simple fields. The term "aggregate" is a common one, and is used in various different contexts (e.g. UML), in which case it does not refer to the same concept as a DDD aggregate.

Source: Martin Fowler

Designing aggregates

The thing that took me years to really get is that aggregates should be way smaller than what we initially think. We often try to model too much in a single aggregate because it feels safer to have everything transactional (i.e. changes affect everything immediately), like we're ensuring consistency across everything. But that's a trap - it leads to these massive objects that are hard to work with and even harder to keep consistent (Eventually a massive tech debt). And you later realize that you are not actually benefiting from using aggregates.

In fact, aggregates are all about transactional consistency. If two things need to be consistent with each other all the time, they probably belong in the same aggregate. If eventual consistency is acceptable, they probably don't. This simple rule has saved me from creating monster aggregates many times. One of the things that you have to push when talking with the business side about entities is that if transactional/strong consistency is really required. They usually think that, but when you understand the problem and the domain, most of the time, eventual consistency is enough.

Since aggregates are a set of entities together, it's important to think about an aggregate's design as a tree of entities. (Similar to how a set is implemented, right?).

Events

Events are probably the most popular (and widely-used) topic from DDD. I'll cover events in detail in a different blog post since they span to a wide range of technical topics. For now, it's important to understand that events in DDD represent a log of what happens in the domain and it's objects. If you scan the logged domain events you have a history of what happens.

Events are basically every where. If you login the netflix app. that's an event. Clicking on a new movie to watch, that probably spans a few events.

Repositories & Factories

These two patterns are probably the most practical and straightforward concepts in DDD. They're designed to handle different aspects of working with domain objects - how we store them and how we create them.

You probably used those thousands of different times. In ORMs, Logging packages, API Frameworks, literally everywhere. It'

Repositories Repositories represent how we store our objects. And no, we are not talking about which database to use. It's more about how you separate the concept of storing our entities. It's more about how do you separate concept of storing our entities and objects from the details of how to actually do it. Repositories are focused on the former, They provide a simple way to store and retrieve domain objects while hiding all the technical details of how this actually happens.

The main value of this if you haven't guessed it already is to keep your domain entities logic clear since you don't want to clutter the domain logic with infrastructure details.

Factories

Factories solve a different problem - they handle object creation when it gets complex. And I don't mean complex in terms of technical complexity. I'm talking about business complexity, where creating a valid domain object requires understanding and applying business rules. When you study DDD, factories might seem like overengineering. And for most cases, they are. But once you work with complex domains, you start to appreciate them. They shine when creating an object requires business rules, multiple steps, or coordination between different parts of your system.

Services

Services in DDD are pretty simple to understand - they handle operations that don't fit into entities or value objects. If you are building a bank and trying to design the logic of moving money from one account to another, this is not the responsibility of either account. This is a domain operation that happens for both of them. Services cover this kind of business operations. It has to be stateless and represented with a business name from the ubiquitous language.

They are different from the backend services we are familair with them. A domain service can be a function, a class, an API, basically any implementation that fits the use case.

Now that we covered most topics in DDD, one of the most common mistakes that I made since I learned it is trying to apply it everywhere. I'm a big fan of DDD, but it's not a silver bullet nor is it easy to do. In fact, domain-driven design is a fantastic framework when you are trying to solve a business problem or implement business rules. I've seen it work very well in that case, even though it takes a lot of time, requires deep understanding of the business, and has a steep learning curve.

But in data systems or data-intensive applications, it doesn't work at all. Because in those types of systems, you are more concerned about managing the data than the logic. This is where the efficiency and correctness of the system is your concern, not the logic.

But the reason I like DDD so much is that you don't have to do it all the way. You can use any of its concepts on its own. For instance, using a ubiquitous and common language is usually a good practice in all cases. The event-driven architecture philosophy is an adjacent topic for DDD - if you understand EDA, you're most likely familiar with DDD.

Resources

Here are some resources to learn more about DDD

• The Domain-Driven Design Book

• Martin Fowler: DDD

• Awesome DDD

I've read this book a few times, and I always recommend it to people whenever we talk about leadership.

Introduction

The book talks about Mr. Ancelotti's leadership style, how he works with different people, how he influences them, and how he manages the team, especially during tough times.

He introduced the concept of Quiet Leadership -- a cool and calm way of management that allows people to do their best work by supporting them with his hands-off approach that enables people to be more comfortable and creative.

The Leadership Arc

In his book, Mr. Ancelotti talked about how he used the concept of leadership arc. It's a concept that resonated with me very well, especially looking back at my career so far.

The Courtship Phase: when you are just starting out a new job. People are excited and eager to have you on board. You are looking at all the opportunities and potential to make a difference.
Then comes the Honeymoon Phase: when everyone gives you the time to establish your presence.

After that comes the Success and Stability Phase: when you are now part of the team and are at the top of the curve.

Later comes the Breakup Phase: when you start feeling out of sync and decide to leave.

Or as Mr. Ancelotti put it:

"Sometimes you leave on your own terms, sometimes you don't. That's football, just as it is in business."

Looking at this, it describes almost every job I've had. Especially being an early engineer of two exited startups. You get to experience the change from just a small team that's working closely together doing tons of different things to a large corporation with complex hierarchy and politics.

Quiet Observations

The book described a core pillar of quiet leadership: Quiet Observations. Mr. Ancelotti stressed the importance of quiet observations before actions. This is one of the most important traits in great leaders. There are a lot of managers and leaders that jump to reactions immediately. They don't take the time to observe and understand the situation.

Personally, I learned this the hard way. Early in my career, I was known for having a short fuse and jumping to reactions immediately. This was usually very tiring and stressful not only for me but for the people around me.

Most of the mentors that I asked about this problem recommended Crucial Conversations. I've read this book twice, and I totally understand why it's a bestseller.

This is the kind of book that HR professionals would recommend. It has a lot of shortcomings. It uses language that if a normal human used it in a "crucial conversation" would be very awkward and pretentious.

In fact, the Quiet Observation concept is by far much more powerful than what this book offers.

Calm in the Chaos

Mr. Ancelotti -- beside all the trophies and the astonishing career -- is known for his calm and composure. He is a massive example of how leaders should stay calm under pressure, especially in high-stakes moments.

In the book, he stated how he uses clear and concise thinking over emotional reactions. While in my opinion, this is a hard emotional regulation skill to master, it's a great one that really affects any leader's ability to make the right decisions.

He also stressed how important it is to set up a transparent and blame-free environment that's focus-oriented. While this usually is hard to set up, especially in large companies, it's a great skill to even apply at a micro-level.

That being said, I've seen people take advantage of such an environment. Which in my opinion is what makes striking the right balance between a blame-free environment and responsible environment while being calm the hardest part of it.

The Quiet Leader

In the movie "The Godfather", Don Vito Corleone was a great example of a quiet leader. He was a great listener and observer; he never reacted to the situation. He always let the situation unfold and then he made his move. Mr. Ancelotti mentioned this as proof that while some people might think that a quiet leader is a weak one, it's actually the opposite. As he beautifully put it:

"When you watch Vito Corleone in The Godfather, do you see a weak, quiet man or do you see a calm, powerful man in charge of his situation?
My approach is born of the idea that a leader should not need to rant or rave or rule with an iron fist, but rather that their power should be implicit. It should be crystal clear who is in charge, and their authority must result from respect and trust rather than fear."

The book also explained how some mentors tried to argue this point and stressed that being tough and strict as a leader is the only way to be respected.

This is actually a very common opinion in the tech industry, especially if you are an IC. People usually choose being tough, strict, and sometimes rude as the way to be respected and enforce boundaries.

I've seen engineers that use their position to gatekeep and make things harder for other teams. This is probably more common than you would think. But I remember a friend of mine once told me that "Every other Principal engineer that he worked with was super rude with very high ego".

I don't think this is a good way to live. I would think approachability and quiet leadership is a much better way to enforce boundaries and get things done while being loved and respected.

And while some might argue that the first way is a more effective way to keep your job and keep people in line, I would think that the second one is a much more sustainable way to lead.

I would rather leave a job where people were happy to work with me than stay while people wish I was gone. And while people usually argue this with the job security point, I would imagine that good leaders (especially in tech) would more or less find a way to keep their job or find a new one.

Adaptability

The book also talked about how leaders should be able to adapt to different situations. This touches on the last point I mentioned. As a leader, I think it's very important to be able to adapt to different situations and setups. It's a hard-earned skill but it allows you to face different challenges and situations without your mental or social capacity taking a big hit.

Conclusion

This book is on the top of my most favorite books. I've read it a few times and I always learn something new from it. It resonates with me very well and I think it's a great book to learn from. Specially when you see Mr. Ancelotti's calm presence, you usually ask your self "how can I be more like him?".

I would highly recommend this book to anyone who is looking to learn how great leaders think and act. As Mr. Ancelotti said in the book:

"Leadership can be learned but cannot be imitated."

I think there are great lessons to be learned from this book. It's not an easy task but with time and practice it could be a natural skill to develop.

Footnotes

The quotes used here are from the book "Quiet Leadership: Winning Hearts, Minds and Matches" by Carlo Ancelotti. All rights reserved to the book author(s).

Last week, I had a talk with one of my mentees and we touched on some authentication concepts. And I realized that while the idea of service accounts might be straightforward to some, it usually takes some time before you grasp how they actually work.

Introduction

You are starting your day and you just typed in your username and password for your cloud console (or maybe through your company SSO). And now based on your permission, you can do certain actions. So you go for the buckets service, and you can view some bucket and you try to delete a bucket and...
shCopyError deleting bucket...Permission Denied!
The reason you're seeing this is that your cloud admin gave you a specific permission that allows you to view and edit buckets, but never delete it. Obvious, right?
Service accounts aren't much different. You create "accounts" for each service with certain permissions.

Identity and Access Management (IAM)

Before diving deeper into service accounts, let's understand IAM - the system that makes all of this possible. IAM is like a security guard at a building who checks IDs and knows exactly what areas each person can access.
In cloud environments, IAM serves two main purposes:

• Authentication: Verifying who you are (or which service is making the request)

• Authorization: Determining what you're allowed to do

When you logged into your cloud console, IAM verified your identity and checked what permissions you have. These permissions are usually organized into roles - collections of permissions that make sense together. For example, a "Storage Viewer" role might include permissions to list and read buckets, but not modify them.

Service Accounts

Cloud Service Accounts

Now you have to write up a new script that will read from the bucket and write it to your local disk. But you realize that you needed to login with your credentials in order to view the bucket.
Here's where service accounts are needed. You need to grant a specific service certain permissions in order for the service to execute the task.
So when you use the service account to authenticate your service, it will go through a very similar workflow like the one you do when you login. It will use the service account provided to talk first to the IAM Auth services to pull credentials and permissions. Then it will use those to talk to the cloud APIs (in our example, it will read and download data files from the bucket services).

The beauty of service accounts in cloud environments is that they're designed for programmatic access. Instead of username/password combinations, they use keys and tokens that can be easily rotated and managed programmatically. This allows you to have a fine-grained control over what your service can do. And you can easily rotate the credentials when they expire.

And all cloud providers have a predefined set of roles that you can use for almost all scenarios. From experience, the less "custom" roles you have, the better and cleaner access you have.
Most likely, you will need to assign multiple roles to a service account. But this is better than having a single role with a lot of permissions.

Kubernetes Service Accounts

Inside Kubernetes, service accounts work similarly but with some key differences in implementation. When a pod starts up in your cluster, it automatically gets assigned a default service account unless you specify one. This service account determines what API operations that pod can perform against the Kubernetes API server.
For example, if you have a monitoring pod that needs to list all pods across namespaces to collect metrics, it would need a service account with permissions to perform those list operations. The workflow looks like this:

Pod starts up and receives the service account token through an automatically mounted volume
When the pod needs to talk to the API server, it uses this token for authentication
The API server validates the token and checks the service account's permissions
Based on the permissions, the request either succeeds or gets denied

What makes Kubernetes service accounts special is how tightly they're integrated with the cluster's role-based access control (RBAC) system. You define what a service account can do by binding it to roles or cluster roles.

The Bridge: Workload Identity

Now here's where things get interesting - what happens when your pod running in Kubernetes needs to talk to cloud services? You have two different authentication systems that need to work together:

• Kubernetes service accounts: for in-cluster authentication

• Cloud service accounts: for cloud API authentication

This is where workload identity comes in. It creates a trust relationship between your Kubernetes service accounts and cloud service accounts. Instead of storing cloud credentials in your cluster (which can be a security risk), workload identity allows pods to acquire cloud credentials dynamically based on their Kubernetes service account.
Here's how it works:

• You configure a mapping between a Kubernetes service account and a cloud service account

• When a pod starts up with that Kubernetes service account, it receives a special token

• The pod can exchange this token for temporary cloud credentials

• The pod can now make authenticated calls to cloud APIs using these credentials

This approach has several benefits:

• No need to manage cloud credentials manually (which is a very big risk)

• Fine-grained control over which pods can access what cloud resources

• Clear audit logs of which workloads are accessing cloud services

Best Practices

Now, to the most common question I always get. What's the best practice for structuring the service account when you have a microserives system that does a lot of cloud interactions?
-- The answer is to use the Principle of least privilege: Give service accounts only the permissions they absolutely need to function. And use Policies and RBAC to abstract and assign required access for service accounts.

Final Thoughts

Service accounts are a fundamental building block of cloud native security. Whether you're working with cloud providers directly or through Kubernetes, understanding how service accounts work helps you build more efficient systems.
With cloud platforms, it's now much more straightforward to build secure systems. Using tools like IAM, Service Accounts and Workload Identity, you can build a zero-trust system with very little effort or security knowledge.

In the previous post, we covered transactions, ACID properties, and isolation levels. We discussed how databases handle concurrent transactions and the different isolation levels available. But we didn't dive into how Postgres actually implements these features.

This post will cover Multi-Version Concurrency Control (MVCC), which is the mechanism Postgres uses to implement transaction isolation. We'll see how Postgres maintains multiple versions of data, how it decides which version a transaction should see, and how it cleans up old versions.

What's MVCC?

Before we dive into MVCC, let's understand why we need it. In the previous post, we talked about locks and how they help manage concurrent access to data without partial failures. But locks have a significant drawback - they make readers wait for writers and vise versa.

Imagine you're building a content platform. You have a post with millions of views, and you want to update the post's content. With a simple locking mechanism, you'd have to:

1. Lock the post

2. Update it

3. Release the lock

During this time, no one can read the post. This is terrible for performance and user experience. This is where MVCC comes in.

The Core Idea

Instead of making readers wait for writers, MVCC maintains multiple versions of the same data. When you update a row, Postgres doesn't overwrite the old version. Instead, it creates a new version and keeps the old one around.

This means:

• Readers can see a consistent snapshot of data without being blocked by writers

• Writers can modify data without blocking readers

• Different transactions can see different versions of the same data based on when they started

How MVCC Works in Postgres

Transaction IDs

Every transaction in Postgres gets a unique identifier called a Transaction ID (XID). This is just a 32-bit integer that increases monotonically. When a transaction starts, it gets an XID that's greater than all currently active transactions.

XID Lifecycle

Transaction IDs in Postgres follow a lifecycle:

• Assigned when transaction starts

• Active while transaction runs

• Moves to committed/aborted state after completion

• Eventually becomes "frozen" (more on this below)

Transaction ID Wraparound

Since XIDs are 32-bit integers, they can eventually wrap around (Resets to 0).

This can cause a problem where a new transaction that just started can have an XID smaller than the recent transactions' XID. This can cause a problem with the visibility semantics (discussed below) and mess all of the isolation levels.

Postgres handles this through treating XIDs as a circular linked list, this allows the comparison operators to handle wraparound transparently:

• Counter starts at 0

• Moves clockwise through safe zone (green)

• Warning zone at ~1B transactions (yellow)

• Critical zone requiring VACUUM (red)

• Final approach to max value (blue)

• Wraps back to 0

If the counter reaches the red zone, Postgres will perform a VACUUM operation to clean up old versions of data. On the other hand, if the number of transactions is small, the wraparound use reset points to reset the counter to a safe value.

Freezing

Very old transactions are marked as "frozen"
Frozen tuples are visible to all transactions
Prevents age-related visibility issues

Let's look at a simple example.

• Transactions gets an XID.

-- Transaction 1 (XID: 100)
BEGIN;
UPDATE users SET status = 'active' WHERE id = 1;
COMMIT;

-- Transaction 2 (XID: 101)
BEGIN;
UPDATE users SET status = 'inactive' WHERE id = 1;
COMMIT;

Each row in Postgres has additional system columns that aren't visible by default:

• xmin: XID of the transaction that created this version

• xmax: XID of the transaction that deleted/updated this version

• cmin: Command identifier for the transaction that created this version

• cmax: Command identifier for the transaction that deleted/updated this version

Example:

BEGIN;  -- Transaction starts (XID: 100)

-- Step 1: Create user
INSERT INTO users (id, name, status) VALUES (1, 'John', 'pending');
-- Row: (1, 'John', 'pending')
-- xmin: 100 (creating transaction)
-- xmax: NULL (active version)
-- cmin: 0 (first command)
-- cmax: NULL (not deleted)

-- Step 2: Update status
UPDATE users SET status = 'active' WHERE id = 1;
-- Previous Row: (1, 'John', 'pending')
-- xmin: 100
-- xmax: 100 (marked by same transaction)
-- cmin: 0
-- cmax: 1 (updated by second command)

-- New Row: (1, 'John', 'active')
-- xmin: 100
-- xmax: NULL (active version)
-- cmin: 1 (second command)
-- cmax: NULL (not deleted)

-- Step 3: Update User name
UPDATE users SET name = 'Johnny' WHERE id = 1;
-- Previous Row: (1, 'John', 'active')
-- xmin: 100
-- xmax: 100 (marked by same transaction)
-- cmin: 1
-- cmax: 2 (deleted by third command)

-- New Row: (1, 'Johnny', 'active')
-- xmin: 100
-- xmax: NULL (active version)
-- cmin: 2 (third command)
-- cmax: NULL (not deleted)

-- Step 4: Delete user
DELETE FROM users WHERE id = 1;
-- Previous Row: (1, 'Johnny', 'active')
-- xmin: 100
-- xmax: 100 (marked by same transaction)
-- cmin: 2
-- cmax: 3 (deleted by fourth command)

COMMIT;

Command # Operation Data xmin xmax cmin cmax 1 INSERT (1, 'John', 'pending') 100 NULL 0 NULL 2 UPDATE (1, 'John', 'active') 100 100 0 1 3 UPDATE (1, 'Johnny', 'active') 100 100 1 2 4 DELETE (1, 'Johnny', 'active') 100 100 2 3

Transaction Snapshots

When a transaction starts, Postgres creates a snapshot of the database's state. This snapshot contains:

• The transaction's own XID

• All currently active transaction XIDs

• The latest committed transaction XID

This information helps Postgres decide which versions of rows a transaction should see. It's like taking a photo of the database at a specific point in time.

Why Snapshots Matter

The snapshot mechanism is crucial for maintaining transaction isolation. Without it, transactions would see different data as other transactions commit changes, leading to inconsistency and unpredictable behavior. This is particularly important in applications that require strict strong consistency.

What's in a Snapshot?

A transaction snapshot in Postgres contains three essential pieces of information. These three pieces of information are used to determine which version of a row a transaction should see.

1. The Transaction's Own XID

• Every snapshot knows its transaction's identifier (XID)

• XID is used to determine if changes are made by the current transaction

• Helps implement "read your own writes" behavior

2. All Currently Active Transaction XIDs

• List of all transactions that were running when the snapshot was taken

• Used to determine which changes should be invisible

• Helps prevent reading "in-progress" modifications

3. Latest Committed Transaction XID

• The most recent transaction that was committed when the snapshot was taken

• Sets an upper bound for visible changes. Since XID is monotonically increasing, it helps quickly filter out future modifications

• Helps implement "repeatable read" behavior

When a transaction starts, Postgres creates this snapshot and uses it throughout the transaction's lifetime. The snapshot never changes, even if thousands of other transactions commit while yours is running. This is what enables consistent reads.
Think of it like taking a picture of a whiteboard. Even if someone changes what's written on the physical whiteboard, your picture stays the same. This is exactly how Postgres maintains consistency - your transaction keeps seeing the database as it was when it started.

Visibility Check Rules

Now comes the interesting part - how does Postgres decide which version of a row a transaction should see?

Postgres uses the information in the snapshot to determine which version of a row a transaction should see. It uses the Visibility Algorithm.

A row version is visible if:

• It was created by a transaction that committed before our snapshot. (xmin < snapshot.xmin)

• It wasn't deleted by any transaction visible to our snapshot. (xmax > snapshot.xmin)

This ensures that a transaction only sees data that has been committed and not deleted by other transactions.

Vacuum Process

Those algorithms are great and very efficient but what happens to old versions that no one needs anymore? This is where VACUUM comes in.

VACUUM is the process that cleans up old versions of data. It removes dead tuples and frees up space. It's very common in the use cases where data is being updated frequently. Otherwise Postgres disk usage will grow indefinitely even if the data is deleted.

Dead Tuples

When you update or delete a row, Postgres doesn't immediately remove the old version. These old versions are called "dead tuples". They stick around because some long-running transaction might still need to see them.

-- This creates a dead tuple
UPDATE users SET status = 'inactive' WHERE id = 1;

Dead tuples take up space and can slow down queries because Postgres has to skip over them. This is where VACUUM helps.

How VACUUM works

VACUUM is a process that:

1. Scans tables for dead tuples.

2. Marks space occupied by dead tuples as reusable.

3. Updates indexes to remove references to dead tuples.(The most expensive part)

4. Updates the free space map.

Postgres also has an AutoVacuum process that runs automatically when:

• A table has accumulated enough dead tuples

• The table has grown significantly since the last vacuum

AutoVacuum is a background process that runs periodically and cleans up dead tuples. It's very efficient and doesn't block the main database operations, but it's not perfect. It can miss some dead tuples and it's not always able to clean up all the dead tuples. There's also a risk of running out of disk space if the vacuum process is not able to clean up dead tuples fast enough to keep up with the data growth.

Final Thoughts

MVCC is an advanced topic and takes a lot of effort in order to implement it correctly. It's not something that you can just throw together. It's a very complex system that requires a lot of thought and planning.

Recently, I've faced a problem where a database table that has very high write throughput had grown to 5x the size of the data even with AutoVacuum running. The problem was that the table was not designed to support HOT (Heap-Only Tuples) updates, specifically that the columns that was being updated frequently was part of multiple indexes.

There's also a lot of scientist and engineers who oppose the use of MVCC in the database world. They argue that it's not a good idea to have multiple versions of the same data and that it's not a good idea to have a system that's not ACID compliant.

This blog post is a very good detailed read about the problems with MVCC and how it's not a good idea to have multiple versions of the same data (in some use cases) and how it's not a good idea to have a system that's not ACID compliant.

In the previous post, we discussed how to study databases and the resources to use. In this post, we will dive into the core of databases: Transactions.

The reason I'm starting with transactions because they are (in my opinion) the most important concept in databases. When you interact with database, you don't need to understand the ins and outs of that database. Most people don't anyway. But in order to write a software that have a predictable behavior and is efficient, you need to understand what the database provides you and what you should expect from it.

This post will cover how transactions work (from the user perspective), What are the ACID properties and why they matter.

And then we will cover Isolation Levels and Concurrency Control, I think this has to be hand-to-hand with transactions in order to understand how they work and build a full example.

The basics

What's a Database?

Before I dive into transactions, I wanted to define what's a database on a very high-level. While an actual database system (like Postgres) is a very complex software. The very simple of view of a database would be a service that allows you to store, retrieve and iterate over data. For instance, if you store a bunch of CSV files on desk and use Python to read and write them, that's a database system.

But if you take this analogy and try to understand it in a more technical way, you will find that you have a few components:

1. Server (aka. Database Server): This will be the front-end component of our system. It's responsible for all user's interactions.

2. File System (aka. Storage Engine): This is the piece that handles how we read and write data on disk.

3. Query Processor: This is the piece that connects the server to the file system. It's responsible for translating user's query (with all of its type). And run the appropriate operation on data on disk.

This makes up a very simple yet complete database system. Right? Well, theoretically yes, but the actual implementation is far more complex than this. But we don't need to understand all of the implementation details in order to be a power user of a database system. This analogy I think is enough to get us started.

What's a Query?

So we established that we have data on disk, we have a front-end API to interact with the data and we have a way to translate that interactions to actual operations and algorithms. And this interactions are what we are focusing on.

Database Queries defines how we should interact with the database. And it's where you define the database behavior in order to build a predictable software.

Queries varies in types, and there are lots of query languages out there. But since we are focusing on Postgres, anywhere in this series where we mention queries, we are talking about SQL queries.

Queries Types

Since queries are used to describe what we need from the database and how the database should behave, we are specially interested in two types of queries:

1. READ Queries: These queries allow us to read specific data from the database. We can read data as is. Or we can do some operations and transformations on the data before we read it in order to get it in a specific shape.

2. WRITE Queries: These queries allow us to add new data to the database. We can add a new row to a table, or we can update an existing row.

The interesting (and sometimes tricky) part is when you have multiple queries happening at the same time. From different and similar types but touching the same row(s). Understanding these basic blocks of queries and transactions helps us grasp why transactions are so important. When you have multiple users or applications all trying to read, write, update, or delete data at the same time, you need a way to keep everything organized and consistent. And depending on your application's scope, this can have major implications.

Transactions

From a very high-level, a transaction is a way of grouping together a set of queries so they could be executed together. This simple definition abstracts away all the details (which we will try to uncover) from the database user.

A single block of work

Everytime you read about transactions, people explain the bank transfer problem. So I'm sticking with this!
Imagine you're building a banking app, and you need to transfer $100 from Adam's account to Alice's. Sounds simple enough, right? But this is actually where things gets a bit tricky.

When you think about it, it's a simple process. Deduct $100 from Adam's account and add $100 to Alice's. But then you realize, it's never that simple, right?

Computer goes down, network goes down, someone might unplug the internet cable while the transfer is in progress. There's infinite possibilities of things going wrong. This means, we need an efficient way to provide us with some sense of safety.

Transactions is what does this. When you think about this operation as a transaction, the worst case scenario is partial failure. Meaning, we deduct $100 from Adam but the money never reached Alice. With a simple transaction, the database guarantees us that both operations will be executed as a single block of work. So no matter what happens, both operations will be executed or none of them will.

This feature of transactions is what people refer to as Atomicity.

A consistent and permanent state

And since we talked about things going wrong. We still need a predictable guarantee that if our transaction succeeds, the changes will reflect. Whether this happens in sub-millisecond or hours, we need that to be predictable and permanent.

This means that the database should provide us with a way to configure our transaction to ensure changes are permanent and consistent.

Durable

This brings us to another important aspect of transactions - durability. When a transaction completes successfully, we need to be confident that those changes are permanent, even if our database crashes right after. It's like saving an important document - you want to be sure your work is actually saved, not just floating somewhere in temporary memory.

In our banking example, once we confirmt a successfull transaction. This change is saved forever, even if the database went down and up again. this transaction is still safe.

Understanding ACID Properties

You've probably heard the term "ACID" thrown around when people talk about database transactions. But the problem with this topic is that most people think they only need to memorize these properties as abstract concepts (mostly to answer questions in an interview).
But when we talk about database transactions, ACID properties are the backbone of what makes them reliable and predictable. And this was the core debate a few years ago between the SQL and NoSQL communities. Since if you don't have strong ACID properties, you don't really know what you will get when you query your database. (Though in the recent years, NoSQL databases improved in this area a lot.)

Atomicity (All or Nothing)

Atomicity might sound like a complex term, but it's actually a very simple (and powerful) concept. Think of it as an "all or nothing" guarantee. When you mark a set of operations as a transaction, the database promises that either all of them will succeed, or none of them will happen at all.

Back to our banking example. Imagine you're building a financial app that handles investment portfolios. When a user wants to rebalance their portfolio, they might need to:

• Sell some stocks

• Wait for the sale to clear

• Buy different stocks with the proceeds

• Update their portfolio record

Without atomicity, you could end up in situations where:

• The stocks are sold but the new purchase fails

• The purchase succeeds but the portfolio record isn't updated

• Some operations succeed while others fail, leaving the system in an inconsistent state

  

Atomicity prevents these partial failures. The entire rebalancing operation either completes successfully or rolls back entirely. If anything goes wrong at any step, all changes are undone automatically. This is why we often say transactions are "rolled back" - it's like rewinding a tape to the beginning if something goes wrong. (Think a time machine)

Consistency (Following the Rules)

Consistency ensures that a transaction can only bring the database from one valid state to another valid state. It's like having a set of rules that must always be followed.

While consistency is a very large topic on its own, and some people argue that it's only part of the ACID transaction to complete the acronym. But Generally speaking, Consistency is set of rules that ensure that every transaction must take your database from one valid state to another valid state.

Consistency ensures that no transaction can break the DB rules, even if multiple transactions are running at the same time. If any rule would be violated, the entire transaction fails and rolls back (because of the atomicity property)

For example, if we have a rule that account balances can't go negative, a transaction that would make a user's balance negative should fail completely. The database won't let us break our own rules.

Isolation (Working Independently)

Isolation is probably the trickiest ACID property to understand. It determines how transactions deal with each other when they're running at the same time.

Imagine you and your friend are both trying to book the last seat on a flight at exactly the same time. Isolation ensures that only one of you can get that seat - the other person's transaction will have to fail or wait.

Different isolation levels provide different guarantees about what concurrent transactions can see. We will touch more on this below.

Durability (Changes that Stick)

We covered this briefly earlier - durability means that once a transaction is committed, it stays committed. Your changes are permanent and survive any subsequent system or power failures.

It doesn't just mean the data is saved - it means it's saved in a way that can survive various types of failures and can be recovered correctly when the system restarts (We will cover disaster recovery in a later post).

Isolation & Concurrency Control

Now that we understand what transactions are and what guarantees they provide, let's talk about how databases handle multiple transactions happening at the same time.

Why Concurrency Control Matters

Assume you're building a realtime data pipeline that handles user activity data from multiple sources. Your system needs to:

• Ingest events from various services.

• Process and aggregate metrics.

• Generate reports.

Without concurrency control, you will most likely hit a lot of data race problems, this can cause:

• Inconsistent aggregations because metrics are being updated while being read

• Lost updates when multiple processes try to modify the same user profile

• Incorrect reports because they're running on data being modified

This is why databases implement sophisticated concurrency control mechanisms. They ensure that concurrent operations work together in a predictable way that allows us to maintain data consistency.

The Building Blocks: Locks

At the heart of concurrency control are locks. Think of locks like access controls in a secure facility - they manage who can access what resources and when. In databases, we have different types of locks:

1. Shared Locks (Read Locks)

These are like multiple analysts reading from the same dataset. Multiple transactions can hold shared locks on the same data simultaneously. For example, when generating different reports from the same dataset, multiple processes can read simultaneously without interference.

2. Exclusive Locks (Write Locks)

These are like a data cleanup job that needs exclusive access to fix inconsistencies. Nobody else should be modifying or even reading the data until the cleanup is complete. When a process needs to modify data, it requires an exclusive lock.

Understanding Isolation Levels

As mentioned before, Isolation is probably the most subtle and complex of the ACID properties. It determines how changes made by one transaction affect other concurrent transactions.

1. Read Committed

This is Postgres's default isolation level, providing a practical balance between consistency and performance.
Under Read Committed:

-- Process 1: Calculating hourly metrics
BEGIN;
SELECT SUM(events) FROM metrics WHERE hour = '2024-02-01 14:00';  -- Sees 1000 events
-- Meanwhile, Process 2 backfills some missing events
SELECT SUM(events) FROM metrics WHERE hour = '2024-02-01 14:00';  -- Now sees 1200 events
COMMIT;

This is usually acceptable for metrics and analytics where small inconsistencies are tolerable.

2. Repeatable Read

Repeatable Read is particularly useful for generating consistent reports or doing complex data analysis. It ensures that if you read something once, you'll see the same data again within the same transaction.

-- Process 1: Generating daily report
BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SELECT COUNT(*) FROM user_profiles;  -- Sees 10000 users
-- Meanwhile, new users are added
SELECT COUNT(*) FROM user_profiles;  -- Still sees 10000 users
COMMIT;

This is valuable when you need consistency across multiple reads, like:

Generating end-of-day reports
Running complex analytical queries
Performing data validation checks

3. Serializable

This strongest isolation level ensures that concurrent transactions behave as if they ran sequentially. It's crucial for:

Critical data updates where consistency is paramount
Complex workflows with interdependent data
Data migration or transformation processes

For example, when updating user hierarchies or permission structures, you might need serializable isolation to ensure the integrity of your access control system.

Practical Applications

Consider these scenarios in a data system:

Event Processing

-- Process raw events into aggregated metrics
BEGIN;
INSERT INTO raw_events SELECT * FROM event_queue;
UPDATE metrics 
SET count = count + (SELECT COUNT(*) FROM raw_events WHERE processed = false);
UPDATE raw_events SET processed = true;
COMMIT;

User Profile Updates

-- Update user preferences while maintaining consistency
BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SELECT * FROM user_profiles WHERE user_id = 123 FOR UPDATE;
UPDATE user_preferences SET settings = new_settings WHERE user_id = 123;
UPDATE user_audit_log SET last_modified = CURRENT_TIMESTAMP WHERE user_id = 123;
COMMIT;

Data Cleanup Jobs

-- Remove stale data while maintaining referential integrity
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
DELETE FROM expired_sessions WHERE last_access < NOW() - INTERVAL '30 days';
UPDATE user_metrics SET active_sessions = 
  (SELECT COUNT(*) FROM sessions WHERE user_id = user_metrics.user_id);
COMMIT;

Summary

Understanding transactions is crucial for building reliable applications that use databases. They give us important guarantees through ACID properties:

• Atomicity ensures our operations succeed or fail as a unit

• Consistency keeps our data valid

• Isolation helps manage concurrent access

• Durability makes our changes permanent

The key is choosing the right isolation level for your needs - balancing between consistency and performance. For most applications, the default Read Committed level works well, but some situations might require stronger guarantees.

In the next post, we'll dive deeper into MVCC in Postgres.

My Desktop Setup

Introduction

Since I was a computer science student, the study of databases has always been the most interesting topic for me. When you study how databases work, you will touch every aspect of computer science, from algorithms, data structures, operating systems, networking, and distributed systems to concurrency.

Postgres is one of the most mature software systems in the world. It is my go-to storage system. It's battle-tested, open source, and has a massive community backing and a lot of literature around it.
And once you use extensions like PostGIS, PGVector, TimescaleDB, and others, you will see Postgres as a complete data platform.

While everything we are going to discuss in this series is focused explicitly on Postgres implementation, the same concepts are common in all other major databases.

Approach

When it comes to a topic as common as databases, there are many resources. This usually puts me in an analysis paralysis mode where I keep jumping from one resource to another.

Lately, I have decided to take a more organized approach. I focus on a handful of resources, pick one topic at a time, and write about it. This way, I can be more organized and accountable.

Since I've already read Database Internals and Designing Data-Intensive Applications (multiple times), I'm starting with the Fundamentals of Database Engineering course.

For every section, I will read (again) the relevant chapters from both books and write my notes about them.
And to make this more Postgres specific. I'll pick the relevant sections from Postgres Internals to make this more sided with Postgres.

While this approach will require more time and effort, it will help get the most out of all resources and give each topic the required attention.
It will also give me the ability to iteratively publish posts about every topic.

Topics to cover

If I learned one thing from being a software engineer. It is that you can never know enough about any topic.

This series will cover topics that are foundational in understanding how databases work. I will update this list with more topics as I work through the series. Below is the current list (With no particular change):

• Topic 1: How to study databases. (This post)

• Topic 2: Transactions and ACID Properties.

• Topic 4: Isolation Levels and Read Phenomena.

• Topic 5: Locking Mechanisms.

• Topic 6: MVCC in PostgreSQL.

• Topic 7: Write-Ahead Logging (WAL) and Recovery.

• Topic 8: Distributed Transactions and Two-Phase Commit

• Topic 9: Indexing and Query Optimization.

Resources

The resources I will be using are usually considered the defacto studies for distributed systems and databases. I will also be using Hussien Nasser's Database Engineering course on Udemy (which in my opinion is the best video course on databases).

• Postgres Documentation

• The Internals of PostgreSQL

• Designing Data-Intensive Applications

• Database Internals

• Fundamental of Database Engineering

Timeline

As we are starting August, my plan is to finish this series by the end of the year with a satisfying number of posts. I will be posting a new post every one or two weeks. Following article will discuss database transactions and ACID properties.

When working with production systems, we usually track metrics like latency and throughput to understand system performance. But, sometime (and I mean most of the time), those metrics can give a false sense of good performance.

Well, not the metrics themselves but how we reason about them and how they get used and visualized.

In the amazing blog somethingsimilar: notes on distributed systems for the young bloods: Jeff Hodges makes a point about metrics and percentiles:

Metrics are the only way to get your job done. Exposing metrics is the only way to bridge the gap between what you believe your system does in production and what it actually does.

Use percentiles, not averages. Percentiles are more accurate and informative than averages. Using a mean assumes that the metric under evaluation follows a bell curve.
“Average latency” is a commonly reported metric, but I’ve never once seen a distributed system whose latency followed a bell curve.

This is a very valid point. Personally, I fell in this issue countless time early in my career, largely from a poor understanding of the statstics part of this and how to interpret system performance data.

Imprecise Metric

Taking a hands-on approach on this, let's assume we just deployed a new API where we serve some calculations and are required to complete these calculations in under 25ms.

[30, 15, 21, 18, 16, 40, 25, 35, 17, 18]

We can easily plot this as a line chart:

We can also calculate the average latency with the following formula:

avg=sum(latency)count(latency)

avg=15+18+17+18+16+21+25+30+35+4010
avg=23.5ms

So, looking at the average for our API, it shows 23.5 ms, which means our API is meeting expectations and everything is fine.

But this is misleading, and once we have a deeper look at the data we will realize we are not even close to meeting the expectations and only percentiles can give us the full picture.

Using Percentiles to Understand Performance

Calculating Percentiles

The concept of percentiles is straightforward. You sort your dataset in ascending order and determine which quantile you are interested in.

This can be calcualted with these formulas

rank=⌊Quantile∗(length(Latency)+1)⌋

PQuantile=Latencyrank

Usually in monitoring systems, we are interested in higher quantiles.

P50, P75, P90, and P99

Sorting our dataset, it now looks like this:

P50

The first percentile we'll calculate is the 50th quantile (median), which is the "middle" value in the sorted dataset.

Applying the numbers on the previous formula:

rank=⌊0.5∗(10+1)floor

rank=5

P50=Latencyrank

P50=18ms

So, the P50 shows us that 50% of our requests are meeting our 25ms latency threshold.

P75

Applying the same formula to calculate the 75th position:

rank=⌊0.75∗(10+1)floor

rank=8

P75=30ms

This tells us that at least 25% of our requests are exceeding our threshold.

P90

By applying the same formula to calculate the P90, P99, and P99.9 quantiles, we gain more insights into our system's performance.

The P90 shows us that we have 10% of our requests are taking at least 40ms which is nearly double of our average latency.

Full Picture

Putting together all the information we got gives us a comprehensive view of our system's performance:

• 50% of our requests are meeting our threshold.

• At least 25% of our requests are exceeding our threshold.

• 25% of requests are over 30ms.

• 10% of requests are nearly double our average latency.

With this information, we can reason about our system's performance and take concrete actions to improve it.

Takeaway

The main takeaway is to use percentiles as much as possible. Generally speaking, when you are monitoring a system, you shouldn't rely on a single angle. The performance data you extract from the system might seem alright (this is the easy bit, right?), but how you represent, interpret, and visualize the data is what truly matters.

If you can't understand the full picture of a system, you will fall into the same trap as many others. Believing that your interpretation of how the system works matches how it actually works.

Recommended Reads

• Timescale blog on Prometheus metrics types

• Prometheus documentation

• How summary metrics work

Or I’m very opinionated about this and I think you should listen to me.

Introduction

For years, I've seen Python code that makes me cringe. It's not because the code is inherently bad, but because it misses the essence of what Python is meant to be. Python is not just a language; it's a philosophy. And yet, many developers write Python in a way that goes against its core principles.

Multi-Paradigm Language

Since Python is a multi-paradigm language, you can write Python code in a variety of ways. You can write procedural code, object-oriented code, or functional code. But Python is designed to be simple and readable. That's probably the only common thing between all these paradigms.

To quote the official Python docs:

Python is an interpreted, interactive, object-oriented programming language. It incorporates modules, exceptions, dynamic typing, very high level dynamic data types, and classes. It supports multiple programming paradigms beyond object-oriented programming, such as procedural and functional programming. Python combines remarkable power with very clear syntax.

So Python at its core can be used in a variety of ways, but it's designed to be simple, readable, and elegant. So why choose complexity over simplicity?

Declarative Programming

Python is a declarative language. I usually use Python in a functional way, but this is about more than just using one paradigm over another. This is about making all Python code look the same. When you write Python in an imperative way, you make the code harder to read and maintain. You make the code harder to reason about.

The core idea behind declarative programming is that you are trying to declare what you want to do, not how you want to do it. That's it. It's that easy.

Pythonic Code

Python is a battery-included language. It has a rich standard library that provides a lot of functionality out of the box. So when you write Python code, you should always try to use the standard library as much as possible. This will make your code more readable, more maintainable, and more efficient.

This is what Pythonic code is all about. It's about minimizing cognitive overhead as you read the code. So, instead of having to read the inside of a for loop to understand that you are just trying to filter a list, you can just use a list comprehension. Instead of having to read the inside of a function to understand that you are just trying to map a function over a list, you can just use a lambda function.

This use of common idioms and patterns (not just in Python, but in any language) is what makes code more readable and maintainable. And what creates a better communication channel between developers as they read and write code.

Vouching for Functional Programming

I'm not a Haskell or Lisp developer. And I'm not trying to explain what's a monad or a functor. But I'm a big fan of functional programming. For me, functional programming is about writing declarative code that's preferably stateless and immutable with minimal side effects. And Python is a great language for that.

When you write Python in a functional way, you make it easier for you (the writer) to think about the code and for others (the readers) to understand it. You make it easier to test and debug. You make it easier to scale and maintain.

Pairing data classes with functional programming is a great way to write Python code. You can write code that's very efficient and self-explanatory. That doesn't require other people to have extensive experience or domain knowledge to understand what's going on.

That's How the Language is Designed

If you look at the Python standard library, large frameworks (like Django and Flask), and popular libraries (like Pandas and Numpy), you'll see that they all follow the same principles. They provide declarative, human-driven APIs that are easy to understand and use. I've seen a lot of excellent engineers and data scientists write very complex algorithms in a declarative way that makes someone as dumb as me understand what's going on in minutes. While I've also seen a lot of engineers who would make me spend an hour trying to understand a simple for loop.

Final Advice

So, my advice to you is this: write Python in a way that's easy to read. Use the standard library as much as possible. Spend time to invest in the quality of your code. Read a lot of Python code from big projects. This will help you understand the value of writing declarative, efficient code.

Throughout, the years, I noticed that what I was lacking is a systematic workflow that help me increase my deep work hours without burning all of my energy.

I was looking for something that checks all of these boxes:

• Lean workflow, that won’t break when something urgent comes in.

• Play nicely with high level of context switching.

• Easy to follow.

• Longer break time.

• Easy to measure.

Now that I understood the requirements, It was easier for me to try different workflows and techniques and tune it for my case.

Pomodoro

The problem with pomodoro was how strict it was in respecting the cycle length. Even if I finished the task, I had to come up with something productive to do until the end.

But the good thing that Pomodoro brings is the idea of work cycle that you could take longer breaks between them.

Flowtime

Flowtime is a more loose technique which eliminates the idea of a timer and time boundaries and focus on you, the individual.

Yet, in Flowtime, you need to document the start time and end time for every task. Which for me felt like an unnecessary step.

At this point, I knew that I had to come up with a technique that works for me. One that mix between Pomodoro’s idea of work cycles and Flowtime flexibility.

Finding my workflow.

After months of trial and error, I finally found a workflow that helped me be the most productive version of myself without being tired all the time.

Like Pomodoro, we divide a single day into smaller focus cycle. A workday is usually between 4 and 6 working cycles.

A cycle could last between 45 mins to 180 mins. This depends on many factors, including task size, interruptions, calendar, and personal matters.

Every cycle begins with a small planning phase, this includes spinning my local development environment, or doing quick due diligence on the task.

And because most of what I do in this phase is a routine work, it takes between 5 and 15 mins, but gets me into the battle rhythm for the next phase.

After a small break, I get into this loop where I work through my task while taking a short break (5-10 mins) between every stopping point.

I keep this rhythm going until I get tired, bored or interrupted. Which is fine because I try to commit to a stopping point every 15 to 30 mins.

In busy days, I take long breaks between cycles. Sometimes up to 30 mins, which I typically spend hanging around with my wife, getting a snack or taking a nap.

In less busy days, I take longer breaks, sometimes up to 2 hours. In which I run home errands, go outside, or play video games.

This help me feel relaxed even in the most stressful days. But most importantly, I can measure how productive I was with 2 different metrics.

1- How much work did I finish in each cycle?

2- How many cycles did I manage to finish every day?

This workflow helped me become very productive and organized without using any specific time management and planning tools. It also gives me full control of how my day will look like.

I get to work hard, while spending time with family and friends.