Modeling Voice Pipelines as Scalable Agents: A Kubernetes Approach In the world of Voice AI agents, one of the key challenges is creating agent architectures such that it can scale to support conc...
Balancing Act: Using Structured Output with Streaming in LLMs When building applications with LLMs, one of a very important trade off comes is - how to use structured responses while using streami...
Large Language Models (LLMs) have become essential tools for various data processing tasks, including data labeling, classification, and enrichment. However, when working with substantial datasets,...
Practical insights from integrating Large Language Models into production systems, including handling non-JSON outputs, evaluation loops, model switching, and more.
Taming Latency: Strategies to Handle Latencies in Scalable Systems The most difficult bugs I have solved are always about latencies. In production systems, based on different factors like traffic...
Good Coding Practices enabling SOC2 Compliance When working on a 0 to 1 product, lot of code specific changes are done to allow moving fast. However, once customer starts using the service, additi...
Tests Over Types: Why Testing Trumps Type Hints in Python In the Python world, static type hints have gained significant popularity since their introduction in PEP 4841. Even with all the benefit...
Resilient Systems: Managing Retries and Traffic Patterns Retry Mechanisms At their core, retry mechanisms are simple: when an operation fails, try again. However, implementing them correctly is c...
Request Hedging: Fighting Tail Latencies In systems with low latency SLA requirements, it becomes very important to address the tail latencies as it can have a significant impact on users. Imagin...
The Power of Two Choices in Systems The “power of two choices” is a useful technique in systems that offers significant improvements over purely random selection while remaining computationally ef...
DORA Metrics: Simple Definitions and What They Measure Introduced by Google, DORA metrics are four key measures for CI/CD maturity of a system and/or team. 1. Deployment Frequency Definition: Ho...
Redis Locks and Fencing Tokens Lot of times out of the box Redis locks are used for ensuring mutual exclusion between workers. However, the redis based locking mechanism actually needs some additi...
Redis Pipelining: The Powerful Feature You’re Probably Not Using Redis is known for its incredible speed. I have seen lot of developers are not aware of another such awesome feature in redis - pip...
Do You Really Need Kubernetes? The Case for Simpler Infrastructure in Early-Stage Products The tech stack choice made by the team directly impacts the productivity of the team. For startups, this ...
When designing scalable systems, it’s a very important factor to implement stateless services to leverage horizontal scaling. However, based on the nature of the problem, adding multiple workers do...
Cuckoo Hashing: Elegant Collision Resolution Recently I came across Monolith paper by ByteDance about their recommendation system[1] Here they talk about cuckoo hashing as a way to manage dictiona...
Clean Database Migrations There are a lot of guidelines about clean code but the database migration is an equally important part of the system usually less talked about. A well-executed migration...
Beyond Booleans: Why Enums Should Be Your Go-To for State Management Representing state can significantly impact code clarity, maintainability, and extensibility. Lot of times certain attributes e...
Consistent Hashing: Balancing Distribution in Dynamic Systems Distributing data across different worker nodes is a very important problem in distributed systems. A simple modulo(%) no of nodes ope...
The FLP Theorem: Understanding the Impossibility of Consensus in Distributed Systems In the world of distributed systems, the FLP theorem stands as a cornerstone principle with far-reaching implic...