HubofML #11: Uber’s Real-time Data Intelligence Platform At Scale, Scaling Datastores at Slack with Vitess And More
This edition features exciting posts from Uber, HelloFresh, GoogleAI, Slack, Postman, LinkedIn, Pinterest and More
Hey there,
As always, below are my best findings for the last month. I hope this edition sparks new ideas in you. If you miss the last one, you can catch up here.
If you enjoy this newsletter, please share it with others who might find it interesting.
Machine Learning/Data Science
Uber’s Real-time Data Intelligence Platform At Scale: Improving Gairos Scalability/Reliability
At Uber, real-time data enables their operations teams to make informed decisions about their services on the Uber platform. In this post, Uber engineers share their journey with Gairos, a real-time data processing, storage, and querying platform to facilitate streamlined and efficient data exploration at scale.
Recognizing Pose Similarity in Images and Videos
Everyday actions, such as jogging, reading a book, pouring water, or playing sports, can be viewed as a sequence of poses, consisting of the position and orientation of a person’s body. This post on the GoogleAI blog describes a new algorithm for human pose perception that recognizes similarity in human body poses across different camera views by mapping 2D body pose keypoints to a view-invariant embedding space.
Rethinking Customer Lifetime Value using Machine Learning at Hellofresh
HelloFresh offers a subscription model where customers pay a weekly fee to receive delicious meals and pause their orders for an indefinite period. But this poses a series of challenges when implementing customer lifetime value (CLV) models to optimize marketing initiatives. This is a good read on how HelloFresh's engineers created an algorithm that uses the latest Machine Learning (ML) techniques to offer weekly customer-level predictions.
MLOps is an intersection of ML, DevOps, and Data Engineering. You can see it as a set of practices that provide determinism, scalability, agility, and governance in the model development and deployment pipeline. Here are ten practices for infrastructure that allows you to deploy ML applications in a durable, hassle-free, and flexible manner.
Engineering Leadership
Being a Chief Technology Officer: Lessons Learned in My First Year
Most engineers get promoted to roles without being handed a manual to guide them in the new position. Shekhar Gulati's case was not different when he became a CTO. In this post, he shares some valuable lessons he learned in his first year as CTO.
What strategy should an R&D department take to be both appealing to future employees and nurturing to existing employees? And how can this be done without losing sight of the competitive nature of business? @payammoghaddam tells how they are experimenting with taking on a more deliberate approach to training and accelerating the talent acquisition process at Galvanize.
Cracking The Engineering Manager Interview
While coding interviews have become more or less standardized across the tech industry, there is still significant variability in hiring for Engineering Manager roles. This post by @chronicblabber covers some common topics to give you an overview of what to expect when interviewing for an engineering manager role.
Three Crucial Skills Leaders Must Develop to Become Executives
“Becoming a great executive requires a set of skills beyond what's needed to be a leader. Executive skills are subtle and can be elusive to managers, demanding a great deal of focus, courage, and dedication”. Nikhyl Singhal, in his post, shares three crucial skills that leaders must develop to become executives.
Software Engineering
Scaling Datastores at Slack with Vitess
Slack started as a simple LAMP stack, and from a dataset layout perspective, they began with a workspace-sharded model. But this model has a number of disadvantages. Here is an interesting read on Slack's journey and how they migrated from MySQL to Vitess to tackle scale problems.
How Postman Engineering Handles a Million Concurrent Connections
Bifrost is a websocket gateway that lets Postman handle millions of concurrent connections. Joyce Lin shares how Postman engineers developed Bifrost by chipping away at a service that grew too big.
Maximizing Developer Effectiveness
One of the questions I asked myself every day is how to maximize my team’s productivity. How can I ensure that each engineer is as effective and productive as they could be? This post on Martin Fowler’s blog identifies key developer feedback loops, including micro-feedback loops that developers do 200 times a day and ways to optimize them. An interesting read for tech leads.
How LinkedIn Turned to Real-time Feedback for Developer Tooling
Michael Berlowitz shares their experience at Linkedin and how they evolved their real-time feedback to evolve tooling and provide a more productive experience for LinkedIn's developers, which helped them double feedback participation and tailor recommendations and improvements.
What People Get Wrong About DevOps
One of the most misunderstood words in software teams is “DevOps”. I get emails from companies looking to hire DevOps Engineers all the time. In this post, I shared why DevOps is not just a title, a role, or a job function, and you can’t hire it. And why It’s more of a culture that your entire organization needs to embrace DevOps for it to work.
Scaling Cache Infrastructure at Pinterest
"Pinterest's distributed cache fleet spans an EC2 instance footprint consisting of thousands of machines, caching hundreds of terabytes of data served at more than 150 million requests per second at peak. This caching layer optimizes top-level performance by driving down latency across the entire backend stack and provides significant cost efficiency by reducing the capacity required for expensive backends". Kevin Lin, a software engineer at Pinterest, gives a technical deep dive into the infrastructure that supports Pinterest's cache fleet at scale.
Thanks for reading! If you like this newsletter and want to support it, please share it with others.
Cheers 🎉,