If you are a software engineer and "Can you review this PR quickly?" is not your most frequently sent message then are you even a software engineer? Maybe, but you're definitely not a 10x engineer or a vibe coder.
Having spent the last 12 years shipping consumer apps to billions of users at companies of various sizes based and operated in very different geographies I have come to realize that most developers absolutely hate code reviews. Though only a small percent of them, publicly admit it and I, for what its worth empathize with them today.
Being very passionate about developer experience, I try talking to engineers fairly regularly both within and outside my team to understand how can we constantly keep improving and evolving our tools and processes so that they feel empowered and excited to do their best work every day.
During a conversation with an engineer over lunch at our team offsite in New York last month, I asked them what are some of the most painful things in our tooling that they deal with on a daily basis. They go I feel really slowed down. I was like, are you out of your Claude budget or is it the CI pipelines? To my surprise, they go nope, I just can't get my PRs approved fast enough and I was like, you mean you can't get your PRs "reviewed" fast enough, they go yeah same thing! While I was still processing, on how to probe further. They go on to show me a text replacement shortcut. The engineer has mapped lfg to Can you review this PR quickly? 🙏. For those that don't know, lfg most commonly stands for let's f**king go! They go, yeah even though I tag people on PRs, or ask for reviews in Slack channels, they don't look at them fast enough, so I DM a few of them to get a quick approval.
You see what's happening here and I don't blame the engineer. We are all producing (generating) code with AI. It is impossible that we can thoroughly review or even read all the code that comes our way on a daily basis without losing our minds. So the status-quo has quietly become, here is my PR, approve it and move on.
Call it a good or a bad habit, but I do like to lurk around on Github more than I should reading people's PRs regardless of whether someone asked for a review or not. I just generally like keeping myself as up to date as I can about the changes happening in my codebase on a day to day basis. I see tons of comments being left by code review agents which get blatantly ignored by folks unless it is marked as critical.
Some people have gone as far as even responding to AI's comments with something on the lines of ignoring this for now, until something breaks. Again, I'll restate the fact that a lot of times comments are just total non-sense and if you start seeing them over and over, the trust in those reviews erodes really fast.
As I mentioned in my previous blog post, recently I read this research paper from Meta titled: Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency where they discuss a few approaches they are taking to keep up with rapidly rising scale of diffs that they merge in their codebase by auto-reviewing some of them and human reviewing others.
In their paper they claim to answer three major questions:
- What is the feasibility of this framework?
- How do they calibrate for risk?
- What is the impact they observe?
While they present quantitative data, I personally don't think it should be relied upon too much because Meta as a company operates very differently than most software companies. So I won't dive into that. But what stands out and makes me really interested in this paper is that they don't solely or overly rely on LLMs as a judge for this.
The basic idea is that given a PR (Pull Request) pass it through a simple decision tree and at each step try to confidently say "this PR does not need a human review"
A lot of these layers are purely based on heuristics and deterministic steps where they use differently signals to classify the whether a PR can be auto approved or does it need a human reviewer.
Without going into too much detail, a key component of this is a machine learning model called Diff Risk Score or DRS. This model predicts given a diff or a code change what is the likelihood that it will cause a Production Incident (PI). I like the fact that they generalize the failure to a PI and not just index on a crash or a bug.
The risk threshold is represented using the Percentile Notation like Px. For example P5 would mean only consider the 5% of the safest PRs. P10 would mean pass only 10% of the safest PRs. The lower the number, the more conservative the output, which means less number of PRs qualifying as safe, and the higher the number, the more aggressive or loose is the pass through rate.
Fun fact: DRS was originally created for a permit safe strategy to allow people to commit certain changes during code freezes. It now powers several other use cases including but not limited to cherry-pick assessments for mobile releases, refactoring prioritization, reviewer recommendations etc.
Due to the nature of the output from this model, an org or team can easily customize the Percentile they are comfortable to use. Someone could say I want to pass top 20% of the safest PRs where as others could use P75.
How does it fit into RADAR?
They use a multi-layered filter based on the author of the diff and some other parameters associated with the code change. Where DRS is one of the layers.
As an example: If a diff was authored by a human, they'll only let Safest 5% of the diffs pass through automatically where as if a diff was generated by an allowlisted bot, they let Safest 50% of the diffs pass through.
For human authored diffs, your tenure, your role, scope etc. affects the score. Meta is great at Stack Ranking so I trust they have way more data than we know about, to calculate this score.
After DRS pass throughs, an Automated Code Reviewer (ACR) which is their LLM agent evaluates a diff for semantic logic, rogue changes, performance degradation risks or bugs.
Performance and Outcomes
As per the authors, they claim RADAR processed over half a million (535K+) diffs, out of which 331K landed, with peak daily throughput touching 25K diffs.
They claim to have reduced the Time to Close a PR by 75% and reduced Review Wall Time by 35%.
The revert rate stood at 1 in 3 auto-approved diffs which is not great but given the volume of PRs it in my opinion seems to be a good achievement. Where it becomes even better is the production incident rate which was only 1 in 50.
Where From Here?
I feel very excited by the idea of creating a pipeline like this myself to understand what sort of code changes pass through and which ones don't especially on Android projects. I still remain highly focused and motivated on solving the problem building an agentic loop to verify the correctness of output of agent end to end that I discussed in this post.
If you are exploring anything in this domain that you'd like to chat or brainstorm about, I'd love to connect. Say hi to me on LinkedIn or email me at ishan at TLD.
