During interviews, technical skills in the industry are still largely vetted through LeetCode-style questions. These are small algorithmic riddles in the form of “I have an array with positive numbers, find the n^th largest”, or “Print nodes in a binary tree in a zig-zag order”. It is pushed to the point where, hired as an engineering manager, three out of my 4 interviews had leet code puzzles. Whether they’re an effective selection criteria is a divisive question, and since recruiting is unfortunately still not a very data-driven practice, one that is very subjective.
I don’t think it’s an all or nothing situation. You should use questions that provide data per whether the candidate will perform in the job they’re interviewing for. If that job is extremely algorithmic driven, e.g. in academia, or involves OS-level optimization, sure, maybe leetcode exercises are relevant.
For 99.9% of the jobs in today’s market, LeetCode is generally irrelevant to the interview. When I see this kind of exercise discarding some of the best developers I’ve been working with, for jobs I know for sure they’d perform in, it gives me pause to whether they make sense at all.
Leet code exercises are bad at producing data on your candidate
First, they’re unrelated with the job of a person on a day to day basis. Very few of our jobs are actually about finding an efficient algorithmic solution on the spot, with no support. It discards existing libraries that would do the job
well1 better. It discards your team. It discards Google, StackOverflow, books and such. A lot of dev jobs today are about data processing, rendering, reporting. They’re further away from the CPU, they’re leveraging existing frameworks. And whenever a hard issue appears, you have time to think the problem. You can use literature, search, ask. And yes, performance and optimization are still important parts of our jobs, but a candidate’s ability to implement binary-search on the spot is not a predictor to whether they’ll optimize the code they write.
Second, this sort of question don’t tell you much about the quality of code the person will produce. At best it will tell you if they use readable variable names and indent their code well. It doesn’t tell how they organize and structure their code, leverage object paradigms, arrange functions or componentize code, which, for the majority of jobs, is much more important. It doesn’t tell you if they test things well. Some argue that it tells you about problem solving, but most problem solving in our days are about structure rather than algorithm. There are other ways to test problem solving that are more relevant to the job at hand. You should aim your question at producing data per relevance of candidate to job, not per whether they’re good at academia-style exercises.
Third, they discriminate against slow-thinkers2. People who need to take a step back and find solutions to problem when they don’t think about it. I recently practiced some leet-code for fun, some of these I got blocked when trying to do them, and found optimal solution while in the shower, or driving my car. Maybe you are looking for a developer who finds the solution right away every-time they are presented with an algorithmic problem. Real life generally requires days of work on a design to figure what the right way to proceed is.
They discriminate against people who have higher stress level / stage fright. Interviews can be tense, even with the chillest of interviewers. If the candidate really wants the job, they perceive a high outcome behind a very small test. They might stress out if they get blocked. And these are exercises that are prone to block people. They require very specific knowledge, and their discrepancy compared to day-to-day work makes it an unfamiliar task. Having someone judging you while you’re trying to think is not the best scenario for such people to shine. Use this if you want to hire developers to code urgent things in extremely tight deadlines (< 1h) in front of an audience.
Another major issue: you’re skewing the data with somebody’s ability to prepare for the interview, rather than their effective skill. Someone who spent days doing all the problems on LeetCode might have seen the precise exercise you’re giving them ; or at least is temporarily refreshed on DFS, BFS, merge-sorting, heaps and such. You will favor these persons, which only brings you the data-point that they’re motivated by the job. It’s an important one, but doesn’t tell you if the person will perform.
My final major problem with LeetCode style questions is that they are heavily biased. As the interviewer, you have the curse of knowledge: you, knowing the answer, assume that the problem should be easy for the candidate. “But I’ve seen others do it easily”, you might tell. This is survivorship bias, because you’re selecting the people who succeeded at your test, therefore strengthening your belief that the test is relevant using data produced by the test itself.
So what to use instead?
Leetcode questions are very comfortable for the interviewer. You give the problem and wait for people to crack the solution, occasionally guiding them. It’s repetitive. You don’t have to think too much.
A first alternative is to look at some of the candidate’s code to begin with. If they provide a GitHub account, looking at that might provide a lot of information about their code. Not having a GH account is definitely not an eliminating stage, but if your goal is getting data on coding abilities, a person’s contribution to open source or personal projects can help.
You might still want to do an exercise. Start with asking yourself what you are actually looking for in a successful candidate. A way of doing so is to look at successful people in your team, and try to categorize why they are successful. Alternatively, look at under-performing people and find what they are lacking. A lot of these might be soft skills, some example when it comes to the technical skills:
- Deal with ambiguity
- Reviewing code, understanding what it does, finding gaps.
- Code structure
- Learning new concepts
From there you can derive your exercise. There are a few exercises that I like to use, usually in combination. My goal is to stick close(r) to the job interviewed for, using relevant skills they should already have. As such, you should not need any preparation in order to complete one of these. All starts with showing some code, a class that does some stuff and its corresponding tests. The code is not glaringly bad, but it’s also not great on purpose.
- Ask people to look at the code and placing themselves in a situation where they would review that code, tell me what they can spot is wrong. This gives me info per what people are looking for in reviews (are they stopping at naming and lack of comments or do they look at structure as well).
- Usually I will leave a few unit tests out, and ask people to identify them and add them. Tells me if they understand someone else’s code, if they understand unit testing, gives me a look at their coding skills.
- Some of these missing tests usually require a bit of refactoring to make the code testable. If the candidate doesn’t see it, I’ll usually try to put my finger on what’s wrong, and progressively try to get them to make code better, then get them to add the tests. Gives me information on code cleanliness, and more advanced view in how they structure it.
- When people are on the lower experience of the scale, I usually try to explain them a pattern or concept that they didn’t see before, say constructor dependency injection, composite, etc. I ask them to apply it to my example. If they get stuck, I show them, I write the code with them, re-explaining along the way. Then once I’m done, I asked them to confirm they understood, then I erase everything I did, and ask them to re-do, this time with minimal guidance. The data that gives me, is their capacity to grasp new concepts, to be coach-able. Everyone has “fast learner” on their resume, this tells you whether it’s true.
- I’ll also keep some of these questions a bit vague to give me some info on “Deal with Ambiguity”.
Is that the best way? I don’t know. I’m sure it’s not generic, it’s not meant to be, it’s meant to filter candidates who correspond to the profiles I’m looking for. I could also ask for an offline exercise. I quite like these, because they remove portion of the stress I mentioned, but I can’t use them here. There are tons of alternatives, you can pick a ticket and pair program. Have them review an actual PR. Etc. That’s just the one I’m using, put there to illustrate a point. If you’re looking for something different, your test will also be very different. The point is just that, with little effort, you can invent your own adventure that tests for what you’re actually looking for, without going the leetcode-way.
If I’m hiring a landscaper, I’m not gonna ask them to tell me about the classification of ficus in Fiji, or the specific reproduction period of Douglas Fir in the West Coast. I’m gonna ask them to trim a tree and see if the result suits me. Why are we using riddles to hire developers? Let’s use stuff that is more relevant!
A lot of use can do LeetCode exercises. It might even allow you to find some geniuses. Maybe it actually produces some data on someone’s relevance. I just think there are much more important things to be tested for, and I don’t think they’re a very effective tool at predicting success.