A Multi-perspective Look at Technical Assessments in Interviews
In early 2015, I was actively seeking a new opportunity as a senior developer or software team leader. During this process, I interviewed with numerous companies ranging from well-established large enterprises, to mid-sized advertising agencies, to startups. The interview processes that these companies employed were just as diverse as the sizes and type of businesses that granted me the interviews, which prompted me to take a multi-perspective look at how technical skills were assessed throughout the hiring process.
Types of Technical Assessment
During my job search, I experienced two different types of technical assessments, coding challenges and deep knowledge questions.
For those unfamiliar with the concept, a coding challenge is given to a candidate before, during, or after an interview so the hiring organization can get a feel for the prospective hire's programming skills. These challenges often involve small problems that are simplified versions of what might be seen in the day-to-day responsibilities of the position.
From my experience, coding challenges were most prevalent at startups and varied in size and complexity. Here is a sample of what I was asked to do:
- Use a public API to create an asynchronous search plugin.
- Use provided sales data to create a list of the top 20 items purchased by frequency.
- Create a file system representation that moves contents around asynchronously using Promises/A+.
- Normalize user submitted data and design a class system to chart it.
- Solve a 15 puzzle using artificial intelligence.
Many of these challenges were a prescreening mechanism, while some occurred after the in-person interview. I also had one that was a remote pair programming session that occurred via Floobits during the interview itself.
Pros of this Approach
Some of the coding challenges were really interesting and helped expose me to new things as a developer. Even if I didn’t progress to the next round of the hiring process, there was an ancillary benefit of getting hands-on experience with some techniques that were new to me, such as Promises/A+. As a candidate, the coding challenge provided insight on the technology stack, programming paradigms, and gave me a better idea of what the job was beyond the posted job description.
Coming from a managerial perspective, the coding challenge can provide a fairly accurate assessment of the candidate’s ability to solve the types of problems encountered in an actual project. It isolates the candidate’s code for critical assessment and helps provide insight to the candidate’s coding style. Given that the candidates are all given the same challenge, it also helps the manager to make an apples-to-apples comparison between different developers and their approach to solving problems.
Cons of this Approach
For the coding challenges that have time limits, an employed candidate is at a disadvantage over a candidate who is unemployed. The employed candidate has the evening hours and maybe a weekend to complete the challenge. An unemployed candidate has the entire workday to solve the problem in addition to all of the time that the employed candidate has, and depending on the scope of the challenge, this time discrepancy could be an issue.
The other con of this approach is that the coding that occurs is very high stakes. With one bad performance or misstep, the candidate can be eliminated from consideration. This phenomenon becomes even more exaggerated in remote pair-programming scenarios where a member of the hiring organization is watching the candidate’s every move in real time. Contrast this to a regular workday; if a developer is stuck on a problem during the course of a normal workday, he or she can revisit the problem at a later point and can usually work on another problem in the meantime.
Finally, if the scope of the coding challenge requires a significant time investment, then that is a deficiency in the hiring process that should be evaluated because it is most likely to be costing a business quality candidates.
Deep Knowledge Questions
Pros of this Approach
One great thing about asking deep knowledge questions during the interview is that each candidate is afforded the same amount of interview time as the next, so everyone has the same amount of time to answer the questions. The same questions are given to each candidate, which also ensures that the candidates are being assessed on a level playing field.
Also, demonstrating a familiarity with the quirks of a language shows the candidate’s preparedness for the interview. This level of preparedness indicates that the candidate cares enough to identify what questions might be asked and reflect upon any gaps in knowledge that may require extra studying.
Cons of this Approach
The biggest con is that this approach is more superficial in its assessment of the candidate’s knowledge. With a minimal amount of prep, anyone can ostensibly answer these deep knowledge questions without actually knowing how that translates to application behavior. I associate studying for the deep knowledge questions as cramming for a test - knowledge will often be forgotten if it is not applied in the context of an actual project.
Another downside to asking deep knowledge questions in interviews is that it cuts into the time for other important details such as establishing a cultural fit. There is a very limited amount of time in each interview, and it can be hard to establish if someone can pass the beer test if the entire interview is spent grilling the candidate on the deep knowledge questions.
After seeing the interview process as both a candidate as well as a hiring manager, I think the sweet spot is use both approaches. Ask a very limited number of deep knowledge questions in the in-person interview and then present the candidate with a brief coding challenge after the in-person interview. This allows the hiring organization to verify that the candidate knows how the answers to the deep knowledge questions translate to code as the organization works toward their hiring decision.