I make apps for other people

Making good technical hires

Posted by Chris Jones
On October 12th, 2011 at 14:16

Permalink | Trackback | Links In |

Comments Off on Making good technical hires
Posted in Google Plus Posts

The people
Interviews involve several people:

  • The candidate, who is applying for a position with the company
  • The recruiter, who acts as a first point of contact with the candidate, selects the candidate from a pool of resumes, referrals, or internal transfer requests, and starts the interview process by gauging the candidate’s willingness to interview, availability, performs a first level of fact gathering (collecting addresses, email, phone numbers), and schedules the first phone screen
  • The interviewers, who ask the candidate questions, answer questions about the company, and provide an evaluation of the suitability of the candidate for employment
  • The hiring manager, who oversees the recruiter, reviews candidates, coordinates interviewer reports and feedback, suggests interview schedules to both candidates and interviewers, closes the candidate when an offer is extended, and troubleshoots the hiring process

The process
A position request is generated, either through growth or attrition. The recruiter searches for candidates from a pool of resumes, referrals, and internal transfer requests.

Typically, the hiring process involves multiple phone screens:

  1. The recruiter contacts the candidate, assesses initial interest, and arranges for the first technical interview phone screen. Have the recruiter indicate that there will be coding during each interview, so the candidate should be prepared with pencils and paper or a PC on which to write a short snippet of code. The recruiter begins with the first interview report, gathering basic information for the first interviewer.
  2. The candidate is interviewed with the first technical phone screen. This phone screen tests general aptitude with skills and proficiencies on the candidate’s resume, asks specific questions about academic or professional work, has a simple coding exercise that is performed over the phone, verifies that the candidate is, indeed, interested in working for this company, and may end with a post-interview coding assignment that must be completed within a certain period of time. After the interview is complete, the interviewer writes an interview report so that his evaluation of the candidate is available to the recruiter, hiring manager, and any future interviewers.
  3. The candidate is interviewed with the second technical phone screen. This interview goes into more technical depth, both with proficiencies expressed by the candidate and also with the candidate’s work history. Data structures, algorithms, and design are appropriate topics and should be covered in this interview if not adequately covered in the first phone screen, as is a slightly more difficult over the phone coding exercise. This interview will end with a coding assignment if the first interview didn’t or if the first coding assignment was inadequate for assessing the candidate’s skills. After the interview is complete, the interviewer writes an interview report so that his evaluation of the candidate is available to the recruiter, hiring manager, and any future interviewers.
  4. Optionally, there may be a third phone screen and interview report, especially in borderline cases.
  5. If the candidate succeeds at passing the phone screens, the hiring manager may invite him to an in-house interview. This may involve travel, and will involve between four and seven interviews in house, usually including one over lunch.
  6. The interviewers write their interview reports and are debriefed by the hiring manager. If the candidate passes the in-house interviews, any critical decision makers who missed the in-house interview can now interview the candidate in one or more phone interviews

If the candidate is accepted, the recruiter or hiring manager closes the candidate and negotiates a starting date, salary and benefits, movement and relocation expenses, etc.

Each interview should be about 45 minutes in duration, including the in-house interviews. The interviewer should avoid the temptation to engage in chatty conversation and should try to guide the candidate to answering the needed questions and demonstrate his technical ability.

As a goal, 50% of candidates should pass the first technical screen. If more than 50% are passing, chances are the technical screen isn’t tight enough, that the questions being asked or quality of the interviews is insufficient and the interviewers need to be reassessed. If fewer than half of the candidates pass, it’s an indicator that either the recruiters are not getting candidates that fulfill the requirements, that the technical interviews are focusing on the wrong criteria, or that the interviewers are performing poorly.

About 30% of the candidates should pass the second technical screen and be considered for flying in to interview. Many candidates will choose to end the process at this point as they weren’t serious to start, or other jobs or considerations have occurred.

About 10% of the candidates will pass the in-house interviews and any (missed) follow-up interviews. Those candidates will be given employment offers.

Thoroughness and clarity in the initial phone screens are essential to this process. The goal is to make good hiring decisions, to save the company money and the technical team’s frustration. A bad hiring decision can cost twelve months or more in salary as the bad hire not only gets paid for poor performance, but also consumes other employees’ time for training, hand-holding, and performing those failed tasks. Make the right hiring decision from the beginning, and err on the side of false negative evaluations, rather than false positive decisions. Over time, the system becomes tuned to prevent bad hires..

Work examples or coding exercises are a high indicator for success. Be reviewing coding style, correctness, and what additional effort is made in relation to the code within a limited period of time, technical interviewers can evaluate both code quality and what kind of code the candidate will likely produce while working.

In-house interviews with eight or more people tend to indicate a bad interview experience. The candidate’s attitude is likely to move from, “This is a tough company to get hired by,” to “this company is in love with interviews and I don’t want anything to do with them.”

If the resume feels wrong, go to the hiring manager or recruiter and find out why the candidate was recommended. This is especially true for the second phone screen: if the candidate doesn’t match what’s expected from the resume or the candidate doesn’t seem like a good match fit, find out why before recommending for or against the candidate.

The interviewers
Different interviewers have different roles. The initial phone screen is a technical interview that explores stated skills and proficiencies. The second phone screen is more theory based, exploring data structures, algorithms, and design. Both interviewers help establish an assessment of the candidate’s talent and potential for growth.

In-house interviews cover specific topics, including (again) algorithms, design, data structures, specific skills (such as database administration, language design and implementation, system administration, or an interesting project or skill from the candidate’s resume), teamwork, planning, or any other skills appropriate for the position.

One of the interviewers will be the bar raiser, who will be responsible for making sure that the candidate is better than half his peers at that level. An ideal candidate will leave the bar raiser feeling that the candidate would be capable of learning and doing the bar raiser’s job — often better than the bar raiser himself.

Preparing for interviews
The interviewer should prepare carefully and thoroughly for the interview. The interviewer should read the resume in advance, and if he has any questions contact the hiring manager or recruiter. Interviewers should understand the job family requirements that the candidate is interviewing for, and if the interviewer is not qualified to assess the candidate, contact the hiring manager to find a replacement interviewer. The interviewer also needs to read feedback on previous interviews (or for internal candidates, management assessments) so that he can prepare an interview plan and cover topics that need more assessment or were missed, and cover new topics that are appropriate to this interview. Finally, the interviewer needs to be able to answer questions about and promote the company, the team, and their work—the interview is not just a chance for the interviewer to assess the candidate, but also a chance for the candidate to assess the company, so the interviewer needs to put his best foot forward.

Performing a phone screen
Every phone screen will be a unique conversation. The tone should be friendly, not adversarial, and the interviewer should encourage the candidate to be at ease. A good phone screen will typically follow this pattern.

The first five minutes of the interview are an opportunity for the interviewer to introduce himself, set the candidate at ease, explain how the interview will work, and answer any immediate questions or objections that the candidate has.

Basic coding
Ten minutes should be spent describing a simple, typical programming problem. The candidate should be encouraged to write the solution on paper or the PC and dictate the code to the interviewer, who will then verify it for completeness and correctness. This tests the candidate’s ability to solve problems given a description and provides and indicator of the kind and quality of solutions will be provided when under pressure and time constraints. If the candidate doesn’t understand the problem, for instance, the interviewer asked for something specific to computer science, such as an “in-order traversal of a binary tree,” the interviewer should have backup questions ready so that time isn’t spent explaining concepts and is instead spent coding the solution.

Algorithms and problem solving
Fifteen minutes should be spent on algorithms and problem solving. The candidate should be encouraged to think out loud, to describe how he would solve the problem. If the candidate already knows the answer, as will be the case with many classic computer science problems, the interviewer should try an alternate problem. The interviewer can use this to gauge how the candidate thinks, if the candidate is creative, talented, and learns, or if the candidate tries the same solutions and doesn’t look beyond what he already knows.

Arguably, the least important section of the interview, the 10 minutes spent here can be used to evaluate the design and architecture abilities of the candidate. If design patterns are important to the position or team, the candidate’s familiarity when them can be tested. Typically, a design problem is provided and the candidate is tasked with providing a solution which will be evaluated by the interviewer after the interview.

Wrap Up
The final five minutes of the phone screen will be spent answering any questions the candidate has about the position, company, or team. The interviewer will let the candidate know that he will be contacted by the hiring manager within a given period of time, usually 48 hours), and that any additional questions can be sent to the hiring manager in email.

The importance of a phone screen
Phone screens are very difficult to perform. The interviewer and candidate both miss subtle clues and body language that can help with the conversation that would normally be present in an in-person interview. What makes a phone screen especially hard is the material covered, highly technical and where it’s difficult to demonstrate proficiency through voice alone. Coding challenges are hard, as the phone screener will typically have to ask the candidate to write out the solution and then read it out loud so it may be transcribed and checked. Additionally, it’s harder to judge the candidate’s confidence: coding over the phone is difficult for both parties which can damage the candidate’s impression on the interviewer, and many candidates won’t do as well without a personal presence. This is unfortunate, but a fact of how phone screens impact the interview.

The phone screen is not an opportunity for the interviewer to assess if the candidate will do a particular job (leveling the candidate), but if the candidate is qualified for a technical position. The actual position will be determined by the manager interview, senior and team interviews, and the feedback after the in-house interview. In a phone interview, don’t discard someone who may be a better fit for another position, such as a developer well qualified for a junior position but interviewed for a senior role, especially if the candidate can grow into the position or is otherwise well qualified for another open position. The question to answer is, “Does this candidate make a good developer, regardless of whether he fits the level requested?” People can be promoted into new positions when they’re ready for the role.

Hiring decisions should be made around talent and the potential for growth. Specific skills are important for specific roles (such as a database administrator), but when a candidate is missing a specific skill set, that isn’t enough to disqualify him. Try to determine if the candidate is capable and eager to learn the skills and to grow into the position.

People matter
The interviewers are critical to the candidate accepting the job, and vice-versa. Candidates and teams need to like each other to work well together. If there are strong personality conflicts or if the candidate isn’t a good match for the team or obviously won’t gel, then it’s a bad hire despite the candidate’s talent and potential.

Interviewers have a natural tendency to want to hire people like themselves, people who share similar interests, look the same, have similar tendencies, etc. This bias will influence which candidates the interviewer responds positively toward and must be something of which the interviewer is aware before reviewing and voting on the candidate after an interview.

One of the best ways to avoid this bias is through the application of standards, stating what is required for the position and what isn’t required. By identifying requirements, an interviewer has a better chance of identifying the skills that are needed rather than skills that he admires. The bar raiser, who should be evaluating the candidate on technical proficiency rather than personality and appearance, may have a more stringent set of requirements but should likewise avoid evaluating the candidate on skills that are not required for the position.

Interviewing events
Not all interviews start on the phone. Interviewing events are special trips to colleges, job fairs, or conventions where the interview process is condensed to one to two days. Candidates are typically self-selected, choosing to start the process and quickly moving through screening and hiring interviews. The outcome of a successful interviewing event will be offers to candidates to fulfill a specific role—but rarely with the input of the team that will be receiving the candidate as a new member, so it’s important to choose people who have generally team-oriented and enthusiastic personalities. An offer will be extended to successful candidates within six to 24 hours of the interview in most cases.

Internal recruiting
Internal recruiting is not done through active sourcing: teams don’t poach members of other teams. Employees are notified of openings and they can choose to apply for those positions. Internal recruiting is not necessarily less formal, but requires a smaller number of people for the actual interviews, possibly only two or three interviews would be required, and much of the determination of the fitness of the position can be taken from current management feedback. Likewise, employees should not be eligible for interviewing for new openings in the company until they have received a performance evaluation or management review.

Asking the right questions
When interviewing a student or someone fresh from school, questions should concentrate on what was performed in school, both as class assignments and as personal projects, rather than the grades achieved in classes. Questions on algorithms, textbook design questions, and classic computer science topics are fair game if the candidate came from a CS background.

When interviewing a candidate with significant work background (one who has been out of school for over a year or more), ask questions about projects that actually went to production, live, or shipped. Some computer science questions are appropriate (what’s a queue, linked list, stack, map; when do you use one or the other). Always cover coding, algorithms, data structures, and design in every technical interview.

Questions are not supposed to be tricky. Microsoft’s legendary mindbenders aren’t helpful in evaluating and only serve to frustrate the candidate while making the interviewer look like a smug, if clever, ass. Riddles are an expression of a step function, showing success by moving from one point to another without demonstrating the path or discovery process. Their “ah ha” moment nature makes them useless for evaluation of how well a candidate works—it’s a bad indicator for engineering talent, as there is no correlation between software engineering and riddles.

Do ask questions about languages, process, or other technical proficiencies listed on a resume. Cover a sampling of basics, intermediate, and advanced topics, and look for accurate, honest answers. Common bad responses include talking around an answer, making up inaccurate answers, hedging, and not admitting when the candidate doesn’t know or isn’t qualified to answer.

Don’t assume that other interviewers have covered basic questions. Read previous interviewer feedback on the candidate, but if a basic question isn’t covered do ask it. The coverage of topics in the phone screen is more important than the total time spent on any phone interview.

There are, generally, two schools of thought about which questions can be asked:

  • those that believe that anything on a resume or mentioned in the interview is fair game and the candidate should have in-depth knowledge of that topic, or
  • those that don’t disqualify a candidate if the depth of knowledge isn’t as strong as claimed.

Generally, if the hiring goal is to find talented and growing professionals, the second school of thought is preferred. However, it may be appropriate for the bar raiser or another interviewer to delve deeply into resume and find out how strong the candidate’s knowledge is.

The interview shouldn’t end before the interviewer can say if the candidate has the talent and growth potential to succeed in a position. Spend less time chatting and discussing general topics at the beginning of the interview, saving those for the end if there is still available time and the important questions have been covered.

If you can’t assess the candidate, you’re not the right interviewer. Find someone else on your team or in the company to interview the candidate. Likewise, without certain languages or skills, it will be difficult or impossible to be successful at a company (for instance, a company that is a Java and C++ shop isn’t necessarily a good match for a developer who only works in Ruby and Perl).

Treat interview questions as if they are a guided discovery process. Ask the candidate to describe what he’s thinking about, how he’s solving a problem, and to think out loud.

Defining standards
For each skill or skill area, certain standards should be applied. This helps the interviewer determine if the candidate is qualified for the position, and to focus on pertinent skills rather than discussing skills that aren’t useful for the position or company.

An example of these standards for a software developer follows:

Skill Required Not Required
General coding Reasonable understanding of either C++ or Java. Includes language syntax (function declarations, control loops, variable scope, class declarations, etc.), basic OO (inheritance, polymorphism, encapsulation), understanding how to break logic into small functions. Coding in other languages like Perl, Python, C#, shell scripting, Visual Basic, HTML, JavaScript, etc. Older candidates who have not worked with OO are not expected to have deep OO knowledge.
C++ coding Memory management—malloc/free or new/delete, stack vs. heap. Define classes, methods, understand constructors and destructors. Const, exception handling/safety, multiple inheritance, rarely used keywords (like volatile), template, STL, compiler optimization.
Java coding Integer vs. int, collection classes. Exception handling. Interfaces vs. Abstract classes. Reflection. Swing. Thread synchronization.
Data structures Knowledge of common data structures—stack, queue, linked list, hash table, array. Candidate should understand how they work, efficiencies, and when it’s appropriate to use each one. Big-O notation doesn’t need to be at the tip of the candidate’s tongue, but he does need to know how each structure works and which one is most efficient in each case.
Algorithms Sorting, binary searches, application of data structure knowledge to solve simple problems. Graph traversal, knowing that quicksort can be O(n^2), complexity analysis of recursion.
Design Design a simple system when given vague requirements. Complex, distributed systems. Getting the best design on the first try.
General computer science Basic OS knowledge (process vs. thread, virtual memory), locks and critical sections, caches. Low level/hardware. Twos-complement math. Interrupts, registers, instruction re-ordering, test and set, multiprocessor systems. Databases.
Linux Specific Linux knowledge is not required. Understanding of common tasks in the shell.
Behavior Candidate should conform to a base code of manners: no swearing, lewd jokes, anger, etc. Besides this basic code, the candidate can act, talk, and dress in a standard manner.
Communication skills Candidate must be able to explain a project to you in less than 10 minutes. You should understand what he did at a high level. Use of whiteboard diagrams, enthusiasm, past work that was actually interesting, good grammar.

These requirements will vary by position. They should be set by the manager or team leader before interviewing begins and should be used to equally evaluate all candidates.

General coding
Writing source code in the interview is considered the single strongest indicator of the candidate’s potential for success. Most failures of candidates are failures to code during the interview (such as at a whiteboard) at all, for which candidates will have a variety of excuses. No technical interview should be entirely code free except possibly an in-person lunch interview.

For phone interviews, because it’s very hard to code over the phone, have the candidate write the code, dictate it over the phone, and have the interviewer verify the code during the interview. The code doesn’t have to be difficult: implementing a simple replacement sort, searching a set for the lowest and highest members, or another typical Computer Science 101 problem is appropriate for both academic and professional candidates.

For in-house interviews, the code assignment can be slightly more difficult, such as coding a class that implements a basic tree. Never let an in-house interview without coding happen.

Avoid implementations that are slick, efficient, or use obscure language constructs. The goal is to write maintainable code that’s obvious in function and accurate in execution. A coder who finds the absolute tightest way to bum his code is probably not a developer that will produce maintainable clean code; having the code work is insufficient, it has to be code that the interviewer would want to own. Well written code is expected, but clever code is a warning sign, so evaluate the code on how clear and how accurate it is.

When asking for a coding problem, never start with anything complex. Don’t shut down the candidate by asking for a difficult solution, certainly not something that makes the interviewer pause and consider for 30 minutes. Refute the “can’t code” attitude that often accompanies a request for coding during the interview.

Offline coding assignments
Before assigning coding to be performed after the interview, ask if the candidate has time to complete a simple assignment. Schedule the assignment in advance if they need time, such as a weekend or a particular free day, to complete the task. Then describe the coding assignment when the candidate indicates that they have time or have scheduled a time to be contacted with the assignment details.

The assignment should be completed within hours to one day maximum. The time factor helps to indicate that the candidate will be able to work accurately under deadline and with some pressure, and also provides an evaluation of coding performance, accuracy, completeness, and style. The results of the candidate’s assignment are included in the interviewer’s report. Speed and quality are an indicator of the candidate’s enthusiasm and experience, such that a typical assignment should be completed within 45 to 60 minutes, if appropriately designed.

Evaluating code
As stressed above, code should be clear and accurate. Specifically, code should be working (that is, it compiles and executes without errors or warnings), is documented (in-house standards may be different than what the candidate uses), and is of adequate quality. Judging code quality helps to determine how often the candidate actually codes (on a daily basis or only occasionally), which is usually evident through the characteristics of a daily coder who will bring a particular style and structure to the code.

Evaluating computer science proficiency
Basic computer science knowledge and good judgment that tells how and when to apply that knowledge is a requirement of a good software developer. Candidates should have knowledge of algorithms, data structures, architecture and design. In all cases, be reasonable and practical with the assessment of a candidate’s knowledge: a recent computer science school graduate is expected to know the implementation of algorithms and data structures by heart through book learning, while a seasoned professional has likely been using them only through library calls for years.

When evaluating the candidate’s proficiency with data structures, try to have the candidate express which data structure us which, and when to use each. This is an indicator of candidates who can do more than use “glue code” (such as J2EE or other component stacks) to create applications.

When a candidate correctly answers a design or algorithm question too quickly, it’s an indicator that he’s heard it before. Ask the candidate if he has, and move on to another question. A good interviewer will have a small stock of these questions available. Websites collect some of these questions and some candidates will have studied them in advance.

Sample questions
These questions are appropriate for a software engineer. Not all questions will be used in a single interview, and some questions may be discarded if the candidate has been asked that question before and already knows the answer.

Understanding strings

  • Write a strrev() function to reverse strings
  • When and how do you use atoi() and itoa() functions?
  • Implement an isPalindrome() function

Understanding numbers

  • Write a roman2Int() function
  • How do you determine the greatest common denominator? Implement it.
  • Implement an isPrime() function

Data structures

  • Reverse a linked list
  • Implement a stack over an linked list
  • How can you determine which pair of numbers sum to x? Implement your solution.


  • Determine if a graph is a directed acyclic graph (DAG)
  • Find duplicates in a list
  • Find the heaviest subsequence in a list or set
  • Find the odd numbers in a randomly distributed list or set

Design (object oriented) — Identify the classes, relationships, and methods of…

  • a parking lot
  • students and professors
  • traffic sensors
  • a deck of cards with a shuffle action
  • the game of Life (define the rules for the candidate if needed)
  • trends in page log

Reporting on the interview
After interviewing a candidate, the interviewer will vote as to whether he feels the candidate would likely pass the next step, giving a vote of Strong Hire, Inclined to hire, Not inclined to hire, or Don’t hire. Writing a review of the candidate and interview soon after helps justify that vote and collect the interviewer’s thoughts, as well as providing a guide to future interviewers. A well written review will include a justification for, or, more appropriately, an explanation and understanding of the interviewer’s vote, the questions asked of the candidate, the candidate’s approximate responses to those questions, and the interviewer’s feelings about the answers and the candidate’s talent and potential for growth. The best reviews would include word-for-word transcripts of the interview.

Subsequent interviewers should read the reviews and tailor their interviews based on what was covered, what should be expanded or elaborated, to cover points that were missed, in addition to performing the appropriate interview type for that phase of the interviewing and hiring process.

A sample structure for interview feedback is:

  • First, the interviewer’s vote and his reasons and examples
  • Second, the pros and cons of the candidate, what his strengths and weaknesses are and how he would benefit or harm the team and company
  • Third, the interviewer should record supporting evidence gleaned from the interview, including quotes, text from the candidate’s resume, and other facts discovered through the interview process
  • Finally, the interviewer needs to express areas of concern, for example, does the candidate seem too junior, or possibly overqualified for this position, or does the candidate not seem serious enough about the interview.

The feedback should be accurate, precise, and appropriate. Personal feelings should be discarded in most cases, except where they are germane to the candidate, such as, “I didn’t feel any chemistry with the candidate and I don’t believe that he will be a good fit for my team.”

Candidates can and should be rejected after the first phone screen, but the reasons for doing so should be backed up with evidence in the interview report.

Comments are closed.