Skip to main content

Developing an effective screening test

  • Updated

Article 1 of 5 in a series on creating your own tests.

This is the first article in a series of five, to help you create your own tests using the TestGorilla platform. The full series includes:

  1. Developing an effective screening test
  2. Choosing a question type
  3. Writing situational judgment questions
  4. How to create your own test
  5. Creating a coding question or test

The option for adding your own tests is available to account owners, admins, and recruiters who are on our Scale and Business plans.

When creating your own test, we recommend outlining it before putting it into the TestGorilla platform. This will help ensure that everything is organized and laid out exactly as you want.

Approx. reading time 7 minutes

In this article

  1. Define skill areas
  2. Define the categories within each skill area
  3. Write questions
  4. Clarify
  5. Final check
  6. Next steps

 

Define skill areas

A test should be designed to measure a candidate’s knowledge about something very specific, such as their proficiency with a certain level of Mandarin, their ability to troubleshoot broken Ruby code, or how familiar they are with a specific product — you would test someone on their knowledge of using an iPhone or a Macbook, but not all Apple products.

It’s important to take time to clarify what exactly you're trying to measure and how it will be structured. This will form the foundation of your test.

 

What this means in practice

Clearly define the skill areas within your test. These sections will break your test down into different domains of knowledge. Each test should contain 3 or 4 skill areas that are mutually exclusive and collectively exhaustive ( MECE ). Using the MECE principle to organize your skill areas means finding the best arrangement in which you organize all the relevant information into clearly distinct groups with no repetition.

It’s helpful to research and review current job descriptions to see what other employers expect from similar listings. Depending on the role your test applies to, you might be able to refer to officially defined standards to help with your organizational choice.

no Organizing a group of people into a few subgroups using nationality — such as European, African, Asian — is not a good method as it is a non-MECE system of organization. People with dual nationality will need to belong to two groups, creating repetition within each group, while those with no nationality won’t fit into any group.
yes Using each person’s age would be a MECE way to organize the group. Every person has an age, and that age is unique; after all, you cannot be both 28 and 29 at the same time. This means every person in the group will belong to one and only one subgroup.

 

Example

Emily wants to develop an intermediate-level customer service test. First, she’ll want to review current industry standards, like the ones from the NOS. She can also look for jobs for customer service representatives elsewhere on the internet. Based on these standards, she could define skill areas that look like this:

pink-1 Ability to understand customer concerns or requests, and uncover root causes
pink-2 Ability to interact with customers in a positive and constructive manner
pink-3 Ability to accurately and efficiently determine appropriate solutions
pink-4 Ability to avoid negative or unintended consequences

Essentially, defining the skill areas for a test is about crafting a high-level structure for the knowledge and skills that the test will measure.

 

Define the categories within each skill area

Each skill area should assess candidates on a specific topic within the overall scope of the test. It's important to define the specific skills, or categories, that each skill area covers. This will give you a framework for writing questions in the next step.

For a test to be fair, the categories within each skill area must still follow the MECE organizational principle. This means that the categories should collectively cover all the information in a specific skill area without creating any repetition within that skill area or with categories in other skill areas.

 

What this means in practice

Clearly define the categories — specific skills — within each skill area. These categories break your skill areas down into different individual skills. Each skill area should have about 5 or 6 categories; they should be mutually exclusive and collectively exhaustive.

yes Let’s go back to our earlier age and nationality examples to better see how this works. Breaking down age groups into specific ages is easy to do with the MECE system; simply choose a starting and ending age for each group, making sure there is no overlap —  such as 25–34 and 35–44. 
no However, breaking nationality into groups isn’t as clear. Should the Egyptian nationality be placed under an African group — based on geography — or under a Middle Eastern group — based on language? 

 

Example

Back to Emily making her customer service test from the first section. She’ll want to further define the key categories that are covered in each of the main skill areas that she previously outlined. Further outlining the first skill area might look something like this:

pink-1 Ability to understand customer concerns or requests, and uncover root causes
  blue-1 Understanding the main issue in a stated problem
  blue-2 Uncovering the root cause of an issue
  blue-3 Empathizing with concerned customers
  blue-4 Understanding the potential impact of an issue for the customer
  blue-5 Understanding the potential impact of an issue on the company
pink-2 Ability to interact with customers in a positive and constructive manner
pink-3 Ability to accurately and efficiently determine appropriate solutions
pink-4 Ability to avoid negative or unintended consequences

The skill areas and categories together form a comprehensive outline of your test. By following this outline, you can create questions with specific objectives in the next step.

 

Write questions

The questions in your test should relate to the skill areas and categories that you've defined in the previous step. Each question should address only one skill area and one category.

An expert in the field should be able to read your questions and immediately understand which skill area each question relates to. Your questions should be based on real-world scenarios that the candidates have to solve, avoiding generic, theoretical questions that can be answered from a manual without having the necessary skills.

 

What this means in practice

To come up with relevant and effective questions, first determine what a test-taker should be able to demonstrate for each defined category.

Ask yourself what knowledge or skill would be good evidence of the candidate's abilities.

You should create at least as many questions as you plan to have your candidates answer, though it’s recommended that you create a larger bank. Those questions must be spread evenly across skill areas. For example, if you plan for your candidates to receive 15 questions, across three skill areas, you’ll need to write at least 5 questions for each skill area.

We recommend making more questions than are needed in a single sitting of a test. This way, TestGorilla can use a different subset of your questions in each assessment. This helps protect the integrity of your test, as it reduces the chance of questions being leaked online.

The difficulty of the questions should vary somewhat within each skill area, while still maintaining the overall level of difficulty determined for the test. That way, the test can accurately distinguish a range of ability levels.

Most of your questions should be scenario-based questions — also called situational-judgment questions — that ask test-takers to solve a specific scenario. This will help evaluate their skills in dealing with a relevant issue or situation, rather than just their theoretical knowledge.

In general, avoid questions of the type "What does X stand for?" or "What does Y do?" that can be easily answered with a simple Google search.

Writing situational judgment questions is discussed at length in the third article of this series: Writing situational judgment questions.

 

Example

Back to Emily's customer service test. A question in the fourth skill area — Ability to avoid negative or unintended consequences — could be:

Showing your compassion about the inconvenience experienced.
Allowing the customer to vent and calm down.
Asking the customer to come back when they have calmed down.
Explaining what the best solution is for this moment.

It's important that you do not copy questions from an existing source, either online or offline, unless you own the property rights to that source — such as a test you've always used for recruiting, that you're digitizing with TestGorilla.

 

Clarify

The goal of each question is to measure a specific skill, ability, or knowledge. Therefore, you should make sure your questions are straightforward and easy to understand. Candidates shouldn't have to spend time figuring out what exactly a question is trying to ask, nor should they be purposely confused by overly ambiguous answers.

 

What this means in practice

blue-1 Be clear and concise in your wording. If not, you're measuring someone's language skills, vocabulary, or reading ability rather than the skill area to which the question belongs.
blue-2 Avoid negatives. Word like not,  false,  least,  worst, and except can cause unnecessary confusion about the action the candidate needs to take.
blue-3 Don't try to trick candidates. For example, multiple-choice questions should have one correct answer and several distractor choices. The distractors should be somewhat tempting, but not overly tricky.
blue-4 Don't give away the correct answer. Don't make one answer much longer or short or very different from others. Avoid adverbs or other qualifiers that signify whether an answer is desirable or not — words like just, only, immediately, or right away. Don't give the answer to one question in another question.
blue-5 Limit questions that have multiple correct responses. This includes limiting the use of all of the above and none of the above as options. You can include a few of these, but should avoid them whenever possible.

 

Perform a thorough final check

Every test should be a good predictor of job performance and should also provide a good experience for job candidates. Thoroughly reviewing your finished test questions is critical for quality control.

 

What this means in practice

Make sure you can say  Yes to each of the following questions:

yes Do your questions cover the skill areas appropriately and completely?
yes Are the questions sufficiently varied — to avoid duplication of knowledge tested?
yes Is every question and answer option crystal clear?
yes Is all provided information relevant to the answer?
yes Are the image, audio, and/or video materials you're using clear and relevant? Are graphs and diagrams properly and correctly labeled?
yes Did you check grammar and spelling for all questions?

 

Next steps

As mentioned above, this is the first article in a series of five, to help you create your own tests using the TestGorilla platform.

The full series includes:

  1. Developing an effective screening test
  2. Choosing a question type
  3. Writing situational judgment questions
  4. How to create your own test
  5. Creating a coding question or test

We recommend you next read article two: Choosing a question type

Related articles