Creating a coding question or test

Updated August 13, 2024 16:40

Article 5 of 5 in a series on creating your own tests.

This is the fifth article in a series of five, to help you create your own tests using the TestGorilla platform. The full series includes:

Developing an effective screening test
Choosing a question type
Writing situational judgment questions
How to create your own test
Creating a coding question or test (optional)

The option for adding your own tests is available to account owners, admins, and recruiters who are on our Pro plans.

When creating your own test, we recommend outlining it before putting it into the TestGorilla platform. This will help ensure that everything is organized and laid out exactly as you want.

Approx. reading time 8 minutes

What is a coding question?

Coding tests and questions are used to assess a candidate’s ability to create or debug code in a specified language within a certain time limit. For most coding languages, we develop tests in the following categories:

Algorithmic thinking (e.g. searching, sorting, recursion, iteration)
Language-specific concepts (e.g. data structures, object-oriented programming)
Debugging

The key difference between a coding test and a coding question is that a test contains a bank of coding questions, and our platform will cycle through them. A coding question is presented alone, as part of the Custom questions section.

Who can create a coding question?

Any user with the Owner, Admin, or Recruiter role is able to create a custom coding test or question. Due to the highly technical nature of coding tests, you are required to have an expert in your required coding language create your questions for you.

Note: TestGorilla cannot assist you with the content of custom tests. Any tests you create yourself must be an original creation.

Available coding languages

We currently have 20 coding languages available, which can be used in both tests and questions:

C: 10.2.0
C# : 6.12.0
C++ : 10.2.0
Go: 1.16.2
Java: 15.0.2
JavaScript: 18.15.0
Kotlin: 1.8.20
PHP: 8.2.3
Python: 3.10.0
R: 4.1.1
Ruby: 3.0.1
Scala: 3.2.2
SQL: SQLite 3.31.1
Swift: 5.3.3
Typescript: 5.0.3

In our language-agnostic coding tests, you can use all of the above languages as well as the following:

Dart: 2.19.6
Elixir: 1.11.3
Erlang: 23.0.0
Julia: 1.8.5
Perl: 5.36.0
Rust: 1.68.2

Note: Only one language can be used per test or question.

Creating a coding question

Coding questions can be used in two ways:

As part of your question bank for a My company test
As a custom question, added in Step 3: Add questions during the assessment creation process

The process of creating your coding question is the same, regardless of which method you are following. Each of the linked guides goes into more detail about how to reach the required stage. This guide will take you through the process of using the question editor to create your coding question.

Select your language

The first screen presented to you after launching the question editor allows you to choose your coding language. There are 15 languages to choose from, choose one from the grid to move to the next screen.

Note:

If you are creating a My company test, the language must remain the same for all coding questions within the test.
If you are using custom questions, you can choose a different language for each question.

Question editor

The question editor is on the next screen. The text box on the left side of the page is where you will write your test description. This will provide the details your candidate needs to be able to complete the exercise. You can format the text by using the toolbar at the top of the text box. This will allow you to highlight key passages of text.

Writing the test description

We recommend breaking down your test description into five steps:

	Give some context and the overall goal of the coding task. This brings the question to life and is necessary for the candidate to understand the intent of the coding task. Here's a good example of what this would look like:
	Articulate in clear terms what the candidate needs to do. For this example, the candidate needs to write a function. You'll want to Specify the output: what should the function return? Specify the arguments of the function (the inputs) Working with our above scenario, here is an example:
	Give an example of input and output that satisfies the requirements. This might look something like this:
	Add relevant details. In our vault example, this would be:
	Optional: Add any potential hints or other considerations. This is not necessary in most cases but it might be something you want to add.

Entering code-specific information

The right side of the page is for you to enter code-specific information. There are 4 tabs for you to complete:

Function signature
Here you can set the type of variable the function returns, its name, and the input parameter(s). You can choose from the following parameter types:

String
Integer
Boolean
Float
Array (entered as comma-separated values such as 1,2,3,4,5)

Note: Objects are not supported as a function parameter

Initial code
The initial code is the starting point for candidates. It shows the function signature. You can add some code as a starting point if your task requires it. Please indicate where candidates are expected to insert their own code.

Note: This is not necessary if you're developing a debugging task

Test cases
Test cases are combinations of input and output parameters that satisfy the requirements you've specified in the question. Under this tab you'll find two headers:

Test cases to be used during the test. These are test cases the candidate has access to during the test. They help guide the candidate to complete the coding task successfully, since candidates can test their code against these test cases during the exercise.

If the candidate clicks run during the coding exercise, the code they have written is executed with the input parameters specified in each of these test cases.

If their code returns a result that matches the results you specified, the test case is passed, and the candidate will be told so. If their code fails, they will also be told so; this enables them to search for errors and make edits before final submission.

These test cases should be straightforward. 4 is the minimum to be able to proceed.

Test cases to be used for validation. These are test cases that will be used to produce the final test score. They are not used during the assessment, so candidates cannot check their code against these test cases.

These test cases should cover all requirements laid out in your scenario, including exceptions, corner cases, etc. We recommend creating 6-10 depending on the complexity of the question.

The Correctness score will be the percentage of passed validation test cases.

You can also add test cases to measure the efficiency of the code. These test cases produce the Performance score, showing the % of performance tests passed.

The performance is defined by the runtime required to execute the test case, in milliseconds. The candidate’s code passes the performance test case if the runtime of the code is fully executed within the time limit you have set.

A performance test case is relevant for coding tests where scalability is important. You can see how much time it took to run a performance test case when you run your verification code (see below):

Verification code
The verification code is basically the model answer. You need to code it and run it to ensure all test cases you entered are passed. Only then can you save the question.

Once the candidate has completed the test, there will be a coding report that shows the candidate's code and the result of the validation test cases that determine the score. You can read more about the coding test report here.

SQL tests
SQL tests are a bit different, given the additional database component in the test. Here you can see the purpose of the different tabs when writing a SQL test:

Database structure. Here, you choose whether you ask the candidate to write a query — using SELECT — or a database update —using INSERT, UPDATE, or DELETE.
This tab is also where you'll script the initial database by creating the table(s) and inserting the appropriate values. In most cases, you should copy your initial database structure as a code block into the test description. Otherwise, candidates have no idea how the database is structured and what’s in it.
Initial code. Here you can include some first lines of code if your task requires it.
Test cases. As above, test cases are split between test cases that are available to the candidate — Test cases to be used during the test — and test cases used to determine the Correctness Score — Test cases to be used for validation.

The biggest difference with writing SQL questions is in the structure of test cases. When writing test cases, you'll be able to insert additional values into the database in the Test case setup. This setup is a script that will run immediately after the creation of the database structure but before the candidate code. The verification code works in the same way as in other coding questions: this is the model answer and should be clean and efficient.

When writing a SQL test case, it's good to keep the following flows in mind.

If you ask the candidate to create a query, you define the expected result of the query given the database structure and the test case setup, as per Figure 1 below

Figure 1: Sequence of SQL scripts being executed when asking a candidate to write a query (using SELECT)

In the event that you ask the candidate to write a database update, you should also write a checking query. This query will be executed after the test taker code.

In the Result you specify what the checking query is supposed to return. See figure 2.

Figure 2: Sequence of SQL scripts being executed when asking a candidate to write a database update (using INSERT, UPDATE, or DELETE)

Note: Every test case recreates the database. Test cases are therefore isolated from each other.

Saving your question

The easiest part of the process! When you have completed entering all of the details, click the Save button in the top-right corner of the screen.

Our platform will run all of your code first to ensure that there are no errors. You will receive an error message if any are detected — you’ll need to go through everything again to correct your error.

If the save is successful, the question editor window will close and return you to your previous screen.

Language-agnostic coding tests

This is available on the Starter and Pro plans

With our language-agnostic coding tests, you can configure which programming languages will be available for your candidates to choose from and may include as many languages as you like from the 20 currently available.

This way you will be able to assess the candidate's ability in a programming language of their choice.

To see a list of all available language-agnostic coding tests, click here.

Simply add a language-agnostic coding test to your assessment, after which you will be presented with a screen that allows you to choose the coding languages that candidates will be able to choose from. There are 20 languages to choose from, and you can select any number of them.

You can edit the programming languages available in a language-agnostic coding test by clicking on the edit button next to the added test.

Note: Candidates can decide the language used during the test, and are provided practice questions as well as onboarding tooltips.

Candidates can also use our configurable IDE (integrated development environment) to personalize the color theme, window size, screen size, and font size to match their needs.

Common questions

Why do I need to create multiple coding questions for my test?
This allows our platform to present a different question each time your test is used. If you have candidates who have taken your test before, they will have an advantage over the other test takers as they will know the content of the test ahead of time. Having multiple questions reduces the risk of this happening.

Should I use a coding test or a custom coding question?
This will depend on your own needs. If you frequently screen candidates for the same developer role, taking the time to make a full company test might be better, so that you don't have to do the process again anytime soon. However, if you only need one or two questions, and have the space within your plan's allotted custom questions, that could be easier.

Is there a limit to how many coding tests or questions I can use in my assessment?
Depending on your plan, your assessment can include up to 5 tests and as many as 20 custom questions. In theory, you could have 5 coding tests and 20 custom coding questions in your assessment. Just bear in mind the overall length of your assessment; a candidate is unlikely to take your assessment if they have to spend 12 hours on it!

Next steps

While not included in this series, we do have other articles that are helpful when making your own test that are worth taking a look at. They are:

Creating a coding question or test