A guide to interpreting the results of a coding test.
Coding tests are technical tests designed to assess the coding skills of a candidate. In addition to providing a percentage score, we give you the ability to see how the candidate actually conducted themselves during the test. This article explains how to read and understand the report. Coding tests are available on all plans.
Approx. reading time 4 minutes
In this article
Overall result
The overall result is found in the Test section of the candidate’s results page:
Here you can see the score the candidate achieved, along with how long they took.
Click View report to view the detailed test report.
Coding test report
The coding test report appears as a new tab in your browser.
|
At the top of the page, you can Email this report to a colleague. The report will be available to them for 14 days.
|
||
|
Test score. The test score represents the % of the validation test cases that are passed. The test score breaks down into a correctness score and a performance score, though most coding tests will only have a correctness score. |
||
|
Correctness score. The correctness score covers all the validation test cases that determine if a given input returns the required output. |
||
|
Question. The coding question that the candidate received. Every coding test in our library has a database of possible coding questions. Only one of these is given to the candidate during the test. The question typically gives some context, specifies the requirements, and gives a few examples to illustrate how the code should work. |
||
|
Timeline. You can play the code back from start to finish using the timeline. This allows you to see how it came together. On the timeline, we label when candidates copy & pasted elements of the code. The pasted parts are highlighted in yellow in the editor. |
||
|
Solution. This window is dynamic — the information displayed will depend on the position of the timeline bar. At the start of the timeline, this window will be largely empty. By the end of the timeline, you will be able to see the candidate’s final entry before submitting their solution. |
||
|
User and validation results. Here you can view the candidate’s responses to each test case along with its result. Clicking on each test case will display the expected result along with the user result based on the candidate’s code. The validation result is the outcome of the candidate’s final submission for the exercise, and determines whether it passes or fails. |
Test cases and validation cases
A test case is a pretend scenario that the candidate’s code is tested against when they choose to run the code during the test. The purpose is for them to ensure the code works. These test cases check if a given input in their code returns the required output. Essentially, the candidate uses test cases to see if they’re headed in the right direction. The results of the test cases have no influence on the test score.
A validation case is the actual coding exercise they must solve, and determines the test score. After submitting their final code, it is fully validated and checked to ensure it produces the expected result. Failing the validation case means failing the exercise. During the test, the candidate cannot see if their code passes the validation cases.
Theoretically, it is possible to fail test cases but pass the validation case. It’s also possible to pass the test cases, then fail the validation case.
Error messages
In the event that the candidate fails the test or validation cases, we'll present an error message. These are some of the typical error messages that may be encountered:
- Task finished with errors. An exception occurred during the execution of at least one test case. In this case, the score is 0%, even if some test cases may have passed, because none of the validation cases were passed.
- Task reached unknown status. The code could not be compiled. An example of when this might happen is if the candidate made changes to the function signature. In this case, none of the test cases pass, and the score on the test is 0%.
- Task timed out. The code could not be executed in the given runtime limit. This leads to a 0% score on the test.
- Unsupported operand type. This indicates the candidate used an operation that isn't supported by the particular code library used for this test scenario. Such as using a mySQL command instead of one for SQLite.
Common questions
Does a 0% mean that the developer is bad and I shouldn’t hire them?
Not necessarily. Remember how your math teacher always encouraged you to show your work? That was because if you didn’t, you couldn’t get partial credit.
Unfortunately, a computer system can’t really give partial credit — code either works or it doesn’t. This is exactly why we share the coding report. It allows any of your in-house developers to review the candidate’s solution, and make a more informed decision.
The code might have been perfect, except for one small typo that the candidate didn’t have time to fix, resulting in a 0% score. You could miss out on a great candidate if you don’t have a developer review the code.
Who can I share the coding report with?
The report can be shared with anyone, whether they are a registered TestGorilla user or not. You simply enter the email of the desired person, and we send them a direct link to the report. They will have to be added to your account and log in to view it.
If you are not a developer — or don’t have a background in coding — we recommend having a qualified person look at the report for you. They will be able to provide further insight into the candidate’s performance and submission.
What is a performance score?
Performance test cases make up the performance score. Those test cases not only require the right output, but the output also needs to be returned within a given timeframe (in milliseconds).
Performance test cases are only used if there is an efficiency requirement in the code. The performance score will be displayed below the correctness score, if applicable.