AI Optimize 2.0 goes into depth on which AI models are used in grading, what the estimated AI score range is per question, the average score given by members of your team, and more! This information will be highlighted within the top of both the Invite and Select stages, via the top statistics modal. This will be visible at all times, with different statuses to help the customer understand if there’s action required.
How do I trigger the AI Optimize modal?
AI Optimize will trigger on new candidate completions, or if you have an older assessment, manual grading of all questions for one candidate's assessment. If you have an assessment that once had the modal but now does not, this typically occurs whenever changes were made to the assessment questions. However, once 3-5 more candidates complete the newest version of the assessment, the modal will activate again.
AI Optimize Modal Features
Statuses
The statuses labeled within the AI Optimize modal allow your team to track and monitor how your AI score is performing across assessment stages. This also allows hirers to see which machine learning models are activating, so they can make direct optimizations in-real time, including optimizing the difference between AI and team scores to build trust.
Upon opening your assessment, you will notice a status representing the state of the AI's health and accuracy. The statuses reflected for an assessment are:
-
Medium: Correct answer samples are added.
Actions to do: needs more grades
The trigger for moving to Medium: 555 model is on for all questions -
High: 555 model is on for all questions
Actions to do: need more grades
The trigger for moving to High: 555 model + Iterative models are ON -
Optimized: Highest status for now, grading is complete
(You may choose to periodically grade as more variety appears in responses however no further grading is required)
By default, most assessments from the library will have the status of Medium, as they will have enough of a dataset behind them to grade with medium accuracy.
New assessments will have the default score of Medium.
Once an assessment reaches the High status, no additional grading is required unless a large volume of new completions occur that the AI is unable to confidently grade, or questions within the assessment are changed.
Please note: if you have performed a significant amount of grading in an assessment and the score is still medium, this happens if there is not a large variance in the grading. For example, all candidates have been graded 8-10.
You can still be confident the AI is performing as expected and candidates are ranked based on their performance. The candidates at the top of the ranking are likely better performers on the jobt than those in the middle - bottom.
Data per question
Estimated Score Range: A range guide has been added to the grading box to assist your team in your score selection. This improvement allows you to reach the optimal score for each grading bucket.
Grading Required: In the format of x/y: X is the number of responses you need to grade in order to optimize the assessment. Y is the amount that have already been graded from this target.
Responses: This is the total number of candidates that have respond to this assessment question.
Average team score gives you the ability to see who gave what scores to each candidate response. This feature should be used as a safety measure to ensure that candidates are being manually graded fairly and without bias. If a user feels a response has been graded incorrectly they can click the flag icon to the right, to mark the response for review.
Here you’ll be able to see individual candidate responses and scores, we’ll anonymize the candidate details. You can flag any outliers or inconsistencies that may introduce bias or affect your AI grading models.
Once grades have been flagged, they will be grouped, and you can filter for the flagged grades for a particular question. These can be periodically reviewed by your team and your account manager.
If it’s deemed that the score should be updated or removed, the Product and Engineering teams can action this request manually.
Question Insights
Question Insights show the AI models used to grade each question. This can be accessed at any time by clicking on the graph icon in the top right corner of the optimize modal.
The panel highlights in green the modals that have been activated and used to grade the question along with what they actually review. The status next to a deactivate modal alerts the user that grading is required to activate that model.
Watch a quick overview here.
Once you've optimized your AI's grading Health, you should now be ready to review candidates to select the best candidate for your role! Take a look at this helpful article that goes over this next step.