1. How does Vervoe’s AI work?
What we have developed is not a simple “one size fits all” approach. Our machine-learning algorithms are multilayered, unique and unlike any other. Not only do we learn about the candidates, but we also learn the unique preferences of each individual employer.
Vervoe's AI feature can predict scores for any type of skills assessment. We have found a way to create bespoke algorithms for every employer and specifically for every one of their open roles. The algorithms adapt to the unique preferences of the employer and don’t require a specific set of questions.
Vervoe's machine learning feature consists of 3 models that help calculate candidate scores:
1. The "how" model: This model analyzes a candidate's behavior on the platform- how they complete the assessment.
Example: it observes how long it takes a candidate to respond to questions, spelling, and grammar, how many times they make changes to an answer, are they copying and pasting, etc. It is looking at many data points to help score their performance. (It's important to note that there are no right or wrong behaviors. Our machine learning models simply learn what kind of behaviors correlate with good or bad grades, and it's different for every type of role.)
2. The "what" model: The system analyses the quality of the candidate's response. All questions require a correct answer sample to compare the candidate's answers to the answers provided. Through natural language processing, a candidate's responses are viewed by looking for words, phrases, sentence structure, and other sentiments that accurately reflect the outcomes required.
Example: Comparing how closely a candidate answers the question vs. the correct answer sample provided within the assessment.
3. The "preferences" model: This model requires input from the user to train it to understand what the scale of bad to good answers look like for their specific use case. This method uses a model called 'iterative' where a user blindly grades a set of candidate responses to individual questions by giving them a score from 0-10. The set of questions that are exposed to the user to grade are the furthest apart from each other. This scoring feedback teaches the AI what you value; things like spelling and grammar or a positive sentiment.
Example: If a user grades one response as a 10 our model will then look for an answer that appears to be completely different to see how you score that answer. This variation in responses helps the model quickly identify and plug the gaps in between the potential score ranges to accurately grade all candidates with your preferences in mind.
How many candidates need to be graded for the Preferences model?
Initially, you’ll be asked to blindly score a small portion of candidate responses to individual questions in your assessment. We need around 20 graded responses to start optimizing your assessment for your own tailored needs. Obviously the more you grade the more we understand about you but we’re aiming to get about five grades from you across the spectrum of 1 – 10.
Please note that for video and audio answers, the AI is not looking at facial features or voice intonation. It reviews and analyzes a transcript for that response.
The AI feature will score and rank the candidates based on the combination of data points compiled from the three models.
2. What does the AI score predict exactly?
The AI score is a prediction on how you will grade the candidate based on your grading preferences that our AI feature has learned.
3. How accurate is the AI score?
Initially, it works with 83% accuracy in predicting top performers (verified via cross-validation). The level of accuracy improves the more candidates you grade manually.
4. How does it help eliminate bias?
- Bias is a very real threat with AI - since typical AI algorithms learn from humans (who are naturally biased), we have made concentrated efforts to avoid this.
- We do not provide our models with any identifying information like gender, race, etc. Our models are blind to anything that could potentially create bias. Our algorithms only detect and analyze candidate performance data.
- Always on monitoring: We monitor manual grading to ensure data integrity. If manually graded responses show a clear bias towards the mention of certain keywords we can detect the drop in score and flag it whilst removing it from our learning set.
5. When do I see an AI score and when do I see my own score?
The AI score will be calculated as soon as the candidate submits their skills assessment. It may take a few minutes for the score to appear on the card. The AI will not show a score on every question instead, the score depends on the overall assessment.
Your score will appear as soon as you grade every response submitted by the candidate.
6. How are the graded candidate cards sorted?
Our system will sort the graded cards based on the highest to lowest score. The employer's score will take precedence, but if no employer grade is provided for a candidate - the AI feature's score will be used to sort that card.
7. How do I turn off the AI score? I don't want it to grade my candidates for me.
This feature is intended to make your job easier, without forcing you to change the way you work. It's just a prediction. Keep grading the way you're used to. Once you score all the responses for a candidate, your score will be available. However, for individual questions you can choose to either turn off the grading or choose to manually grade the question. This means that you will not receive a grade for any candidate who completes until this question or questions are graded.