The Bongo AI coach is designed to provide tonal analysis and coaching tips for learner videos. These will help drive practice opportunities for learners in a safe-space. Once videos have been submitted for review, instructors can use the AI tips and analysis to cover the basics while they can focus on providing feedback based on their subject-matter expertise.
Administrators can also generate Learning Objectives which will be used to provide targeted coaching tips and to optionally be used to automatically score a learner’s video.
What is the underlying technology for Bongo AI Coach?
The Bongo AI Coach utilizes a large-language model and generative AI to produce coaching tips and tonal analysis. The Bongo product currently interfaces with an Open AI GPT model though new versions and other models may be considered in the future as the technology evolves.
How is the Bongo AI Coach different from ChatGPT?
While ChatGPT utilizes the same underlying AI model that the Bongo AI Coach will be using, there are key differences that users can expect. ChatGPT is an open dialog and can take a significant amount of trial-and-error to get the desired results. It also has limitations on how much data a user can send in.
The Bongo product builds prompts that are tailored to our use case of providing learners with actionable feedback directly from within the Bongo product. Users will never need to enter prompts to get the results they need. Bongo will also handle complex prompt engineering that will allow for feedback on videos that would otherwise exceed the limits one would encounter when interfacing directly with ChatGPT.
What are the different elements of Tone Analysis?
The Bongo AI Coach will automatically assess the tone of a learner’s video submission across the following four dimensions:
- Confident vs. Nervous
- Persuasive vs. Informative
- Humorous vs. Serious
- Casual vs. Formal
The coach will provide insight as to where a video sits on the ranges above and provide examples that reinforce that analysis. Because AI Coach operates off of the video transcript, the tone analysis focuses on the types of words and phrases that a learner uses in their video.
What can I expect from Coaching Tips and Feedback?
The Bongo AI Coach will review a learner’s video and share tips and feedback around, opportunities for improvement, areas of excellence, and unique facets to what they have presented. The Bongo AI Coach will provide specific examples from a learner’s transcript to support each of those areas.
What information is Bongo sending to OpenAI?
In order to retrieve coaching tips and tonal analysis, Bongo will be sending the video transcript and Auto Analysis metrics. The actual video and any information about the user will not be sent. This content will not be used to train any language models to avoid exposure to proprietary or private information.
How is my personally identifiable information (PII) being protected?
No PII will be directly sent to OpenAI's GPT model, only data related to the video transcript generated by AWS Transcribe. The transcript data will be encrypted both in transit and at rest. Bongo will send the data to a version of the model that is appropriately located relative to a user’s region. Finally, by using OpenAI models through an API, the transcript will not be used for training the models which will avoid exposure to proprietary or private information.
How does the Bongo AI Coach handle bias?
The Bongo team is aware that AI models can be subject to bias based on the data set on which they have been trained. While there is no magic bullet, we are building in some safety measures. First, closely controlling how the OpenAI model is prompted will ensure more consistent results as compared to an open-ended experience. In the first release and until further testing confirms reliability, we are relying on a separate service to transcribe the speech in the video and will not be sending the actual audio track to provide an objective assessment.
Can I turn off the Bongo AI Coach?
If your institution or organization does not want the AI Coach activated for their learner videos, please contact our support team to discuss options for turning it off.
Can I customize the tonal ranges evaluated by the Bongo AI Coach?
For the initial release, all users will have a standard set of 4 tonal ranges that the AI Coach will evaluate. The AI Coach is an area of strategic investment for the Bongo team and the ability to customize tonal ranges is an item for future consideration.
Can I customize how the Bongo AI Coach will review videos for my assignment?
Yes. By providing course materials at the time of assignment configuration to develop Learning Objectives, the AI Coach will provide feedback specifically around how well those Learning Objectives have been met in addition to the standard coaching feedback.
Do the Bongo AI Coach results impact the Auto Analysis score?
The coaching tips and tone analysis produced by the Bongo AI Coach are designed to be useful insights on areas of improvement, but they will not impact the Auto Analysis score.
If I record multiple videos for an assignment, will the Bongo AI Coach evaluate them against one another?
No. The Bongo AI Coach will provide feedback on each video in isolation.
Are there any situations where I will not get Tips, Feedback, and Tone Analysis from the Bongo AI Coach
If there are less than 30 words in your video, Tips, Feedback, and Tone Analysis will not be available as the AI model does not have enough information to produce quality results in those circumstances.
Can I create my own Learning Objectives from scratch?
No. Bongo’s proprietary process to analyze reference materials is designed to build Learning Objectives and criteria that can be effectively used in the feedback and scoring process. While edits are supported, creating Learning Objectives from scratch has been found to produce lower quality feedback and scoring results.
Can I edit the Learning Objectives that Bongo creates for me?
Once Learning Objectives have been generated, administrators have the ability to edit or delete both the objective and its sub-criteria. Below are a few key considerations when editing Learning Objectives and their criteria:
- The generated text in the Learning Objectives and their criteria are designed to build custom prompts to the AI model that generate targeted coaching feedback and automatically score videos (if Smart Scoring is enabled). Drastic changes to this text could adversely impact the quality of the feedback and scoring.
- Learning Objectives and their criteria are designed to work together. For best results, maintain a consistent theme between the objectives and the criteria. For example, if a Learning Objective is about “basketball” and one of its criteria is changed to be about “dogs”, the quality of feedback will be compromised. of feedback will be compromised.
- Changing the action verb in the Learning Objective or criteria is a good way to customize feedback and scoring results to meet your needs. For example, changing the following text from “Describe how a counselor should use empathy when meeting with clients” to “Demonstrate empathy when meeting with clients” will fundamentally change how AI Coach will evaluate learner videos.
Does the Auto Analysis score impact the Smart Score results?
No. Those scores are entirely independent. If you choose Smart Scoring as your Evaluation Type, we will automatically hide the Auto Analysis scores from learners to avoid any confusion.
How reliable are the Smart Score results?
The Bongo team has built tools to run the Smart Score algorithm at high-scale. By repeatedly running the same videos, the scoring results were confirmed to have reliability within targeted guidelines similar to that of human evaluation of subjective matter.
How accurate are the Smart Score results?
Smart Scoring has undergone a battery of tests to compare its results with those of videos assessed by human instructors. At this time, the Smart Score results typically fall within 5% of the human evaluations.
Can I make adjustments to the Smart Score results?
Yes. Instructors have the ability to make adjustments to the score provided for each Learning Objective. These adjustments will be tracked to inform future improvements to Bongo’s AI capabilities. A flag is available at the time of configuration to manage learner access to Smart Score results. Results can either be immediately published and available to learners or they can be hidden from learners until an instructor reviews and makes adjustments if necessary.
Can I turn off Smart Scoring?
If your institution or organization does not want Smart Scoring available as an evaluation type, please contact our support team to discuss options for turning it off.
How many Learning Objectives and criteria can I have?
There are no defined limits in the system on the number of learning objectives or criteria. However, testing seems to indicate that the time required to complete scoring when there are more than 8 criteria under a specific learning objective can increase quite a bit due to multiple cycles of querying the language model.
What languages does Bongo AI Coach support?
Learner’s will receive general feedback from AI Coach in any currently supported Bongo
language. The language of the feedback will be determined based on the browser settings of the user who is recording or uploading a video. However, generation of Learning Objectives, specific coaching tips and feedback based on Learning Objectives, and Smart Scoring are currently only verified in English.
What Bongo assignment types support Smart Scoring?
Smart Scoring is an available Evaluation Type for the Individual Project and Q&A assignment types.
How can Bongo help me build a Q&A assignment?
Assignment authors can provide reference materials in text format which Bongo will utilize to build a variety of questions for an assignment. Bongo will also analyze the material to provide answer criteria for each question. Authors can choose which best meet their needs and import them into their assignment.
Can I add my own questions or edit those generated by Bongo?
Yes. Assignment authors can create their own questions and answer criteria or edit those generated from the imported reference materials.
How does Smart Scoring for Q&A work?
Bongo will automatically score learner responses to Q&A questions based on the answer criteria configured. Each question can be configured for a total possible number of points that will be utilized in the scoring process. As with Smart Scoring for Individual Project, evaluators can optionally edit the Smart Score results.
Comments
Article is closed for comments.