Which AI agent Platform to Use? ¶
For individuals and organizations who are unfamiliar with the development of conversational AI solutions, this seems like a daunting question to answer because there are hundreds if not thousands of agent options available out there.
Like adopting a child, investing in an agent solution is not a light decision nor a short-term investment. We would like to share with you the tips on whether and how to make a wise AI investment. In particular, we recommend three quick rounds of due diligence plus a deep dive.
Round 1: Decide on whether you'd need an agent at all¶
Just like making any investment, it's always good to start with a justifiable investment rationale and a clear investment goal. The same principle should also be applied to an agent investment. First, use this quick assessment to decide whether it is worth investing in an agent solution.
Round 2: Decide on which type of agent you'd need¶
Once you decide that you do want to invest in an AI agent solution, a simple Google search will present you with hundreds if not thousands of AI agent platforms! Almost all of them claim that they are powered by generative AI and all seem to offer similar features, giving the first impression that any of them would do the job.
In reality, not every agent platform is created equal or is a perfect fit for the agent solution you have in mind. For example, many of our customers mentioned to us before they found Ascent, they had explored 20+ agent platforms but failed to find the one that meets their needs.
To help you expedite your agent platform selection process, in this round, we help you identify the type of agents you need before diving into the nitty-gritty details of an AI agent platform. This step is utterly important since different types of agents require different types of technologies and support, which in turn affects the type and size of investment you'd need to make.
1. Question-Answering agents (QA agents)¶
The main task for these agents is to answer user questions based on public and/or proprietary information. For example, universities may use an agent on their website to answer student questions about learning programs (see Image 1.1 below), or hospitals may use a agent in their mobile app to answer patient questions on medical services. Very similar to the behavior of ChatGPT, this type of agent is typically driven by user questions and assumes that all questions can be answered in a SINGLE response, which is not the case in many real-world applications.
2. Question-Asking agents (Survey/Interview agents)¶
The main task for these agents is to ask user questions and gather information from an intended audience. Very similar to an online survey, this type of agent has been used to gather information for research studies, personality assessments, and job interviews (Image 1.2 provides an example of an interview agent). Typically, this type of agent is NOT capable of answering user questions including clarification questions during the process.
3. Proactive, Two-Way Conversational agents (Proactive agents)¶
In real-world applications, agents often need to perform multiple tasks simultaneously to deliver desired user experience and accomplish the business objectives. For example, a QA-only agent on a university website may be inadequate because the same agent must also ask questions to gather necessary information from the visitors before it can answer their questions (see Image 1.3 below). Similarly, a survey-only agent for job interviews may be unsatisfactory because it must also answer candidate questions regarding the positions and the organization during the interview. This third type of agent is a hybrid of the QA and survey agents, capable of both asking and answering questions during a conversation, known as Proactive Chatbots.
Round 3: Evaluate AI agent Platforms by the Type of agent Needed¶
Once you decide the type of AI agent you need, you can then use the relevant agent KPIs to evaluate AI agent platforms. Below lists the agent KPIs relevant to each type of agent.
agent KPIs for evaluating AI platforms for QA agents¶
- Answer quality and quality control
- Handling complex Q&A
- Real-time agent behavioral monitoring and improvements
agent KPIs for evaluating AI platforms for Interview agents¶
- Abilities to elicit quality user responses
- Abilities to handle user digressions
- Abilities to generate interview summary reports and analytics
agent KPIs for evaluating AI platforms for Proactive agents¶
- Interactional intelligence
- Personal intelligence
- Prudence
- Plug-n-play
For a detailed description of these KPIs, please visit this blog
Take a Deep Dive: Compare agent Options by 3 Factors¶
The the first 3 rounds could help you quickly decide whether it is worth investing in an AI agent solution and weed out most of the agent options that will not help you fulfill your agent goals. Assume now that you have 3-5 agent platforms to do a deep dive on. We recommend that you evaluate and compare these selected platforms along three factors as listed below.
I. User Experience and Business ROI¶
an agent or conversational AI agent is meant to scale out human-human engagements (e.g., university enrollment counselors with prospective students and college academic advisors with existing students). Since an agent acts on the behalf of an organization, a key criterion to evaluate an agent platform is to assess how well an agent engages with a user and delivers satisfactory user experience while maximizing business ROI, such as improving prospective student conversion and existing student retention rate. When evaluating how well an agent option can deliver satisfactory user experience and business ROI, you want to compare available agent options on the following aspects:
-
How well can an agent drive an empathetic and productive two-way conversation?
-
How well can an agent deeply understand the psychographic characteristics (e.g., interests, unspoken needs, and personality) of its users from the conversations and use such insights to personalize machine or human advices?
II. Time to Value and Cost of Ownership¶
If not careful, investing in an agent solution could become a nightmare as if investing in a bottomless pit that requires constant new investments (e.g., time and money) but with little return of value. To avoid the pitfalls, it is important to deep dive into agent options and compare the following:
-
How much AI training do I need to do up front to teach a agent before it can function and deliver value for me?
-
What kind of customizations are supported?
-
What resources are required to do the customization (e.g., non-IT subject matter experts, dedicated IT personnel, or professional services)?
-
How long will it take to make all the customizations?
-
How do I test, iteratively improve, and deploy an agent solution? How easy and fast can I do this?
-
What kind of reporting and analytics are available for me to access? Are they real-time data?
-
How can I monitor and maintain or upgrade my agent? Will the maintanence or upgrades interrupt ongoing conversations?
III. Enterprise-Scale Adaptibility and Extensibility¶
When an organization adopts an agent solution, it is often the case that the organization wishes to build upon the current solution and support new users of the agent. Thus it is important to evaluate whether such a solution can be easily and quickly adapted to support new tasks or be extended to support multiple tasks. For example, when an educational institute starts with an agent solution to support student recruitment, it may want to extend the agent to help prospective students in their application process. Likewise, the educational institute may want to add additional agents to help student retention or alumni engagement. To evaluate how quickly and easily an agent solution can be easily extended or reused in whole or partially to support new tasks, ask the following:
-
How quickly and easily can I reuse and adapt an existing agent to support a new task?
-
How quickly and easily can I reuse and adapt an existing agent to support multiple tasks?
-
How can I reuse just part of an agent (e.g., Q&A only) in another agent?
-
How can I connect an agent to external resources, such as CMS or CRM system?
-
How can I extend an agent with external functions (e.g., my own algorithms or third-party functions such as language translations)?
-
How can I white-label an agent or embed an agent into my propreitary solutions?
Build Your Own agent Comparison Table¶
To help your Round-2 evaluation of multiple agent options, we have created a set of detailed comparison criteria by the 3 factors mentioned above:
For your convenience, we list all the evaluation dimensions in a table below, supplemented with concrete examples wherever possible to help explain each evaluation dimension. While we use building an educational agent solution as an example, the evaluation criteria should be broadly applicable for building any agent solutions.
You can also download the comparison template in an Excel file to build your own comparison sheet and evaluate different agent options by checking off each dimension for each agent option.
Click on the "+" button to zoom in on the comparison table. You can also download the pdf for offline review.
What's Next¶
Ready to create your own AI agent? Check out these design tips for creating super helpful and user-pleasing AI agents to help you scale your business and delight your audience.