Agentic AI: publishers in search of maturity, according to AI Builders

What is a Large Action Model? The engine of an autonomous agent

These new types of models appearing in 2022 are called “Large Action Models” or LAM. For the moment, these are rather fine-tuned variants of LLM or large language models to which part of the training has been devoted (in particular using reinforcement learning techniques) to carry out these tasks. These are the foundations of what publishers call “autonomous agents”.

“An agent is the combination of an orchestrator LLM, capable of breaking down tasks and assigning them, with a dedicated code fragment,” describes Stéphane Roder, CEO of AI Builders. “This piece of code takes care of each subtask, understands how the tool works, interacts with it, retrieves the produced result and transmits it to the orchestrator.” This would be the evolution of RPA. Unlike an RPA bot that must be manually programmed or transmitted the recording of a series of tasks, an agent “will find the actions to carry out on its own”.

The proliferation of these agents suggests a shift from prescriptive analytics, which LLMs promised to improve – to planned actionability. “This is a trend that Gartner predicted in 2014,” notes Stéphane Roder.

The first use cases for these more or less autonomous agents are “application assistants”. AI Builders distinguishes between two: application assistants integrated into office suites and those infused into business suites. “We are observing a basic trend among all software publishers who offer or will offer these application assistants integrating at least one agent,” comments the CEO of AI Builders.

Salesforce Agentforce and Microsoft Copilot stand out from the crowd

To help companies see more clearly in this jungle of rapid growth, the consulting firm has concocted its AI Decision Matrix.

Like Gartner’s magic square, AI Builders has defined four categories: AI Next Gen, AI Best-In-Class, AI Rising Star and AI Safe Bet. The position of the solution on the ordinate materializes its performance and on the abscissa its maturity.

Performance is evaluated based on the quality of the responses obtained, the possible customization of the assistant, security and data management, the number of features and the complexity of the tasks performed. Maturity is rated according to four criteria, namely the level of implementation of the source solution on the market, the extent of internal and external integrations, ease of deployment and use, as well as scaling.

Solutions labeled Next Gen and Rising Star are on the left of the table. The “rising stars” are under development – ​​therefore inefficient and unreliable – but promised to gain ground on the market. Next Gen solutions are considered efficient, but not very mature. The Safe Bet category, as the name suggests, brings together high-performance assistants with multiple uses and more reliable than average. “Best-in-Class” solutions are supposed to be the best on the market and are distinguished by their ability to integrate with companies’ existing IS. Office application assistants are indicated by an orange dot, while those dedicated to business tools are in red.

As a result, Zia from Zoho, Muse, Konverso and Work Intelligence from Wrike are the “Rising Stars”. Conversely, Salesforce Agentforce, Microsoft Copilot, Gemini from Google and Now Assist from ServiceNow are the most efficient and mature assistants. Safe Bet solutions bring together office tools from Dust, Notion AI as well as the business assistants Agent Lumi, Adobe Sensei and Amazon Q.

Priority to performance

“Across all categories of assistant, we consider Agentforce to be the most powerful solution to date,” said Dimitri Calmand, Data/AI analyst at AI Builders Research. “What makes the difference in our opinion is the Agent Builder capability which allows you to create your own agents with existing or additional actions via MuleSoft, among others, which saves considerable time.”

“Across all categories of assistant, we consider Agentforce to be the most powerful solution to date.”

Strangely, GitHub Copilot, yet one of the most popular generative AI tools among developers, is categorized as “Next Gen”, with Joule from SAP. “As much as GitHub Copilot is effective for generating code and tests, its integration into the programming environment and its ability to assist certain tasks specific to categories of developers are not very advanced,” explains Stéphane Roder.

As for Joule from SAP, “the most advanced functionalities are planned for 2025”. On the other hand, the fact that the German publisher is also developing agentic assistants would be revealing, according to the CEO of AI Builders. “If a player like SAP starts offering agents, it’s because we are seeing a standard being established.”

If a player like SAP starts offering agents, it’s because we are seeing a standard being established.”

It should also be noted that certain evaluation criteria were overweighted by the firm. Thus, the quality of the responses obtained, the complexity of the tasks carried out and the extent of the integrations are highlighted in this benchmark. “These are the criteria that will have the greatest impact on our client’s ROI,” explains Pauline de Lavallade, director of AI Builders Research. Conversely, securing solutions is more complex to judge since it often depends on a model of shared responsibility. Especially since there remain many unknowns about how to secure agents’ interactions with third-party tools.

Pricing: again, lots of experimentation

The decision matrix does not seem to take into account the pricing of the solutions. “Assistant prices vary depending on the options and levels of customization,” answers Dimitri Calmand. “Several tools, including Gemini and Notion, offer free trials, then the pricing increases to around 20 euros per month per user. However, more personalized business options significantly increase the cost. For example, Copilot for Microsoft 365 is priced at 30 euros per month per user, while Copilot Studio is billed at 200 dollars for 25,000 messages per month,” he illustrates.

In addition to different pricing depending on the modules chosen, Salesforce offers existing Sales, Services, Marketing and Commerce Cloud customers 1,000 free conversations and 250,000 free data credits for Data Cloud. Beyond that, the CRM giant intends to charge two dollars for each conversation. The CRM giant also offers another personalized pricing model.

For Agentforce Service Agent, it provides an ROI simulator taking into account the cost of customer service employees, the number of conversations they handle per day and the volume of support tickets to be transferred to Agentforce. This tool provided for illustrative purposes does not take into account implementation costs, but taking the client into account will reduce part of their payroll.

“There are also the beginnings of an economy of verticalized agents which is being established. Some agents will be developed by partners or companies before being marketed as on an App Store,” adds Stéphane Roder.

A changing landscape

The AI ​​Builders matrix which has just been released will be updated in six months. “It’s a market that moves very quickly. The solutions will continue to evolve every month, new players will emerge and existing tools will be renamed. Typically, at Salesforce, Einstein GPT has become Agentforce,” recalls Pauline de Lavallade.

“Imagine, an AI project is born and a week later, someone comes to me for an opinion,” exclaims Stéphane Roder.

In any case, the actors cited are in fierce competition. “There is the fear of disintermediation. Take the example of Microsoft which wants to connect its Copilot to everything, including Salesforce and SAP, and Salesforce which opposes this desire by offering its own agents,” mentions the CEO of AI Builder. At the annual Dreamforce conference, Marc Benioff was vehement towards Microsoft.

In addition to “generic” LLM or LAM, the consulting firm foresees the emergence of universal assistants, embodied by functions such as “Computer Use” and is closely monitoring the vision of “agentic automation” imagined by UiPath. “The concept of agents is expanding into many modalities, but we have not yet studied this part,” concedes Stéphane Roder. “Our customers are less interested in these solutions that interface with systems and they are still technically inefficient. We are at the very beginning of this ability to decompose reasoning.”