Why 95% of custom-built GenAI projects fail

Users reported that they like the flexibility and ease-of-use of general-purpose LLMs, as opposed to custom-built solutions, which “can’t adapt to [companies’] workflows.” (Translation: General purpose LLMs allow users to bypass the guardrails that are built into custom solutions.)

OK. That’s intuitive enough. Perhaps more surprisingly, MIT found that one of the reasons custom solutions performed so poorly is because organizations are biased towards funding lower ROI projects over higher ROI projects.

That’s far less intuitive (on the surface at least). The teams that can secure funding more easily are Sales and Marketing, because they are more directly tied to revenue generation. But the teams that are most likely to see high ROI from custom-built solutions are “back office” functions like Legal, Finance, and Operations.  So, the high ROI projects don't get funded because they aren't directly tied to revenue generation.

So, you might say that the success rate of custom AI solutions is unnecessarily or artificially low as a result.

This seems like a logical question based on the report's results, but I think the report is missing something major. It jumps straight to recommending that companies use more complex Agentic AI solutions to help solve problems. 

But let’s go back to the part where 90% of employees are using general-purpose LLMs on a regular basis. If general-purpose LLMs are primarily good for “easy” tasks, yet almost all employees feel the need to use them, perhaps we should slow down and ask why that is. Can we count on employees to troubleshoot any potential issues in a complex Agentic AI workflow if they’re still struggling with the basics? Can we even count on a smaller group of employees to design the Agentic AI solutions? 

Personally, I think we need to consider a couple of clear trends before jumping head-first into Agentic AI:

First, it’s a lot harder than necessary for users to find information. Gartner has found that nearly half of employees struggle to find the content and data they need to do their jobs, which means completing even the most basic tasks can be challenging. AI can absolutely help with this, but only if it’s set up correctly. If organizations limit access to information due to cost, implementation difficulties, or an incomplete understanding of all the information sources employees need, the usability will suffer. This holds true for Agentic AI as well.

Secondly, employee workloads are getting bigger and harder. Sentry found that 81% of execs are asking employees to take on work above their role and training level. In this environment, “easy” is a relative term, since employees are undertrained and overloaded. Agentic AI might be able to help here, but again, only if it’s set up correctly. Ideally, humans should verify the output of AI workflows. But if the humans who are verifying them are not properly trained themselves, the system fails.

IMHO, I think there’s a major opportunity for organizations who are willing to address these issues more holistically. If employees need so much help with easy tasks, perhaps AI isn’t the answer.

Instead, companies may need to step back and reevaluate their job descriptions, organizational structures, or time allocation for training. Additionally, organizations can audit their systems and workflows to reduce the amount of “institutional knowledge” required to complete simple tasks.

Lastly, organizations can implement more nuanced frameworks for measuring the value of projects. It could include potential revenue generated, multi-attribution sales metrics, potential damage if tasks aren’t done correctly, and operational efficiencies. These measures could form a weighted metric that accounts for potential risks, rewards, and efficiencies while ensuring that even teams who don’t directly contribute to revenue are recognized. The weighted metric could then be used to inform the overall importance of projects.

This type of holistic thinking is much harder to do than simply implementing some new tech, but for companies who are willing to do the work, the potential upside is huge. Not only can companies save money by working more efficiently, they can minimize the amount of processing dollars required for AI projects while freeing up employees for higher value work.