CIOs are increasingly examining generative AI projects with a greater focus on business value, paying particular attention to the costs of GenAI technology as well as its potential benefits.
This is a departure from the early days of generative AI, when companies were primarily concerned with exploring the possibilities of the technology and developing countless ideas for use cases. The business case for the technology is now becoming increasingly important as companies look to expand generative AI beyond the initial pilots. Two related needs are emerging among enterprise users: identifying use cases with the best ROI prospects and recognizing generative AI costs that could erode financial gains.
In this context, GenAI follows the trajectory of traditional IT implementations, which are ideally based on financial considerations and integrated cost controls.
“Last year we did a lot of experimentation,” said Juan Orlandini, CTO for North America at Insight Enterprises, a solutions integrator based in Chandler, Arizona. “This year we are finally looking at GenAI as another capability.” “We still need the justification and ROI of a traditional enterprise application.”
This is especially important for companies with small GenAI teams, limited budgets, and little margin for error.
Danielle Conklin, CIO at Quility, an online insurance or insurtech company based in Swannanoa, North Carolina, said the company has a two-person data science team, including herself. Rather than exclusively using off-the-shelf Large Language Models (LLMs ) for GenAI, Quility aims to develop its own advanced models, Conklin said. Early use cases include customer loyalty and CRM. However, she added that cost and ROI were crucial factors.
“Reaching to a sophisticated level requires time, personnel and resources,” Conklin said. “With two people we can only focus on one or two things. We have to make sure that the one we choose is the one thing that gives a high return on investment.”
She said the cost is much more than the initial investment in two people’s time: “Do we need to use other providers? Or third-party data? Do we need data cleaning tools and data quality tools? And there are long-term maintenance costs.” the model [and] Refreshing the model.”
Revealing the costs of generative AI: managing changes and preparing data
IT leaders will likely see higher-than-expected expenses as they examine the economics of generative AI. Aamer Baig, senior partner at McKinsey & Company, said companies may be attracted to GenAI’s relatively low startup costs. McKinsey research found that GenAI models just drive about 15% of the cost of a typical project.
We all grew up with certain orthodoxies when it comes to estimating costs. And we’re finding that many of these orthodoxies turn out not to be true when it comes to generative AI.
Aamer BaigSenior Partner, McKinsey & Company
But other, less obvious costs can also drive up the price of a project compared to traditional IT initiatives.
“We all grew up with certain orthodoxies about how costs are estimated,” Baig said while speaking at the 2024 MIT Sloan CIO Symposium earlier this year. “And we find that many of these orthodoxies turn out not to be true in generative AI.”
Baig pointed to the example of change management, a large budget item for digital transformation projects and an even greater need for GenAI.
“We made a big splash a few years ago when we said you need to budget as much for change management as you do for development,” he said, referring to digital transformation efforts. “Now, [with GenAI]we find up to three times [the development cost] for investments in change management are required.”
Generative AI, like digital transformation, requires changes in workflows, business processes, policies and KPIs, Baig said. But GenAI also includes new considerations for change management, such as: B. timely engineering and special AI training.
Mike Mason, chief AI officer at Thoughtworks, a technology consulting firm in Chicago, also noted the importance of change management in GenAI projects.
“Change management is something we don’t think companies are paying enough attention to,” he said. “You talk about changing the way people do their work – you can’t ignore the change management aspect.”
Mason also cited AI readiness as a cost that companies should factor into their calculations. This also includes the readiness of the data to support AI applications. Data needs to be available, rather than locked in storage silos, and cleaned before being fed into a GenAI system, he said. An IT department may need to upgrade the infrastructure to make this possible. Steps could include a cloud migration and adopting a modern data platform, Mason added.
He shared the example of a Thoughtworks life sciences customer seeking data modernization to make data more available and support the use of GenAI in drug discovery. The company had data from preclinical studies scattered across numerous data repositories, Mason explained. As a result, drug researchers had difficulty finding information about the company’s previous experiments and incurred unnecessary costs in repeating tests. The life sciences company used a data network that provides a uniform platform for accessing data on experiments and trials.
“Viewing existing data can provide a very powerful ROI factor,” Mason said.
Quility is now also focusing on data as part of its GenAI efforts. The insurer uses Snowflake as an enterprise data warehouse and Apache Kafka, an open source distributed event streaming platform that supports data pipelines and data integration across organizations.
“We want to be a data-driven company,” Conklin said. “We want to give [employees] Information at the time of decision.”
Look beyond traditional operating costs versus construction costs
The ongoing costs of running a generative AI application could be another unexpected cost impacting ROI. A generally accepted prediction is that the cost of running applications will be between 15% and 30% of the cost of building them given digital transformation, Baig said.
“At GenAI, I humbly suggest you ditch that,” he said, noting that the cost of running generative AI models could be equal to the cost of building them, depending on the use case.
CIOs should consider hidden costs when deploying GenAI at scale.
Mason said the cost of running a model and the inference process where the model interprets new data would typically dwarf the cost of training a model. In addition, some of these costs may prove difficult to predict. For example, the way GenAI providers price API calls to their LLMs makes price and cost predictions difficult. Providers use a token system to set prices for these calls, with longer text responses using more tokens.
“Token-based pricing is new for businesses and, in my opinion, less predictable,” Mason said.
When a user gives input to an LLM, the result can be a short or long answer, he said. Accordingly, token-based pricing makes it difficult for companies to determine the true cost of running an application to production, he added.
Vamsi Duvvuri, technology, media and entertainment and telecom AI lead for the Americas at consultancy EY, cited cost uncertainty as one of the lessons learned from the first wave of generative AI projects.
“Companies still struggle to manage and predict costs around GenAI,” he noted.
Duvvuri said most current cost models do not provide economies of scale under pay-as-you-go scenarios. PAYG is the approach many companies are taking when starting to use generative AI, he added.
A positive cost development: The price of generative AI models such as ChatGPT 4o or Claude 3.5 has recently fallen due to competitive prices and efficient architectures, said Duvvuri. However, technology users should continue to focus on controlling GenAI costs.
“Companies must do the hard work of optimizing the unit cost of the work performed by AI models,” Duvvuri said.
This task for IT leaders is to scale the underlying technical and functional patterns of an AI system, he emphasized. Technical patterns include retrieval-enhanced generation and multiple model chaining. RAG is used to increase the accuracy of LLMs, while model chaining aims to improve the quality of model output. Functional patterns include summarization/classification, translation and composition, Duvvuri said.
Technical decisions at the start of a generative AI deployment can significantly impact cost and ROI. The cost variance could be between 10x and 20x for GenAI, compared to 1x to 5x for digital transformation, Baig noted.
“I personally believe strongly in the power of great technical decisions,” he said. “A very strong business model, making the right decisions from a technology and business model perspective – that can lead to huge cost fluctuations. So the big decisions up front are really important.”
John Moore is a TechTarget editorial writer covering the role of the CIO, economic trends and the IT services industry.
