The GenAI strategy determines the ROI challenges for IT leaders

Watch SaaS GenAI costs

A variety of SaaS providers are bundling generative AI functions into their products to monetize the technology. Orlandini categorized companies that purchase such off-the-shelf features as GenAI “consumers.” For example, consumer organizations could purchase Copilot through the Microsoft 365 product suite or Adobe’s Firefly image generator through its applications, he noted.

The consumption model is perhaps the easiest way to advance GenAI, but it is not without financial challenges.

Prasad Ramakrishnan, who retired as Freshworks’ CIO last month but continues to advise the company, said CIOs need to carefully review vendors’ pricing and be prepared to negotiate contract terms. Based in San Mateo, California, Freshworks offers SaaS-based CRM and IT service management tools.

These discussions should be about how much IT leaders are willing to pay for generative AI and its net benefits beyond a vendor’s core product, Ramakrishnan said. It’s also important not to get too caught up in a new and largely untested technology, he added.

“Don’t go for a long contract,” he said. “No more than a year.”

But there are also savings that go beyond the license price. John Buccola, CTO at E78, a service provider based in Oak Brook, Illinois, acknowledged that companies can negotiate discounts on software licenses as part of their GenAI business strategy. However, he places greater emphasis on the cost-cutting potential of license management. E78 provides advisory and managed services to private equity firms and their portfolio companies.

Companies are over-provisioning their SaaS-based GenAI licenses and thereby overpaying for the technology, Buccola claimed. The task: Determine who is using the licenses and compare this with the company’s expenditure on them.

“SaaS offers many opportunities to ensure that the amount of licenses in inventory matches the usage of those licenses,” he said. “The savings will not be so much in the purchase prices, but rather in the administration.”

Assessing the business benefits of SaaS GenAI

However, license optimization is only part of the ROI equation. Added to this is the business benefit, which in the case of SaaS-based GenAI revolves around productivity improvements.

Employee utilization helps document the business value of the technology. But companies also need to quantify whether the tool saves employees time compared to previous technologies, Orlandini said. Measuring the before-and-after effects of a provider’s GenAI offering typically requires an employee survey, he said.

Such surveys become particularly important when generative AI vendors do not offer tools usage tracking capabilities.

Mike Mason, chief AI officer at Thoughtworks, a Chicago-based technology consulting firm, said AI assistants like GitHub Copilot don’t provide per-developer usage statistics. He believes this is because the vendor doesn’t want to appear to be providing spyware to developers.

Ultimately, however, an organization must decide whether the benefits of GenAI justify the additional fee, which can be $25 to $40 per user per month, depending on the offering.

Mason said he talks to executives who calculate $30 a month times the number of users and conclude that the cost “couldn’t be worth it in any way.” However, this assessment does not take into account what employees could ultimately achieve with GenAI, he added.

“I personally make well over $30 a month with the AI ​​tools I use for work,” Mason said.

To reinforce this point, Mason used ChatGPT to quickly and easily calculate how much time a Fortune 500 employee needs to save over the course of a month to break even at a GenAI fee of $30 per month to reach. His estimate: An average worker making $54 an hour needs to save about 34 minutes per month to recoup the monthly fee.

Using SaaS GenAI is a moving target

While some industry executives view such savings as easily achievable, the business value of GenAI can vary significantly from product to product. According to Buccola, this is the case with the Copilot tools embedded in various Microsoft products. Value, as measured by utilization, becomes a moving target, he said, noting that employees can work with Copilot in one product but move to an instance of another product’s technology six weeks later.

Copilot’s accessibility within products plays an important role in sustainable tool use, said Buccola. It’s at the heart of Outlook and offers users the ability to compose messages quickly, he said. Copilot in Excel, on the other hand, poses more of an accessibility challenge. Excel users must save spreadsheets to a Microsoft OneDrive location and format them in a specific way to use Copilot, he added.

“It remains to be seen which tools will best utilize AI,” Buccola said. “We see a lot of variation across the Microsoft suite in terms of the frequency with which Copilot is used.”

Rising infrastructure costs with GenAI refinement

Enterprise GenAI customers could focus on avoiding the pitfalls of overprovisioning licenses or misjudging business value. However, organizations taking a more customized approach to GenAI have different ROI considerations.

Companies that Orlandini called GenAI “adapters” take basic GenAI models and modify them for specific use cases. Most mid- and high-end companies end up doing some form of customization when developing AI-based products, he noted. Such models can be hosted in public or private clouds. GenAI’s need for such resources – cloud computing and storage – can add significantly to the cost of scaling the technology beyond the pilot phase.

Accrete, a New York-based enterprise AI company, runs its Nebula platform on AWS. The company offers AI services through Nebula, which is based on modified transformers and large language models. Accrete uses GenAI services from OpenAI in addition to its proprietary technology. Peter Bierfeldt, Accrete’s CISO, said the company’s cloud infrastructure costs have started to rise, which has led to cost control.

“We had a pretty significant burn rate where we were probably looking at $2 million a year on an annualized basis,” he said.

Cost drivers included Amazon Elastic Compute Cloud (EC2) instances, Amazon SageMaker and Amazon Elastic Block Store. Accrete’s first cost-cutting project was to move a “tremendous amount” of EC2 instances into containers and the Kubernetes management platform, Bierfeldt said. This migration took place in 2023. This year, the company deployed Kubecost, a Kubernetes cost monitoring and management application.

The combination of Kubernetes and Kubecost has reduced Accrete’s AWS spend by 40% to 45%, Bierfeldt said. He estimated this would result in just over $1 million in cost savings this year.

While costs are still present on the container side, they are easier to track in this environment compared to EC2, he added.

“It basically burns money 24/7 today,” he said of an EC2 instance. “Now we will watch this a little more closely with Kubernetes and Kubecost.”

Optimizing cloud resources for cost efficiency

The ability to optimize the use of resources through containers also contributes to cost efficiency. Accrete can add containers to handle a surge in demand and shrink them again within 10 to 15 minutes if the load slacks, Bierfeldt said. This flexibility allows the company to quickly turn off unused resources.

The costs for the model run can increase quickly. We are seeing a whole range of architectural patterns and new deployment strategies.

Accrete has also reduced the cost of operating SageMaker, which companies use to customize and deploy AI models. Some processing workloads don’t require the computing resources that SageMaker provides, Bierfeldt noted. These workloads were instead moved to container instances.

This “allowed us to rely less on SageMaker,” he said.

Bierfeldt said Accrete used some of its cloud cost savings to hire more engineers, researchers and salespeople. The spending reduction will also help the company keep the prices of its AI services low as it adds more customers, he added.

Accenture’s Tay said companies offering GenAI services are rethinking their infrastructure as costs rise. He has seen growing interest among customers in exploring deployment options along a continuum of cloud infrastructure: private, public and hybrid. IT leaders must decide which models work best in which environments, taking cost and accuracy into account, among other things.

“The cost of model runs can increase quickly,” Tay said. “We’re seeing a whole range of architectural patterns and new deployment strategies.”

John Moore is a TechTarget Editorial writer covering the role of the CIO, business trends, and the IT services industry.