Amazon's AI chips are 'not as good as' Nvidia's; startups claim

Amazon’s AI chip customers may not be happy with the company’s offering. According to a Business Insider report, internal documents from the e-commerce giant reveal that some AI startups have said that Amazon’s chips are no match for Nvidia's GPUs.

The AI startup Cohere found that Amazon's Trainium 1 and 2 chips were "underperforming" Nvidia's H100 GPUs. The report also noted that access to Trainium 2 was "extremely limited" and affected by frequent service disruptions. However, "performance challenges" with Cohere were under investigation by Amazon and its chip group Annapurna Labs, but progress on these issues was "limited,” the report added. Meanwhile, AI image generation startup Stability AI has also expressed similar concerns with Amazon’s chips. The document suggested that Stability AI concluded that Amazon's Trainium 2 chips didn’t match the latency of Nvidia's H100 GPUs, making them "less competitive" in terms of speed and cost.

What is Amazon’s plan with these in-house AI chips

Amazon is relying on these in-house AI chips to support its future growth. The company’s in-house Trainium chips play a key role in its strategy to stay competitive in the AI cloud market. The profitability of Amazon Web Services (AWS) was initially driven by designing its own data centre chips rather than relying on costly Intel components. In the current phase of generative AI, Amazon aims to reduce its reliance on Nvidia GPUs while continuing to offer advanced AI cloud services.

However, some AWS customers prefer Nvidia hardware over Trainium, which could affect Amazon’s upcoming cloud margins by increasing GPU-related expenses. Internal reports of customer dissatisfaction highlight the company’s difficulty in matching Nvidia’s performance and maintaining strong engagement among startup clients, a crucial segment for AWS.Other AWS customers have also raised concerns about Amazon’s AI chips, citing performance and cost issues.

A startup, Typhoon, found Nvidia’s A100 GPUs up to three times more cost-efficient than AWS’s Inferentia 2 chips, while AI Singapore reported better results using AWS’s Nvidia-powered G6 servers. Clients have also mentioned “challenges adopting” Amazon’s custom chips, leading to low usage. According to Omdia, Nvidia holds over 78% of the AI chip market, with Google and AMD each around 4%, while AWS ranks sixth with 2%.

What Amazon said about the feedback from AI startups

In an emailed statement to Business Insider, an Amazon spokesperson said the company is “grateful” for customer feedback that helps make its chips “even better and more widely used.”The spokesperson clarified that the Cohere case is “not current” and added that Trainium and its other in-house AI chip, Inferentia, “have achieved great results” with customers including Ricoh, Datadog, and Metagenomi.“We’re very pleased with the growth and adoption we’re seeing for Trainium 2, which at this stage is primarily used by a small number of very large customers like Anthropic,” the spokesperson said.AWS stated that its in-house AI chips offer 30% to 40% better price performance than the current generation of GPUs and confirmed that it continues to invest in new chip designs and future versions of these components.“We expect to accommodate more customers starting with Trainium 3, previewing later this year. We’ll accomplish that as we always do by listening to our customers and remaining vocally self-critical with each other as we continue innovating to give more customers access to chips with the best possible price and performance,” the company added.During Amazon’s recent earnings call, CEO Andy Jassy said Trainium 2 chips are “fully subscribed” and now represent a “multibillion-dollar” business.