Working on Elon Musk’s ‘smartest AI on Earth’: a Vietnamese scientist’s early experience

"A few months at xAI have given me immense value, opportunities and emotions," the 33-year-old said on his Facebook page on Feb. 18, following the launch of Grok 3, which Elon Musk describes as the "smartest AI on Earth."

"Grok 3 is the result of months of research and development by my colleagues and me, as well as months of sacrifices by the families of xAI employees, who had little time with their loved ones," Hieu says.

He signed a contract with xAI in July 2024 and quickly realized how fast-paced the company was. Though scheduled to start work the following week, he was called in early to collect his laptop and set up his system in preparation. Shortly after receiving the laptop, a senior colleague contacted him about a critical issue involving electromagnetic oscillations in Colossus, xAI’s supercomputer.

The problem was that when too many GPUs ran simultaneously, their energy consumption would resonate, potentially damaging the power turbines. The colleague asked if he could solve the issue that morning. Musk later referenced this issue in a livestream introducing Grok 3, comparing Colossus to a symphony orchestra and its 100,000 GPU chips to musicians. When they all "play or go silent together," the resonance effect becomes immense.

To mitigate the issue, Hieu wrote a GPU kernel program that reduced oscillations but required higher energy consumption. "The essence of this kernel is that when the system enters a low-power phase, I force it to perform additional tasks, thus consuming more electricity," he explains.

Vietnamese scientist Pham Hy Hieu. Photo courtersy of Hieu

The next day Musk was briefed on the increased power consumption and concluded that oscillations were unavoidable. To address this, he deployed a Tesla megapack battery system as an energy buffer. Turbines charge the batteries, which in turn power the CPUs, reducing direct strain on the turbines.

"This demonstrates Musk's incredible problem-solving ability," Hieu gushes. "A genius idea. How many billionaires can still think so deeply about a technical problem?"

While acknowledging Musk’s controversial reputation, he believes his strengths—broad vision and unconventional thinking—are "immensely worth learning from."

Hieu has been well known in the global AI community. In 2011 he received full scholarships from five prestigious U.S. universities. He graduated from Stanford University in 2015 and received job offers from Microsoft, Facebook and Google. He chose Google though it rejected his first application in 2012.

At Google Brain, he worked on neural network applications. In 2017 Google partnered with the U.S.’s Carnegie Mellon University to establish a joint PhD program, allowing him to conduct research while working at Google. In his first year as a PhD student he made significant breakthroughs, but later struggled to achieve similar success. Realizing that complex, impractical research yielded little impact, he adopted a simpler approach to problem-solving, which led to greater achievements in his final years at Google.

In 2021, at age 29, he successfully defended his PhD thesis at Carnegie Mellon and presented six research papers at top international conferences, adding 5,000 citations to his research portfolio.

His research on applying AI to discrete mathematical problems helped Google significantly reduce AI training costs by minimizing reliance on expensive supercomputers with thousands of processing chips and vast data storage requirements.