The startup will be led by Musk, already the CEO of electric car maker Tesla, CEO of rocket launch company SpaceX and owner of Twitter, who has said on several occasions that the development of AI should be paused and that the sector needed regulation. Musk has repeatedly voiced concerns about AI's potential for "civilizational destruction."
In a Twitter Spaces event Wednesday evening, Musk explained his plan for building a safer AI. Rather than explicitly programming morality into its AI, xAI will seek to create a "maximally curious" AI, he said.
"If it tried to understand the true nature of the universe, that's actually the best thing that I can come up with from an AI safety standpoint," Musk said. "I think it is going to be pro-humanity from the standpoint that humanity is just much more interesting than not-humanity."
Musk also predicted that superintelligence, or AI that is smarter than humans, will arrive in five or six years.
Musk co-founded OpenAI, the company behind ChatGPT, in 2015, but stepped down from the company's board in 2018.
Microsoft is an investor in OpenAI.
The website for xAI said it will hold a Twitter Spaces event on July 14.
The team at xAI includes Igor Babuschkin, a former engineer at Google's DeepMind; Tony Wu, who worked at Google; Christian Szegedy, who was also a research scientist at Google; and Greg Yang, who was previously at Microsoft.
Musk in March registered a firm named X.AI Corp, incorporated in Nevada, according to a state filing. The firm lists Musk as the sole director and Jared Birchall, the managing director of Musk's family office, as a secretary.
Musk had said in April that he would launch TruthGPT, or a maximum truth-seeking AI to rival Google's Bard and Microsoft's Bing AI that tries to understand the nature of the universe.
Generative AI caught the limelight with OpenAI's launch of popular chatbot ChatGPT, which came in November last year, ahead of the launch of Bard and Bing AI.
Dan Hendrycks, who will advise the xAI team, is currently director of the Center for AI Safety and his work revolves around the risks of AI.
Musk's new company is separate from X Corp, but will work closely with Twitter, Tesla and other companies, according to the website.
xAI said it is recruiting experienced engineers and researchers in the San Francisco Bay Area.