Thousands had packed into the Hindu temple in Kerala state's Kollam district on Saturday night for the show when a stray firework apparently landed on a stockpile, triggering a huge blast that tore through concrete buildings.
"Five workers have been detained for questioning," Kollam police chief P. Prakash told AFP by phone.
"These are not formal arrests. Once they are questioned, only then we will know their involvement and take further steps."
Police will investigate who was responsible for the fireworks display going ahead even though local authorities had refused permission, he said.
The Kerala government has already ordered a judicial inquiry into the disaster and police have registered a criminal case against temple authorities.
Footage of the disaster which occurred in the early hours of Sunday showed a series of explosions with flames and huge plumes of smoke.
Firefighters and police battled to douse the fire and rescue those trapped at the complex, but some victims were charred beyond recognition and more than 30 of the 106 dead have yet to be identified.
Prime Minister Narendra Modi rushed to the scene and a team of specialist doctors were deployed from New Delhi to treat the horrific burn injuries.
Witnesses told how the force of the explosion sent concrete slabs and other building parts slamming into the crowd.
Local lawmaker N.K. Premachandran said a fireworks competition traditionally held every year between different groups had been staged at the temple, drawing thousands to watch.
One firework landed on a building that was storing the rest of the pyrotechnic material, triggering the explosion, he said.
Fires and stampedes are not uncommon at temples and during religious occasions, often because of poor security arrangements and lax safety standards.