The incident took place during a prayer service at the city's Golden Temple on Saturday, according to local media.
The man allegedly barged into the inner sanctum, where Sikhism's holy book, the Guru Granth Sahib, is kept.
He then grabbed a ceremonial sword placed next to the book, but was overpowered by guards and worshippers.
Sikhs regard the Guru Granth Sahib as not just the holy scripture of their religion but also as the living Guru, and believe it should be treated with respect as a human might be.
The scuffle took place at around 17:45 local time (11:45 GMT), and was captured on camera as evening prayers were being broadcast on television.
It is unclear exactly what happened next. Police said the man was found dead once officers arrived at the scene, and an investigation is under way.
Less than 24 hours later, another man was beaten to death by Sikhs accusing him of sacrilege after he allegedly tried to remove the Sikh flag, Nishan Sahib, from a temple in Kapurthala, also in Punjab state.
Indian media report police initially took the man into custody but locals then clashed with police and ended up killing the man.
Punjab state's police chief tweeted that "Stern action will be taken against all those disturbing the law and order in Punjab" after the "unfortunate incidents" in Amritsar and Kapurthala.