There a many reservations from other religions about the worship of deity’s (form’s of God’s and Goddesses in the temple) in Hinduism. Its called idolatry and is not recommended in Christianity, Islam and perhaps other beliefs.
The ‘Jaiva Dharma’ book – compiled by Srila Bhaktivinoda Thakura has a chapter (Nitya-Dharma & Idolatry) where a Islamic Mullah ask questions and brings arguments like this:
Isn’t meditating on a form of the Lord within one’s mind superior to imagining a form with the help of the material elements?
In our Koran it is written that the Supreme Entity has a divine all-conscious form, so we are compelled to accept this fact. However, any image of that spiritual form is material; that is what we call but. The worship of but is not the worship of the Supreme Being.
Matter is distinct from God. I think that it is better not to worship material objects, because it is said that Satan introduced the worship of matter to keep the living entities bound in the material world.
Than the Vaishnava Pandit Goracandra Das Babaji of Mayapur gave some points and views on this topic: (its shortened here to bring some interesting points only)