What should the imam do when there are people praying behind him who do innovated actions?

Innovations are not all
of the same degree. Some of them have to do with beliefs, some have to do
with minor practical issues of sharee’ah; some put a person beyond the pale
of Islam whereas others do not. In some cases the daa’iyah may think that
something is an innovation when it is not, or there may be a scholarly
opinion which is less correct than the opinion with which he is familiar, or
there may be another opinion on the matter of which he is not aware.  

Usually questions are asked about an imam who is doing
innovated actions and the ruling thereon, and the answers are well known. 
If his bid’ah is one that constitutes kufr, then it is not permissible to
pray behind him; if it does not constitute kufr, then it is valid to pray
behind him, but it is better to pray behind an imam who is of Ahl al-Sunnah.
Concerning such cases al-Hasan al-Basri (may Allaah have mercy on him) said:
“Pray, and he is accountable for his bid’ah.” 

But if the question is about a bid’ah which the members of
the congregation are doing and the question is coming from the imam, there
is something a little odd about that. The job of the imam is not only to
lead the people in prayer, rather he also calls them, teaches them and
advises them. Any imam who leads the people in prayer will usually have
behind him people who shave their beards, deal in riba, disobey their
parents, backbite and spread gossip, and so on… they are the target audience
of the imam’s da’wah in the mosque.  

Calling people to Allaah and teaching them is the mission of
the Prophets; they are also obligatory upon everyone to whom Allaah gives
knowledge, even if it is only a little. 

This was the way of the Prophet (peace and blessings of
Allaah be upon him); if he saw a mistake he would correct it. 

For example, when he saw a person praying badly and not
bowing or prostrating correctly, he taught him how to pray. Agreed upon. 

When Mu’aawiyah ibn Hakam spoke during the prayer, the
Prophet (peace and blessings of Allaah be upon him) taught him that no
speech of man was appropriate during the prayer. Narrated by Muslim. 

When the Bedouin urinated in the mosque, the Prophet
(peace and blessings of Allaah be upon him) explained to him that these
mosques are not the place for any of this filth. Agreed upon…. And there are
many more examples. 

If the imam realizes that what some of the people are doing
is an innovation that goes against sharee’ah, then he should call them in
the manner that is best. He can do that by making the most of the best
opportunity to advise them and guide them, in numerous ways, such as the
following: 

Reciting specific verses of Qur’aan, then explaining their
meanings after the prayer. 

Inviting some daa’iyahs and seekers of knowledge to give
lectures or lessons on the Sunnah and following it. 

Distributing leaflets and pamphlets. 

Setting up an audio library and encouraging people to borrow
from it. 

In all cases, the imam should be knowledgeable, gentle and
wise, and above all he should be sincere in his love for people, so that he
may be granted divine help and success. 

Shaykh ‘Abd al-‘Azeez ibn Baaz (may Allaah have mercy on him)
was asked: 

If the people to whom the call (da’wah) is addressed are
influenced by a particular culture or society, what is the best way to call
them? 

He replied: 

The one who is calling them to Allaah should explain to them
the mistakes and bid’ahs etc that are present in the ways that they are
following and the cultures to which they belong, and the environments in
which they live. He should explain to them the things that are contrary to
sharee’ah in their societies in which they live, and call them to examine
everything about which they are confused against the most just standard,
which is the Book of Allaah and the Sunnah of His Messenger (peace and
blessings of Allaah be upon him). Whatever is in accordance with one or both
of them may be regarded as legitimate according to sharee’ah, and whatever
goes against them should be rejected, no matter who advocates it. Thus the
scholars used to examine controversial matters against the shar’i evidence;
whatever is in accordance with sharee’ah must stay, and whatever goes
against sharee’ah must be rejected, even if the one who says it is great,
because the truth prevails over all. This is what must be done with any
customs or traditions that go against sharee’ah; they must be given up, even
if they are the traditions of our forefathers, shaykhs, ancestors and so on.
Everyone must adhere to that which has been enjoined by Allaah and His
Messenger, because that is the path to salvation, as Allaah says
(interpretation of the meaning): 

“And verily, this is My straight path, so follow it, and
follow not (other) paths, for they will separate you away from His path.
This He has ordained for you that you may become Al‑Muttaqoon (the pious)”

[al-An’aam 6:153]

 And Allaah is the Source of strength. End quote. 

Majmoo’ Fataawa al-Shaykh Ibn Baaz,
4/240, 241 

So seek the help of
Allaah, and learn from the experience of your fellow imams and be sincere
towards those are praying behind you, and you will find much good and be
rewarded greatly.