What is the ruling on gesturing to someone who is speaking during the khutbah to tell him to be quiet?

Firstly:

The hadith to which you refer in your question was narrated by Muslim, and tells us that the Prophet (blessings and peace of Allah be upon him) said: “Whoever touches the pebbles has engaged in an idle action.” Narrated by Muslim (857).

Al-Qadi ‘Iyad (may Allah have mercy on him) said: Regarding the words “Whoever touches the pebbles has engaged in an idle action,” that is because by moving them and distracting himself with them, he becomes engaged in idle action and distracts others from listening to the khutbah with the noise that he makes by moving them."(Ikmal al-Mu‘allim bi Fawa’id Muslim  3/253).

Clearer than that is the hadith of Abu Hurayrah, according to which the Prophet (blessings and peace of Allah be upon him) said: “If you tell your companion to be quiet and listen on Friday when the imam is delivering the khutbah, you have engaged in idle talk.” Narrated by al-Bukhari (892).

These hadiths do not mean that someone should not alert the one who is talking or doing something during the khutbah [and tell him to stop] by means of gestures, as Ibn ‘Umar (may Allah be pleased with him) did.

Ash-Shirazi said: Speaking is not allowed, which means either that it is better not to speak or that one must not speak, according to the two different scholarly views. Rather the way to do that is to indicate by means of a gesture telling him to be quiet."(Al-Mafatih fi Sharh al-Masabih  2/322).

It is soundly narrated from the Sahabah that they would gesture to one who spoke during the khutbah.

Ibn Qudamah (may Allah have mercy on him) said: If a person hears someone talking – that is, during the khutbah – he should not tell him to stop by speaking, because the Prophet (blessings and peace of Allah be upon him) said: “If you tell your companion to be quiet and listen on Friday when the imam is delivering the khutbah, you have engaged in idle talk.” But he may gesture to him, as was stated by Ahmad, by putting his finger on his mouth.

Among those who thought that one should gesture and not speak were: Zayd ibn Suhan, ‘Abd ar-Rahman ibn Abi Layla, ath-Thawri, al-Awza‘i, and Ibn al-Mundhir…

This is indicated by the fact that when someone said to the Prophet (blessings and peace of Allah be upon him) [i.e., during the khutbah]: When will the Hour be? The people gestured to him in the presence of the Messenger of Allah (blessings and peace of Allah be upon him) to be quiet.

And as gesturing is permissible even in the prayer, which is invalidated by normal talk, it is more apt that it should be permissible during the khutbah."(Al-Mughni  3/198).

Secondly:

What you referred to about Ibn ‘Umar throwing a pebble was narrated by Ibn Abi Shaybah, from Ibn ‘Umar (may Allah be pleased with him), that he saw a man talking when the imam was delivering the khutbah on Friday, so he threw a pebble at him, and when the man looked at him, he put his hand over his mouth. Narrated by Ibn Abi Shaybah (5324) and via him by Ibn al-Mundhir in al-Awsat (4/78). It was classed as sahih by the commentator Shaykh Sa‘d ash-Shatri.

See: Ma sahha min Athar as-Sahabah fi’l-Fiqh (1/477).

When Ibn ‘Umar did is not included in the prohibition on touching the pebbles, because if it is permissible to alert someone by means of a gesture, that should also apply to using pebbles, even though it is not what is most appropriate, and it is not what people usually do, as Imam Malik said.

Ibn al-Mulaqqin (may Allah have mercy on him) said: It was narrated from Malik that people may be told to be quiet by saying ‘Subhan Allah’ or by gestures, but not by throwing pebbles, because the Prophet (blessings and peace of Allah be upon him) said: “Whoever touches the pebbles has engaged in an idle action.” Ibn ‘Umar threw a pebble, but this is not the option to choose. Ibn al-Mundhir also narrated from Malik: There is nothing wrong with a gesture."(At-Tawdih li Sharh al-Jami‘ as-Sahih 7/607).

Perhaps Ibn ‘Umar (may Allah be pleased with him) only did that to get the man’s attention as the man was not near him or facing him, such that he would see his gesture. So he threw the pebble in order to get his attention, then he told him not to speak by means of his gesture.

And Allah knows best.