The Messenger of Allah ﷺ said, "Whoever touches the pebbles has engaged in idle conduct."
حدثنا أبو بكر بن أبي شيبة حدثنا أبو معاوية عن الأعمش عن أبي صالح
قال رسول الله ﷺ من مس الحصى فقد لغا
Muslim reported it in the Book of Jumuah, chapter on the virtue of one who listens attentively and remains silent during the sermon, no. 857, from Ibn Abu Shayba and others, at greater length. See also no. 1090.