We have the letter of Muaz from the Prophet ﷺ stating that he only took zakat from wheat, barley, raisins, and dates.
حدثنا عبد الرحمن بن مهدي حدثنا سفيان عن عمرو بن عثمان يعني ابن موهب
عندنا كتاب معاذ عن النبي ﷺ أنه إنما أخذ الصدقة من الحنطة والشعير والزبيب والتمر
Its isnad is sahih. Haythami said in 3/75, "Its narrators are the narrators of the Sahih." Hakim graded it sahih in 1/401, number 1457, and Dhahabi agreed with him.