He was riding behind the Messenger of Allah ﷺ and continued reciting the talbiyah until he threw the pebbles at Jamrat al-Aqaba.
أنبأنا كثير بن هشام قال حدثنا فرات حدثنا عبد الكريم عن سعد بن جبير عن ابن عباس
أنه كان رديف رسول الله ﷺ فلم يزل يلبي حتى رمى جمرة العقبة
Its isnad is sahih. Furat is Ibn Abu Abdul-Rahman Qazzaz, and he is trustworthy. Bukhari mentioned him in Tarikh al-Kabir 1/4/129. Abdul-Kareem is Ibn Malik Jazari, and Furat narrates from Saeed bin Jubayr directly, but here he narrated from him through an intermediary. Saeed bin Jubayr is the well-known trustworthy and reliable follower of the companions, whom Hajjaj unjustly killed in the year 95, when he was 49 years old. In kh it reads "Saad bin Jubayr," which is a clear error. This hadith is an abridgment of the one before it.