He attributed it to the Prophet ﷺ, who said, "Do not prevent anyone from circling this House and praying at any hour he wishes, by night or by day."Fadl said that the Messenger of Allah ﷺ said, "Children of Abd Manaf, do not prevent anyone."
حدثنا ابن السرح والفضل بن يعقوب وهذا لفظه قالا حدثنا سفيان عن أبي الزبير عن عبد الله بن باباه
يبلغ به النبي ﷺ قال لا تمنعوا أحدا يطوف بهذا البيت ويصلي أي ساعة شاء من ليل أو نهار قال الفضل إن رسول الله ﷺ قال يا بني عبد مناف لا تمنعوا أحدا
[Its chain of transmission is sahih] Tirmidhi transmitted it, Hajj, chapter on what has been narrated concerning prayer after Asr, etc., no. 868, and Nasai, no. 2927, and Ibn Majah, no. 1254, from the hadith of Sufyan through this route. Abu Zubayr explicitly stated that he heard it directly, as recorded by Nasai, no. 586, and Hakim graded it sahih according to the criteria of the two Shaykhs (Bukhari and Muslim), 1/448, and Dhahabi agreed with him.