The Messenger of Allah ﷺ prayed two rakahs at Mina, and Abu Bakr did the same after him, as did Umar after Abu Bakr, and Uthman during the early part of his caliphate. Then Uthman later prayed four rakahs. So whenever Ibn Umar prayed with the imam, he would pray four, but when he prayed alone, he would pray two.
صلى رسول الله ﷺ بمنى ركعتين. وأبو بكر بعده وعمر بعد أبي بكر، وعثمان صدرا من خلافته، ثم إن عثمان صلى بعد أربعا. فكان ابن عمر إذا صلى مع الإمام صلى أربعا، وإذا صلاها وحده صلى ركعتين.
Bukhari related it in the chapter on shortening prayer (1082), and Muslim in the prayer of travelers (17/694), both from the hadith of Ubaydullah, from Nafi, from Ibn Umar, and he mentioned the hadith. The wording given is Muslim's. In another narration in Muslim, through the route of Hafs bin Asim, from Ibn Umar, he said, "ﷺ prayed at Mina the prayer of a traveler, and so did Abu Bakr, Umar, and Uthman, for eight years, or he said, six years." Hafs said, "Ibn Umar used to pray two rakahs at Mina, then go to his bed. I said, 'Uncle, if only you prayed two more rakahs after them.' He said, 'If I did that, I would have prayed the prayer in full.'"Muslim said, "Haritha bin Wahb Khuzai is the maternal brother of Ubaydullah bin Umar bin Khattab."