Josephus (Wars 5:4:2) writes amount the east wall of Yerushalayim, that at its end, "it was joined to the eastern stoa of the temple". Some (R' Yehosef Shwartz, in Tevu'os Ha'aretz chapter 7, Michael Avi-Yonah, in his famous model; the Holyland model, and others,) have understood this to mean that it joined the southeast corner of the Har Habayis, and the east wall of Har Habayis was also part of the east wall of the city, like it is today. However, this cannot be, as Rashi (סוכה נא ע"א ד"ה שלא היתה מאירה מאור בית השואיבה) seems to imply that (at least part) of Yerushalayim was to the east of the Har Habayis. Furthermore, the Gemarah (Shavuos 16a) talks about two areas on Har Hazeisim, which is located east of Yerushalayim, that were added to Yerushalayim.