And the homeworld gems are from a different planet

They live on Earth in Beach City, but they came from Homeworld / Kindergarten

To get into detail, originally there are 2 types of gems: Home world gems and Kindergarten gems

After Rose Quartz (Who was a home world gem at the time) arrived on earth, she saw the beauty of the planet and formed a rebellion against the Home world gems

Rose quartz and her army became the Crystal Gems (Known members at the time are Pearl, Garnet (Ruby & Sapphire) and The Cluster (Which is a cluster of broken gems forcefully fused together by the Home world gems)(And maybe amethyst too)).

After the rebellion only few of the crystal gems survived and 5.000 years later (5 Millennium (50 Centuries)) the Crystal gems now live in the Temple on beach city, and in while they were there Rose Quartz had a son named Steven Q. universe (Who is half gem and half human) with Greg universe, but sadly Steven and Rose can't exist both at the same time, since Rose is pretty much half of Steven. (btw if you didn't realize Steven's middle name is Quartz as stated in Too many birthdays).

and then...... wait why am i telling you people the history of Steven universe, you all pretty much know the lore already...

Well the point is they live in the Temple which is located in Beach city, they don't live IN Beach city, no it's not like they have a house or something (well they do but it's specifically made for Steven).

Beach City is where they live thats where the temple is at least!

Why

Don't they live in that temple house thing with Steven?