Gems are a luxury resource, and building material. They are used in the production of Jewelry. They can also be used to generate revenue by selling them to villagers in markets using a Luxury Goods Stall.
They can be found in multiple different mountain outcrops, including Iron Deposits, Gold Deposits, and Stone Deposits. They generate quite slowly however, 42 active miners will generate between 1-3 Gems per week, though it seems to be determined by RNG. Some have observed that they generate more often in Gold Deposits, but it has not been confirmed.
They can also be acquired through Military Missions.