Gem Fragmentss are materials required for the crafting of Elixirs and Incense by the Alchemist. They are collected from veins dotted around the map and as loot from monsters, chests and caches rewarded for completing Whispers of the Dead.
They can be upgraded at the Jeweler into gems to socket into items.
The other types of Fragments you can find are: