The noun frozen yogurt can be countable or uncountable.
The singular form of frozen yogurts is frozen yogurt.