Loot Luck will increase your chances of getting anything in the loot table of the NPCs you last hit on mercy mode, so the more loot luck the more trinkets you will loot of the bodies of the stuff you are farming
Rare Trinket Gem Luck will give you a chance to upgrade the gem tier of trinkets given by NPCs you last hit on mercy mode, only once when they drop, this stat has not any effect when identifying or rolling the trinkets.
This means that Loot Luck and Rare Trinket Gem luck scale of each other, the flat base numbers are obviously unknown to me, but lets set this made up napkin math example: you need to kill 1000 mobs to get 100 trinkets and of a 100 trinkets you get one with topaz on average. with 20% loot luck you will get 100 trinkets in 834kills witch gives you one with a topaz on average while with 20% gem luck you will get a topaz trinket on 83.4 trinkets witch also takes 834 kills, but if you have 10% on each to get a topaz trinket you need 91 trinkets and to get those you need 827 kills, so as you can see 10/10 is better than 20/0 or 0/20 here, this being said i still get the felling feeling that Gem luck performs better than loot luck and i would take +9/+5 over +18/0, either way if you plan to farm trinkets in the long run or you are the one doing the last hit on your guild raids and you see yourself playing this game in the long run you should aim to have 18/18 asap, it makes quite the difference as i think that my napkin example is the worst case scenario, these stats may be stronger than i think.