Transmuting skill works like any other skill, such as your tradeskills or your piercing skill. It's 5 skill points per level (so, 5 at level one, 10 at level 2, etc.) You can find what your transmuting skill is under your character tab, Skills, General. The formula for finding out what level of stuff you can transmute is Skill number divided by five plus five. So if your transmuting skill is at 100, you could transmute items up to level 25 (100/5+5=25).
Cheapest method that I have found is buying adepts and transmuting them. So if you're at 100 transmuting, try to buy some adepts in the 20-25 range.