As you gain levels in your guild, that will unlock items for sale at special merchants. Some of the significant millestones are levels 15, 20, and 30 for discounted and new mounts.
Check out
this for a complate breakdown of goodies you get at various levels.
If I remember, at guild level 5 your guild name will be visible to others that click on you.