Food disguised to look like other food, or some other object altogether. A modern analogy for a subtleties would be character cakes, but in period they were either sweet or savoury.
A favourite example of a subtelty was skinning a peacock (keeping the feathers in tact), roasting the bird, and then replacing the skin back over it so it looked like the live thing. Other common subtelty creations are dragons, castles, birds, heraldic devices.
Also note that there are about 15 different ways of spelling subtelty.