Brown is not a normal heraldic colour - it is normally used only as the "proper colour" of items (eg treetrunks).
Brown dye is a somewhat simpler colour to obtain in period, and thus brown fabric could be worn by the poor, for workday clothes and by monks sworn to poverty.
see also "period brown"