Since #sewing folk seem fairly active on here, looking for #advice. Taking this down a size and the top fabric is doing some mega fraying, when doing the resizing should I use the lining or the top layer to get the fit right? Worried about handling things too much