It is with great trepidation that I wander into a dispute I don't fully understand but, looking at it from the outside, it seems to me that it's a bit late to worry about doing stuff that's "politically polarizing" or that might cause people to lose trust in the Supreme Court, since both those trains left the station long ago -- politics in the US appears, from this side of the Atlantic, polarized between, on the one hand, people who believe in liberal democracy and, on the other, those who believe in various forms of authoritarian nationalism and Christo-fascism.
Similarly, the Supreme Court has, with this weeks' rulings, demonstrated pretty conclusively that it's a now a packed and extremist partisan body, with a clear and frightening agenda, that can't be relied upon to deliver impartial judgments.
Probably there isn't any immediate solution to the problem, which has been twenty or thirty years in the making, and to my mind any long-term solution will have to be achieved at the state level, undoing the last 20 years of gerrymandering and voter suppression in which the Republicans have engaged to secure their positions.