I get what you are saying. But you Court Pack when you can. You don't Court Pack every single year. Then once the Court is Packed, you back off.
It's much less likely that a divided Exec/Senate will confirm justices because of McConnell's Precedent. So your statement is irrelevent because it's literally impossible to pack the court in divided government. The question is, given the chance, should a Dem Pres+Senate pack the Court?
Term limits is about increasing the turnover of the SC, however more Turnover does not mean a more representative Court when the Senate is so biased towards Red States. Does an expiring Justice induce the Senate to approve a President's nominee? McConnell proved that answer to be "negative." So increasing the turnover is not a solution. It's a non-answer by Booker.