I have a bunch of community lots saved to my library (and houses of different styles) and I want all the lots (or houses) of the same type to show up next to each other. How can I achieve this?

By saving them to the library in the same order you want them to show up. The library is sorted chronologically.

By naming them how you want them to appear and sorting by name

OK, that's doable.

Smidgen, do you mean naming them in Edit Town or somewhere else like S3PE?

I edit my lot names by extracting the package file from the Library folder in the The Sims 3 folder. This can be a bit tricky as all the lots in the Library folder have a ridiculously long number code, despite naming the lots in Edit Town. So when I create, or place, a new lot I always extract the last few numbered package files from my Library folder, sorting by Date Modified is your friend here! When I place/create a lot (ultimately saving it to the Library folder) I note the date, time and lot name, this makes it easier to work out what lot is what. I open the package in S3PE to check that it's the correct lot, just by looking at the image, and rename the ridiculously long code to the require name I want.

You can then organise your lots however you like, and when sorted by name it makes it simple to transfer lots around if you have various Sims 3 folders for specific worlds/gameplay etc or seasons. Whether it be by lot creator name, season style, community, residential, I find this works for me.