That really sounds fun, and it also ensures that guests can mingle with one another since they have a goal to meet.
By the way, what happens if you still have more than 1 color at the end of the night? Another question, what happens to your clothes? Do you really exchange clothes (not return them after)?