Having a costume contest at your Halloween party seems like an obvious fit, but there are a few pitfall that are important to avoid. The contest should be one of the highlights of your party and for that to happen you are going to need to have a plan.
The first and arguably most important step for a successful Halloween costume contest is to clearly announce the contest well before your event. Make sure that every guest knows the not only are you going to have a costume contest but that you plan for it to be one of the focal points of the party. The more you hype it, the more your guests will commit to wearing an outstanding costume.
If you are going to have more than around 15 guests consider awarding prizes for costumes in multiple categories. Pick a winner with the scariest, funniest, sexiest and most creative costume. After you have chosen the category winners, award a best overall costume from the group of category winners.
The voting for the winners should include all of your party guests. You can set up a table with pens and voting ballots with the category names and a blank space for them to write who they think should win. An alternate option is to vote by crowd. Bring the guests in front of the party and one at a time ask the crowd to cheer for who they think should be the winner. The person who gets the loudest cheers in each category wins.
As far prizes a simply rule to remember is the better the prizes the better the costumes. But that does not mean you have to spend a fortune. You should know your guests pretty well, select prizes that they will find appealing or just plain fun. This is definitely a situation where creativity can trump the dollar value of the prize.