Pasty - that depends totally upon what you think of your fellow employees.
It is polite to invite everyone you have worked with to your leaving doo, regardless of whether you liked them or not.
You could try the tactic that I used when leaving my last job, went for lunch with my section and...