In a large saucepan, stir cocoa powder into 1/2 cup of the eggnog until completely combined. Add the remaining 3 1/2 cups eggnog, milk and instant espresso powder. Heat and stir until very hot and simmering. Do not boil. Remove from heat and stir in vanilla.
To serve, pour into 8 mugs. Top each serving with whipped cream and serve with cinnamon sticks. If you like, each guest can drizzle a little of his/her favorite liqueur over whipped cream and, if you like, garnish with chocolate curls.