Stopping draft from fireplace?

my basement fire place has a shutter door just up in the chimney but it isn't air tight and the cold air is rushing in constantly. I tried to back insulation in there but its an awkward opening at best and big, hard to make it air tight. any suggestions, i saw a balloon you could blow up in there but not sure that will be easy or doable either