he wants to be heads up, he's sighing and counting chips as if to say 'I'm calling anyway' to hopefully get the rest of you to put yours hands down.
if you go all in he probably folds.
Sounds about right - so the fold is correct. Let him risk the chips knocking the shorty out rather than you.