I totally believed Dean would, being angry/afraid over Jack being an unpredictable loose cannon, convince himself that sealing Jack in the box was the only choice they had.

I also think, after Dean's emotions settled down/Sam worked on him, he would have let Jack out for another chance.

Where he screwed up was in abandoning Jack in the locked room. Had they stayed in the vicinity to assure Jack they had not left him to rot (even if they were going to do that) time could have been bought to come up with another solution.