Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in Registration Statements on Form S-8 (No. 333-105946, No. 333-117973, No. 333-139736, and No. 333-145030) and Form S-3 (No. 333-157196) of First Security Group, Inc. of our report, dated March 16, 2010, except for the effects of the restatement described in Note 2, as to which the date is November 16, 2010, with respect to the consolidated financial statements of First Security Group, Inc. and subsidiary, and our report dated March 16, 2010, except for the effects of the material weakness described in the sixth paragraph of such report, as to which the date is November 16, 2010, with respect to the effectiveness of internal control over financial reporting, which reports are included in this Annual Report on Form 10-K/A (Amendment 2) of First Security Group, Inc. for the year ended December 31, 2009.

/s/ Joseph Decosimo and Company, PLLC

Chattanooga, Tennessee

November 16, 2010