Exhibit 5.1

LAW OFFICES

SILVER, FREEDMAN & TAFF, L.L.P.

A LIMITED LIABILITY PARTNERSHIP INCLUDING PROFESSIONAL CORPORATIONS

3299 K STREET, N.W., SUITE 100

WASHINGTON, D.C. 20007

PHONE: (202) 295-4500

FAX: (202) 337-5502

WWW.SFTLAW.COM

July 8, 2013

First PacTrust Bancorp, Inc.

18500 Von Karman Avenue, Suite 1100

Irvine, California 92612

Ladies and Gentlemen:

We have acted as special Maryland counsel to First PacTrust Bancorp, Inc., a Maryland corporation (the “Company”), in connection with the issuance and sale by the Company on July 8, 2013 of an aggregate of 210,000 depositary shares (the “Depositary Shares”), pursuant to the exercise of the over-allotment option granted to the Underwriters (as defined below), each such Depositary Share representing ownership of 1/40th of a share of 8.00% Non-Cumulative Perpetual Preferred Stock, Series C of the Company, par value $0.01 per share and a liquidation preference of $1,000 per share (the “Underlying Preferred Shares”). The Underlying Preferred Shares will, when issued, be deposited by the Company against delivery of depositary receipts (“Depositary Receipts”) to be issued by Registrar and Transfer Company (the “Depositary”) evidencing the Depositary Shares, under a Deposit Agreement, dated June 12, 2013 (the “Deposit Agreement”), among the Company, the Depositary, and the holders from time to time of the Depositary Receipts issued thereunder. Each Depositary Receipt will evidence one or more Depositary Shares. The Depositary Shares are being sold pursuant to the Underwriting Agreement, dated as of June 5, 2013 (the “Underwriting Agreement”), by and among the Company, Goldman, Sachs & Co. and Keefe, Bruyette & Woods, Inc., as representatives of the several underwriters named in Schedule A thereto (collectively, the “Underwriters”). The Depositary Shares and the Underlying Preferred Shares have been registered under the Securities Act of 1933, as amended (the “Securities Act”), pursuant to a registration statement on Form S-3 filed by the Company with the Securities and Exchange Commission (the “Commission”), which became effective on November 23, 2010 (File No. 333-170622) (the “Registration Statement”), including a base prospectus dated November 23, 2010 (the “Base Prospectus”) and a prospectus supplement dated June 5, 2013 filed with the Commission on June 6, 2013 pursuant to Rule 424(b) under the Securities Act (the “Prospectus Supplement,” and together with the Base Prospectus, the “Prospectus”).

In connection with our opinion, we have examined originals, or copies, certified or otherwise identified to our satisfaction, of the Deposit Agreement, the Underwriting Agreement, the form of certificate evidencing the Underlying Preferred Shares, the Registration Statement, the Prospectus, the Company’s charter, including the articles supplementary for the Underlying Preferred Shares, the Company’s bylaws, resolutions of the Company’s Board of Directors and committees thereof, certificates of public officials, certificates of corporate officers and such other documents and corporate records as we have deemed appropriate for the purpose of rendering this opinion. We have assumed without investigation the genuineness of all signatures, the legal capacity of natural persons, the authenticity, accuracy and completeness of all documents submitted to us as originals, the conformity to authentic and complete original documents of all documents submitted to us as certified, conformed or photostatic copies and the authenticity, accuracy and completeness of the originals of such copies. We have further assumed the due authorization of such documents by all parties other than the Company and the taking of all requisite action respecting such documents by all parties other than the Company, the due execution and

First PacTrust Bancorp, Inc.

July 8, 2013

Page 2

delivery of such documents by each party other than the Company and have additionally assumed that all agreements are the valid and binding agreements of all parties to such agreements, other than the Company. In addition, we have assumed the accuracy of certifications of public officials, government agencies and departments, corporate officers, and individuals and statements of fact, on which we are relying, and have made no independent investigations thereof.

Based upon the foregoing, and subject to the limitations, qualifications, exceptions and assumptions set forth herein, we are of the opinion that:

In rendering the opinion set forth herein, we express no opinion as to the laws of any jurisdiction other than the General Corporation Law of the State of Maryland, as currently in effect. This opinion is limited to the facts bearing on this opinion as they exist on the date of this letter. We disclaim any obligation to review or supplement this opinion or to advise you of any changes in the circumstances, laws or events that may occur after this date or otherwise update this opinion.

We hereby consent to the filing of this opinion as an exhibit to the Company’s Current Report on Form 8-K filed on July 8, 2013, and to the reference to our name under the heading “Legal Matters” in the Prospectus. In giving our consent, we do not thereby admit that we are in the category of persons whose consent is required under Section 7 of the Securities Act or the rules and regulations of the Commission thereunder. The opinions expressed herein are matters of professional judgment and are not a guarantee of result.