Exhibit 10.5

EXCHANGE AGREEMENT

This Exchange Agreement (this “Agreement”) is made and entered into as of June 6, 2012, by and among Cole Family Holdco, LLC, a Delaware limited liability company (“Family LLC”), KCP Holdco, Inc., a Delaware corporation and wholly-owned subsidiary of Family LLC (“Parent”), and KCP Acquisitions, Inc., a Delaware corporation and wholly-owned subsidiary of Parent (“Intermediate Holdco”, and together with Family LLC and Parent, the “Parties”).

WHEREAS, concurrent with the execution and delivery of this Agreement, Parent, KCP MergerCo, Inc., a New York corporation and wholly-owned subsidiary of Parent (“Merger Sub”), and Kenneth Cole Productions, Inc. (the “Company”) are entering into an Agreement and Plan of Merger (the “Merger Agreement”), which provides, among other things, for the merger of Merger Sub with and into the Company, with the Company surviving as a wholly-owned subsidiary of Parent (the “Merger”);

WHEREAS, concurrent with the execution and delivery of this Agreement, Family LLC and the stockholders (“Family Stockholders”) of the Company parties thereto are entering into a Rollover Agreement (the “Rollover Agreement”), which provides, among other things, for the exchange of (i) (A) that number of shares of Class A Common Stock, par value $0.01 per share, of the Company (“Class A Stock”) and (B) that number of shares of Class B Common Stock, par value $0.01 per share, of the Company (“Class B Stock” and together with Class A Stock, “Rollover Shares”) described therein, for (ii) limited liability company interests in Family LLC;

WHEREAS, subject to the conditions set forth herein, immediately after the Rollover Closing (as defined in the Rollover Agreement), (i) Family LLC desires to contribute the Rollover Shares to Parent and (ii) Parent desires to issue to Family LLC, in exchange for the contribution of such Rollover Shares, shares of common stock, par value $0.01, of Parent (the “Parent Common Stock”), representing 100% of the outstanding capital stock of Parent; and

WHEREAS, subject to the conditions set forth herein, immediately after the Family LLC Contribution (as defined below), (i) Parent desires to contribute the Rollover Shares to Intermediate Holdco and (ii) Intermediate Holdco desires to issue to Parent, in exchange for such contribution of the Rollover Shares, shares of common stock, par value $0.01, of Intermediate Holdco (the “Intermediate Holdco Common Stock”) representing 100% of the outstanding capital stock of Intermediate Holdco.

NOW, THEREFORE, in consideration of the mutual promises, covenants, representations and warranties contained herein, the Parties hereto agree as follows:

1. Contributions.

(a) Immediately after the Rollover Closing and prior to the Effective Time, (i) Family LLC will contribute, assign, transfer, convey and deliver the Rollover Shares to Parent and, in exchange for the contribution of such Rollover Shares, Parent shall issue and deliver to Family LLC 100 shares of Parent Common Stock (the “Family LLC Contribution”) representing 100% of the outstanding capital stock of Parent and (ii) all Parent Common Stock previously issued shall cease to be outstanding and shall automatically be cancelled and shall cease to exist.

(b) Immediately after the Family LLC Contribution and prior to the Effective Time, (i) Parent will contribute, assign, transfer, convey and deliver the Rollover Shares to Intermediate Holdco and, in exchange for such contribution of the Rollover Shares, Intermediate Holdco shall issue and deliver to Parent 100 shares of Intermediate Holdco Common Stock (the “Parent Contribution” and, together with the Family LLC Contribution, the “Contributions”) representing 100% of the outstanding capital stock of Intermediate Holdco and (ii) all Intermediate Holdco Common Stock previously issued shall cease to be outstanding and shall automatically be cancelled and shall cease to exist.

(c) In the event that the Contributions are consummated but the Merger Agreement is terminated in accordance with its terms, then the Contributions will be void ab initio and deemed not to have occurred and each of Family LLC and Parent, respectively, will deliver to each of Parent and Intermediate Holdco, respectively, the number of shares of Parent Common Stock or Intermediate Holdco Common Stock received pursuant to paragraphs (a) or (b) of this Section 1 and each of Intermediate Holdco and Parent, respectively, will deliver to each of Parent and Family LLC, respectively, the Rollover Shares previously delivered by Parent and Family LLC, respectively, to Intermediate Holdco and Parent.

2. Closing.

(a) The closing of the transactions contemplated by this Agreement (the “Contribution Closing”) will take place at the offices of Willkie Farr & Gallagher LLP, 787 Seventh Avenue, New York, New York 10019, immediately prior to the Effective Time.

(b) At the Contribution Closing, each of Family LLC and Parent, respectively, will deliver to each of Parent and Intermediate Holdco, respectively, stock certificates duly endorsed for transfer to each of Parent and Intermediate Holdco, respectively, or accompanied by stock powers duly endorsed in blank, representing the Rollover Shares, and each of Parent and Intermediate Holdco will reflect on their books and records each of Family LLC’s and Parent’s ownership, respectively, of 100 of shares of Parent Common Stock or Intermediate Holdco Common Stock, respectively.

3. Representations and Warranties of the Parties. Each of the Parties hereto represents and warrants, severally but not jointly, as follows:

(a) Binding Agreement. Such Party has the capacity to execute and deliver this Agreement and to consummate the transactions contemplated hereby. Such Party has duly and validly executed and delivered this Agreement and this Agreement constitutes a legal, valid and binding obligation of such Party, enforceable against such Party in accordance with its terms, except as such enforceability may be limited by applicable bankruptcy, insolvency, reorganization or other similar laws affecting creditors’ rights generally and by general equitable principles (regardless of whether enforceability is considered in a proceeding in equity or at law).

2

(b) No Conflict. Neither the execution and delivery of this Agreement, nor the consummation of the transactions contemplated hereby, nor the performance of such Party’s obligations hereunder will (a) result in a violation or breach of, or constitute (with or without due notice or lapse of time or both) a default (or give rise to any right of termination, cancellation, or acceleration) under any contract, agreement, instrument, commitment, arrangement or understanding to which such Party is a party, or result in the creation of a security interest, lien, charge, encumbrance, equity or claim with respect to the Rollover Shares, or (b) require any material consent, authorization or approval of any person, entity or governmental entity, or (c) violate or conflict with any writ, injunction or decree applicable to such Party.

4. Conditions Precedent. The obligations of each Party to consummate the transactions contemplated hereby are subject to the conditions set forth in Sections 6.01 and 6.02 of the Merger Agreement being satisfied or waived by Parent (other than any conditions that by their nature are to be satisfied at the Closing, but subject to the prior or substantially concurrent satisfaction of such conditions), and such other conditions set forth in the Equity Commitment Letter, dated of even date herewith, among the Family Stockholders, Family LLC, Parent and the Company (the “Equity Commitment Letter”).

5. Tax Matters.

The Parties hereto (i) shall treat each of the Contributions as a tax-free contribution under Section 351 of the Internal Revenue Code of 1986, as amended, for all federal, state and local income tax purposes, and (ii) will not take any position on any return that is inconsistent with such treatment.

6. Miscellaneous.

(a) Notices. All notices and other communications hereunder shall be in writing and shall be deemed to have been duly given if delivered personally or sent by telecopy, overnight courier service or by registered or certified mail (postage prepaid, return receipt requested), to the attention of Family LLC, Parent or Intermediate Holdco, respectively, at the following address or at such other address as shall be specified by Family LLC, Parent or Intermediate Holdco by like notice:

c/o Kenneth Cole Productions, Inc.

603 West 50th Street

New York, NY 10019

Telecopier: 1-866-698-7042

Attention: Kenneth D. Cole

3

with a copy to (which shall not constitute notice):

Willkie Farr & Gallagher LLP

787 Seventh Avenue

New York, New York 10019

Telecopier: (212) 728-9129

Attention: Adam M. Turteltaub

(b) Binding Effect; Benefits.

(i) This Agreement will be binding upon the successors, heirs, executors and administrators of the Parties hereto. Except as set forth in Section 6(b)(ii) and other than as set forth in the Equity Commitment Letter, nothing in this Agreement, express or implied, is intended or will be construed to give any person other than the Parties to this Agreement and their respective successors or permitted assigns any legal or equitable right, remedy or claim under or in respect of any agreement or any provision contained herein.

(ii) Notwithstanding anything to the contrary in this Agreement, each of the Parties acknowledges and agrees that the Company is hereby made a third party beneficiary of this Agreement for the purposes of enforcing (including specific performance of) Parent’s rights to cause the Contributions to be consummated in accordance with the terms hereof, solely to the extent expressly provided and permitted by the terms and conditions of Section 8.08(b) of the Merger Agreement and the terms and conditions hereof.

(c) Amendments. This Agreement may not be modified, amended, altered or supplemented except upon the execution and delivery of a written agreement executed by all of the Parties hereto and the Company and in accordance with the terms of the Equity Commitment Letter.

(d) Assignability. Neither this Agreement nor any right, remedy, obligation or liability arising hereunder or by reason hereof will be assignable by any Party, without the prior written consent of the other Parties and the Company and in accordance with the terms of the Equity Commitment Letter.

(e) Governing Law. This Agreement shall be governed by and construed in accordance with the laws of the State of New York, regardless of the laws that might otherwise govern under applicable principles of conflicts of laws thereof, except to the extent that mandatory provisions of federal law apply. Each of the Parties hereto hereby irrevocably and unconditionally submits, for itself and its property, to the exclusive jurisdiction of the courts of the State of New York and any appellate court thereof and the United States District Court for the Southern District of New York and any appellate court thereof, in any action or proceeding arising out of or relating to this Agreement or the agreements delivered in connection herewith or the transactions contemplated hereby or thereby or for recognition or enforcement of any judgment relating thereto, and each of the Parties hereby irrevocably and unconditionally (i) agrees not to commence any such

4

action except in such courts, (ii) agrees that any claim in respect of any such action or proceeding may be heard and determined in such courts, (iii) waives, to the fullest extent it may legally and effectively do so any objection which it may now or hereafter have to venue of any such action or proceeding in any such courts, and (iv) waives, to the fullest extent permitted by law, the defense of any inconvenient forum to the maintenance of such action or proceeding in any such courts. Each of the Parties hereto agrees that a final judgment in any such action or proceeding shall be conclusive and may be enforced in other jurisdictions by suit on the judgment or in any other manner provided by law. Each of the Parties to this Agreement irrevocably consents to service of process in any such action or proceeding in the manner provided for notices in Section 7(a) of this Agreement; provided, however, that nothing in this Agreement shall affect the right of any Party to this Agreement to serve process in any other manner permitted by law.

(f) Counterparts. This Agreement may be executed by facsimile and in two or more counterparts, each of which will be deemed to be an original, but all of which together will constitute one and the same instrument.

(g) Severability. If any term or other provision of this Agreement is invalid, illegal or incapable of being enforced by any rule of law, or public policy, all other conditions and provisions of this Agreement will nevertheless remain in full force and effect so long as the economic or legal substance of the transactions contemplated herein are not affected in any manner materially adverse to any Party hereto. Upon such determination that any term or other provision is invalid, illegal or incapable of being enforced, the Parties hereto will negotiate in good faith to modify this Agreement so as to effect the original intent of the Parties as closely as possible in a mutually acceptable manner.

(h) Waiver. Any Party to this Agreement may waive any condition to their obligations contained herein.

(i) Termination. This Agreement will terminate on the earlier of the termination of the Merger Agreement in accordance with its terms or the expiration or termination of the obligations of Family LLC and the Family Stockholders under the Equity Commitment Letter. Termination will not relieve any Party from liability for any breach of its obligations hereunder committed prior to such termination.

(j) Further Assurances. Each Party to this Agreement shall execute such documents and other papers and take such further actions as may be reasonably required or desirable to carry out the provisions hereof and the transactions contemplated hereby.

(k) Effectiveness. The obligations of the Parties under this Agreement shall not be effective or binding upon the Parties until such time as the Merger Agreement is executed and delivered by the parties thereto.

(l) Definitions. Capitalized terms not otherwise defined herein have the meanings ascribed to such terms in the Merger Agreement.

[Signatures on the following page]

5

IN WITNESS WHEREOF, the Parties hereto have executed this Agreement as of the date first above written.