Exhibit 10.4

ROLLOVER AGREEMENT

This Rollover Agreement (this “Agreement”) is made and entered into as of June 6, 2012, by and among Cole Family Holdco, LLC, a Delaware limited liability company (“Family LLC”), and the stockholders (“Family Stockholders”) of Kenneth Cole Productions, Inc. (the “Company”) listed on Annex A attached hereto.

WHEREAS, concurrent with the execution and delivery of this Agreement, KCP Holdco, Inc., a Delaware corporation and wholly-owned subsidiary of Family LLC (“Parent”), KCP MergerCo, Inc., a New York corporation and wholly-owned subsidiary of Parent (“Merger Sub”), and the Company are entering into an Agreement and Plan of Merger (the “Merger Agreement”), which provides, among other things, for the merger of Merger Sub with and into the Company, with the Company surviving as a wholly-owned subsidiary of Parent (the “Merger”);

WHEREAS, as of the date hereof, each Family Stockholder is the beneficial owner of, and has the sole or shared right to vote and dispose of, (i) that number of shares of Class A Common Stock, par value $0.01 per share, of the Company (“Class A Stock”) and (ii) that number of shares of Class B Common Stock, par value $0.01 per share, of the Company (“Class B Stock” and together with Class A Stock, “Rollover Shares”), set forth opposite such Family Stockholder’s name on Annex A hereto; and

WHEREAS, subject to the conditions set forth herein, immediately prior to the Effective Time (as defined in the Merger Agreement), (i) each Family Stockholder desires to exchange that number of Rollover Shares set forth opposite such Family Stockholder’s name on Annex A hereto and (ii) Family LLC desires to issue to such Family Stockholder, in exchange for such Rollover Shares, membership units in Family LLC (“Family LLC Units”) as set forth opposite each Family Stockholder’s name on Annex A hereto.

NOW, THEREFORE, in consideration of the mutual promises, covenants, representations and warranties contained herein, the parties hereto agree as follows:

1. Rollover.

(a) Immediately prior to the Effective Time, (i) each Family Stockholder will assign, transfer, convey and deliver such Family Stockholder’s Rollover Shares to Family LLC, and in exchange for such Rollover Shares, Family LLC shall issue and deliver to such Family Stockholder the number of Family LLC Units set forth opposite such Family Stockholder’s name on Annex A (collectively, the “Rollover”) and (ii) any and all Family LLC Units previously issued shall cease to be outstanding and shall automatically be cancelled and shall cease to exist. No Family Stockholder shall be required under any circumstance to contribute, or cause to be contributed, to Family LLC a number of Rollover Shares in excess of the number of Rollover Shares set forth opposite such Family Stockholder’s name on Annex A hereto.

(b) In the event that the Rollover is consummated but the Merger Agreement is terminated in accordance with its terms, then the Rollover will be void ab initio and deemed not to have occurred and each Family Stockholder will deliver to Family LLC

the number of Family LLC Units received by such Family Stockholder pursuant to paragraph (a) of this Section 1 and Family LLC will deliver to each Family Stockholder the Rollover Shares previously delivered by such Family Stockholder to Family LLC.

2. Closing.

(a) The closing of the transactions contemplated by this Agreement (the “Rollover Closing”) will take place at the offices of Willkie Farr & Gallagher LLP, 787 Seventh Avenue, New York, New York 10019, immediately prior to the Effective Time.

(b) At the Rollover Closing, each Family Stockholder will deliver to Family LLC stock certificates duly endorsed for transfer to Family LLC, or accompanied by stock powers duly endorsed in blank, representing each such Family Stockholder’s Rollover Shares, and Family LLC will reflect on its books and records such Family Stockholder’s ownership of the number of Family LLC Units set forth opposite such Family Stockholder’s name on Annex A.

3. Representations and Warranties of the Investors. Each Family Stockholder represents and warrants, severally but not jointly, as follows:

(a) Binding Agreement. Such Family Stockholder has the capacity to execute and deliver this Agreement and to consummate the transactions contemplated hereby, and (i) in the case of each Family Stockholder that is an individual, the execution and delivery of this Agreement does not require any consent from such Family Stockholder’s spouse or any other person, and (ii) in the case of each Family Stockholder that is a trust, Robyn Transport is such Family Stockholder’s trustee and has the authority to execute and deliver this Agreement on behalf of such Family Stockholder. Such Family Stockholder has duly and validly executed and delivered this Agreement and this Agreement constitutes a legal, valid and binding obligation of such Family Stockholder, enforceable against such Family Stockholder in accordance with its terms, except as such enforceability may be limited by applicable bankruptcy, insolvency, reorganization or other similar laws affecting creditors’ rights generally and by general equitable principles (regardless of whether enforceability is considered in a proceeding in equity or at law).

(b) Ownership of Shares. Such Family Stockholder is the “beneficial owner” (as defined in Rule 13d-3 under the Securities Exchange Act of 1934, as amended, which meaning will apply for all purposes of this Agreement) of, and has the sole (or shared with one or more other Family Stockholders) power to vote and dispose of the number of Rollover Shares set forth opposite such Family Stockholder’s name in Annex A hereto free and clear of any security interests, liens, charges, encumbrances, equities, claims, options or limitations of whatever nature and free of any other limitation or restriction (including any restriction on the right to vote, sell or otherwise dispose of such Rollover Shares), except as may exist by reason of this Agreement or pursuant to applicable law. Except as provided for in this Agreement, there are no outstanding options or other rights to acquire from such Family Stockholder, or obligations of such Family Stockholder to sell or to dispose of, any of such Rollover Shares.

2

(c) No Conflict. Neither the execution and delivery of this Agreement, nor the consummation of the transactions contemplated hereby, nor the performance of such Family Stockholder’s obligations hereunder will (a) result in a violation or breach of, or constitute (with or without due notice or lapse of time or both) a default (or give rise to any right of termination, cancellation, or acceleration) under any contract, agreement, instrument, commitment, arrangement or understanding to which such Family Stockholder is a party, or result in the creation of a security interest, lien, charge, encumbrance, equity or claim with respect to such Family Stockholder’s Rollover Shares, or (b) require any material consent, authorization or approval of any person, entity or governmental entity, or (c) violate or conflict with any writ, injunction or decree applicable to such Family Stockholder or such Family Stockholder’s Rollover Shares.

(d) Accredited Investor. Such Family Stockholder is an “accredited investor” as such term is defined in Rule 501(a) promulgated under the Securities Act of 1933, as amended (the “Securities Act”).

(e) Investor’s Experience. Such Family Stockholder’s financial situation is such that the Family Stockholder can afford to bear the economic risk of holding the Family LLC Units to be received by such Family Stockholder, and such Family Stockholder can afford to suffer complete loss of its investment in such Family LLC Units. Such Family Stockholder’s knowledge and experience in financial and business matters are such that the Family Stockholder is capable of evaluating the merits and risks of the Family Stockholder’s investment in such Family LLC Units.

(f) Investment Intent. Such Family Stockholder is acquiring Family LLC Units solely for the Family Stockholder’s own account for investment and not with a view to or for sale in connection with any distribution thereof. The Family Stockholder agrees that the Family Stockholder will not, directly or indirectly, offer, transfer, sell, pledge, hypothecate or otherwise dispose of any Family LLC Units (or solicit any offers to buy, purchase or otherwise acquire or take a pledge of any of the Family LLC Units), except in compliance with (i) the Securities Act and the rules and regulations of the Securities and Exchange Commission thereunder, (ii) applicable state and non-U.S. securities or “blue sky” laws and (iii) the provisions of this Agreement and any other agreement among the Family Stockholders.

4. Conditions Precedent. The obligations of each Family Stockholder to consummate the transactions contemplated hereby are subject to the conditions set forth in Sections 6.01 and 6.02 of the Merger Agreement being satisfied or waived by Parent (other than any conditions that by their nature are to be satisfied at the Closing, but subject to the prior or substantially concurrent satisfaction of such conditions), and such other conditions set forth in the Equity Commitment Letter, dated of even date herewith, among the Family Stockholders, Family LLC, Parent and the Company (the “Equity Commitment Letter”).

5. Certain Approvals. Each Family Stockholder hereby acknowledges that the Principal Stockholder, as the managing member of Family LLC, may take any and all actions on behalf of Family LLC in its sole discretion, including voting the shares of Parent held by Family LLC, before the Effective Time.

3

6. Operating Agreement. Simultaneous with the Effective Time, each Family Stockholder shall enter into the Amended and Restated Limited Liability Company Agreement of Parent, in such form to be agreed among the Family Stockholders prior to Closing.

7. Tax Matters.

The parties hereto (i) shall treat the Rollover as a tax-free partnership contribution under Section 721 of the Internal Revenue Code of 1986, as amended, for all federal, state and local income tax purposes, and (ii) will not take any position on any return that is inconsistent with such treatment.

8. Miscellaneous.

(a) Notices. All notices and other communications hereunder shall be in writing and shall be deemed to have been duly given if delivered personally or sent by telecopy, overnight courier service or by registered or certified mail (postage prepaid, return receipt requested), to any Family Stockholder at the address of such Family Stockholder set forth on Annex A (or at such other address as shall be specified by such Family Stockholder by like notice) and to Family LLC, at the following address or at such other address as shall be specified by Family LLC by like notice:

to:

Cole Family Holdco, LLC

c/o Kenneth Cole Productions, Inc.

603 West 50th Street

New York, NY 10019

Telecopier: 1-866-698-7042

Attention: Kenneth D. Cole

with a copy to (which shall not constitute notice):

Willkie Farr & Gallagher LLP

787 Seventh Avenue

New York, New York 10019

Telecopier: (212) 728-9129

Attention: Adam M. Turteltaub

(b) Binding Effect; Benefits.

(i) This Agreement will be binding upon the successors, heirs, executors and administrators of the parties hereto. Except as set forth in Section 8(b)(ii) and other than as set forth in the Equity Commitment Letter, nothing in this Agreement, express or implied, is intended or will be construed to give any person other than the parties to this Agreement and their respective successors or permitted assigns any legal or equitable right, remedy or claim under or in respect of any agreement or any provision contained herein.

4

(ii) Notwithstanding anything to the contrary in this Agreement, each of the parties acknowledges and agrees that the Company is hereby made a third party beneficiary of this Agreement for the purposes of enforcing (including specific performance of) Parent’s rights to cause the Rollover to be consummated in accordance with the terms hereof, solely to the extent expressly provided and permitted by the terms and conditions of Section 8.08(b) of the Merger Agreement and the terms and conditions hereof.

(c) Amendments. This Agreement may not be modified, amended, altered or supplemented except upon the execution and delivery of a written agreement executed by all of the parties hereto and the Company and in accordance with the terms of the Equity Commitment Letter.

(d) Assignability. Neither this Agreement nor any right, remedy, obligation or liability arising hereunder or by reason hereof will be assignable by any Family Stockholder without the prior written consent of Family LLC and the Company and in accordance with the terms of the Equity Commitment Letter; provided that, without the prior written consent of the Company, a Family Stockholder may assign its rights hereunder in connection with any Permitted Transfer (as defined in the Voting Agreement) under the terms of the Voting Agreement (a “Permitted Assignee”), provided that any Permitted Assignee shall, as a condition to any such assignment, execute a joinder agreement, in form and substance reasonably satisfactory to the Company, whereby the Permitted Assignee agrees to be bound by the provisions of this Agreement applicable to the Family Stockholders.

(e) Governing Law. This Agreement shall be governed by and construed in accordance with the laws of the State of New York, regardless of the laws that might otherwise govern under applicable principles of conflicts of laws thereof, except to the extent that mandatory provisions of federal law apply. Each of the parties hereto hereby irrevocably and unconditionally submits, for itself and its property, to the exclusive jurisdiction of the courts of the State of New York and any appellate court thereof and the United States District Court for the Southern District of New York and any appellate court thereof, in any action or proceeding arising out of or relating to this Agreement or the agreements delivered in connection herewith or the transactions contemplated hereby or thereby or for recognition or enforcement of any judgment relating thereto, and each of the parties hereby irrevocably and unconditionally (i) agrees not to commence any such action except in such courts, (ii) agrees that any claim in respect of any such action or proceeding may be heard and determined in such courts, (iii) waives, to the fullest extent it may legally and effectively do so any objection which it may now or hereafter have to venue of any such action or proceeding in any such courts, and (iv) waives, to the fullest extent permitted by law, the defense of any inconvenient forum to the maintenance of such action or proceeding in any such courts. Each of the parties hereto agrees that a final judgment in any such action or proceeding shall be conclusive and may be enforced in other jurisdictions by suit on the judgment or in any other manner provided by law. Each of the parties to this Agreement irrevocably consents to service of process in any such action or proceeding in the manner provided for notices in Section 8(a) of this Agreement; provided, however, that nothing in this Agreement shall affect the right of any party to this Agreement to serve process in any other manner permitted by law.

5

(f) Counterparts. This Agreement may be executed by facsimile and in two or more counterparts, each of which will be deemed to be an original, but all of which together will constitute one and the same instrument.

(g) Severability. If any term or other provision of this Agreement is invalid, illegal or incapable of being enforced by any rule of law, or public policy, all other conditions and provisions of this Agreement will nevertheless remain in full force and effect so long as the economic or legal substance of the transactions contemplated herein are not affected in any manner materially adverse to any party hereto. Upon such determination that any term or other provision is invalid, illegal or incapable of being enforced, the parties hereto will negotiate in good faith to modify this Agreement so as to effect the original intent of the parties as closely as possible in a mutually acceptable manner.

(h) Waiver. Any party to this Agreement may waive any condition to their obligations contained herein.

(i) Termination. This Agreement will terminate on the earlier of the termination of the Merger Agreement in accordance with its terms or the expiration or termination of the obligations of Family LLC and the Family Stockholders under the Equity Commitment Letter. Termination will not relieve any party from liability for any breach of its obligations hereunder committed prior to such termination.

(j) Further Assurances. Each party to this Agreement shall execute such documents and other papers and take such further actions as may be reasonably required or desirable to carry out the provisions hereof and the transactions contemplated hereby.

(k) Effectiveness. The obligations of the Family Stockholders under this Agreement shall not be effective or binding upon the Family Stockholders until such time as the Merger Agreement is executed and delivered by the parties thereto.

(l) Definitions. Capitalized terms not otherwise defined herein have the meanings ascribed to such terms in the Merger Agreement.

[Signatures on the following page]

6

IN WITNESS WHEREOF, the parties hereto have executed this Agreement as of the date first above written.

[Signature Page to Rollover Agreement]

Annex A