Exhibit 4.1

SECOND AMENDMENT

TO THE

RIGHTS AGREEMENT

THIS SECOND AMENDMENT TO THE RIGHTS AGREEMENT (this “Amendment”) is made as of May 1, 2014, by and between R. G. Barry Corporation (the “Company”) and Broadridge Corporate Issuer Solutions, Inc. (the “Rights Agent,” which term shall include any successor Rights Agent hereunder). Capitalized terms not otherwise defined herein shall have the meanings given to them in the Rights Agreement (as such term is defined in the recitals below).

W I T N E S S E T H:

WHEREAS, the Company and The Bank of New York Mellon Corporation (the “Former Rights Agent”) executed and entered into a Rights Agreement dated as of May 1, 2009 (the “Rights Agreement”);

WHEREAS, the Company, the Former Rights Agent and the Rights Agent executed and entered into a First Amendment to the Rights Agreement dated August 15, 2011, to remove the Former Rights Agent and to appoint the Rights Agent as successor under the Rights Agreement;

WHEREAS, Section 27 of the Rights Agreement provides that, prior to the Distribution Date, the Company and the Rights Agent shall, if the Company so directs, supplement or amend any provision of the Rights Agreement without the approval of any holders of certificates representing Common Shares;

WHEREAS, as of the date hereof a Distribution Date has not occurred and Rights Certificates have not been issued;

WHEREAS, the Board has determined that an amendment to the Rights Agreement as set forth herein is necessary and desirable to extend the expiration date of the Rights Agreement and has authorized and approved this Amendment, and the Company and the Rights Agent desire to evidence such amendment in writing.

NOW, THEREFORE, the Company and the Rights Agent hereby amend the Rights Agreement as follows:

1. AMENDMENT OF SECTION 7(a). Section 7(a) of the Rights Agreement is hereby amended by deleting “(i) 5:00 p.m., New York City Time, on May 1, 2014 (the “Final Expiration Date”)” and replacing it with the following:

“(i) 5:00 p.m., New York City Time, on December 31, 2014 (the “Final Expiration Date”)”

2. INDEMNIFICATION. The Company will indemnify, defend, protect and hold harmless the Rights Agent from and against any and all losses, liabilities, costs, damages or expenses, including, without limitation, reasonable attorneys’ fees and expenses, incurred or made, arising out of or in connection with this Amendment.

3. EFFECTIVENESS. This Amendment shall be effective as of the date first written above, as if executed on such date. Except as specifically amended by this Amendment, all other terms and conditions of the Rights Agreement shall remain in full force and effect and are hereby ratified and confirmed.

4. DIRECTION TO RIGHTS AGENT; WAIVER OF NOTICE. By its execution and delivery hereof, the Company hereby directs the Rights Agent to execute this Amendment. The Rights Agent and the Company hereby waive any notice requirement under the Rights Agreement in connection with the entering into and delivery of this Amendment or the amendments contemplated hereby.

5. MISCELLANEOUS. This Amendment shall be deemed to be a contract made under the laws of the State of Ohio and for all purposes shall be governed by and construed in accordance with the laws of such State applicable to contracts made and to be performed entirely within such State. This Amendment may be executed in any number of counterparts and by facsimile and electronic transmission (including by .pdf) and each of such counterparts shall for all purposes be deemed to be an original, and all of such counterparts shall together constitute one and the same instrument. If any term, provision, covenant or restriction of this Amendment is held by a court of competent jurisdiction or other authority to be invalid, void or unenforceable, the remainder of the terms, provisions, covenants, and restrictions of this Amendment shall remain in full force and effect and shall in no way be affected, impaired or invalidated.

[Remainder of page intentionally blank. Signatures follow.]

2

IN WITNESS WHEREOF, the parties hereto have caused this Second Amendment to the Rights Agreement to be duly executed and attested, all as of the day and year first above written.

3