Exhibit 10.4

PLEDGE AND SECURITY AGREEMENT

This Pledge and Security Agreement (the “Agreement”), dated October 31, 2012, is given by San Rafael Distributing, Inc., an Arizona corporation (“San Rafael”) to Calavo Growers, Inc., a California corporation (“Calavo”) in connection with San Rafael’s purchase of Calavo’s membership interest in Maui Fresh International, LLC, a California limited liability company (the “Company”) pursuant to the Sale of LLC Interest Agreement dated as of October 31, 2012 (the “Sale Agreement”).

RECITALS

A. Calavo and San Rafael entered into the Sale Agreement on October 31, 2012, pursuant to which Calavo sold to San Rafael all of Calavo’s membership interest in the Company, and San Rafael delivered to Calavo one promissory note in the amount of Nine Hundred Ninety-One Thousand, Five Hundred Twenty Dollars ($991,520) (the “Equity Note”) and one promissory note in the amount of One Million, Three Hundred Thirty-Seven Thousand, Six Hundred Ninety-Four Dollars ($1,337,694) (the “Goodwill Note”). Collectively, the Equity Note and the Goodwill Note are referred to as the “Promissory Notes”, and collectively, the sale of Calavo’s interest is referred to as the “Sold Interest”.

B. As security for the payment of the Promissory Notes and in order to induce Calavo to accept the Promissory Notes in consideration of the sale of the Sold Interest, San Rafael has agreed to give Calavo a first-priority security interest in all of the membership interests in the Company (the “Entire Interest”) upon the terms and conditions described in this Agreement. Notwithstanding the existence of security for the payment of the Promissory Notes, San Rafael shall at all times remain liable to Calavo for the full and punctual payment of all principal, accrued interest and other amounts that are owed under the Promissory Notes.

NOW, THEREFORE, San Rafael agrees as follows:

1. Grant of Security Interest. As security for the full and timely payment of all principal, accrued interest and other amounts that are owed by San Rafael under the Promissory Notes and this Agreement (collectively, the “Debt”) and the full and timely performance of all other obligations under this Agreement, San Rafael hereby grants to Calavo a continuing and first-priority security interest (the “Security Interest”) in the following (collectively, the “Collateral”): all right, title and interest of San Rafael in and to the Entire Interest and all rights and privileges pertaining to the Entire Interest, including, without limitation, all voting rights, all securities receivable in respect of or in exchange for the Entire Interest, all rights to subscribe for securities incident to or arising from ownership of the Entire Interest, all cash, stock and other dividends or distributions paid or payable on the Entire Interest, all of San Rafael’s books and records pertaining to the foregoing and all proceeds from sales, transfers or other dispositions of the Entire Interest and whatever is received when any of the foregoing is sold, exchanged or otherwise transferred. Without limiting the generality of the foregoing, if San Rafael receives any additional membership interests of the Company or of any successor to the Company, such additional membership interests shall be considered part of the “Entire Interest” for purposes of this Agreement and shall be subject to the Security Interest.

- 1 -

2. Rights of San Rafael. Prior to the occurrence of an Event of Default (as defined below in Section 9) and subject to the restrictions on the transferability of the Entire Interest described below in Section 3, San Rafael shall have all of the rights of a member of the Company with respect to the Entire Interest (including, without limitation, voting rights and the right to receive any cash dividends that may be declared and paid by the Company). Following the occurrence of an Event of Default, San Rafael’s rights with respect to the Entire Interest (including, without limitation, its voting rights and rights to dividends) shall be subject to all of Calavo’s rights and remedies upon the exercise of its Security Interest. San Rafael agrees that, during the continuance of an Event of Default and unless otherwise determined by Calavo: (i) all dividends and other distributions that it would otherwise be entitled to receive with respect to the Entire Interest shall instead be withheld by the Company and applied to payments that are owed by San Rafael on the Promissory Notes, and (ii) Calavo shall have the voting rights with respect to the Entire Interest.

3. Restrictions on Sale or Transfer; Continuing Operations. Except as provided below, until the payment in full of the Debt or unless the Debt will be paid as part of such transaction, San Rafael shall not: (a) sell, assign, pledge or otherwise transfer (whether voluntarily, involuntarily, by operation of law, by gift or for consideration) any of the Entire Interest, or sell the Company; and (b) without the prior written consent of Calavo, allow the Company to sell any of its assets, except in the ordinary course of business, or issue any new or additional membership interests or admit any new members to the Company, or amend its operating agreement. Any such prohibited sale, pledge or other transfer or issuance of new membership interests shall be null and void, and the Company shall not be required to transfer or enter on its books any new or transferred membership interests in the Company until the Debt has been paid in full. Until such time, San Rafael shall continue to operate and manage the Company and its assets in a good and prudent manner, in accordance with past practice. Notwithstanding the foregoing, all or part of the membership interest of Buyer in the Company may be assigned and transferred to Francisco Clouthier, or any entity in which Francisco Clouthier (or a grantor trust in which Francisco Clouthier is the sole trustee or a co-trustee with his spouse) is the sole owner thereof (subject to any community property interest); provided, however, that Francisco Clouthier shall confirm in writing at the time of the transfer the continuing validity and enforceability of the Pledge Agreement and the Guaranty. In the event of such assignment and transfer, and conditioned upon Calavo’s receipt of the above-described confirmation by Francisco Clouthier, the obligations of Buyer shall be assumed by Francisco Clouthier or the transferee entity and Buyer shall be relieved of any further obligation under the Promissory Notes.

4. Restrictive Legends. Substantially the following legends shall be placed on the membership certificates of the members and on the Operating Agreement of the Company: “THE OWNERSHIP INTERESTS REPRESENTED BY THIS CERTIFICATE ARE SUBJECT TO A SECURITY INTEREST IN FAVOR OF CALAVO GROWERS, INC. AND TO RESTRICTIONS ON TRANSFER DESCRIBED IN A PLEDGE AND SECURITY AGREEMENT BETWEEN THE REGISTERED OWNER OF THE MEMBERSHIP INTERESTS AND CALAVO GROWERS, INC., A COPY OF WHICH IS ON FILE AT THE PRINCIPAL EXECUTIVE OFFICE OF THE COMPANY. NO TRANSFER, SALE, ASSIGNMENT, PLEDGE, HYPOTHECATION OR OTHER DISPOSITION OF THE OWNERSHIP INTERESTS REPRESENTED BY THIS CERTIFICATE MAY BE MADE EXCEPT IN ACCORDANCE WITH THE PROVISIONS OF SUCH PLEDGE AND SECURITY AGREEMENT.”

- 2 -

5. Preservation and Protection of the Security Interest. San Rafael shall preserve and protect Calavo’s first-priority security interest in the Collateral and shall cause the Security Interest to be perfected and to continue to be perfected until the Debt is paid in full. Concurrently with the execution and delivery of this Agreement, San Rafael shall execute and deliver to Calavo a UCC financing statement for filing with the California Secretary of State. San Rafael shall execute and deliver to Calavo (within ten days after receipt of Calavo’s request) such other security agreements, endorsements, pledges, assignments and other documents (including, without limitation, financing statements and continuation statements and amendments thereto) as Calavo may request from time to time to effectuate the grant to Calavo of the Security Interest, and Calavo is authorized to file and/or record such documents with the California Secretary of State and other appropriate regulatory authorities. Within ten days after receipt of Calavo’s request, all certificates and instruments representing or evidencing the Collateral shall be delivered to Calavo for retention pursuant to this Agreement and shall be in suitable form for transfer by delivery or, as applicable, shall be accompanied by San Rafael’s endorsement, where necessary, of duly executed instruments of transfer, all in form and substance satisfactory to Calavo. Without limiting the generality of the foregoing, concurrently with its execution and delivery of this Agreement to Calavo, San Rafael shall deliver to Calavo the membership certificates evidencing San Rafael’s ownership of the Entire Interest and an undated power in blank executed by San Rafael with respect to the Entire Interest.

6. Title to the Collateral. San Rafael shall at all times maintain good and marketable title to the Collateral free and clear of all liens, encumbrances and other security interests. San Rafael shall pay in full any tax that is imposed on any of the Collateral prior to its delinquency and, within ten days after any other lien or encumbrance is imposed on any of the Collateral, San Rafael shall pay and discharge such lien or other encumbrance in full.

7. Cash Dividends Paid Prior to an Event of Default. Notwithstanding anything to the contrary in this Agreement, any and all cash dividends that are paid on the Entire Interest by the Company prior to the occurrence of an Event of Default shall not be treated as Collateral that is subject to the Security Interest, and San Rafael shall be entitled to retain and/or transfer such cash dividends in its discretion free from any restrictions imposed by this Agreement. However, any and all cash dividends that are paid on the Entire Interest by the Company during the continuance of an Event of Default shall be treated as Collateral that is subject to the Security Interest and the terms and conditions of this Agreement that pertain to the Security Interest and the Collateral.

8. Power of Attorney. San Rafael hereby appoints Calavo as its attorney-in-fact (with full power of substitution) to execute, deliver and file, effective upon the occurrence of an Event of Default, on San Rafael’s behalf and at San Rafael’s expense (i) any financing statements, continuation statements or other documents required to perfect or continue the Security Interest and (ii) any other documents and instruments that Calavo determines are necessary or appropriate in order to enable it to exercise its rights and remedies that are provided hereunder and by applicable law upon the occurrence of an Event of Default. This power, being coupled with an interest, shall be irrevocable until the Debt is paid in full.

- 3 -

9. Event of Default. An “Event of Default” under this Agreement means (i) San Rafael’s failure to pay when due any principal, accrued interest or other amount that is owed under either or both of the Promissory Notes or this Agreement, or (ii) San Rafael’s failure to perform any other agreement contained in this Agreement, the Sale Agreement or either (or both of) the Promissory Notes or the failure of any representation or warranty of San Rafael that is contained in this Agreement, the Sale Agreement or the Promissory Notes to be true, if the failure under either (i) or (ii) is not remedied within ten (10) days after receipt of written notice from Calavo.

10. Remedies on an Event of Default. Upon the occurrence of an Event of Default, Calavo shall have the immediate right to take control of all or any part of the Collateral, with or without judicial process, and without advertisement, and without demand of performance or notice to San Rafael, except as otherwise provided in Section 9 above, all of which are (except as set forth in Section 9) expressly waived by San Rafael; provided, however, that if any notice is required by law in connection with the exercise by Calavo of its rights and remedies, San Rafael agrees that ten days’ prior written notice is a reasonable time and manner for notice (which ten days’ notice shall be concurrent with, and not in addition to, the notice required under Section 9). Furthermore, Calavo may exercise all of the other rights and remedies that are provided to it under this Agreement and to a secured party by the California Uniform Commercial Code and otherwise by applicable law. Calavo’s rights and remedies shall include, without limitation, the power (i) to transfer into Calavo’s name or into the name of its nominee any or all of the Entire Interest or other Collateral and thereafter to receive and retain all cash and other dividends, distributions and payments made on account of the Entire Interest and other Collateral, and otherwise act with respect thereto as though it were the absolute owner thereof, and (ii) to sell all or any portion of the Entire Interest and other Collateral at a public or private sale at such place and time and at such prices and other terms as Calavo may determine. San Rafael recognizes that Calavo may be compelled to resort to one or more private sales of any or all of the Entire Interest constituting part of the Collateral to a restricted group of purchasers who will be obliged to agree, among other things, to acquire such securities for their own account for investment and not with a view to the distribution or resale thereof. San Rafael acknowledges and agrees that any such private sale may result in prices and other terms less favorable to the seller than if such sale were a public sale and, notwithstanding such circumstances, agrees that any such private sale shall not, for such reason alone, be deemed to have been made in a commercially unreasonable manner. Calavo shall not be under any obligation to delay a sale of any or all of the Entire Interest for the period of time necessary to permit the registration of the Entire Interest for public sale under the Securities Act of 1933, as amended, or under applicable state securities laws. At any sale, Calavo may, to the extent permissible under applicable law, purchase the whole or any part of the Entire Interest or other Collateral, and Calavo shall be entitled to use and apply any or all of the Debt as a credit on account of the purchase price of the Entire Interest or other Collateral. Calavo and any other purchaser of any portion or all of the Entire Interest or other Collateral at any such sale shall hold the purchased Entire Interest or other Collateral free from any claim or right on the part of San Rafael, and San Rafael hereby waives any right of redemption, stay or appraisal that it might otherwise have under applicable law.

11. Application of Proceeds. Any Collateral or the proceeds of the Collateral held or realized upon at any time by Calavo following an Event of Default shall be applied in satisfaction of the Debt, in such order of application as Calavo shall determine in its reasonable discretion, until the Debt is fully paid, and thereafter any balance shall be distributed to San Rafael or as otherwise required by applicable law.

- 4 -

12. No Implied Waivers; Cumulative Remedies. No delay or failure of Calavo in exercising any right or remedy under this Agreement shall operate as a waiver thereof, nor shall any single or partial exercise thereof or any abandonment or discontinuance of steps to enforce such a right or remedy preclude any further exercise thereof or of any other right or remedy. The rights and remedies of Calavo under this Agreement are cumulative and not exclusive of any rights or remedies which it might otherwise have under the California Uniform Commercial Code or other applicable law. Any waiver, permit, consent or approval of any kind or character on the part of Calavo of any Event of Default or any such waiver of any provision of this Agreement must be in writing and shall be effective only to the extent specifically set forth in writing. San Rafael acknowledges and agrees that the exercise by Calavo of its rights under this Agreement and the acquisition or sale by Calavo of any portion or all of the Entire Interest or other Collateral will not operate to release San Rafael from its obligation to pay the Debt until full payment of any deficiency on the Debt has been made in cash. Furthermore, San Rafael acknowledges and agrees that Calavo is not obligated to exercise any of the rights or remedies provided by this Agreement, and that Calavo shall be deemed to have exercised reasonable care in the custody and preservation of any Collateral in its possession if such Collateral is accorded treatment that is substantially similar to that which Calavo accords its own similar property.

13. Reimbursement of Fees and Expenses. If, after not less than 30 days after an Event of Default which has not been cured, an attorney is engaged by Calavo to undertake collection, or enforce or construe any provision of this Agreement, the Sale Agreement, or one or both of the Promissory Notes, with or without the filing of any arbitration proceeding or legal action by Calavo, then San Rafael shall pay on demand all reasonable attorneys’ fees and other costs and expenses incurred by Calavo in connection therewith. If an action (arbitration or court proceeding) is brought to enforce the terms of this Agreement, the Sale Agreement, or one or both of Promissory Notes, or otherwise in connection with the transaction described herein and relationship between the parties in connection with such transaction, then the prevailing party shall be entitled to recover its reasonable attorneys’ fees and other costs and expenses incurred in connection therewith.

14. Termination of the Security Interest. The Security Interest shall terminate only if and when the Debt has been paid in full. Upon termination of the Security Interest, Calavo shall, at the request of San Rafael, file with the California Secretary of State and any other applicable regulatory authority a statement indicating that the Security Interest has terminated.

15. San Rafael’s Representations and Warranties. San Rafael hereby represents and warrants to Calavo that it is purchasing the Sold Interest for its own account for investment purposes and not for the purpose of the public distribution of the Sold Interest. No other individual, corporation, partnership or other person, entity or group will have any direct or indirect security interest or other beneficial ownership interest in the Entire Interest during the period that they remain subject to the Security Interest.

16. Reporting Requirements. San Rafael agrees to deliver to Calavo complete and accurate quarterly financial reports (balance sheet, cash flow statement, income statement), maintained on a consistent basis and fairly showing the operations of the Company, within thirty (30) days following the end of each quarter.

- 5 -

17. Miscellaneous Provisions.

(a) Further Assurances. San Rafael shall from time to time at the request of Calavo, and without further consideration, execute and deliver to Calavo such further instruments of assignment, transfer, conveyance and confirmation and take such other action as Calavo may reasonably request in order to more effectively fulfill the purposes of this Agreement.

(b) Severability. The provisions of this Agreement will be deemed severable and the invalidity or unenforceability of any provision will not affect the validity or enforceability of the other provisions hereof. If any provision hereof is determined by a court of competent jurisdiction or an arbitrator to be invalid or unenforceable, such provision shall be limited to the extent necessary to make it valid and enforceable, or if necessary, severed from this Agreement, and the remainder of the Agreement shall be in full force and effect.

(c) Complete Agreement. This Agreement, the Promissory Notes, and the Sale Agreement constitute the complete and exclusive agreement between San Rafael and Calavo with respect to the subject matter herein and thereof and replace and supersede any and all prior understandings, agreements, negotiations and discussions, both written and oral, between the parties hereto with respect to the subject matter hereof and thereof.

(d) Successors and Assigns. Subject to the provisions of this Agreement and the Sale Agreement relating to the transferability of any portion or all of the Entire Interest, this Agreement shall be binding upon and inure to the benefit of San Rafael and Calavo and their respective successors and assigns. Whenever appropriate in this Agreement, references to San Rafael or Calavo shall be deemed to refer to such company’s successors or assigns.

(e) Notices. Any notice required or permitted to be given to Calavo or San Rafael must be in writing and personally delivered or sent by registered or certified United States mail (postage prepaid and return receipt requested), by overnight delivery service or by facsimile transmission, addressed to the address shown below or to such other address as such party may designate in the foregoing manner to the other party. Any such notice that is sent by San Rafael or Calavo in the foregoing manner shall be deemed to have been delivered upon actual personal delivery or actual receipt by facsimile transmission (with telephonic confirmation of receipt) or delivery by the United States mail or an overnight delivery service.

- 6 -

(f) Amendment and Termination. This Agreement may be amended or terminated only upon a writing executed by both Calavo and San Rafael.

(g) Counterparts. This Agreement may be executed in two counterparts, each of which shall be deemed an original, but both of which shall constitute one and the same instrument.

(h) Attorneys’ Fees. If either party brings a claim or lawsuit against the other party to this Agreement to interpret or enforce any of the terms of this Agreement, or to interpret or enforce the Equity Promissory Note, the Goodwill Promissory Note, or the Sale Agreement, the prevailing party shall, in addition to all other damages, be entitled to reasonable attorneys’ fees and costs, costs of witnesses, and costs of investigation from the non-prevailing party.

(i) Governing Law. This Agreement shall be construed, interpreted and enforced in accordance with, and shall be governed by, the laws of the State of California without reference to, and regardless of, any applicable choice or conflicts of laws principles.

(j) Dispute Resolution. All disputes concerning this Agreement shall be settled by arbitration, before one arbitrator, in accordance with the commercial arbitration rules of the American Arbitration Association then in effect. The arbitrator shall be selected in accordance with such commercial arbitration rules. A party is entitled to initiate an arbitration proceeding if a dispute cannot be resolved amicably within ten days after the other party has been notified of the existence of the dispute. The arbitrator is authorized to grant injunctive relief and/or specific performance in addition to monetary relief. The arbitrator hereby is instructed to interpret and enforce this Agreement in strict accordance with its terms and in accordance with California law. All arbitration proceedings shall be held in Los Angeles, California.

Notwithstanding the foregoing, each party is entitled to bring an action for temporary or preliminary injunctive relief at any time in any court of competent jurisdiction in order to prevent irreparable injury that might result from a breach of this Agreement. Furthermore, upon the occurrence of an Event of Default, Calavo is entitled to exercise all of the rights and remedies described in this Agreement and, at any time, to bring an action in a court of competent jurisdiction (or, at its election, to initiate an arbitration proceeding) for purposes of enforcing the Security Interest.

The award of the arbitrator in any arbitration proceeding shall be final and may be enforced in any court of competent jurisdiction, and an action to compel arbitration may be brought in any court of competent jurisdiction. The unsuccessful party to any arbitration

- 7 -

proceeding or to any court action that is permitted by this Agreement shall pay to the successful party all costs and expenses, including, without limitation, reasonable attorneys’ fees and the fees of the arbitrator, incurred therein by the successful party. EACH PARTY AGREES THAT, TO THE EXTENT PERMISSIBLE BY LAW, ALL RIGHTS TO A TRIAL BY A JURY OF ANY CLAIM CONCERNING THIS AGREEMENT ARE ABSOLUTELY AND FOREVER WAIVED.

IN WITNESS WHEREOF, the parties hereto have executed and delivered this Agreement as of the day and year first above written.

- 8 -