Exhibit 10.4

INVESTMENT ALLOCATION AGREEMENT

This INVESTMENT ALLOCATION AGREEMENT, dated as of September 29, 2009, is made and entered into by and among COLONY CAPITAL, LLC, a Delaware limited liability company (“Colony Capital”), COLONY FINANCIAL MANAGER, LLC, a Delaware limited liability company (the “Manager”), and COLONY FINANCIAL, INC., a Maryland corporation (the “Company”).

WHEREAS, the Company is a newly organized corporation that is engaging in various related transactions pursuant to which the Company will effect an initial public offering of its common stock (the “Initial Public Offering”);

WHEREAS, Colony Capital is the sole managing member of the Manager;

WHEREAS, concurrently with the execution of this Agreement, the Company will execute a Management Agreement (the “Management Agreement”) with the Manager and Colony Financial TRS, LLC, a Delaware limited liability company (the “TRS”), pursuant to which the Manager will provide for the day-to-day management of the operations of the Company and its subsidiaries, including the TRS, and will be responsible for the selection, purchase and sale of the Company’s portfolio investments, the Company’s financing activities, and providing the Company with investment advisory services;

WHEREAS, also concurrently with the execution of this Agreement, the Manager will execute an Investment Advisory Agreement with Colony Capital, pursuant to which Colony Capital will perform, among other things, portfolio management, asset valuation, risk management and asset management services as well as certain administration services that the Manager may deem necessary for the performance of its duties under the Management Agreement; and

WHEREAS, in order to address certain potential conflicts arising from the Company’s relationship with Colony Capital and the Manager, the parties hereto desire to establish certain policies relating to the allocation of investment opportunities by Colony Capital, the Manager or any of their respective affiliates in assets that are substantially similar to the types of assets described as the Company’s target assets in the Company’s Registration Statement on Form S-11 (No. 333-160323) relating to the Initial Public Offering (collectively, the “Target Assets”).

NOW, THEREFORE, for the mutual promises made herein and in the other agreements executed by the parties concurrently herewith or contemplated hereby, and other good and valuable consideration, the receipt and adequacy of which are hereby acknowledged, the parties agree as follows:

ARTICLE I

Investment Allocation Policy

1.01. Initial Co-Investment Allocation Policy. (a) For as long as the Management Agreement is in effect, neither Colony Capital, the Manager nor any of their respective affiliates other than the Company or any of the Company’s subsidiaries may sponsor or manage (i) any additional publicly traded investment vehicle that will primarily acquire or originate assets secured by U.S. collateral that are substantially similar to the Target Assets or (ii) any publicly traded investment vehicle that will primarily acquire or originate assets secured by non-U.S. collateral that are substantially similar to the Target Assets or any private investment vehicle that will primarily acquire or originate assets that are substantially similar to the Target Assets (collectively, the “Investment Vehicles” and, together with Colony Distressed Credit Fund, L.P., Colony Investors VIII, L.P., Colony Parallel Investors VIII, L.P.,

Colony Parallel NA-RE Investors VIII, L.P. and Colyzeo II, L.P., the “Co-Investment Vehicles”) unless the Company is entitled (but not obligated) to contribute, subject to the Company’s investment guidelines, availability of capital and maintaining its qualification as a real estate investment trust (“REIT”) for U.S. federal income tax purposes and its exemption from registration under the Investment Company Act of 1940, as amended (the “1940 Act”), at least one-third of the capital to be funded by such Co-Investment Vehicles in assets that are substantially similar to the Target Assets; provided, however, that from and after the termination of the commitment period of Colony Distressed Credit Fund, L.P., the Company shall be entitled (but not obligated) to contribute, subject to the Company’s investment guidelines, availability of capital and maintaining its qualification as a REIT for U.S. federal income tax purposes and its exemption from registration under the 1940 Act, at least one-half of the portion of the capital to be funded by the Co-Investment Vehicles in assets secured by U.S. collateral that are substantially similar to the Target Assets; provided, further, that to the extent that the Company does not have sufficient capital to contribute at least one-third (or one-half, as applicable) of the capital required for any such proposed investment by the Co-Investment Vehicles, Colony Capital, the Manager or their respective affiliates, as applicable, shall provide for a fair and equitable allocation of investment opportunities among the Company and the Co-Investment Vehicles, taking into account the suitability of each investment opportunity for the particular Co-Investment Vehicle and the Company and each such Co-Investment Vehicle’s and the Company’s availability of capital for investment (the “Initial Co-Investment Allocation Policy”).

1.02. Revised Co-Investment Allocation Policy. If at any time in the judgment of Colony Capital and the Manager, the Initial Co-Investment Allocation Policy will not result in fair and equitable allocations among the Company and any Co-Investment Vehicle, Colony Capital and the Manager may propose a new policy or a deviation from the Investment Allocation Policy set forth in Section 1.01 with respect to a particular investment providing for the fair and equitable allocation of investment opportunities or such particular investment between the Company and the Co-Investment Vehicles (the “Revised Co-Investment Allocation Policy”); provided, however, that such Revised Co-Investment Allocation Policy shall not be effective unless and until it has been approved by a majority of the Company’s independent directors.

1.03. Investment Allocation Policy Regarding Other Investment Vehicles. For as long as the Management Agreement is in effect, with respect to public or private investment vehicles sponsored or managed by Colony Capital or its affiliates that do not primarily acquire or originate assets that are substantially similar to the Company’s Target Assets (collectively, the “Other Investment Vehicles”), Colony Capital, the Manager or their respective affiliates, as applicable, shall provide for a fair and equitable allocation of investment opportunities in assets that are substantially similar to the Company’s Target Assets among the Company and the Other Investment Vehicles, in each case taking into account the suitability of each investment opportunity for the particular Other Investment Vehicle and the Company, each such Other Investment Vehicle’s and the Company’s availability of capital for investment and the sourcing of such investment.

ARTICLE II

Term

2.01. Term. This Agreement shall be effective as of the date first written above and shall terminate on the first to occur of any of the following events:

(a) Immediately upon the execution by all parties hereto of a written agreement to terminate this Agreement (or upon the effective date of such termination as specified in such written agreement); provided, however, that such termination shall not be effective unless and until it has been consented to by a majority of the Company’s independent directors; or

2

(b) At such time as either (i) the Management Agreement is terminated by any party thereto, for any reason, or (ii) the Manager ceases to be the manager of the Company and its subsidiaries for any reason.

2.02. Rights of Termination. If this Agreement is terminated, such termination shall be without any further liability or obligation of either party to the other, except as provided in Section 3.07.

ARTICLE III

Miscellaneous Provisions

3.01. Assignment. This Agreement may not be assigned by any party hereto without the prior written consent of the other parties.

3.02. Notice

(a) All notices, demands or requests provided for or permitted to be given pursuant to this Agreement must be in writing, to the following addresses:

If to Colony Capital, to:

Colony Capital, LLC

2450 Broadway, 6th Floor

Santa Monica, California 90404

Attention: Joy Mallory

Fax No.: 310-407-7416

If to the Manager:

Colony Financial Manager, LLC.

2450 Broadway, 6th Floor

Santa Monica, California 90404

Attention: Joy Mallory

Fax No.: 310-407-7416

If to the Company:

Colony Financial, Inc.

2450 Broadway, 6th Floor

Santa Monica, California 90404

Attention: Chief Financial Officer

Fax No.: 310-407-7430

(b) All notices, demands and requests to be sent to a party hereto pursuant to this Agreement shall be deemed to have been properly given or served if: (i) personally delivered, (ii) deposited for next day delivery by Federal Express, or other similar overnight courier services, addressed to such party, (iii) deposited in the United States mail, addressed to such party, prepaid and registered or certified with return receipt requested or (iv) transmitted via facsimile or other similar device to the attention of such party.

3

(c) All notices, demands and requests so given shall be deemed received: (i) when personally delivered, (ii) twenty-four (24) hours after being deposited for next day delivery with an overnight courier, (iii) forty-eight (48) hours after being deposited in the United States mail, or (iv) three (3) hours after being transmitted via facsimile or otherwise transmitted and receipt has been confirmed.

3.03. Binding Nature of Agreement; Successors and Assigns. This Agreement shall be binding upon and inure to the benefit of the parties hereto and their respective heirs, personal representatives, successors and permitted assigns as provided in this Agreement.

3.04. Entire Agreement. This Agreement contains the entire agreement and understanding among the parties hereto with respect to the subject matter of this Agreement, and supersedes all prior and contemporaneous agreements, understandings, inducements and conditions, express or implied, oral or written, of any nature whatsoever with respect to the subject matter of this Agreement. The express terms of this Agreement control and supersede any course of performance and/or usage of the trade inconsistent with any of the terms of this Agreement.

3.05. Amendments. This Agreement may be amended or modified only by an agreement in writing signed by all parties hereto; provided that any such amendment shall not be effective unless and until it has been approved by a majority of the Company’s independent directors.

3.06. No Implied Waivers; Remedies. No failure or delay on the part of any party in exercising any right, privilege, power, or remedy under this Agreement, and no course of dealing shall operate as a waiver of any such right, privilege, power or remedy; nor shall any single or partial exercise of any right, privilege, power or remedy under this Agreement preclude any other or further exercise of any such right, privilege, power or remedy or the exercise of any other right, privilege, power or remedy. No waiver shall be asserted against any party unless signed in writing by such party. The rights, privileges, powers and remedies available to the parties are cumulative and not exclusive of any other rights, privileges, powers or remedies provided by statute, at law, in equity or otherwise. Except as provided in this Agreement, no notice to or demand on any party in any case shall entitle such party to any other or further notice or demand in any similar or other circumstances or constitute a waiver of the right of the party giving such notice or making such demand to take any other or further action in any circumstances without notice or demand.

3.07. Governing Law. THIS AGREEMENT SHALL BE CONSTRUED IN ACCORDANCE WITH AND GOVERNED BY THE LAWS OF THE STATE OF CALIFORNIA. EACH OF THE PARTIES HEREBY IRREVOCABLY AGREES THAT THE COURTS OF THE STATE OF CALIFORNIA SHALL HAVE EXCLUSIVE JURISDICTION IN CONNECTION WITH ANY ACTIONS OR PROCEEDINGS ARISING BETWEEN THE PARTIES UNDER THIS AGREEMENT. EACH OF THE PARTIES HEREBY IRREVOCABLY CONSENTS AND SUBMITS TO THE JURISDICTION OF SAID COURTS FOR ANY SUCH ACTION OR PROCEEDING. EACH OF THE PARTIES HEREBY WAIVES THE DEFENSE OF AN INCONVENIENT FORUM TO THE MAINTENANCE OF ANY SUCH ACTION OR PROCEEDING IN SAID COURTS.

3.08. Headings. The headings contained in this Agreement are for convenience only and shall not affect the construction or interpretation of any provisions of this Agreement.

3.09 Severability. If any provision of the Agreement shall be held to be invalid, the remainder of the Agreement shall not be affected thereby.

4

3.10. Counterparts. This Agreement may be executed in any number of counterparts, each of which shall be deemed to be an original as against any party whose signature appears thereon, and all of which shall together constitute one and the same instrument. This Agreement shall become binding when one or more counterparts of this Agreement, individually or taken together, shall bear the signatures of all of the parties reflected hereon as the signatories.

5

IN WITNESS WHEREOF, the parties have caused this Agreement to be executed by their representatives on the date first written above.

6