UNITED STATES

SECURITIES AND EXCHANGE COMMISSION

Washington, D.C. 20549

FORM 10-Q

Quarterly Report Pursuant to Section 13 or 15(d) of

the Securities Exchange Act of 1934

CASUAL MALE RETAIL GROUP, INC.

(Exact name of registrant as specified in its charter)

(781) 828-9300

(Registrant’s telephone number, including area code)

Indicate by check mark whether the registrant: (1) has filed all reports required to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during the preceding 12 months (or for such shorter period that the registrant was required to file such reports), and (2) has been subject to such filing requirements for the past 90 days.    Yes  x    No  ¨

Indicate by check mark whether the registrant has submitted electronically and posted on its corporate Web site, if any, every Interactive Data File required to be submitted and posted pursuant to Rule 405 of Regulation S-T (§232.405 of this chapter) during the preceding 12 months (or for such shorter period that the registrant was required to submit and post such files).    Yes  x    No  ¨

Indicate by check mark whether the registrant is a large accelerated filer, an accelerated filer, a non-accelerated filer or a smaller reporting company. See the definitions of “large accelerated filer,” “accelerated filer” and “smaller reporting company” in Rule 12b-2 of the Exchange Act.

Indicate by check mark whether the registrant is a shell company (as defined in Rule 12b-2 of the Exchange Act).    Yes  ¨    No  x

The number of shares of common stock outstanding as of July 30, 2011 was 48,460,493.

PART I. FINANCIAL INFORMATION

CASUAL MALE RETAIL GROUP, INC.

CONSOLIDATED BALANCE SHEETS

(In thousands, except share data)

The accompanying notes are an integral part of the consolidated financial statements.

2

CASUAL MALE RETAIL GROUP, INC.

CONSOLIDATED STATEMENTS OF OPERATIONS

(In thousands, except per share data)

(Unaudited)

The accompanying notes are an integral part of the consolidated financial statements.

3

CASUAL MALE RETAIL GROUP, INC.

CONSOLIDATED STATEMENTS OF CASH FLOWS

(In thousands)

(Unaudited)

The accompanying notes are an integral part of the consolidated financial statements.

4

CASUAL MALE RETAIL GROUP, INC.

CONSOLIDATED STATEMENTS OF CHANGES IN STOCKHOLDERS’ EQUITY

For the six months ended July 30, 2011

(In thousands)

(Unaudited)

The accompanying notes are an integral part of the consolidated financial statements.

5

CASUAL MALE RETAIL GROUP, INC.

Notes to Consolidated Financial Statements

In the opinion of management of Casual Male Retail Group, Inc., a Delaware corporation (the “Company”), the accompanying unaudited consolidated financial statements contain all adjustments necessary for a fair presentation of the interim financial statements. These financial statements do not include all disclosures associated with annual financial statements and, accordingly, should be read in conjunction with the notes to the Company’s audited consolidated financial statements for the fiscal year ended January 29, 2011 included in the Company’s Annual Report on Form 10-K, which was filed with the Securities and Exchange Commission on March 18, 2011.

The information set forth in these statements may be subject to normal year-end adjustments. The information reflects all adjustments that, in the opinion of management, are necessary to present fairly the Company’s results of operations, financial position and cash flows for the periods indicated. The preparation of financial statements in conformity with accounting principles generally accepted in the United States requires management to make estimates and assumptions that affect the reported amounts of assets and liabilities and disclosure of contingent assets and liabilities at the date of the financial statements and the reported amounts of revenues and expenses during the reporting period. The Company’s business historically has been seasonal in nature, and the results of the interim periods presented are not necessarily indicative of the results to be expected for the full year.

The Company’s fiscal year is a 52- or 53- week period ending on the Saturday closest to January 31. Fiscal 2011 is a 52-week period ending on January 28, 2012. Fiscal 2010 was a 52-week period ending on January 29, 2011.

Segment Information

The Company reports its operations as one reportable segment, Big & Tall Men’s Apparel, which consists of its three operating segments – B&T Factory Direct, Casual Male XL and Rochester Clothing. The Company considers its operating segments to be similar in terms of economic characteristic, production processes and operations, and have therefore aggregated them into a single reporting segment. The Company’s DXL™ store format carries merchandise from all three of the Company’s operating segments. The operating results and assets of the Company’s other direct businesses, Living XL, Shoes XL and the Company’s International Web Stores, are immaterial.

Other Intangibles

The Company’s trademarks are considered indefinite-lived intangible assets and must be tested annually for potential impairment. At January 29, 2011, both the Casual Male and Rochester trademarks were tested for potential impairment. Utilizing an income approach with appropriate royalty rates applied, the Company concluded that the Casual Male trademark, with a carrying value of $29.2 million, and the Rochester trademark, with a carrying value of $1.5 million, were not impaired. During the first six months of fiscal 2011, no events or circumstances exist which could potentially cause a reduction in the fair value of the Company’s reporting units, thus requiring interim testing of the Company’s trademarks. Other intangibles, which include customer lists and favorable lease commitments, were $1.3 million at July 30, 2011.

Stock-based Compensation

All share-based payments, including grants of employee stock options and restricted stock, are recognized as an expense in the statement of operations based on their fair values and vesting periods. The fair value of stock options is determined using the Black-Scholes valuation model and requires the input of subjective assumptions. These assumptions include estimating the length of time employees will retain their vested stock options before exercising them (the “expected term”), the estimated volatility of the Company’s common stock price over the expected term and the number of options that will ultimately not complete their vesting requirements (“forfeitures”). The Company reviews its valuation assumptions at each grant date and, as a result, is likely to change its valuation assumptions used to value employee stock-based awards granted in future periods. The values derived from using the Black-Scholes model are recognized as an expense over the vesting period, net of estimated forfeitures. The estimation of stock-based awards that will ultimately vest requires significant judgment. Actual results, and future changes in estimates, may differ from the Company’s current estimates.

6

For the first six months of fiscal 2011 and fiscal 2010, the Company recognized total stock-based compensation expense of $0.8 million and $0.9 million, respectively. Approximately $0.5 million of the $0.8 million of stock-based compensation expense for the first six months of fiscal 2011 and $0.2 million of the $0.9 million for fiscal 2010 related to the expense associated with the grant of equity awards pursuant to the Company’s Long-Term Incentive Plan (“LTIP”).

The total compensation cost related to non-vested awards not yet recognized as of July 30, 2011 is approximately $1.8 million which will be expensed over a weighted average remaining life of 30 months.

Valuation Assumptions for Stock Options and Restricted Stock

During the first six months of fiscal 2011, the Company granted 569,661 shares of restricted stock of which 538,661 shares were granted to members of management as a result of the Company achieving certain performance targets pursuant to its LTIP for fiscal 2010. In addition, the Company granted stock options to purchase 79,902 shares of common stock, of which 72,576 stock options were granted as part of the LTIP for fiscal 2010.

During the first six months of fiscal 2010, the Company granted 308,550 shares of restricted stock which included 283,550 shares to members of management as a result of the Company achieving certain performance targets pursuant to its LTIP for fiscal 2009. During the first six months of fiscal 2010, stock options to purchase 368,172 shares of common stock were granted. Of this amount, stock options to purchase 20,606 shares of common stock were issued pursuant to the LTIP for fiscal 2009.

Each restricted share of common stock was assigned a fair value equal to the closing price of the Company’s common stock on the date of grant. The fair value of each option granted is estimated on the date of grant using the Black-Scholes option-pricing model. The weighted average grant date fair value of stock options granted during the first six months of fiscal 2011 was $1.53 per share.

The following assumptions were used for grants for the first six months of fiscal 2011 and fiscal 2010:

Expected volatilities are based on historical volatilities of the Company’s common stock; the expected life represents the weighted average period of time that options granted are expected to be outstanding giving consideration to vesting schedules and historical exercise patterns; and the risk-free rate is based on the U.S. Treasury yield curve in effect at the time of grant for periods corresponding with the expected life of the option.

During the first six months of fiscal 2011, options for 169,141 shares of common stock were exercised with an intrinsic value of approximately $64,000.

Recently Issued Accounting Pronouncements

The Company has reviewed accounting pronouncements and interpretations thereof that have effective dates during the periods reported and in future periods. The Company believes that the following impending standards may have an impact on its future filings. The applicability of any standard will be evaluated by the Company and is still subject to review by the Company.

In May 2011, the Financial Accounting Standards Board (“FASB”) issued Accounting Standards Update (“ASU”) No. 2011-04—Fair Value Measurement (Topic 820)— Amendments to Achieve Common Fair Value Measurement and Disclosure Requirements in U.S. GAAP and IFRSs. The amendments in this update change the wording used to describe

7

the requirements in U.S. GAAP for measuring fair value and for disclosing information about fair value measurements to ensure consistency between U.S. GAAP and IFRS. This update is effective for interim and annual periods beginning after December 15, 2011. The Company does not expect the adoption of ASU No. 2011-04 to have a material impact on the Company’s financial statements.

In June 2011, the FASB issued ASU No. 2011-05, Comprehensive Income (Topic 220)—Presentation of Comprehensive Income, to make the presentation of items within other comprehensive income (“OCI”) more prominent. The new standard will require companies to present items of net income, items of OCI and total comprehensive income in one continuous statement or two separate consecutive statements, and companies will no longer be allowed to present items of OCI in the statement of stockholders’ equity. This new update is effective for interim and annual periods beginning after December 15, 2011. The adoption of this new standard may change the order in which certain financial statements are presented and will provide additional detail in those financial statements when applicable, but will not have any other impact on the Company’s financial statements.

Credit Agreement with Bank of America Retail Group, Inc.

The Company has a credit facility with Bank of America, N.A., most recently amended on November 10, 2010 (the “Credit Facility”).

The Credit Facility provides for a maximum committed borrowing of $75 million, which, pursuant to an accordion feature, may be increased to $125 million upon the request of the Company and the agreement of the lender(s) participating in the increase. The Credit Facility includes a sublimit of $20 million for commercial and standby letters of credit and a sublimit of up to $15 million for Swingline Loans. The maturity date of the Credit Facility is November 10, 2014.

Borrowings made pursuant to the Credit Facility will bear interest at a rate equal to the base rate (determined as the highest of (a) Bank of America N.A.’s prime rate, (b) the Federal Funds rate plus 0.50% and (c) the one month LIBOR rate) plus a varying percentage, based on the Company’s borrowing base, of 1.00-1.25% for prime-based borrowings and 2.00-2.25% for LIBOR-based borrowings. The Company is also subject to an unused line fee. At July 30, 2011, the Company’s prime-based interest rate was 4.25%.

The Company’s obligations under the Credit Facility are secured by a lien on all of its assets. The Company is not subject to any financial covenants pursuant to the Credit Facility.

At July 30, 2011, the Company had no borrowings outstanding under the Credit Facility. Outstanding standby letters of credit were $2.3 million and documentary letters of credit were $9.2 million. Unused excess availability at July 30, 2011 was $63.6 million. Average borrowings outstanding under this facility during the first six months of fiscal 2011 were less than $25,000, resulting in an average unused excess availability of approximately $65.4 million. The Company’s ability to borrow under the Credit Facility is determined using an availability formula based on eligible assets, with increased advance rates based on seasonality.

The fair value of the amount outstanding under the Credit Facility at July 30, 2011 approximated the carrying value.

Long-Term Debt with Banc of America Leasing & Capital, LLC

Pursuant to two Equipment Security Notes with Banc of America Leasing & Capital, LLC for equipment financing, the Company had $5.1 million outstanding at July 31, 2010. During fiscal 2010, the secured notes were fully repaid. Both notes accrued interest at a per annum rate of 1.75% plus the rate of interest equal to the 30-day published LIBOR rate.

The Company has no long-term debt outstanding at July 30, 2011.

8

Earnings per Share

Basic earnings per share are computed by dividing net income by the weighted average number of shares of common stock outstanding during the respective period. Diluted earnings per share is determined by giving effect to unvested shares of restricted stock and the exercise of stock options using the treasury stock method.

The following table provides a reconciliation of the number of shares outstanding for basic and diluted earnings per share:

The above options, which were outstanding at July 30, 2011, expire from July 31, 2011 to May 31, 2021.

At July 30, 2011, the Company had total deferred tax assets of approximately $46.7 million, with a corresponding valuation allowance of $46.7 million and a total deferred tax liability of approximately $1.9 million. The deferred tax assets include approximately $17.0 million of net operating loss carryforwards that expire through 2029 and approximately $8.4 million of deferred gain on sale-leaseback and, to a lesser extent, other book/tax timing differences.

The ability to reduce the valuation allowance of $46.7 million is dependent upon the Company’s ability to achieve sustained taxable income to realize its deferred tax assets. Because fiscal 2011 earnings will be a significant factor in determining the amount of the valuation allowance that can be reversed, the deferred tax assets remain fully reserved at July 30, 2011. However, based on the Company’s results of operations for the past two fiscal years as well as its forecasted earnings for fiscal 2011 and beyond, the Company anticipates reversing a substantial portion of the valuation allowance in the fourth quarter of fiscal 2011.

The Company’s effective tax rate for the first six months of fiscal 2011 has been reduced from the statutory rate due to the utilization of the Company’s fully reserved net operating loss carryforwards. The Company expects its effective rate to return to a rate of approximately 40% for periods subsequent to reversal of the valuation allowance.

A tax position is recognized as a benefit only if it is “more likely than not” that the tax position would be sustained in a tax examination, with a tax examination being presumed to occur. The amount recognized is the largest amount of tax benefit that is greater than 50% likely of being realized on examination. For tax positions not meeting the “more likely than not” test, no tax benefit is recorded. The charge for taxation is based on the results for the year as adjusted for items that are non-assessable or disallowed. The charge is calculated using tax rates that have been enacted or substantively enacted by the balance sheet date. Pursuant to Topic 740, Income Taxes, the Company will recognize the benefit from a tax position only if it is more likely than not that the position would be sustained upon audit based solely on the technical merits of the tax position. At July 30, 2011, the Company had no material unrecognized tax benefits.

9

The Company is subject to U.S. federal income tax as well as income tax of multiple state and foreign jurisdictions. The Company has concluded all U.S. federal income tax matters for years through fiscal 1998, with remaining fiscal years subject to income tax examination by federal tax authorities.

The Company’s policy is to recognize accrued interest and penalties related to unrecognized tax benefits in its income tax provision. The Company has not accrued or paid interest or penalties which were material to its results of operations for the first six months of fiscal 2011.

10

FORWARD-LOOKING STATEMENTS

Certain statements contained in this Quarterly Report on Form 10-Q constitute “forward-looking statements” within the meaning of the United States Private Securities Litigation Reform Act of 1995. In some cases, forward-looking statements can be identified by the use of forward-looking terminology such as “may,” “will,” “estimate,” “intend,” “plan,” “continue,” “believe,” “expect” or “anticipate” or the negatives thereof, variations thereon or similar terminology. The forward-looking statements contained in this Quarterly Report are generally located in the material set forth under the heading “Management’s Discussion and Analysis of Financial Condition and Results of Operations,” but may be found in other locations as well. These forward-looking statements generally relate to plans and objectives for future operations and are based upon management’s reasonable estimates of future results or trends. The forward-looking statements in this Quarterly Report should not be regarded as a representation by us or any other person that our objectives or plans will be achieved. Numerous factors could cause our actual results to differ materially from such forward-looking statements. We encourage readers to refer to Part I, Item 1A of our Annual Report on Form 10-K for the year ended January 29, 2011, filed with the Securities and Exchange Commission on March 18, 2011, and Part II, Item 1A of our Quarterly Reports, including this Quarterly Report, which identify certain risks and uncertainties that may have an impact on our future earnings and the direction of our Company.

All subsequent written and oral forward-looking statements attributable to us or to persons acting on our behalf are expressly qualified in their entirety by the foregoing. These forward-looking statements speak only as of the date of the document in which they are made. We disclaim any obligation or undertaking to provide any updates or revisions to any forward-looking statement to reflect any change in our expectations or any change in events, conditions or circumstances in which the forward-looking statement is based.

BUSINESS SUMMARY

Casual Male Retail Group, Inc. together with our subsidiaries is the largest specialty retailer of big & tall men’s apparel with retail operations in the United States and London, England and direct businesses throughout the United States, Canada and Europe. We operate under the trade names of Casual Male XL, Casual Male XL Outlets, DestinationXL™, Rochester Clothing, B&T Factory Direct, Shoes XL and Living XL. At July 30, 2011, we operated 377 Casual Male XL retail stores, 60 Casual Male XL outlet stores, 6 DestinationXL (“DXL™”) stores and 15 Rochester Clothing stores. In August 2011, the Company opened two additional DXL stores in Murray, Utah and Fairless Hills, Pennsylvania. Our direct business includes several catalogs and e-commerce sites which support our brands and product extensions.

Unless the context indicates otherwise, all references to “we,” “our,” “ours,” “us” and “the Company” refer to Casual Male Retail Group, Inc. and our consolidated subsidiaries. We refer to our fiscal years which end on January 28, 2012 and January 29, 2011 as “fiscal 2011” and “fiscal 2010,” respectively.

When discussing sales growth, we refer to the term “comparable sales.” Comparable sales for all periods include our retail stores that have been open for at least one full fiscal year together with our e-commerce and catalog sales. Stores that have been remodeled, expanded or re-located during the period are also included in our determination of comparable sales. Our Destination XL stores are considered relocations and comparable to all the closed stores in each respective market area. We include our direct businesses as part of our calculation of comparable sales since we are a multi-channel retailer, offering our customers convenient alternatives for their shopping. The method of calculating comparable sales varies across the retail industry and, as a result, our calculation of comparable sales is not necessarily comparable to similarly titled measures reported by other companies.

RESULTS OF OPERATIONS

Financial Summary

For the second quarter of fiscal 2011, our earnings were $0.14 per diluted share as compared to $0.12 per diluted share for the second quarter of fiscal 2010. For the first six months of fiscal 2011, our earnings were $0.22 per diluted share as compared to $0.21 per diluted share for the first six months of fiscal 2010.

11

Our comparable sales for the second quarter of fiscal 2011 were up 4.9% over the prior year’s second quarter with our gross margin rate increasing 190 basis points over the second quarter of fiscal 2010. Even though we are still experiencing negative store traffic (down 1% for the second quarter of fiscal 2011 as compared to the second quarter of fiscal 2010), comparable sales have increased as a result of not only modest product price increases, which we started to implement during the first quarter of fiscal 2011, but also from continued improvements in sales productivity driven by the Company’s customer service program. Through maintained initial markups and controlled markdown activity we have been able to continue to improve our merchandise margin, which has helped to offset the expected increases in selling, general and administrative (“SG&A”) expenses.

At July 30, 2011, we have $18.3 million in cash and cash equivalents, no outstanding debt and full availability under our credit facility of $63.6 million. We continue to generate positive free cash flow (see “Presentation of Non-GAAP Measure” below) and manage our inventory, which has been a key component to optimizing our merchandise margins, enabling us to avoid excessive promotional and clearance activity.

During the second quarter of fiscal 2011, we opened two DXL™ stores in Atlanta, Georgia and Phoenix, Arizona. Subsequent to the end of the quarter in August 2011 we opened two additional DXL stores in Murray, Utah and Fairless Hills, Pennsylvania. In aggregate, the DXL stores continue to show improved sales within their respective markets of approximately 20%, on a comparable basis versus the second quarter of fiscal 2010, with all stores showing some improvement. We are looking forward to the launch of our new DXL website, which was originally scheduled to launch during the second quarter but will be launching early in the third quarter of fiscal 2011. Our DXL website, with a multi-channel solution similar to the DXL store experience, will have state-of-the-art functionality and will enable our customers to shop across all brands and product extensions with ease. During the second half of fiscal 2011, we expect to open an additional 11 to 12 DXL stores for a total of 19 to 20 DXL stores operating by the end of fiscal 2011.

Fiscal 2011 Outlook

We expect earnings for fiscal 2011 to be between $0.40 to $0.45 per diluted share. This is based on an increase in sales of approximately 3.0%-4.0%, resulting in total sales of $405.0-$410.0 million. Our gross margin rate is expected to improve by 105 to 150 basis points from fiscal 2010’s gross margin rate (an increase from the previously reported guidance of 75 to 125 basis points). We have reinstated our employer match for our 401K Plans and have given modest salary increases to some of our employees, resulting in an expected increase in SG&A of approximately 3.0% to $156.0 million. The modest salary increases are the first increases since the economic difficulties began in fiscal 2008.

From a liquidity perspective, we have slightly revised our expected cash flow from operating activities to a range of $37.5 to $40.0 million (an increase from the $34.0 million previously reported), resulting in free cash flow (as defined below under “Presentation of Non-GAAP Measure”) of approximately $17.5 to $20.0 million (up from the $16.0 million previously reported). We expect our cash balances to increase to $22.0-$25.0 million by the end of fiscal 2011. Our capital expenditures for fiscal 2011 will be approximately $20.0 million (up from the $18.0 million previously reported). These expenditures will primarily be spent on investing in the infrastructure of our direct channel, in connection with the launch of our new DXL™ website in the third quarter of fiscal 2011, and capital incurred with our planned opening of 15 to 16 new DXL stores during fiscal 2011. As we open new DXL stores, we will be closing existing stores in the area. For fiscal 2011, we currently expect to close 15 to 20 existing stores, as leases expire. By the end of fiscal 2011, our total store count is expected to be approximately 450 stores.

For fiscal 2012, we plan to open 25-30 additional DXL stores, resulting in approximately 50 DXL stores operating at the end of fiscal 2012, with at least one store located in most of the major metropolitan cities across the United States. At the same time, we expect to close approximately 40-45 existing stores, as leases expire, in connection with opening these DXL stores, resulting in an estimated store count of approximately 435 stores at the end of fiscal 2012.

Presentation of Non-GAAP Measure

The presentation of non-GAAP free cash flow is not a measure determined by generally accepted accounting principles (“GAAP”) and should not be considered superior to or as a substitute for net income or cash flows from operating activities or any other measure of performance derived in accordance with GAAP. In addition, all companies do not

12

calculate non-GAAP financial measures in the same manner and, accordingly, “free cash flows” presented in this report may not be comparable to similar measures used by other companies. We calculate free cash flows as cash flow from operating activities, less capital expenditures and discretionary store asset acquisitions. We believe that inclusion of this non-GAAP measure helps investors gain a better understanding of our cash flow performance, especially when comparing such results to previous periods. The following table reconciles our non-GAAP free cash flow measure:

Sales

For the second quarter of fiscal 2011, total sales increased by 3.8% to $100.9 million when compared to total sales of $97.3 million for the second quarter of fiscal 2010. Comparable sales for the second quarter increased 4.9% when compared to the same period of the prior year. On a comparable basis, sales from our direct business increased by 4.1% and sales from our retail business increased 5.0%. As planned, during the first quarter of fiscal 2011, we started to increase prices within certain merchandise categories. For the second quarter of fiscal 2011, our average unit retail increased approximately 8% over the prior year’s second quarter. This increase is due partly to product price increases, and partly to the result of improved sales productivity by our selling associates.

For the first six months of fiscal 2011, total sales increased by 2.3% to $196.7 million as compared to $192.2 million for the first six months of the prior year. Comparable sales for the first six months increased 3.5% when compared to the first six months of fiscal 2010. The increase consisted of a 4.4% increase in our direct business and a 3.4% increase from our retail business.

Overall, we continue to experience improvement in our “dollars spent per transaction” and “average retail” metrics, which have contributed to mitigating the low traffic flow.

As expected, consumer spending has started to improve, albeit slowly. Each quarter, we experience gradual improvements in our store metrics as overall sales productivity continues to increase. Although we do not expect traffic levels to return to pre-recession levels this year, we are planning for a modest increase in our dollars per transaction as a result of price increases on some of our merchandise product, as well as continued improvement in sales productivity from better serving the needs of our customers. As such, we are expecting fiscal 2011 sales volumes to increase by approximately 3.0-4.0% as compared to fiscal 2010, with total sales of between $405.0-$410.0 million. We expect comparable sales growth to approximate between 4.0-4.5%.

Gross Profit Margin

For the second quarter of fiscal 2011, our gross margin rate, inclusive of occupancy costs, was 48.3% as compared to a gross margin rate of 46.4% for the second quarter of fiscal 2010. The increase of 190 basis points for the second quarter of fiscal 2011 was the result of an increase of 95 basis points in merchandise margins plus an increase of 95 basis points related to the leveraging of occupancy costs against higher sales. Our merchandise margin continues to benefit from our improved inventory management and managed markdowns. On a dollar basis, occupancy costs for the second quarter of fiscal 2011 decreased 2.6% when compared to the second quarter of fiscal 2010.

For the first six months of fiscal 2011, our gross margin rate, inclusive of occupancy costs, was 47.7% as compared to a gross margin rate of 46.1% for the first six months of fiscal 2010. The increase of 160 basis points for the first six months of fiscal 2011 was the result of an increase of 100 basis points in merchandise margins plus an increase of 60 basis points related to the leveraging of occupancy costs. On a dollar basis, occupancy costs for the first six months of fiscal 2011 decreased 1.5% when compared to the first six months of fiscal 2010.

13

For fiscal 2011, we are expecting that our occupancy costs, on a dollar-basis, will remain flat to fiscal 2010. As a result, we expect to leverage occupancy costs by approximately 30 to 50 basis points in fiscal 2011. In addition, we are planning on continued improvement of 75 to 100 basis points in merchandise margins (an increase from previously reported guidance of 45 to 75 basis points). Accordingly, for fiscal 2011, we are expecting gross margin will improve by approximately 105 to 150 basis points (an increase from previously reported guidance of 75 to 125 basis points).

Selling, General and Administrative Expenses

SG&A expenses for the second quarter of fiscal 2011 were 38.0% of sales as compared to 36.4% of sales for the second quarter of fiscal 2010. On a dollar basis, SG&A expenses increased $2.9 million, or 8.3%, for the second quarter of fiscal 2011 as compared to the second quarter of fiscal 2010. This increase is primarily due to payroll-related expenses, such as modest salary increases, bonus accruals, severance payments, reinstatement of the 401K employer match, as well as increased staffing in our global sourcing and merchandise areas.

For the first six months of fiscal 2011, SG&A expenses were 38.4% of sales as compared to 37.0% for the first six months of fiscal 2010. On a dollar basis, SG&A expenses increased $4.4 million, or 6.2%, for the first six months of fiscal 2011 as compared to the first six months of fiscal 2010. The first six months of fiscal 2011 include an increase in the bonus accrual of approximately $1.9 million, or $0.04 per diluted share, over the prior year. For fiscal 2010, the bonus was not accrued until the third quarter when achievement of the bonus became probable. This shift of expense contributed to our 6.2% increase for the first six months of fiscal 2011. This increase in SG&A will adjust itself during the second half of fiscal 2011, resulting in our SG&A expenses being in line with our annual forecast for the full year.

While SG&A expense management is a significant priority for us, we are expecting our SG&A expenses to increase by approximately 3% for fiscal 2011. This increase is primarily related to our incremental marketing costs associated with targeting our new DXL stores as well as the reinstatement of our 401K employer match and modest salary increases to some of our associates. Overall, we expect to limit our SG&A growth rates, except where necessary to support our growth activities or where there are unanticipated costs that are necessary to support our overall activities.

Interest Expense, Net

Net interest expense was $0.1 million for the second quarter of fiscal 2011 as compared to $0.2 million for the second quarter of fiscal 2010. For the first six months of fiscal 2011 net interest expense was $0.2 million as compared to $0.3 million for the first six months of fiscal 2010. The interest expense for the second quarter and first six months of fiscal 2011 primarily relates to the unused line fee on our credit facility as a result of having essentially no borrowings during the first six months of fiscal 2011.

Income Taxes

At July 30, 2011, our total deferred tax assets were approximately $46.7 million, with a corresponding valuation allowance of $46.7 million and a deferred tax liability of approximately $1.9 million. The deferred tax assets include approximately $17.0 million of net operating loss carryforwards that expire through 2029 and approximately $8.4 million of deferred gain on our sale-leaseback and, to a lesser extent, other book/tax timing differences.

The effect of the weakened economy on our retail business, especially in fiscal 2008, had a significant impact upon our revenue and profitability. Further, the conditions of the economy also negatively impacted our market value as a result of the deterioration of the capital markets and resulted in substantial impairments in fiscal 2008. Accordingly, due to our cumulative operating losses as well as our uncertainty regarding the economy and our ability to generate future taxable income to realize all of our deferred tax assets, in the fourth quarter of fiscal 2008, we established a valuation allowance against our deferred tax assets.

The ability to reduce this valuation allowance of $46.7 million is dependent upon our ability to achieve sustained taxable income to realize our deferred tax assets. Because our fiscal 2011 earnings will be a significant factor in determining the amount of the valuation allowance that can be reversed, our deferred tax assets remain fully reserved at July 30, 2011.

14

However, based on our results of operations for the past two fiscal years as well as our forecasted earnings for fiscal 2011 and beyond, we anticipate reversing a substantial portion of the valuation allowance in the fourth quarter of fiscal 2011.

Our effective tax rate for the second quarter and first six months of fiscal 2011 has been reduced from the statutory rate due to the utilization of fully reserved net operating loss carryforwards. We expect our effective rate to return to a rate of approximately 40% for periods subsequent to reversal of the valuation allowance.

Net Income

For the second quarter of fiscal 2011, we had net income of $6.6 million, or $0.14 per diluted share, as compared to net income of $5.6 million, or $0.12 per diluted share, for the second quarter of fiscal 2010. For the first six months of fiscal 2011, net income was $10.8 million, or $0.22 per diluted share, as compared to net income of $9.8 million, or $0.21 per diluted share, for the first six months of fiscal 2010.

Inventory

At July 30, 2011, total inventory was $95.0 million compared to $92.9 million at January 29, 2011 and $94.2 million at July 31, 2010. Because of the upcoming Fall selling season, our inventory levels are typically higher when compared to year-end balances.

Inventory at the end of the second quarter of fiscal 2011 increased slightly by approximately $0.8 million as compared to July 31, 2010. We continue to make a concerted effort to manage our inventory levels and, as a result, our merchandise margins continue to improve.

SEASONALITY

Historically, and consistent with the retail industry, we have experienced seasonal fluctuations as it relates to our operating income and net income. Traditionally, a significant portion of our operating income and net income is generated in the fourth quarter, as a result of the “Holiday” season.

LIQUIDITY AND CAPITAL RESOURCES

Our primary cash needs are for working capital (essentially inventory requirements), capital expenditures and our growth initiatives. As discussed below, our capital expenditure program for fiscal 2011 is $20.0 million, which is greater than fiscal 2010 primarily due to the opening of 15 to 16 DestinationXL stores as well as the upgrading of our e-commerce sites to a multi-brand format.

We currently believe that our existing cash generated by operations together with our availability under our credit facility will be sufficient within current forecasts for us to meet our foreseeable liquidity requirements and capital needs to support our DestinationXL expansion and growth plans. For the first six months of fiscal 2011, free cash flow, which we define as cash flow from operating activities, less capital expenditures and discretionary store asset acquisitions, if any, improved by $4.4 million to $13.4 million from $9.0 million for the first six months of fiscal 2010. See “Presentation of Non-GAAP Measure” above regarding non-GAAP free cash flow. The improvement in free cash flow of $4.4 million in the first six months of fiscal 2011 was due to improved operating income and the timing of working capital.

In addition to cash flow from operations, our other primary source of working capital is our credit facility, which we amended and restated with Bank of America, N.A. (the “Credit Facility”) during the fourth quarter of fiscal 2010. The Credit Facility provides for a maximum committed borrowing of $75 million, which, pursuant to an accordion feature, may be increased to $125 million upon our request and the agreement of the lender(s) participating in the increase. The Credit Facility includes a sublimit of $20 million for commercial and standby letters of credit and a sublimit of up to $15 million for Swingline Loans. The maturity date of the Credit Facility is November 10, 2014. Our Credit Facility is described in more detail in Note 2 to the Notes to the Consolidated Financial Statements.

We had no borrowings outstanding under the Credit Facility at July 30, 2011. Outstanding standby letters of credit were $2.3 million and outstanding documentary letters of credit were $9.2 million. The average monthly borrowing

15

outstanding under this facility during the first six months of fiscal 2011 was less than $25,000, resulting in an average unused excess availability of approximately $65.4 million. Unused excess availability at July 30, 2011 was $63.6 million. Our obligations under the Credit Facility are secured by a lien on all of our assets. The facility contains no financial covenants.

Capital Expenditures

The following table sets forth the open stores and related square footage at July 30, 2011 and July 31, 2010, respectively:

Total cash outlays for capital expenditures for the first six months of fiscal 2011 and fiscal 2010 were $4.8 million and $3.6 million, respectively.

For fiscal 2011, our capital expenditures are expected to be approximately $20.0 million. The budget includes approximately $12.0 million related to the opening of 15 to 16 additional DestinationXL stores and approximately $5.0 million for continued information technology projects, including the launch of our enhanced multi-branded e-commerce site, with the remainder for general overhead projects. In addition, for fiscal 2011, we expect to close approximately 15 to 20 existing stores (5 of which have already been closed during the first six months of fiscal 2011).

DXL store openings

Based on the strong performance of the four DXL stores opened in fiscal 2010, we plan to open up to 15 to 16 new stores during fiscal 2011. Depending on the real estate and market demographics for each of these store locations, we expect the size of each store to be between 6,000 to 12,000 square feet, to accommodate each market.

The launching of our DXL website was postponed until the third quarter of fiscal 2011. This website will combine all of our existing e-commerce sites into one enhanced website, with state-of-the-art features and best practices. This will enable our customers to shop across all of our brands and product extensions with ease and will bring all of our customers under one concept. Their classification as a “Rochester” customer or a “Casual Male” customer will no longer limit their ability to access our full product assortment.

Store Count

Below is a summary of store openings and closings since January 29, 2011:

CRITICAL ACCOUNTING POLICIES

There have been no material changes to the critical accounting policies and estimates disclosed in our Annual Report on Form 10-K for the year ended January 29, 2011 filed with the SEC on March 18, 2011.

16

In the normal course of business, our financial position and results of operations are routinely subject to a variety of risks, including market risk associated with interest rate movements on borrowings and foreign currency fluctuations. We regularly assess these risks and have established policies and business practices to protect against the adverse effects of these and other potential exposures.

Interest Rates

We utilize cash from operations and from our Credit Facility to fund our working capital needs. Our Credit Facility is not used for trading or speculative purposes. In addition, we have available letters of credit as sources of financing for our working capital requirements. Borrowings under the Credit Facility, which expires November 10, 2014, bear interest at variable rates based on Bank of America’s prime rate or LIBOR. At July 30, 2011, we had no outstanding borrowings. Because our average outstanding borrowings during the first six months of fiscal 2011 were less than $25,000, any increase in interest rates would have been immaterial to the financial results for the first six months of fiscal 2011.

Foreign Currency

Our Sears Canada catalog operations conduct business in Canadian dollars and our Rochester Clothing store located in London, England conducts business in British pounds. Our international e-commerce sites conduct business in Euros and British pounds. If the value of the Canadian dollar, British pound or Euro against the U.S. dollar weakens, the revenues and earnings of these operations will be reduced when they are translated or remeasured to U.S. dollars. Also, the value of these assets to U.S. dollars may decline. As of July 30, 2011, sales from our Sears Canada operations, our London Rochester Clothing store and our international e-commerce sites were immaterial to consolidated sales. As such, we believe that movement in foreign currency exchange rates will not have a material adverse affect on our financial position or results of operations.

Evaluation of Disclosure Controls and Procedures

As required by Rule 13a-15 under the Exchange Act, our management, under the supervision and with the participation of our Chief Executive Officer and Chief Financial Officer, evaluated the effectiveness of our disclosure controls and procedures (as defined in Rules 13a-15(e) and 15d-15(e) under the Exchange Act) as of July 30, 2011. Disclosure controls and procedures include, without limitation, controls and procedures designed to ensure that information required to be disclosed by a company in the reports that it files or submits under the Exchange Act is accumulated and communicated to the company’s management, including its principal executive and principal financial officers, as appropriate to allow timely decisions regarding required disclosure. Management recognizes that any controls and procedures, no matter how well designed and operated, can provide only reasonable assurance of achieving their objectives and management necessarily applies its judgment in evaluating the cost-benefit relationship of possible controls and procedures. Based on this evaluation, our Chief Executive Officer and Chief Financial Officer concluded that, as of July 30, 2011, our disclosure controls and procedures were effective.

Changes in Internal Control over Financial Reporting

No change in our internal control over financial reporting (as defined in Rules 13a-15(f) and 15d-15(f) under the Exchange Act) occurred during the quarter ended July 30, 2011 that has materially affected, or is reasonably likely to materially affect, our internal control over financial reporting.

17

PART II. OTHER INFORMATION

We are subject to various legal proceedings and claims that arise in the ordinary course of business. We believe that the resolution of these matters will not have an adverse impact on our operations or financial position.

There have been no material changes to the risk factors as previously disclosed in Part I, Item 1A (“Risk Factors”) of our Annual Report on Form 10-K for the year ended January 29, 2011 filed with the SEC on March 18, 2011.

None.

None.

None.

18

SIGNATURES

Pursuant to the requirements of the Securities Exchange Act of 1934, the Registrant has duly caused this report to be signed on its behalf by the undersigned thereunto duly authorized.

19