Exhibit 99.1

CONTACT:

Jeff Unger

Vice President Investor Relations

(561) 482-9715

Casual Male Retail Group, Inc. Reports Third-Quarter 2012 Financial Results

Delivers Strong Destination XL Retail Store and E-commerce Platform Growth Despite Market Weakness

CANTON, Mass., November 16, 2012 — Casual Male Retail Group, Inc. (NASDAQ: CMRG), the largest multi-channel specialty retailer of big & tall men’s apparel and accessories, today reported operating results for the third quarter of fiscal 2012.

Highlights

Management Comments

“While overall third quarter results did not meet our expectations, Destination XL retail stores and direct sales through DestinationXL.com performed well with excellent comps,” said President and CEO David Levin. “Our lower year-over-year revenues and the resulting earnings decline were due to the mild fall weather that affected sales of seasonal apparel and lower sales from the direct channel as we transition our catalog customers to our e-commerce platform. On the positive side, our new DXL store comps were up 13.8% year over year with a flat comp for our Casual Male XL stores. Even though lower catalog sales resulted in a year-over-year decline for the direct channel, sales from our more profitable e-commerce business grew 11%. We believe that the increased focus on digital marketing will offset the top-line weakness in the long term and our direct operating margins will improve as a result of the reduced catalog circulation.”

“The early success of Destination XL gives us confidence in our plan to accelerate the opening of 225 to 250 DXL stores in major metropolitan areas and to close all of our traditional Casual Male XL locations across the country by the end of 2015,” said Levin. “We are currently testing a comprehensive marketing program in five target regions. The early feedback has been encouraging and we plan to

launch a national campaign in the spring to generate more awareness of the DXL brand. As we execute the transition to DXL, we expect to grow our share of end-of-rack customers, increase overall dollars per transaction, expand our market share, and ultimately substantially grow sales and achieve double digit operating margins.”

Third-Quarter Fiscal 2012 Results

Sales

For the third quarter of fiscal 2012, total sales were $88.7 million compared with $89.0 million in the third quarter of fiscal 2011. Comparable sales for the third quarter increased 1.5% compared with the same period of the prior year. On a comparable basis, sales from the retail business increased 2.5% while the U.S. direct business decreased 3.0%. The increase in the retail business of 2.5% was primarily driven by the new DXL stores. Comparable sales for DXL stores for the third quarter increased 13.8%, and represented 14% of the Company’s retail business.

The 3.0% decrease in comparable direct sales during the third quarter was primarily related to a 33% decline from catalogs, which was partially offset by an 11% increase in e-commerce sales. Throughout fiscal 2012, there has been a shift in the direct business resulting from the Company’s customers responding less to traditional catalogs. In response to this trend, the Company has intensified its digital marketing efforts which include emails, web searches, Internet banners, and affiliate sites. During the third quarter the Company reduced the number of catalogs distributed by 30% with 50% less impressions.

Gross Profit Margin

For the third quarter of fiscal 2012, gross margin, inclusive of occupancy costs, was 44.0% compared with gross margin of 45.0% for the third quarter of fiscal 2011. The decrease of 100 basis points was the result of lower merchandise margins and increased occupancy costs. On a dollar basis, occupancy costs for the third quarter of fiscal 2012 increased 3.2% over the prior year. This increase is partially due to the timing of the DXL store openings and the associated pre-opening occupancy costs incurred.

Selling, General & Administrative

SG&A expenses for the third quarter of fiscal 2012 were 42.5% of sales, flat compared with the third quarter of fiscal 2011. On a dollar basis, SG&A expenses decreased $0.2 million, or 0.4%, for the third quarter of fiscal 2012, compared with the prior-year quarter.

Depreciation and Amortization

Depreciation and amortization for the third quarter of fiscal 2012 grew to $3.8 million from $3.0 million for the third quarter of fiscal 2011, primarily due to increased amortization of $0.5 million related to the Company’s Casual Male trademark, as a result of its becoming a definite-lived asset.

DXL Transition Costs

The Company is incurring transition costs as it moves to its DXL format, which include pre-opening rent and payroll, store training, infrastructure costs and increased marketing. These costs are incremental to last year and are start-up costs associated with store openings that will not continue once a DXL store is open.

The Company’s results for the third quarter and first nine months of fiscal 2012 include total incremental costs of $3.0 million, or $0.04 per diluted share after-tax, and $6.6 million, or $0.08 per diluted share after-tax, respectively. See “Non-GAAP Measures” below.

Income Taxes

As a result of its valuation allowance being substantially reversed in the fourth quarter of fiscal 2011, the Company has returned to a normal tax provision for fiscal 2012. Accordingly, for the first nine months of fiscal 2012, the effective tax rate was 40.5% compared with 8.8% for the first nine months of fiscal 2011. The effective tax rate for the first nine months of fiscal 2011 was reduced from the statutory rate due to the utilization of fully reserved NOL carryforwards.

Income (Loss) from Continuing Operations

The loss from continuing operations for the third quarter of fiscal 2012 was $1.6 million, or $(0.03) per diluted share, compared with a loss from continuing operations of $1.0 million, or $(0.02) per diluted share, for the third quarter of fiscal 2011.

Income from continuing operations for the first nine months of fiscal 2012 was $3.8 million, or $0.08 per diluted share, compared with income from continuing operations of $10.7 million, or $0.22 per diluted share, for the first nine months of fiscal 2011. Assuming a normal tax rate of 40.0% for fiscal 2011, adjusted income from continuing operations for the first nine months of fiscal 2011 was $7.1 million, or $0.14 per diluted share. See below for a reconciliation of adjusted income from continuing operations, a non-GAAP measure.

Net (Loss) Income

The net loss for the third quarter of fiscal 2012 of $1.6 million, or $(0.03) per diluted share, was flat compared with the third quarter of fiscal 2011. The results for the third quarter of fiscal 2012 include costs of $3.0 million, or $0.04 per diluted share, related to the DXL store growth initiative.

Net income for the first nine months of fiscal 2012 was $1.9 million, or $0.04 per diluted share, compared with $9.2 million, or $0.19 per diluted share, for the first nine months of fiscal 2011. Assuming a normal tax rate of 40.0% for fiscal 2011, adjusted net income for the first nine months of fiscal 2011 was $5.5 million, or $0.11 per diluted share. The lower earnings, on a comparable non-GAAP tax basis, of $0.07 per diluted share is primarily due to flat sales and approximately $6.6 million, or $0.08 per diluted share, in higher costs to support the DXL roll-out. See below for a reconciliation of adjusted net income, a non-GAAP measure.

Cash Flow

Cash flow from operations was $8.4 million for the first nine months of fiscal 2012 compared with $11.7 million for the first nine months of fiscal 2011. Free cash flow from operations (as defined below under “Non-GAAP Measures”) decreased by $13.6 million to $(12.9) million from $0.7 million for the first nine months of fiscal 2011, largely due to the seasonal increase in inventory, an increase in capital expenditures associated with the new DXL store openings, higher occupancy and SG&A expenses also associated with the DXL roll-out, and a prepayment penalty related to the closure of the European direct business included in discontinued operations.

Balance Sheet & Liquidity

At October 27, 2012, the Company had cash and cash equivalents of $5.2 million, outstanding borrowings of $7.6 million, and $64.4 million available under its credit facility.

Inventory was $116.1 million compared with $104.2 million at the end of fiscal 2011 and $114.9 million at the end of the third quarter of fiscal 2011. Because of the upcoming “Holiday” selling season, inventory levels are typically higher when compared with year-end balances.

The increase in inventory year-over-year is primarily the result of cost increases and shifting of product mix. On a unit basis, inventory decreased 9.2% compared to the third quarter of fiscal 2011. Unit inventories in branded product have increased by approximately 35% over the prior year to support the DXL store product assortments, which has a greater brand assortment than the Casual Male XL stores.

Retail Store Information

As disclosed in the second quarter of fiscal 2012 news release, the Company is in the process of significantly transforming its business as it accelerates the DXL store openings and the closure of Casual Male XL stores. The DXL stores outperform the Casual Male XL stores and, as the chain is converted, the Company believes that the sales growth will improve. However, during the transition, the Company is experiencing some sales erosion among its Casual Male XL stores located near its DXL stores. Sales from DXL stores represented 14% of the Company’s retail sales for the third quarter and 11% of retail sales for the first nine months of fiscal 2012. By the conclusion of fiscal 2012, the Company expects that the penetration of DXL stores will approach approximately 25% of retail sales.

Fiscal 2012 Outlook

As a result of the lower-than-expected third-quarter results and the negative impact of Hurricane Sandy on the Company’s Northeast business, guidance has been revised for the fiscal year ending February 2, 2013 as follows:

Conference Call

The Company will hold a conference call to review its financial results and business highlights today, Friday, November 16, 2012 at 9:00 a.m. ET. Those who wish to listen to the live webcast should visit the “Investors” section of the Company’s website. The live call also can be accessed by dialing: (888) 455-2296. For interested parties unable to participate live, an archived version of the webcast may be accessed by visiting the “Events” section of the Company’s website for up to one year.

During the conference call, the Company may discuss and answer questions concerning business and financial developments and trends. The Company’s responses to questions, as well as other matters discussed during the conference call, may contain or constitute information that has not been disclosed previously.

Non-GAAP Measures

In addition to financial measures prepared in accordance with generally accepted accounting principles (GAAP), the above discussion refers to non-GAAP adjusted income from continuing operations per diluted share and adjusted diluted earnings per share (“non-GAAP” or “adjusted”). These measures

should not be considered superior to or as a substitute for income from continuing operations per diluted share and diluted earnings per share derived in accordance with GAAP. The Company believes that these non-GAAP measures are useful as an additional means for investors to evaluate the Company’s operating results, when reviewed in conjunction with the Company’s GAAP financial statements. The Company believes the inclusion of these non-GAAP measures enhances an investor’s understanding of the underlying trends in the Company’s business and provide for better comparability between different periods in different years.

The above discussion refers to free cash flow, which also is a non-GAAP measure. The presentation of non-GAAP free cash flow is not a measure determined by GAAP and should not be considered superior to or as a substitute for net income or cash flows from operating activities or any other measure of performance derived in accordance with GAAP. In addition, all companies do not calculate non-GAAP financial measures in the same manner and, accordingly, “free cash flows” presented in this release may not be comparable to similar measures used by other companies. The Company calculates free cash flows as cash flow from operating activities less capital expenditures and less discretionary store asset acquisitions, if applicable.

The above discussion also includes the earnings per share impact of incremental costs that have been incurred in connection with the Company’s DXL growth initiative of $3.0 million, or $0.04 per diluted share, and $6.6 million, or $0.08 per diluted share, for the third quarter and first nine months of fiscal 2012, respectively. The $0.04 per diluted share was calculated, using the third quarter effective tax rate of 40.1%, by taking the net of $3.0 million less $1.2 million of tax divided by outstanding diluted shares of 48.1 million. The $0.08 per diluted share was calculated, using the first nine months’ effective tax rate of 40.5%, by taking the net of $6.6 million less $2.7 million of tax divided by outstanding diluted shares of 48.3 million.

Below are tables showing the reconciliation of all GAAP measures to non-GAAP measures.

About Casual Male Retail Group, Inc.

Casual Male Retail Group, Inc. is the largest multi-channel specialty retailer of big & tall men’s apparel with operations throughout the United States, Canada and Europe. The retailer operates under six brands: Destination XL^®, Casual Male XL, Rochester Clothing, B&T Factory Direct, ShoesXL and LivingXL. Several catalogs and e-commerce sites, including www.destinationxl.com, make up the Company’s direct-to-consumer business. With more than 2,000 private label and name-brand styles to choose from, customers are provided with a unique blend of wardrobe solutions not available at traditional retailers. The Company is headquartered in Canton, Massachusetts. For more information, please visit the Company’s investor relations website: http://investor.casualmale.com.

Forward-Looking Statements

Certain information contained in this press release, including cash flows, operating margins, store counts, revenue and earnings expectations for fiscal 2012, constitute forward-looking statements under the federal securities laws. The discussion of forward-looking information requires management of the Company to make certain estimates and assumptions regarding the Company’s strategic direction and the effect of such plans on the Company’s financial results. The Company’s actual results and the implementation of its plans and operations may differ materially from forward-looking statements made by the Company. The Company encourages readers of forward-looking information concerning the Company to refer to its prior filings with the Securities and Exchange Commission, including without limitation, its Annual Report on Form 10-K filed on March 16, 2012, that set forth certain risks and uncertainties that may have an impact on future results and direction of the Company.

Forward-looking statements contained in this press release speak only as of the date of this release. Subsequent events or circumstances occurring after such date may render these statements incomplete or out of date. The Company undertakes no obligation and expressly disclaims any duty to update such statements.

CASUAL MALE RETAIL GROUP, INC.

GAAP TO NON-GAAP EPS RECONCILIATION

CASUAL MALE RETAIL GROUP, INC.

GAAP TO NON-GAAP FREE CASH FLOW RECONCILIATION

CASUAL MALE RETAIL GROUP, INC.

CONSOLIDATED STATEMENTS OF OPERATIONS

(In thousands, except per share data)

CASUAL MALE RETAIL GROUP, INC.

CONSOLIDATED BALANCE SHEETS

October 27, 2012 and January 28, 2012

(In thousands)