In JDJ's Interview with Amy Fowler, we have this awful quote:

Let's nail this right here. Due to tremendous efforts by the Java 2D, AWT, and Swing teams, and the kindness of Moore's Law, Swing performance is no longer a valid excuse for not using Java on the client.

This is from an employee of Sun whose background includes:
I've been at Sun forever and have been an engineer on the J2SE client team since the days we called it the "JDK" and there were a total of eight packages. Most of that time I've spent on the Swing team, with a year-long tour of duty in J2EE as the JSF spec lead, trying to define a component model in the otherwise amorphous Web tier.

So this is a big quote, and a concerning one. Does Swing deliver usable GUIs now? Sure. Does my Celeron 533 tower deliver acceptable performance? Sure.

Can any user, from Granny Smith to John Q. Developer, tell a huge speed dip when moving from a native app to Swing? Yes, every bit as much as I can tell that my 533 isn't performing at the same level as my 2 GHz Pentium 4.

Here are Swing/Java's GUI issues when it comes to speed:
1.) Start-up for a Swing app is far slower than that for a natively written app.
2.) Response time for app widgets/controls written in Swing is noticibly longer than most natively written apps' controls (like my 533 to 2 GHz comparision). It is my anecdotal experience on Mac OS 9, X, and Windows XP & 2k that using Swing over AWT contributes greatly to this slowdown, everything else equal (so only when an application uses widgets available in both).
3.) A Java app written with Swing (at least with Metal L&F) regularly has its entire pane go grey on Windows in my experience. From JNN to my crappy app to Netbeans, the windows will blank out and will take seconds to reappear after alt-tabbing.
4.) Not everyone has [the latest VM installed], so don't assume those improvements.
5.) Not every programmer is going to go the extra mile to fine-tune performance of their GUIs. It's easier to switch to VB6 or VB.NET (or REALbasic or straight Cocoa (which many Java devvers for Mac lay on top of Java code or access with Java) or other lang of your choice).

Until you can fix #1, nothing else matters. Until you can fix #2, nothing else matters. Until you can fix #3, nothing else matters. That's three dealbreakers right there. Then you've got to think of a way to fix #5 (which is part of what the interview's about) without compromising the hard fact of #4.

What does Swing do well? It provides an incredibly good, cross-platform GUI toolset that behaves exactly the same (all other things equal; you can, of course, use code that captures what platform you're on and change things) regardless of platform. That's not what a typical client (which, as I've pointed out, apparently includes even Java developers) wants. Until Sun & Sun's Swingers realize they're not developing what people want, Java desktop application developers will continue to feel the pain.