When I first started doing this, I was making the picks entirely according to my own subjective opinon. I'm a stat-head, so yeah, I used stats, but I didn't have any "magic formula." And it got increasingly difficult to balance short versus long careers, different eras, sentimentality versus real achievement, etc... in any way that I felt I could justify to an actual fan of any given team. Now, I'm a Red Sox fan. But beyond the Red Sox, Yankees and my new hometown Tigers, I am in no position to argue with a Giants fan or a Marlins fan over who belongs on their dream-team and who's an over-rated bum. So I went out looking for a magic bullet and I learned about WAR.
For those that don't know, WAR stands for Wins Above Replacement. It basically takes a player's entire season, uses sabermetic formulas to figure out the Runs they created offensively or prevented defensively, combines those and then, based on the league's offensive numbers that year, credits that player with the number of Wins that he gave his team, over what they would have had if they'd had a Triple-A Player in his place. (Assumed to produce around 80% of the league's average.) Since it's a Bill James creation, it is boosted by the kind of things that Sabermatricians like: Walks, Power, Speed, Fielding; but also on good old fashion playing time: Games Played and Innings Pitched.
And one of the big advantages with using WAR was that it was now pretty easy, trivial in fact, for me to compare players, not only from different era's (because a Win in 1930, is worth the same as a Win in 1968 or a Win today: ONE WIN) but with different career lengths as well. Quick, who's better: Warren Spahn or Sandy Koufax? Hint: You can't answer unless you know whether I mean career value or peak value. Career value, it's clearly Spahn. Peak value it's every bit as clearly Koufax. So how do you pick between two such players trying to fill the last roster spot at a given position? It also helps by balancing Defense vs. Offense - something I've never really felt comfortable doing. I don't fully understand exactly HOW they do all the figuring on defense but I know they do; because Ozzie Smith had 64.6 WAR in his career while Alan Trammell had 66.9. So you KNOW that defense is being taken into account! (And if you don't? If you think that Ozzie Smith is that close to Alan Trammell on a purely offensive basis? Go home and watch some Football. You don't belong here!)
And while any stat is bound to have it's detractors, I find WAR to be an elegant and holistic way to balance all of the different variables - era, defense vs. offense, career length vs. peak value, speed vs. power vs. average - into a single number and come up with a justifiable ranking that I can use to defend my teams. After that, it's just a question of doing a little research (the easy and fun part, BTW) and placing the eligible Players in the available Roster spots so as to optimize / maximize WAR.
What can I say? There's lots of stats out there that try or claim to do this. WAR just happens to be the one I think does the best job.
1) Allows comparison across era's
2) Puts the value of offense and defense in the same units.
3) Is a peak-weighted, career statistic. (So short, great careers can be compared to longer, merely good ones.)