A model cannot be perfect; some are better reflections of reality than others. A model provides a well-defined framework for discussion and research and establishes guideposts for discovering exceptions and anomalies. As these are documented, the model evolves.

It is this likelihood for change, the thought that all the "bugs" haven’t been worked out, that frightens trustees most. They forget traditional approaches now in use are being modified continually, too.

In fact, trustees might not realize that what is most commonly considered a model’s biggest weakness—the absence of or a limited exposure to actual trading—is its greatest strength. For a model to be effective in generating excess returns, its use cannot be widespread. Once a model begins to trade successfully, others will try to copy it; it’s only a matter of time until someone succeeds. Once too many are on the bandwagon, a model’s effectiveness is lost.

One class of models used in investing, and represented by the work of Ben Graham, defines the decision-making process. Believing in the need for both diversification and an objective means of measuring value, Mr. Graham formulated rules for buying and selling stocks. Because he believed the merits of any one stock cannot be accurately evaluated, security selection works only on the portfolio and not at the individual level.

Computer analysis of huge databases enables other models to identify and capture previously unknown relationships. In some cases there is no apparent conceptual basis for them. It is simply that they exist but had gone unnoticed. If a portfolio manager is aware of these little-known relationships, he or she can exploit them and capture excess returns.

"Non-quant" trustees are faced with the dilemma of trying to distinguish between a creative approach that will add value to their fund and one that sounds intriguing but does not execute well. Trustees will increase the likelihood of making an informed decision if they address a number of key issues that follow.

Does the model make sense both in concept and in application?

The importance of active asset allocation heightens the concern over the usefulness of any quantitative model. It is widely accepted some 90% of a fund’s total return comes from its asset mix. Owning the right assets at the right time and reallocating the assets at the appropriate time makes sense, but can it be done with success?

For example, if $1 million had been invested in the Standard & Poor’s 500 stock index from Jan. 1, 1975, to the close of 1984, it would have grown to $3.95 million. Had an allocation model indicated a move out of stocks for just three quarters—the first half of 1975 and the last quarter of 1982—the stock market’s three biggest quarters would have been missed.

That market-timing portfolio would have grown to just $2.48 million, just slightly more than the $2.38 million that would have been earned if an investor had kept the $1 million entirely in Treasury bills for the 10 years.

Several studies have demonstrated the odds are against successful active allocation over longer periods of time. If a plan’s trustees decide to use market timing, they must determine why a model under consideration has the exceptional predictive powers needed to turn the odds in their favor.

If the future doesn’t resemble the past, how much predictive power can be attributed to computer simulations? This question, while commonly asked of quants, should be directed to all money managers regardless of their track record. If the future is a whole new ball game, then no one’s track record is relevant.

Does the model include measures that minimize the implementation shortfall? This is the difference in return between tracking a portfolio on paper (or back-testing on a computer) and managing it in the real world. The shortfall measures the degree a manager is unable to exploit his or her stock selection skills.

One example of the implementation shortfall is the Value Line ranking system and the Value Line funds that make use of the system. For example, between 1965 and 1986 the Value Line Fund has outperformed the market by 2.5% a year. A paper portfolio based upon the Value Line ranking with weekly rebalancing outperformed the market almost 20% a year. A possible explanation for at least some of the shortfall is that the fund managers had to compete with Value Line Investment Survey subscribers for trades and the fund needed to maintain cash balances to facilitate transactions.

What is the worst case scenario? How often and what magnitude should the model be expected to underperform the benchmark portfolio? If trustees can’t accept the downside risk, then the model, regardless of its strengths, is inappropriate.

Does the model have too many variables in it? The goal ought to be to work with a minimum number of variables, often only one, each of which can be more easily controlled.

The fewer variables the model contains, the greater the likelihood future performance will replicate test results.

How reliable is the input? If the input is incorrect, the output cannot be meaningful. Models that use earnings and dividend projections often fall short on this point.

How was the model formulated and tested? Does it appear the quant developed an algorithm to optimize the fit to the original data without an independent validation on a separate data set? If he or she mined the data, forget it.

Did the developer obey the model slavishly during testing or could his or her judgment have affected the results on any given day? If the model is followed strictly, only one disciplined approach—the one trustees evaluate—was used. If human judgment was involved, you don’t know what, if any, fudge factors were used.