PERFORMANCE MEASUREMENTS—DESIGNING THE GUIDANCE SYSTEM FOR YOUR ORGANIZATION

By James D. Tarr, CPIM

INTRODUCTION

Your business is changing. JIT, TQM, Reengineering, Information Technology, Teams, Time Based Competition, the supply chain and the decline of the Command/Control organization structure are just some of the changes taking place. The critical success factor is knowledge, not physical work. Competitive advantage can no longer be perpetually and automatically maintained. In many industries continuous change and improvement is no longer an option, it is a way of life.

But what are we measuring? Often organizations are measuring the same things they were 10, 20 and even 50 years ago. Most organizations are still using a cost accounting structure developed in the 1920’s. Many are still focusing their measurement efforts on direct labor, even though it is typically less than 10% of product cost and the key to business success is no longer "make and move", but knowledge work. Many businesses still measure function (e.g. purchasing department) when the focus of the business is cross functional process (e.g. material acquisition process).

Steps have been taken to address these shortcomings—quality measures, cycle time measures and cross functional process performance measures. Activity Based Costing models that mirror the business value chain have been developed and used. But many of these are merely measures added to existing measures without a consideration of the effect on the organization. A few companies on the "cutting edge" have developed Balanced Scorecard systems whose design is driven by business strategy and implementation is performed in parallel with other systems initiatives.

This presentation will discuss the multiple roles a performance measurement system plays in today’s organization—control, information and motivation. It will also discuss the various dimensions in which a performance measurement system must work effectively and their interrelationship. The presentation will then describe some of the pitfalls of a poorly designed measurement system and how to avoid them. Finally, it will present a generic framework for a performance measurement system that can help the attendee evaluate and validate the organization’s present measurement system and assist in designing a more effective one. Examples and graphical representations of multidimensional performance measurement will be shown.

PROBLEMS WITH TRADITIONAL PERFORMANCE MEASUREMENT

Why has performance measurement system development lagged behind business process system development? Why did it not parallel the changes in the business organization it intended to measure? There are many reasons, some or all of which apply to organizations whose measurement system has not kept up. Among them are:

• In most organizations, the primary measurement system is financial, capped by the financial statement (Profit & Loss Statement and Balance Sheet). The main purpose of this systems has changed from management information to an information system for external stakeholders such as Investors, Lenders, Government Entities and the like, with the structure defined by Generally Accepted Accounting Principles (GAAP). While this system serves a valuable external purpose, it is poorly designed to provide the information necessary to guide the direction of the organization on any but the macroscopic level.
• Managers have learned how to "manage" the performance measurement system (as opposed to managing the business) to return numbers that meet goals, secure bonuses and generally reflect satisfactory performance. Changing the measurement system in this environment meant risking the security of the existing situation. Thus, in most organizations, there is tremendous resistance to performance measurement change unless it will lead to higher bonuses.
• The organizational change from action orientation to knowledge orientation has been gradual and is, even today, often not recognized in many organizations. Hence, the need for new measures is not always readily apparent. New measures periodically added to old measures seem "good enough".
• Computer technology has allowed organizations to collect increasing amounts of data more rapidly than ever before. The computer can provide information in excruciating detail with precision and rapidity, giving the illusion that performance measurement has been improved. The real result is that it is increasingly difficult to separate the "vital few" from the "trivial many".
• Historically, the relationship and effect of performance measurement on organizational performance were not well understood. With the research that led to techniques such as Activity Based Costing/Activity Based Management (ABC/ABM) and Balanced Scorecard, measurement systems are beginning to parallel organizational performance. However, there is still much to learn. There are still many misapplications of these techniques. Prominent among the problems:
• Measurement systems identified as ABC/ABM often are nothing more than a detailed analysis and reallocation of the General Ledger expense accounts called Cost Decomposition. While this can be a useful improvement, it still retains the limitations and functional bias of the financial accounting system rather than the process orientation of the business organization. This can yield improved product costing, but it is of little use in measuring organizational performance and identifying opportunities for improvement.
• Balanced Scorecard systems often reduce performance measurement to statistically determined performance indices. While this leads to simple comparison of performance against a goal, it has two shortcomings:
• Most performance measures are indirect at best. To make them even more indirect by combining them into a weighted average index is to reduce them to a mere number, eliminating virtually all meaning.
• Within the organization, managers are tempted to "manage to the index", rather than managing the business, thus perpetuating a shortcoming of existing measurement systems.

To affect organizational behavior, the factors in the organization that create and encourage that behavior must be changed. As Shoshana Zuboff (1988) states "...if managers are to alter their behavior, then methods of evaluation and reward that encourage them to do so must be in place."

PERFORMANCE MEASUREMENT AS A SYSTEM

To improve the effectiveness of performance measurement, it must be defined and developed as a system, not just a collection of measures the organization has historically used to control individual behavior. The characteristics of a performance measurement system are that it must be:

• Purposeful--It must support the organization strategy and, as a whole, 1) motivate people within the organization to behave in a way that maximizes the strategy and 2) truly and accurately measure progress toward strategic goals and objectives at all organizational levels.
• Unified--Measurement at all levels of the organization must motivate behavior supporting goals and objectives at higher organizational levels. Direct measures motivate behavior, while next higher level measures and measures in related areas of the organizational supply chain determine successful performance.
• Integrated--Measurement must exist in all functions and processes of the organization. In addition, these measures must support superior performance in all functions and processes related to the area being measured. Attaining superior performance in one function must not be accomplished at the expense of performance in another.
• Fluid--A performance measurement system is a set of metrics that indicates to organization participants status and direction of the critical success factors of the organization. Since the organization is constantly changing, the metrics and targets of the performance measurement system must be constantly changing as well. With each process improvement or supply chain modification the following questions should be asked:
• How will this change affect what’s measured?
• How will this change affect expected performance?
• What metrics will best reflect the revised business system performance?
• What are the underlying assumptions connecting the measure to performance?
• What are the intended consequences of the measure?
• What are the possible unintended consequences of the performance measure and how can they be minimized?

PURPOSE OF PERFORMANCE MEASUREMENT

Performance measurement has several purposes in the business organization, both internal and external. Performance measurement for external purposes is an extensive topic in itself and outside the scope of this paper. In a supply chain, internal performance measures may extend beyond the traditional boundaries of the organization. Internal purposes can be further broken into three general categories with some overlap, as shown below:

• Controlling and redirecting individuals and departments; This is the most short range and oldest use of performance measurement. It carries over from the hierarchical command/control style of organization and management. It was very effective when the work of the organization was primarily action--clearly visible and easily measurable. However, today most work is knowledge based and performance measurement is less a matter of counting and more a matter of judgment. Many organizations still expend too much of their performance measurement effort measuring action.
• Feedback to adjust performance and/or targets; These are mid range measures such as monthly targets, quarterly performance reporting and the like. They are often incorrectly used as command/control tools. However, to the extent they are used to make mid course corrections, (e.g. feedback in an MRP system) they can provide valuable information ensuring that mid range planning and execution are adjusted to continue support of long term strategies. They also serve as an early warning system indicating when strategy needs to be rethought. It must be understood, however, that either the planned target or the execution of the plan can be in error. Results are measured for the purpose of adjusting behavior.
• Performance against strategic and continuous improvement goals; These are long range results measures, used to compare to the business plan and strategic goals in order to verify that the plan is appropriate, or to adjust or redirect the strategy of the organization.
• The quality and effectiveness of performance measures at all levels depend on both the quality of the measurement system and the use to which it is put. The system must be congruent with the business strategy and the organization that it measures. Even the best intentioned management cannot make good strategic decisions based on a performance measurement system that is incomplete or measuring the wrong things. Also, useful performance measurements improperly applied can also lead to incorrect decisions and actions.

CHARACTERISTICS OF MEASUREMENT

All effective measurement systems have a multitude of characteristics. These characteristics apply in varying degree to each measurement, but must be taken into account if a measurement system is to have maximum value. The characteristics are:

• Assumptions--All measurements contain assumptions about the measurement, its purpose and its relationship to the variable being measured. These assumptions can be explicit or implicit, but they always exist. For example, the implicit assumptions of a machine utilization measurement are: 1) all production has equal value and 2) anything produced will have greater value if produced now than if produces later. Once these assumptions become explicit, it becomes clear that they are true only under very specific circumstances, if at all.
• Precision vs. Accuracy--Calculating a measurement to five decimal places is precision. Accuracy is the difference between observed and actual value in the case of existing discrete quantities (e.g. inventory quantities) or the amount of standard deviation of a probability distribution for forecasts, standards or goals (e.g. budgets, performance standards, etc.). Precision and accuracy are not substitutable.
• Congruence—Since most measurements are surrogates for the critical success factor being measured (e.g. on time delivery as a measure of customer service), it is important that the measure varies in relative direction and magnitude with the critical success factor. Since this is difficult to determine (after all, the critical success factor cannot be measured directly), several surrogate measures can be used together for validation.
• Static vs. Vector Measures--Static measures are a measure of the position of a variable at an instant in time, while vector measures indicate the velocity and direction in which a variable is moving. Traditional performance measurement systems emphasize static measures because: 1) they are easier, both to understand and to measure, 2) they are more easily quantified and compared, 3) less information is required. Conversely, vector measures require: 1) a baseline from which to measure, 2) a goal toward which the variable should be moving and 3) a time series of measurement in order to measure direction and velocity. The minimum number of measurements is two if the variable is linear and more if the variable is non linear, probabilistic or the velocity is undergoing acceleration or deceleration.

• Soft vs. Hard measures--Hard measurements are those that can easily be quantified. Soft measures are those that can be measured only in relative terms. Soft measures fall into two categories: 1) relative and 2) statistical. Relative measures are those that can be bracketed between two more concrete measures. For example, a business can know (e.g. by survey) that its customer service has improved over a baseline, such as last year, but that it has not yet attained the goal. Assigning a quantifiable value may be possible, but only very indirectly (e.g. number of customer complaints). However the business can be fairly certain that customer service is improving via the vector measurement.

• Results measures vs. Behavior measures—Most organizations today are knowledge based and most people are hired for what they know, not what they do. To hire someone for what they know and then measure how they do it is, at best, meaningless and, at worst, destructive. The only valid and useful measures for most jobs and activities today are results measures. Knowing that people are doing things right is not nearly as important as knowing that they are doing the right thing. There continue to be, of course, important and required expected behaviors (e.g. courtesy and respect). However, success in maintaining these within the organization culture, while reflected in results measures, are not measured by them.
• Intended vs. Unintended Consequences--Peter Drucker (1973) has indicated that performance measurement in a social system can be neither objective nor neutral. A primary purpose of performance measurement is to get people and organizations to improve in the direction of a certain standard or goal. If the performance measurement is an indirect measure of the goal, as it often must be, treating the measure a hard number often makes maximizing the measure a substitute for the goal itself. This can lead to the unintended consequence of meeting the measure while ignoring the goal.

Performance measurement systems that have certain characteristics run the risk of not measuring anything meaningful and leading to unintended consequences that can interfere with good performance. Most at risk are those performance measurement systems that focus on: 1) precise, hard numbers that are used to measure specific performance to specific indirect parameters, rather than: 2) using vector, relative and statistical measurements combined with management judgment.

In short, type 1) above is the very performance measurement system many companies have carried over from their hierarchical command/control days and are now trying to use to coax quality, participative, continuous improvement from their organization. It is not hard to understand why many organizations meet what seems like intentional resistance to organizational change and continuous improvement initiatives, both internally and within the supply chain.

PERFORMANCE MEASUREMENT AND THE BUSINESS PROCESS

As mentioned above, performance measurement is still a stepchild of financial measurement in many organizations. Metrics are added haphazardly as needed to clarify or control without regard for integration with organization goals and objectives. Many organizations still believe that if each individual and function is measured to some quantifiable standard, the sum of the results will be organizational effectiveness.

Performance measurement system characteristics were discussed above. Here are some points of integration:

• Performance measurement must be an integral part of corporate strategy.
• Each measurement should be traceably shown to support overall corporate purpose.
• Measurements and methodologies must be aligned with corporate cultural values.
• Vector measures will predominate in a continuous improvement strategy.
• There must be a clear understanding of the difference between performance measures that are deterministic (e.g. matching physical inventory counts to "book" balance) and those that are statistical in nature (e.g. standards, forecasts, budgets).
• The system must focus on measurement as information, not measurement as control. The system should measure results, not behavior.
• Measurement systems must leave room for management judgment.
• Measurement systems must be constantly re-evaluated:

BEHAVIOR SPACE MEASUREMENT

How does a measurement system design fulfill all these criteria? An important attempt has been made by Kaplan and Norton (1996) in the Balanced Scorecard approach. It is a comprehensive, systematic methodology that measures groups and individuals on a variety of performance metrics, usually tied to corporate goals. However, Balanced Scorecard uses a weighted average method to reduce a variety of measures to a performance index. This leads groups and employees to "perform to maximize the index", often with little regard for the desired performance. This occurs because the measurable factors that make up the index are not substitutable for one another.

A more effective method is to define a "Behavior Space" bounded by several measurement dimensions that define individual, department and organizational performance. Although this is somewhat more complex than the index, it is well within the ability of most systems to collect, analyze and report the information involved. The increase in usable information, both to the individual and to the organization more than offsets any additional cost and effort required.

The example shown below reflects a performance measurement framework for only two levels in the organization--the Operations Manager or facility level (see Table One) and the Supervisory or department level (see Table Two). A comprehensive system would encompass all levels of the organization, from business planning to shop floor control. The model contains six attributes to be measured:

• Quality -- Products and Processes.
• Service--Both internal and external customer.
• Cost -- Products and Processes.
• Velocity--Cycle time of Products and Processes.
• Knowledge--Organizational resources.
• Investment Effectiveness.

Although other organizations might have additional significant attributes to their business processes, these are probably basic to all. Also, the critical success factors, goals and metrics selected are representational and not intended to be appropriate for all situations.

Table One

Table Two

In using the framework to measure performance, both static and vector characteristics of all measurement are tracked and compared to both Short Term and Long Term goals. Use of the system is primarily informational rather than control oriented or punitive. As such it is used to identify 1) problem areas such as lack of resources and/or training and 2) process improvement opportunities for continuous improvement teams. The system must be constantly re-evaluated by management to insure that it 1) still represents and supports the strategy of the organization, 2) provides correct and adequate information to manage and improve the business and 3) is not contributing, directly or indirectly, to the creation of unintended consequences or waste.

REPORTING BEHAVIOR SPACE PERFORMANCE

Rather than reducing the performance report to an index which requires judgmental weighting and makes the measurement even more indirect, a graphical representation provides a more satisfying and accurate method for the following reasons:

• Each performance measure can be clearly seen in comparison to long and short term goals.
• Areas of good and deficient performance are clearly visible.
• Graphical representation clearly shows actual behavior compared to the expected behavior space.

Below, Table Three shows actual performance against the managerial measures shown in Table One followed by a set of graphical representations of the performance against goals. The percent variance is calculated by dividing the value over or under the short term goal by the range between the short term and long term goals. For example, for the Cost measure:

(17% - 10%) / (30% - 10%) = +35% Variance from short term goal.

TABLE THREE

Figure One shows the table in graphical form. Expected results, the Expected Behavior Space, lies between the inner short term goal boundary and the outer long term goal boundary. The black area represents measures where performance exceeded short term goals and gray areas represent measures where performance fell short. The graph clearly indicates the employee’s performance profile and shows where and how employee resources need to be redirected.

FIGURE ONE

The above representation has several advantages over the weighted average index that is often used in the Balanced Scorecard approach:

• It shows strengths and weaknesses in performance against metrics, which the index doesn’t do.
• It does not rely on subjectively determined "weights" which tend to be arbitrary and easily manipulated.
• It gives a clear indication of performance versus expected results for each critical success factor.
• Individual metrics (e.g. Quality) can be measured vertically from level to level in the organization, an activity that is meaningless for indices.
• It eliminates the "apples and oranges" quality in an index of trying to combine unlike critical success factors. Can you really substitute Quality for Cost?

Figure Two shows the vector component of the Quality metric. It indicates performance against a constantly increasing short term goal and the gap that exists between the current short term goal and the long term goal. The Figure Two graphic would exist for each of the six metrics that defines the Behavior Space.

FIGURE TWO

This is only a sample of the information that can be provided using the Behavior Space Measurement model. Each individual and any team with goals can be measured. Additional formats and methodologies have been developed to incorporate indices, where appropriate, that accurately reflect actual performance results against goals.

CONCLUSION

Designing, developing and implementing performance measurement as a system, integrated into and changing with the business process is a critical challenge for organizations that hope to move into the next millennium successfully. Traditional measures are inadequate. While the Balanced Scorecard approach is a giant step forward, it too is flawed and must be improved. Multi dimensional Behavior Space performance measurement systems move beyond Balanced Scorecard to a more robust system tied to business strategies and critical success factors that will allow the organization to attain and maintain world class performance.

SELECTED BIBLIOGRAPHY

Brown, Mark Graham, (1989) Keeping Score, Quality Resources, New York.

Drucker, Peter F., (1992) Managing for the Future, Truman Talley Books/Dutton, New York.

Maskell, Brian, (1996) Making the Numbers Count, Productivity Press, Portland OR.

Zuboff, Shoshana, (1988) In the Age of the Smart Machine, Basic Books, Inc., New York.