Naeem Siddiqi is a global product manager for
SAS® credit scoring
The impact of the subprime mortgage meltdown on the world’s economy is a cause for
concern for financial institutions. Could better, more complex analytics have helped, or
was it a failure to follow basic risk management principles? Did institutions take on too
much risk? All these questions not only put risk managers in the spotlight, but also raise the
most critical question of all: how can financial institutions continue to lend money without
setting up undue hurdles or taking on more bad debt?
In the drive to lend quickly and economically, banks have sought to automate the lending
process. Customers love the quick extension of credit. Financial officers clamor for the lower
per-unit processing costs. Risk managers find themselves under pressure to create methods
to quickly ascertain creditworthiness.
In today’s climates, the stakes are even higher. At the customer management level,
companies are striving ever harder to keep their existing clients, by offering them additional
products and enhanced services. Risk managers are called upon to help in selecting the
‘right’ (i.e. low-risk) customers for these favored treatments. Conversely, when customers
exhibit negative behavior (non-payment, fraud), risk managers need to devise strategies to
not only identify them, but also to deal with them effectively, in order to minimize further
loss and recoup any monies owed as quickly as possible.
It is in this environment that risk scorecards offer a powerful, empirically derived solution to
business needs. Risk scorecards have been used by a variety of industries for uses including
predicting delinquency, i.e. non-payment, bankruptcy, fraud, claims (for insurance) and for
recovery of amounts owed for accounts in collections.
In the past, financial institutions acquired credit risk scorecards from a handful of vendors.
The financial institution provided their data to the vendors, and the vendors then
developed a predictive scorecard for delivery. While some advanced companies have had
internal modeling and scorecard development functions for a long time, the trend towards
developing scorecards in-house has become far more widespread in the last few years. This
has happened for various reasons.
Application software now available allows users to develop scorecards without investing
heavily in advanced programmers and infrastructure. Complex data mining functions are
available at the click of a mouse, allowing the user to spend more time applying business
and data mining expertise to the problem rather than debugging complicated programs.
Once the tools became available, in-house development became a viable option for many
smaller and medium-sized institutions. The industry could now realize the significant
return on investment (ROI) that in-house scorecard development could deliver for the right
players. Experience showed that in-house credit scorecard development could be quicker,cheaper and far more flexible than before. The cost of maintaining an in-house credit
scoring capability was less than the cost of purchased scorecards. Internal development
capability also allowed companies to develop far more scorecards (with enhanced
segmentation) for the same expenditure. Scorecards could also be developed faster by
internal resources using the right software – which meant custom scorecards could be
implemented more quickly, leading to lower losses. In addition, companies realized that
their superior knowledge of internal data and business insights led them to develop betterperforming
scorecards. The same result arose from having the flexibility to experiment
with segmentation, and following through by developing the optimum number and
configuration of scorecards.
The key to the scorecard development process is to see them as a business solution to a
business problem. Good scorecards are not built by passing data solely through a series of
programs or algorithms; they are built when the data is passed through the analytical and
business-trained mind of the user – and with input from various stakeholders and end users.
Creating scorecards collaboratively
Ever develop or buy a scorecard that couldn’t be used? Or develop a strategy that caused
negative consequences? Ever paid a vendor hundreds of thousands of dollars to develop 13
segmented scorecards, only to find out their IT department can only implement half of those
in the required time frame? These are only some examples of potential problems during the
process of scorecard development and implementation. None of these are related to the
actual process of analyzing data, or looking at fit statistics. The problems all arise due to a lack
of communication between entities, or the ignorance of business issues by those responsible
for scorecard and strategy development. How do we avoid such disasters?
The process of scorecard development needs to be a collaborative one between
information technology (IT), data mining and operational staff. This not only creates better
scorecards, it also ensures that the solutions are consistent with business direction, and
enables education and knowledge transfer during the development process. Scorecard
development is not a ‘black box’ process and should not be treated as such.
Experience has shown that developing scorecards in isolation can lead to problems. For
example, the inclusion of characteristics that are no longer collected, legally suspect, or
difficult to collect operationally, or devising of strategies that result in ‘surprises’ or cannot be
implemented. The level of involvement of staff members varies, and different staff members
are required at various key stages of the process. By understanding the types of resources
required for a successful scorecard development and implementation project, one will also
start to appreciate the business and operational considerations that go into the design of
scorecards and strategies.
A guide to the major players
The most useful scorecards are developed when companies take a team approach involving
not only the scorecard developer, but also risk or credit scoring managers, product managers,
operational managers, IT staff, enterprise risk managers and the legal department. Project
managers can facilitate where a significant number of scorecards are being developed.
A scorecard developer is expected to have data manipulation and data mining skills and
must have an in-depth knowledge of their data, but should also be a ‘big picture’ person,
who can understand that there are operational issues that must be considered during
scorecard development. The scorecard developer, for instance, may not necessarily have
the background that the product and operations managers have – particularly as in relation
to business concerns, but should be able to communicate with others effectively, to get an
idea of those business issues.
Product managers, for example, offer key insights into the client base and assist during
segmentation selection, selection of characteristics and gauging impact of strategies on
the banks clients and prospects. They also coordinate design of new application forms
where new information is to be collected, or dropping something from the form if shown
to be of no statistical or operational use. Segmentation offers the opportunity to assess risk
for increasingly specific populations; the involvement of marketing in this effort can ensure
that scorecard segmentation is in line with the organization’s intended target markets.
This approach produces the best results for the most valued segments and harmonizes
marketing and risk directions. For example, if a company is actively targeting young people
who live in cities, the operationally ‘optimal’ segmentation would be for this particular
segment, instead of a statistically optimal segmentation that is operationally illogical.
In addition, product managers can help identify preferred segments of customers, for
example the most profitable, those with significant deposits, or target markets. Therefore,
the impact on these segments can be anticipated before cut-off strategies are developed,
to avoid negative consequences.
The operational managers are also critical players in scorecard development, especially
in today’s climate, where branches are seen as sales channels, not places where risk
adjudication is practiced. These employees can alert the scorecard developers on issues
such as difficulties in data collection and interpretation by front-line staff, impact on the
portfolio of various strategies, and other issues relating to the implementation of scorecards
and strategies. Staff from adjudication, collections, and fraud departments can also offer
experience-based insight into factors that are predictive of negative behavior. This helps
greatly when selecting characteristics for analysis.
A best practice for delinquency scorecard development is to interview adjudication/
collections staff during the project to obtain their input. A good question to ask is, “What
characteristics do you see in bad accounts?” The objective here is to tap into experience,
and discover insights that may not be obvious from analyzing data alone, or running
variable selection algorithms. Collections staff can have great ideas on characteristics or
variables that define bad accounts. They see default accounts daily – those belonging
to exactly the type of customer the scorecard is being developed to identify. Gathering
insights from experienced co-workers is especially helpful for those who are developing
scorecards for the first time and want to look beyond data analysis. These interviews will
enable them to tap into existing experience to identify generally predictive characteristics
and get ideas on new ones. This exercise also provides an opportunity to test and validate
experience within the organization. The ideas suggested by adjudicators and others can be
validated through statistical analysis – showing visual relationships between variables and
the bad rate or weight of evidence is a good way to do this- and their hunches can then
either be confirmed or corrected. This sort of feedback loop can provide a valuable way to
enhance the quality of adjudication or collections.
Working with adjudicators and branch staff can also help identify sources of bias in
performance data. Credit performance data is always biased due to ‘cherry picking,’ policy
rules and manual adjudication. Studying both official policy rules and personal behavior
of adjudicators will help those identifying biased characteristics. These can later be fixed,
via binning, if needed.
In any situation where branch staff – or anyone else with an incentive to sell, such as
auto dealers or contractors – are responsible for data entry, spending some time in their
environment can be useful in identifying possible data manipulation. Unfortunately, those
with an incentive to sell will often ‘adjust’ data to maximize chances of credit applications
being approved. This is most common with demographic variables which are known to be
unconfirmed by lenders. Any data known to be manipulated can therefore be identified,
and ideally, left out of scorecards.
Finally, the scorecard development team needs to reach out to IT/IS managers to get a
sense of the capabilities of the underlying software and hardware. In particular, where
scorecards are being developed using complex transformations or calculations that need
to be implemented on real-time software, the IT department may be able to advise if these
calculations are beyond the capabilities of the software. The same is true for derived bureau
variables where the derivations have to be done on credit bureau interfaces or using other
software. Where corporate data repositories or enterprise data warehouses are used, the
appropriate staff responsible for the maintenance of those warehouses can also provide
information on any data issues including limitations.
Involving these resources in the scorecard development and implementation project
helps to incorporate collective organizational knowledge and experience, and produces
scorecards that are more likely to fulfill business requirements. Most of this corporate
intelligence is not documented; therefore, the only effective way to introduce it into credit
scoring is to involve the relevant resources in the development and implementation
process itself. This is the basis for intelligent scorecard development.
In these turbulent times, financial institutions need well-crafted, ‘intelligent’ scorecards more
then ever in order to remain competitive and reduce bad debt. The optimal way to achieve
this is to create scorecards in-house, using the collective wisdom of staff, enterprise-wide.
