The relative importance of tender evaluation and contractor selection criteria PDF

Preview

Summary

Available online at www.sciencedirect.com International Journal of Project Management 28 (2010) 51–60 www.elsevier.com/locate/ijproman The relative importance of tender evaluation and contractor selection criteria D.J. Watt a,b,*, B. Kayis a, K. Willey c a University of New South Wales, Faculty of E...

Description

Accelerat ing t he world's research.

The relative importance of tender evaluation and contractor selection criteria Keith Willey International Journal of Project Management

Cite this paper

Downloaded from Academia.edu 

Get the citation in MLA, APA, or Chicago styles

Related papers

Download a PDF Pack of t he best relat ed papers 

Ident ifying key fact ors in t he evaluat ion of t enders for project s and services Keit h Willey Built environment procurement pract ice: Impediment s t o innovat ion and opport unit ies for changes Craig Furneaux Correct T ION implement at ion and adopt ion of e-procurement Umar Faruk mumuni

Available online at www.sciencedirect.com

International Journal of Project Management 28 (2010) 51–60 www.elsevier.com/locate/ijproman

The relative importance of tender evaluation and contractor selection criteria D.J. Watt a,b,*, B. Kayis a, K. Willey c a

University of New South Wales, Faculty of Engineering, Barker Street, Kensington, 2052 New South Wales, Sydney, Australia b DownerEDI Rail, 2B Factory Street, Granville, 2142 New South Wales, Australia c University of Technology, Sydney, Australia Received 3 January 2009; received in revised form 20 April 2009; accepted 21 April 2009

Abstract Research in identifying the relative importance of criteria used to select a preferred supplier has, for the most part, relied on subjective lists of criteria being presented to respondents. The research reported here uses an experimental design approach to quantify the importance of nine common criteria used in an actual evaluation and selection of a contractor/supplier. Unique choice sets were constructed, each comprising three tender evaluation outcomes (alternatives) described in terms of all criteria, but with varying levels. Respondents simultaneously evaluated all three alternatives within each choice set and selected the most preferred. Utility estimates for each criterion level were determined as was the overall contribution made by the individual criterion. Results indicate Past Project Performance, Technical Expertise and Cost are the most important criteria in an actual choice of contractor with Organisational Experience, Workload, and Reputation being the least important. Ó 2009 Elsevier Ltd and IPMA. All rights reserved. Keywords: Contractor selection; Discrete choice models; Experimental designs; Tender evaluation criteria

1. Introduction Tender evaluation and contractor selection continues to be an area of significant importance and interest to organisations responsible for delivering project outcomes. Occurring early in the project life cycle, it is perhaps one of the most critical undertakings performed by clients, the effectiveness of which is directly related to project success and the achievement of specified objectives [2,13,17]. The environment for making judgments about suppliers and their ability to deliver is complex, comprising high levels of ambiguity and uncertainty, competing stakeholder values and complicated relationships as a result of multiple conflicting objectives [10,15,21,28]. Further complications arise in identifying suitable and relevant criteria and assigning * Corresponding author. Address: University of New South Wales, Faculty of Engineering, Barker Street, Kensington, 2052 New South Wales, Sydney, Australia. Tel.: +61 2 98051770. E-mail address: [email protected] (D.J. Watt).

0263-7863/$36.00 Ó 2009 Elsevier Ltd and IPMA. All rights reserved. doi:10.1016/j.ijproman.2009.04.003

appropriate weights, all of which are likely to vary as a function of many factors, least of which are the organisational objectives and experience of the evaluator. Given the complexities and underlying issues surrounding contractor selection, and the variety of criteria available, how then do clients choose suppliers and what is the relationship between the criteria used in an evaluation? Which criteria influence choice? Is price a more important criterion than experience, capability, expertise, or performance? Does the relative importance vary as a function of industry, position, experience or project complexity? These questions form the basis of our continuing research to investigate which factors influence the actual choice of a contractor for major projects and the relative importance of the criteria used. Despite its importance, this aspect of contractor selection remains largely unexplored, as evidenced by the very few studies reported. The importance of the criteria used to evaluate and select contractors or suppliers has been examined under various industrial purchasing situations. These include

52

D.J. Watt et al. / International Journal of Project Management 28 (2010) 51–60

the supply of professional management services and procurement of capital equipment and systems, through to the delivery of large scale projects [2,4,6,7,9,10,12,13]. Criteria included those in which evaluators could gauge contractors and their likely performance across key project dimensions; relevant experience, track record, quality, expertise, capability, cost, safety record, and capacity to name a few. In terms of the their importance in evaluating and selecting contractors, these studies showed that no individual criteria or group of criteria are consistently reported as being more important than others. Dempsey’s [6] study on vendor selection supported Dickson’s [7] earlier research that investigated the purchasing behaviour of managers in selecting suppliers. Together, cost, quality and delivery performance were identified as the most important criteria used during the evaluation and selection process. Similar studies by Cardozo and Cagley [4] and, Hakansson and Wootz [9] investigated the relative importance of service or product quality, performance in delivery and quality criteria, and others to select suppliers for the delivery of industrial equipment and services. The results, whilst reporting from different industries were consistent in that quality of product or service was deemed the most important criteria in the selection of contractors. Cost was considered amongst the least important. In the study by Gustin et al. [8], flexibility and ease of use were identified as the most important criteria used in the selection of contractors to supply an Integrated Logistics Information system. Whilst cost was considered moderately important, it appeared that operability and adaptability were of primary importance under this scenario. Proverbs et al. [22] sought to establish the relative importance of criteria used in the supply of concrete for large construction projects. Results indicated that cost, quality, speed of production and quantity of concrete required were considered, in order of importance the four most important criteria. The two least important identified were, respectively operator/labour availability followed by company practice. Hatush and Skitmore [11] investigated the perceived importance of 20 criteria and the relationship between project success factors in terms of time, cost and quality. From this, past failures, financial status, along with management knowledge and capability were identified as the most important contributors to project success. Despite the comprehensive nature of these studies, none except that provided by Hensher et al. [12] studied how clients actually choose contractors. All relied on attitudinal surveys in which respondents were asked to directly and independently rate the perceived importance of specified criteria. Ranking studies, whilst useful in identifying relevant criteria do not represent an actual tender situation. The iterative nature and mechanistic process in which respondents consider and rank individual criteria within a defined group provides little insight into the decision making behaviour. An actual choice of contractor requires evaluators to consider each contractor simultaneously as a function of all specified criteria and their assigned weightings.

Our research uses a Discrete Choice Experiment (DCE) similar to that used by Burge et al. [3] Crouch and Louviere [5], Hensher et al. [12] in which respondents simultaneously evaluate the characteristics of contractors as a function of the level or value assigned to individual criterion. The main advantage of the approach is that respondents do not rate the importance of specified criteria directly. Rather, each alternative within a set is considered wholistically. The structure of the experiment is such that no individual tenderer (alternative) dominated across all criteria, necessitating respondents to make conscious trade-offs. The paper is structured as follows. The next section sets out the research design methodology, along with an overview of experimental design and its application to this study. Results are then presented along with an analysis and discussion of the relative importance of the criteria used in evaluating contractors. The final section summarises and concludes the article, along with suggestions for future research. 2. Research design methodology We established an empirical study to investigate the contractor evaluation and selection process for the delivery of projects and the importance placed on criteria used. Previous studies on contractor selection tended to be industry specific, with a particular emphasis on construction. Acknowledging that contractor selection criteria and their importance vary as function of organisational objectives and industry sectors, it was considered appropriate to include a cross-section of industries so as to gain a greater appreciation of the factors that influence an actual choice of contractor. The sample group included Engineering Project Contract Management (EPCM) companies known to have experience in delivering large scale projects or in the provision of management services. The population included several international organisations and Australian companies within the mining and exploration, construction, defence and aerospace, manufacturing and processing and telecommunications sectors. Executives, programme and project managers and engineering managers were identified and contacted by telephone or e-mail seeking their participation in the survey. In all 288 prospective respondents were identified, of which 255 agreed to participate. A questionnaire and covering letter setting out the research objectives and a brief description was distributed between September 2007 and April 2008. Of the 255 respondents who agreed to participate, and after some follow up, 222 completed surveys were returned, giving a response rate around 87%. Respondents, when initially contacted, expressed a genuine interest in the research topic, which is believed to be the main factor contributing to such an unusually high response rate. The survey instrument comprised two main sections. The first asked respondents to describe their organisation and the industry sector within which they work, along with

D.J. Watt et al. / International Journal of Project Management 28 (2010) 51–60

their experience, role, and characteristics of projects they had previously worked on over the past ten years. In addition, respondents were also asked to describe a previous project in terms of budget, schedule, industry and complexity. This served as a reference project for the Discrete Choice Experiment (second) component of the study. It ensured the experiment was context-dependent relative to each respondent and that results could be generalised. The Discrete Choice Experiment contained 1 block of 16 scenarios, or choice sets, each comprising three alternatives that represented a typical tender evaluation outcome. Each alternative was defined in terms of eight criteria (attributes) identified from research undertaken by Watt et al. [28], plus tendered price, giving a total of nine attributes. Prior to implementing the survey, a pilot study was conducted to ensure clarity and understanding of the questionnaire, comprehension of the Discrete Choice task and to suggest improvements where necessary. Fifteen senior managers from the construction, defence and mining industries were interviewed all of who suggested minor formatting changes to enhance clarity. Two (2) respondents suggested including definitions for each of the nine (9) criteria to remove any ambiguity. Average time to complete the survey was approximately 30 min with respondents indicating no difficulty in performing the task. Analysis and modelling utilised the Multinomial Logit based on the technique of maximum likelihood estimation as described in Louviere and Timmermans [18] and McFadden [20]. Estimates were initially calculated for all levels of all criteria relative to their lowest (base) level and the model log-likelihood recorded. The process was repeated a further nine times after removing, in turn, one criterion and all its associated levels. This provided a common metric where the difference in log-likelihoods between the full model (all criteria included) and those absent a given criteria, quantified the relative effect of each attribute on the dependent variable, choice. The following section outlines and describes the development and construction of the experimental design undertaken for this study. 2.1. Discrete choice experiments Discrete Choice Experiments provide an effective means to investigate the factors (criteria) that effect client’s choice of contractor. It is an elicitation technique that assists in understanding how individuals value different criteria (attributes) applicable to the evaluation and selection of contractors. For details on the theory and background of Discrete Choice Experiments and common models, refer to Louviere et al. [19]. Derived from mathematical psychology and largely used within the disciplines of marketing and transportation, their use is increasing to other areas where decision-makers are required to determine the relative value of various characteristics of services and/or products [3,5]. Alternatives are described in terms of multiple criteria in which the level of each varies across two or

53

more alternatives. Choice sets containing alternatives are presented to respondents who then identify the preferred alternative within a given choice set. Respondent data are captured and parameters determined through quantifying the joint effect of two or more attributes, in our case evaluation criteria, on the dependent variable, choice. Of the parameter estimation techniques available to process Discrete Choice Experiment data, Maximum Likelihood Estimation (MLE) is the most common [19]. A Discrete Choice Experiment involves several steps, commencing with (1) identification of relevant attributes and levels, or values, which are considered a priori that influence choice. The remaining steps in a Discrete Choice Experiment are (2) development of statistical designs and questionnaire, (3) presentation and data collection, and (4) processing, modelling and analysis. Due to availability and technical support, SAS software was used to generate the randomised experimental design for this study and analysis of response data. Kuhfeld [16] provides a comprehensive reference on experimental design techniques and several applied examples using SAS Macros. A key consideration with experimental designs and Discrete Choice Experiments is the number of criteria and levels assigned. As either increase, so to does the complexity and size (number of combinations) of the design, exponentially, as do the number of alternatives, choice sets, or respondents required. As such, designs are often constrained to balance the requirements for estimability, and the use of large designs, with those of practical smaller designs. That is, the design must be sufficiently large to establish parameter estimates for the most complicated model anticipated, inclusive of any defined interactions, yet small enough to minimise the number of choice sets required. For studies involving many criteria and levels, fractional factorial techniques are employed to restrict the number of combinations and subsequent choice sets to a proportion (fraction) of that required for a full factorial design. It enables the development of highly efficient designs without compromising the response data and model estimates, whilst lessening the cognitive burden placed on respondents, since less choice sets are presented. In terms of the number of choice sets, Adamowicz et al. [1] and Louviere and Timmermans [18] recommend between 8 and 24. Relevant criteria were identified from previous research undertaken by Watt et al. [28]. The basis of that research was a comprehensive literature review combined with an industry survey which identified eight principal criteria. Although cost (tendered price) was not identified as a principal criterion, it was included in this study. Various numbers of levels and types were trialled, including all numerical values (weights), all qualitative descriptors, or combinations of both. Numerical values (weights) were preferred, but not possible due to difficulties in constructing a suitable randomised design that ensured the sum of the normalised weights across all criteria within a given choice set would always equate to unity. Consequently,

54

D.J. Watt et al. / International Journal of Project Management 28 (2010) 51–60

Table 1 Design structure – criteria (attributes) and defined levels. Criteria/attribute

Level 1

Level 2

Level 3

Level 4

Organisational experience Project management expertise Tendered price

10 years

0 1 1 1

– 0.19574 0.29254 0.31035

– 0.06588 0.07152 0.06271

– 8.8275 16.7325 24.4963

– 0.0030...