Unit 10 - Evaluation of Learning and Development PDF

Preview

Summary

Download Unit 10 - Evaluation of Learning and Development PDF

Description

INTRODUCTION TO HUMAN RESOURCES MANAGEMENT. UNIT 1: Evaluation of Learning and Development

Introduction In Unit 8, you were introduced to the training cycle. To remind you, this consists of four stages which are represented in the diagram below:

Foot and Hook (2011:302) are of the view that “evaluation is an extremely important stage in the learning and talent development cycle, and one that is often neglected by organisations…if no evaluation of learning and development is carried out at all then the organisation does not know whether it has been enjoyed or been successful…so it may have wasted money and resources” (Foot and Hook, 2011:302). Evaluation can be defined as being “any attempt to obtain information (feedback) on the effects of the training/learning programme and to assess the value of the training/learning in the light of that information” (Bee and Bee, 2003:135) cited in Marchington and Wilkinson, 2005:250). Armstrong (2012:299) highlights that “in essence, [evaluation] is the comparison of objectives with outcomes to answer the question of how far the event has achieved its purpose”. A useful and straightforward definition is provided by Gibb (2002) who suggests that “Evaluation is defined as the activity of determining the merit, worth or value of things” (Gibb, 2002). In short, evaluation is concerned with establishing the success or otherwise of development

activities, and with assessing whether the associated benefits justify the investment (Stewart, 1999) There are various questions that need to be answered in order to determine the most suitable evaluation strategy: a. b. c. d. e.

Why evaluate? What to evaluate? Who should evaluate? When to evaluate? How to evaluate?

These issues will be addressed in turn within this unit.

Learning Outcomes On completion of this unit you should be able to:    

Explain the purposes of evaluating training, using key evaluation models Examine the issues affecting effective evaluation of training Explain the concepts of return on investment and contribution of evaluation Plan the evaluation process for learning and development.

10.1 Processes Associated with the Review of Learning and Development As discussed, training evaluation is about measuring and analysing various aspects of L&D activities, with a view to determining the effectiveness and value of the activities. It also includes suggestions on how they could be improved. Two other separate processes provide inputs into the overall evaluation report. These are discussed below. 10.1.1 Validation This process aims to establish the truth of, confirm or corroborate findings and is focused on establishing whether the learning activity achieved its stated objectives. Did the training do what it said it would? Evaluation includes validation i.e. the product of the validation part of the process feeds into the wider evaluation activity as this can include other factors like value for money and whether resources were used effectively. 10.1.2 Assessment

In a training context, assessment is a term generally applied to individual progress and achievement. It involves the measuring, by a range of methods, of how well someone can perform an activity or demonstrate knowledge, usually against the pre-set criteria (i.e. objectives of the training). Assessment can get undertaken at various stages throughout the learning programme. It can highlight learning gaps and helps provide evidence for the evaluation process which is concerned with establishing if training delivery has been effective.

10.2 Why Evaluate - What is the Purpose of Evaluation? Easterby-Smith (1994) suggests that there are four main strands for the purposes of training evaluation:  

 

Proving – that the training worked or had measurable impact in itself Controlling – for example, the time needed for training courses, access to costly off-the-job programmes, consistency or compliance requirements Improving – for example, the training, trainers, course content and arrangements etc. Reinforcing – using evaluation efforts as a deliberate contribution to the learning process itself.

The CIPD Learning and Development Survey (2015) indicated that evaluations are mostly used to update the L&D intervention and review the delivery method with “more than half always or frequently (sharing) the results of evaluations internally, while just under a fifth do so externally” (as part of an external benchmarking exercise, for example).

10.3 What to Evaluate Once the purpose of the evaluation is established, the next stage is to determine what should be evaluated. Warr et al (1970) as cited in Harrison (2005), designed a four stage framework to facilitate this review, known as CIRO. Context – This involves establishing such things as analysis, culture, structure, support Inputs – Identifying the resources and how they were used to achieve the learning objectives Reactions – Discovering learners reactions to the event

Outcomes – Assessing what actually happened as a result of the event Some other evaluation approaches, as reviewed by Tamkin et al(2002), include Pulley (1994) who describes responsive evaluation which focuses on what decision-makers in the organisation would like to know, and how this might be met i.e. the senior team/board of management and the kinds of data they require for forward planning. Newby (1992) describes a context evaluation approach, which focuses on the different contexts of evaluation — evaluation activities need to take account of the environment and contextual features of the development. Preskill and Torres (1999) describe evaluative enquiry, which approaches evaluation as a learning experience in itself, using dialogue, reflection, and challenge to distil learning opportunities and create an environment where enquiry skills might be used. 10.3.1 Topics of Evaluation Linked to The Processes Involved Traditionally, evaluation approaches have mostly focused on the process aspects of the learning and development i.e. looking at the aspects of the design and the supporting activities. These include aspects such as: 

The Trainers

Here, the competency of the trainers is reviewed. Do the trainers have the right knowledge, skill and attitudes to deliver the session? Another aspect is around the professionalism of the trainers, are the trainers prepared and punctual? Are the trainers free from bias, treating all trainees equally and fairly? An important element is rapport which looks at whether the trainers establish a rapport with the trainees. 

The Resources

Here elements such as the quality of the materials comes under scrutiny. Do the materials require to be revised, updated or replaced? In terms of audio visual support, questions focus on whether the AV support was appropriate. For example, the sound quality of speakers if using video. Are there appropriate sized screens for the presentation (s)? Another area to consider is whether the facilities were suitable in terms of size, comfort level and quality.

The issue of cost effectiveness of resources used is also important. Have the resources been deployed cost effectively? What savings might have been made? 

The Design

There are a wide range of issues to consider here as the original design rationale is in effect re-visited. These considerations include looking at the range of opportunities for learning in the design and asking if these were sufficient. In terms of the choice of methods, questions such as the suitability of the selected methods i.e. were they the right ones to choose. There needs to be questions around the mix of methods and the training sequence and timings too. The extent that the location suited the choice of method is another consideration. Another area to review is around content. Here, issues relate to how appropriate was the content and whether it was pitched at the right level for the trainees needs to be reviewed. Of course, there needs to be a focus on the cost effectiveness of the design with questions being asked about whether other choices might have improved this. 

The Objectives

As part of the evaluation process, it is important also to revisit the original performance objectives and review these asking if they were the right ones in the first place and whether they were pitched appropriately. This validation of objectives is often missed as the focus is often more on were the objectives achieved. It is important though to ask such questions as were they at the right standard? Were they derived from a systematic identification of training needs and were the needs identified correctly? A more usual area of focus is to ask were the organisation’s objectives achieved and were the trainee’s objectives achieved. 

Administration & support

In this section, the evaluation process looks at the way that the opportunities for training were communicated as it reviews the effectiveness of these activities. Questions such as was the administration and support effective and indeed was it sufficient are usual in this context.

Clearly, an important aspect of the review is identifying areas which could be improved. It is important to examine the processes whereby training records are kept to review their accuracy and ask if they are fit for purpose. It is important to ensure that any training records kept are compliant with the Date Protection Act. 

The Workplace

Here, the focus is on the learning being transferred to the workplace and checking if the learning was at the required level. Questions such as “Was it the right learning?” and “Have performance standards improved?” are also relevant.



Trainees

An important element to consider are the factors affecting trainees. Topics such as “Were the right trainees selected?”, “Were the numbers appropriate?” and ‘’Were the trainees appropriately prepared?’’ are all relevant here. Level 3 evaluation is concerned with reviewing if trainees had opportunities to put training into practice and if their work performance has improved. Most often, it is only the reaction level that is used where trainees are asked what they thought about the training. 

Classroom management

This is where the role of trainer in establishing an effective learning environment is reviewed. This is concerned with questions such as “Did the trainer establish rapport between the trainees? Were problems between trainees identified and resolved effectively? Considerations such as health and safety and equal access are also important. An obvious area of concern is around the trainer’s management of time and resources. Were these managed appropriately and what savings might be made in the future are important considerations as well. 

Organisation

Here, the impact overall (Level 4) is considered. Questions such as ‘has efficiency and effectiveness improved?’ are important in this content. Looking at the impact on productivity is part of the process.

In addition, there should be a consideration of what other effects the training has had on organisational performance.

10.3.2 The RAM Model (CIPD, 2014) The CIPD (2014) developed an approach to learning evaluation known as RAM (relevance, alignment, measurement) which focuses “on the need for: Relevance: How existing or planned training provision will meet new opportunities and challenges for the business. Alignment: If the plan is to deliver a changed L&D offer , it is critical for HR to talk to key managers and other stakeholders about what they are seeking to deliver and how the function can help them achieve it. It is also important to ensure that L&D is aligned to other key strategies such as reward, organisational development, engagement and other aspects of the management of human resources. Alignment with organisational strategy and its marketing and finance strategies and other dimensions of corporate strategy gives focus, purpose and relevance to L&D. Measurement: It is also critical that the HR function effectively and consistently measures and evaluates its interventions. It may be helpful to use a mixture of evaluation methods such as return on investment (ROI) and broader measures of expected change and improvement such as return on expectation, and to link L&D outcomes to key performance indicators (KPIs)”. They maintain that this approach is useful in terms of “maintaining a focus on the outcome, rather than the process itself”. 10.3.3 Return on Investment (ROI) Every activity that takes place in an organisation involves a cost, such as staff, wages, overheads etc. Organisations are either designed to make a profit or operate within the constraints of a budget, consequently it is important that all costs are managed and appreciated. Every activity initiated with an organisation should contribute towards the overall objectives of the business. Hence, every pound invested in training activity, should theoretically produce its equal worth of return in terms of profit or reduced costs. ROI is a method that organisations use to gauge whether the investment into L&D activities is actually contributing to the organisation return. It is the financial or economic benefit that is attributable to the learning intervention in relation to the cost of the investment in learning programmes.

To arrive at the ROI, the costs of providing the training are calculated and measured against the financial impacts (benefits) of the training. Return on Investment = Financial benefit from training – Total cost of training x 100 (%)

Total cost of training

To show how this looks in practice, take the example of a Financial Services Call Centre with a training programme for their Customer Sales Advisers. Over the past 12 months, 14 courses were delivered at a total cost of £357,000. During this period, the advisers who completed the learning activities produced revenue from sales achieved of £628,408. The ROI is calculated as

£628,408 - £357,000 x 100 = 76.02 £357,000 This shows that there was a return on investment of 76%. This training was clearly worthwhile and directly contributed to increased turnover for the company. Despite the appeal of this approach as it provides a framework for quantifying training investment in monetary terms, Wall and Wood (2005) argued there remains a lack of understanding as to the true contribution of training to business performance. Evidence for the relationship between training and financial performance is not well proven and does not go beyond reporting positive association between participation in training and a company’s financial performance (Fleetwood and Hesketh, 2006). Such evidence does not show how much profitable companies invest in training, or what type of training activities they pursue. Logically, the context of the firm and its competitive strategy should be a key factor in detrmining the true benefits of investment in training in relation to performance (Ashton & Sung, 2006; Keep et al., 2006).

10.4 Who Should Evaluate Leslie Rae coined the expression ‘The Evaluation Quintet’ to cover the five groups of people that he believes should be involved in evaluating training.

Senior Management They help structure a culture where evaluation is seen as an essential activity – encouraging managers to allocate an appropriate resource for evaluating training. They will also be instrumental in driving change and hopefully see evaluation results as a useful source of information used to support and guide these. It is important to present information to Senior Management in a way that enables them to quickly assimilate results and make the appropriate strategic connections. Often information dashboards or informatics are used to help with this. The Training Manager This is the key role in ensuring that training evaluation occurs within the organisation and should be responsible for the design of any training evaluation strategy. Training managers will also need to liaise with senior managers over evaluation results as well as engaging with line managers on how they can support and inform the evaluation process. The Line Manager Is an essential partner in order to conduct evaluation and critically review the impact through transfer of learning, for example. A key stakeholder, line managers should be encouraged to participate in determining what type of evaluation activity occurs for different types of training. Their support in terms of encouraging their employees to participate is also crucial and should be seen as part of

creating a culture where evaluation is seen as routine as part of the strategic management of the organisation. The Trainer Individual trainers would normally be involved in discussions around the methodologies that have been used to evaluate the training that they deliver. Whilst in the training environment, trainers should aim to create an open environment where participants feel able to provide fair and honest feedback. They will also be integral to conducting the initial review of reactions (often via a questionnaire). Trainers will need to be provided with the results of the evaluation so that they can be involved in developing and implementing any necessary action plans. The Learner Eason (2015) suggests that the learner, as well as having to learn, also “needs to contribute to the evaluation process. Learners need to provide their initial feedback on the training, providing their views on the relevance and opportunity to transfer the learning into the workplace after the event. Without this interaction the process will lack the foundation data upon which the evaluation results will be formed”.

10.5 When to Evaluate A key model in this area that helps us to answer the question about ‘when to evaluate’ is that of Kirkpatrick (1967) as cited in Marchington and Wilkinson (2012). Kirkpatrick identified four levels of evaluation: Reaction (Level 1) – this is often used during or at the end of a training programme to establish the views of the learners; it has been colloquially called the ‘happy sheet’. Immediate (Level 2) – at this level there is an attempt to evaluate learning outcomes achieved in terms of knowledge, skills and attitudes; techniques include assessments, exams projects, case studies and discussions.

Intermediate (Level 3) – at this level evaluation is related to the impact of the training/learning on job performance, evaluating the transfer of the learning. Evaluation at this level is less common and harder as it is difficult to isolate the impact of training from other factors that may affect job performance. Ultimate (Level 4) – at this level the evaluation is attempting to assess the impact of training on departmental and organisational performance. Once it is established what is going to be evaluated and the levels of evaluation that are to be used, decisions need to be made regarding who will conduct the different levels of evaluation, when this will occur and how the evaluation will be carried out. Although monitoring and evaluation are seen as the final stage in the cycle this should be planned at the outset and agreement sought as to the purpose and how and when they should be carried out. With each of these four stages, there are drawbacks. For example, Kirkpatrick (1977) himself stated that Stage 1 evaluations are dependent on the honesty of the learners who complete the questionnaire and whether they feel comfortable in being able to tell the truth. One approach to overcome this is to omit names encouraging employees to complete them anonymously. This does reduce the fear of any comebacks but reduces the potential for the trainer to provide additional support for learners where additional learning needs arise or in situations where the training does not meet the anticipated need. There is also the criticism that the evaluation focuses too heavily on enjoyment rather than on how useful it has been (Hamblin, 1...