The Client

Our client, ACME Learning (pseudonym), is a digital learning agency that creates custom training programs and technologies for learning and performance support. Launched in 1981, ACME creates solutions for corporate brand training, onboarding simulations, comprehensive training content platforms, and beyond. 

Project teams at ACME are made up of dedicated instructional designers and a flex team of writers, graphic artists, and programmers (see Figure 1). Because ACME is a fast-paced agency that works across multiple industries and clients, their instructional designers (IDs) can work on numerous projects of various sizes and scopes. ACME expects IDs to take ownership of a project from start to finish from the time of hiring. They work closely with writers, artists, project managers, and consultants to develop custom solutions. The responsibilities of the ID include: gathering learning content, creating storyboards, collaborating on design solutions, writing instructional materials, and meeting timeline and quality expectations. 

Our client for this needs assessment was the Chief Learning Officer (CLO) at ACME. In addition to the CLO, there is a Manager for Instructional Designers (ID Manager) who was a key contact for data collection. As of the time of this needs assessment, there were 14 IDs within the organization. IDs are a group under the CLO that includes Instructional Writers, Proposal Team, and Performance Consultants. See Figure 2 for staff members and organization chart. For this needs assessment, our team interacted primarily with the CLO, ID Manager, and a selection of employees within the ID team.

The ChallenGE

ACME Learning was experiencing an increase in turnovers within the Instructional Design department. Since 2018, ACME had seen tenure decrease from 1.7 years to the historic low of 1.5 in 2020 and was averaging 44% turnover in that same time, with a high of 53% in 2020. ACME was concerned about employee morale and the quality of products they could produce, given the high turnover rate. 

In an unstructured interview held with the client, the client stated that up to 65% of the work came from repeat customers, so the company must be able to continue to show its value through its instructional design expertise. They must meet or exceed customer expectations. And because projects are high visibility, they need to have a “wow” factor. Meeting the scope, budget, and timelines set for each project was a critical company objective. Decreasing the turnover rate of instructional designers was critical to ACME’s organizational goal to provide the highest quality of training solutions to clients, both repeat and new. The ability of the company to achieve each of its organizational objectives is disrupted when instructional designers leave in the middle of a project, when teams are short-staffed, and when instructional designers are new and still learning ACMEs processes

Additionally, low retention rates result in low productivity, decreased employee satisfaction, suffering project quality, and overall lost profits. A conservative estimation of the cost of losing a worker in the U.S. is $15,000 (Mahan, Nelms, Bearden, & Pearce, 2020, p. 9). Other estimates have found that replacing a salaried employee costs closer to 33% of the employee’s salary (Sears, Nelms, & Mahan, 2018, p. 9). The most recent data estimates an instructional designer’s salary to be $80,182, meaning replacement costs to the company could be as closer to $26,460 per employee (Zip Recruiter, 2021). Companies across the globe look to increase the tenure of employees by addressing reasons for employees resigning. In the year 2019, it was estimated that 76.8% of turnover was preventable by the employer (Mahan et al., 2019, p. 14). 

ACME had an opportunity to address performance issues that affected retention rates to increase project quality, client satisfaction, employee motivation, and profits. Additionally, doing so could positively impact the tenure of instructional designers, meeting ACME’s goal of 2+ years of tenure for instructional designers.

Gap Analysis

Following Van Tiem’s Performance Improvement HPT Model, we sought to complete an organizational analysis and an environmental analysis. The organizational analysis studies the company’s espoused vision of the future, its values, and its goals. If an intervention (or any change, really) does not align itself to the organizational culture and goals, it is doomed to fail. This analysis helps identify the desired performance (where the organization wants to be and where it should be). The environmental analysis (considering the world, workplace, work, and worker) helped the team identify the actual performance (the reality, what was really happening).

Data Collection

The team incorporated several data collection methods during each phase of our analysis. 

• Exit interviews, turnover numbers, job descriptions, interview selection criteria, onboarding materials, on-the-job training materials, comp data, and online feedback were provided by the client and reviewed by the team. 

• The team sought out industry retention numbers and average industry wages from literature reviews.

• The team gathered elicit data by conducting semi-structured individual interviews with current IDs and ID managers.

Comparing the results of the organizational and environmental analyses, we developed a gap analysis statement to compare the actual performance to the desired performance (see Table 1).

Table 1 Gap Analysis

Cause Analysis

The next step in Van Tiem’s Performance Improvement HPT Model is cause analysis. We utilized the Behavior Engineering Model (BEM) created by Thomas Gilbert in 1978 and updated by Roger Chevalier in 2003. This tool is a systematic way to evaluate both the environmental and individual causes of performance gaps. The decision to use this model was based on early data sets that indicated the performance gap might be both environmental and individual. The model allows analysts to investigate both areas within the organization for a more holistic understanding of where gaps may exist. Using the updated Behavioral Engineering Model (Chevalier, 2008) as a guide, interview questions were created, and individual interviews were conducted with employees. Interviews were transcribed and coded based on keywords from the BEM. Respondent data was also triangulated with extant data provided by the client, data pulled from literature reviews, and online web research. After confirming the performance gaps, we determined multiple causes. The results of this analysis can be found in Table 2.

Table 2 Cause Analysis Findings Based on Chevalier’s Updated BEM

A synopsis of the environment and individual findings indicated themes emerging in all three areas of the Environment: information, resources, and incentives. Instructional designers didn’t always know what the job was about when they were interviewed and hired for the position because they come from varying educational degrees and experiences, most of which do not include a background in instructional design. The tools and resources for them to competently do their job were not readily available to them, and the organization had not presented any incentives for them to stay with the organization for anything other than building experience on their resume. These environmental factors come together to create low morale and decreased loyalty to the organization.

The data further indicated some gaps exist in all three areas of the individual section of the BEM; however, the team chose to primarily focus on knowledge/skills as it had the most consistent data indicating a gap existed. The employees, although highly prepared for the workforce based on experience and education, were not always given the proper training, tools, or support to perform the job well. This often left them feeling frustrated and unsupported.

Intervention Selection

The cause analysis indicated connection points between all factors, but a “one size fits all” solution to close the gaps did not exist, so the team narrowed in on interventions that could have the greatest impact on the performance issue with the least amount of effort from the organization.

The team used Van Tiem, Moseley, and Dessinger’s (2012) Intervention Selection Process, including the Survey Phase and Selection Phase (p. 200). 

Survey Phase

Using the results of our cause analysis, our Survey Phase explored Van Tiem et al.’s (2012) Intervention Selector (pp. 203-206, 213-241) and Hale’s (2006) Intervention Families (pp. 178-189). Van Tiem et al's (2012) model was chosen because of its ease of use and its depth of open-ended suggestions. Hale (2006) was also considered because of its concrete list of specific interventions. Because our cause analysis revealed numerous profound causes, we sought many examples of interventions to explore all solutions. Our cause analysis led us to consider nine different intervention families from Van Tiem et al. (2012) and seven families from Hale (2006). The results of this analysis can be found in Table 3a and Table 3b.

Selection Phase

After narrowing down potential intervention families, we moved on to the selection phase in which, using both tools again, we brainstormed interventions that could fit into the categories outlined in the tables above. We then chose specific interventions from the list we wanted to present to the client for further analysis. Our focus was on finding the high-value, easily attainable solutions that we could leverage in the short-term and also offer them some high-value, harder-to-accomplish interventions that may take more money, time, and commitment to implement. The results of this analysis can be found in Table 4.

Table 4 Interventions Considered

Feasibility Analysis

For the feasibility analysis, we used Watkins' (2012) Weighted Multicriteria Analysis model (p. 179). This was chosen because it systematically assigns ratings to interventions allowing practitioners to see which is a better fit for the organization to address the performance issue. Once the cause analysis findings were discussed with the client, they chose criteria that an ideal intervention should meet and weighted them one through four accordingly, one being the least important criteria and four being the most important. These criteria are displayed in the top row of the table. We then discussed each proposed intervention in detail, assigning quantitative value to how well each intervention met each criterion. The results of this analysis can be found in Table 5, with the top-scoring interventions bolded.

Table 5 Weighted Multicriteria Intervention Analysis

Recommended Solutions

Working closely with our clients to gather their input, opinions, and expertise, we were able to narrowly consider four interventions that rated favorably during our analysis phase,  - 1) provide transparent incentives, 2) implement a formal performance management system, 3) create and organize materials and resources, and 4) create consistent on-the-job training. These four interventions can be applied on their own, or in any combination, and allow the client to tackle the performance gap within short, medium, and long-term timelines. 

Reflection

While this project did follow and systematic and systemic approach to analyzing and assessing performance issues, we did document and track risks and issues we faced along the way.

We recognize that the project would have been better served without the time constraints of the semester. The issues at play are much larger than a group of students could feasibly tackle. With more time devoted to this project, a team would have been able to gather additional data through surveys or additional interviews throughout the organization. It was evident that this was much bigger than was originally anticipated.

During data gathering, we were limited to a small sample of the ID team. For the data to be more thorough, we would have ideally had access to more employees across the organization, including former employees. Our client had no formal exit interview process, so we had to rely on piece-mailed comments from other extant data. And because we only had the time to conduct interviews, the lack of objective data from surveys or other tools made it difficult to provide objective interventions.

Finally, my largest takeaway from this project was how important setting expectations with the client was, and how vital it is to keep an open vein of communication. Our client was taken aback by our suggestions and became somewhat defensive. I know now the importance of keeping clients informed about emerging themes and findings so that they are not surprised by any development.