White Paper: Networked Interactive Multimedia Training
Table of Contents
1. The Core Requirements of a Successful
Training System
-
1.1 Overview
-
1.2 New training methodologies
-
1.2.1 Accelerated learning
-
1.2.2 Just-in-Time training
-
1.3 Training: the competitive edge
-
1.3.1 The value of human resources
-
1.3.2 Training and profitability
-
1.3.3 Limitations of conventional
training systems
-
1.4 The changing face of training
-
1.4.1 Changes in job descriptions
and requirements
-
1.4.2 Changes in the workforce
-
1.5 Conclusion: The core requirements
of a successful training system
2. The Benefits of Networked Interactive
Multimedia Training
-
2.1 Overview
-
2.2 Why is networked interactive
multimedia training the optimal training solution?
-
2.2.1 Advantages for learners
-
2.2.2 Advantages for the training
department
-
2.2.3 Advantages for management
-
2.3 Summary: The benefits of networked
interactive multimedia training
-
2.3.1 For trainees:
-
2.3.2 For trainers:
-
2.3.3 For management:
3. The Benefits from an ROI Perspective
-
3.1 Overview
-
3.2 Why is a new approach to cost-benefit
analysis necessary?
-
3.3 How can the cost of training
be measured and analyzed?
-
3.3.1 Trainee costs
-
3.3.2 Instructional delivery costs
-
3.3.3 Instructional content costs
-
3.3.4 Management costs
-
3.4 How can the benefits of training
be measured and analyzed?
-
3.5 Conclusion: Evaluating the ROI
on a networked interactive multimedia training system
4. References
1. The Core Requirements
of a Successful Training System
1.1 Overview
In many organizations, formal training remains either physically
isolated from the workplace when it is delivered in classrooms and training
centers, or severely constrained by the inflexibility of textbooks, videocassettes,
and CD-ROMs. Additional limiting factors (such as the need to schedule
access to instruction or instructional media in advance) further widen
the gap between training and day-to-day work processes. These characteristics
are not in line with current business training needs arising from leaner,
flatter organizations, high-speed communications, rapidly changing information,
diffused decision making, and team-based management. Traditional training
systems also fail to reflect current knowledge about the importance of
self-directed learning, Just-in-Time training, and on-the-job instruction
[1].
The high-performance workplace - where increases in productivity
are sustained over time - demands a training system that supports a flexible
'learning' organization which can respond and adapt quickly to change [2].
Training information, like all other information types, must be delivered,
shared, and integrated in a seamless manner. These needs can now be satisfied
due to a recent innovation in server technology which allows for interactive
multimedia training incorporating learner interactivity and high-quality
video to be distributed over a network, providing an opportunity for implementing
effective training systems on an organization-wide scale. This development
means that training need no longer be an isolated and relatively inflexible
entity within an organization; instead, it can become an integral part
of that organization's ability to competitively respond to market change.
In order to assess a training strategy's viability and
maximize the return on training investments, a broad understanding of the
forces which have altered the demands on training is required. This section
of the white paper addresses the core requirements of a successful training
system in today's business environment.
1.2 New training methodologies
New approaches to training have been developed in response
to changes in the workforce and the workplace. These methodologies utilize
developments in psychological and organizational research to improve upon
the effectiveness of instruction and enlarge the range of organizational
needs that training systems can address.
1.2.1 Accelerated learning
Recent research in educational technology [3,4] has identified
several factors that determine the efficiency with which employees acquire
new skills and knowledge. This research indicates that learning can occur
at a greatly accelerated rate as compared with traditional approaches if
a training system possesses certain critical characteristics:
-
A low-stress learning environment: the first aspect
in accelerated learning is stress reduction. People learn more efficiently
when they are relaxed and studies have shown that adults are under less
stress when permitted to learn independently versus in a group setting
[5]. As well, frequent slight disturbances in classroom environments impede
efficiency in the learning process - individual instruction, where all
of the learner's senses can focus on the task at hand, removes this impediment.
-
Multimodal instructional delivery: instruction that
utilizes several sensory channels in parallel - such as images, text, and
sound - harnesses much more of the cognitive capabilities of the learner
than unimodal delivery [6]. When these modes are combined, a higher retention
rate results; although learners retain on average only 20 % of what they
hear, simply adding visual elements to the instruction doubles retention
rates [7]. As well, since details are one of the most important elements
of any effective training program, the visuals that accompany text-based
or oral instruction are crucial because they permit learners to learn by
experience rather than simply by lecture [8].
-
Interactivity: instruction that provides the learner
with the ability to interact with the content rather than passively absorb
large chunks of information stimulates more active learner involvement
and engagement. Research has shown that learners comprehend much more information
when they can practice new skills as they acquire them [9], and can retain
75% of the new information they are exposed to provided that they both
see and hear the information as well as do something with the information
when they first encounter it [10].
When the method of instruction combines these three characteristics,
the synergistic results far exceed those of traditional methods. For example,
reports [11,12] indicate that after a two-week period, only 10-20% of the
information communicated in a classroom is remembered. When learners individually
use computer-based multimedia incorporating sound, high-quality video,
and interactivity, retention rates vary from 50 to 95%. Furthermore, training
time is frequently half that necessary using traditional approaches. Training
systems that possess all three characteristics of accelerated learning
enable employees not only to learn more quickly, but also to remember and
apply more of what they have learned.
1.2.2 Just-in-Time training
Performance problems (and training needs) can arise from
the inability of an employee to perform a specific task at hand (such as
creating a chart in a computer spreadsheet application), or they can be
associated with a more general requirement for skill upgrading that will
be applicable to a broad range of current and future tasks (such as interpersonal
and time management skills). The strength of the match between training
and task is a prime factor in determining the impact of training on current
and future workplace performance. A common constraint on the effectiveness
of classroom education in groups is the disparity between what is communicated
in the classroom to the group as a whole and the actual tasks that are
encountered by each individual employee in the workplace. This constraint
can also reduce the effectiveness of videotapes and CD-ROMs delivering
generic content which employees can only access as a whole rather than
enabling them to quickly identify and receive a single yet critical piece
of information.
New training strategies that are geared towards quickly
solving employees' current performance problems are collectively referred
to as Just-in-Time training. Many companies already utilize aspects of
Just-in-Time training within their organizations:
-
Computer software applications with built-in 'help' screens
provide Just-in-Time support for both novice and experienced users - the
ability to perform unfamiliar operations with the software can be acquired
while using the software itself.
-
Impromptu information exchanges between employees, such as
when one employee asks a more experienced peer for guidance while performing
a task, are an example of informal Just-in-Time training.
Both of these partial solutions suffer from limitations.
Although computer software manufacturers have recognized both the need
for and effectiveness of Just-in-Time online help functions in business
software, and lowered their documentation costs by transferring the majority
of their tutorial information from expensive print-based documents to the
software application itself, employees frequently need more instruction
than the manufacturer provides. As well, this solution provides no support
for employees with performance problems that are not software-related.
Although on-the-job training with a peer is a possible alternate approach,
it can result in costly interruptions, distractions, and poor work habits.
In addition, many employees with expertise are unable to effectively transmit
their knowledge and skills to others. From a cost standpoint, this approach
can be as, or more, expensive than classroom training, due to the many
hidden costs [13]. Another fundamental drawback of this informal approach
is that it creates variance in business policy and procedures, which can
easily undermine corporate efficiency. An organization that is striving
towards a 'best practice' model, wherein optimal processes are executed
by all employees, cannot rely on a training solution that does not support
the institutionalization of 'best practice' within both the training environment
and the workplace.
New training technologies utilizing computer networks
can rapidly deliver effective training content directly to the desktop
or work environment, and the selection of the training content can be made
quickly by employees themselves in response to their individualized needs
and the tasks at hand. These technologies create training systems that
can systematically address both Just-in-Time and conventional training
within one integrated solution.
1.3 Training: the competitive
edge
1.3.1 The value of human
resources
"As every advanced economy becomes global, a nation's most
important competitive asset becomes the skills and cumulative learning
of its workforce." (Reich, Harvard Business Review) [14]
"Never before has there been such a focus on the human
resources in our society and never before have we so seriously questioned
the performance of our learning institutions, training programs in the
workplace and the basic skills required of individuals." (Geis, As Training
Moves Toward The Next Decade) [15]
In recent years, corporate strategy for maximizing productivity
has focused on reducing direct labor costs through downsizing, increased
automation, and the reengineering of inefficient processes. As a result
of these interventions, individual employees are now required to quickly
accomplish tasks that previously required groups of workers and lengthy
time periods. These paradigm shifts in organizational structure and function,
however, have also had an effect on the amount and quality of information
employees must have access to in order to perform effectively. Overhead
costs - the costs associated with such processes as problem-solving and
organizational communication of information - are where labor and related
costs have become concentrated. The importance of these 'overhead' processes
has also been heightened by the rapidly changing market environment that
corporations must now compete within; having the best information and making
optimal decisions based on this information are the two prerequisites for
profitability and survival. In order to reduce overhead costs and be able
to respond quickly to market changes, an organization must have employees
with refined and flexible skill bases. These realities have forced a reevaluation
of training's role and importance in the workplace.
1.3.2 Training and profitability
An efficient and effective formal training system is now
a mission-critical aspect of a corporation's overall strategy, and this
development is reflected in industry budgeting trends. A recent survey
of 444 private-sector organizations [16] determined that training budgets
have increased net of inflation in past years, and a substantial mean increase
of 11% was expected in the future by these firms. Even in sectors hardest
hit by the economic slowdown during the early 1990's, training was not
a primary target for budgetary cutbacks [17]. Given the prevalence of cost-cutting
measures across all areas of today's corporate environment, it is noteworthy
that expenditures on training are actually on the rise. This counter trend
indicates that companies perceive a concrete and justifiable value in developing
their internal human resources. This perception is not unwarranted:
-
An analysis of 200 U.S. manufacturing businesses [18] indicated
that companies with efficient production systems but no formal training
programs had significantly lower levels of labor productivity and lower
stock market values; training was isolated as the "critical component"
for profitability.
-
Companies in the Top 50 by ROI in Canada [19] spent an average
of 20% more per capita on training than other companies.
1.3.3 Limitations of
conventional training systems
The decision faced by corporations is no longer whether or
not to implement a training system; the decision now is to select the optimal
training system to match organizational needs. All forms of training, as
strategic interventions, possess several generic advantages [20]:
-
Lower overall cost. When compared with other performance
interventions, such as replacing inadequately skilled employees through
automation or the hiring of new staff, training requires a lower investment
that can be spread out over time.
-
Discretion. As training activity isn't visible to
competitors, it can be utilized as part of competitive positioning without
the risk of revealing corporate strategy during a transition period.
-
Ease of adjustment. Training can be altered relatively
quickly as organizational needs change.
-
Improvement of internal relations. Training is an
investment in existing staff, and employees perceive this investment as
an expression of confidence by management.
Conventional training systems, however, based on lecture-oriented
classrooms and textbooks, or even on more modern delivery media such as
videotapes and CD-ROMs, are predicated on an instructional model that is
in many ways insufficient to meet current demands [21]. Without utilizing
new developments in networking technology, these approaches incur several
critical disadvantages:
-
Lengthy time between an intervention and its results.
For example, in technical jobs with steep 'learning curves', conventional
training may be a relatively slow-response intervention compared to hiring
new staff that are already trained.
-
Lost production. While in classrooms, or awaiting
delivery of instructional media, employees are not able to perform their
job functions.
-
Potential for ineffectiveness. Employees may respond
negatively to group-based training that they perceive as ineffective or
irrelevant to their individual work environments.
1.4 The changing face
of training
Although some of the causes underlying the inadequacy of
conventional training systems are the result of inherent limitations, new
factors have also emerged that limit the ability of these systems to effectively
meet the needs of modern organizations. These factors stem from changes
in the nature of the workforce and the workplace.
1.4.1 Changes in job
descriptions and requirements
To reach new levels of productivity and performance, complexity
is needed. For example, office automation enables one administrative assistant
using a personal computer to do the work that previously required four
people: an office manager, a secretary, a clerk typist, and a bookkeeper.
In order to satisfactorily perform these disparate functions at a heightened
level of productivity and performance, the administrative assistant requires
the support of training in both these four skill areas and in the computer
software that supports the work tasks [22]. These types of changes in job
performance requirements have been noted in numerous studies:
-
"Instead of doing simple procedural and predictable tasks,
(employees are becoming increasingly) responsible for inferences, diagnoses,
judgments and decision making, often under severe time pressure" [23].
-
"The major effects of technological change will be a redistribution
of work assignments and a new division of labor...this will in turn significantly
alter the nature of work and the skill structure of employment" [24].
-
"New and qualitatively different jobs will continue to develop
at a rapid pace. Many previously stable elements of the work environment
are rapidly changing" [25].
-
"Many workers receive brief training in the skills and procedures
they are expected to cover. Comprehensive training (in lengthy seminars)
is not the answer, as not only would this consume a great deal of time,
but much of the information would be outdated by the time the worker re-entered
the workplace. If a wide range of topics are covered, months might go by
before employees have a chance to utilize new skills, reducing the likelihood
that they will remember them" [26].
-
"Changes in the workplace will dramatically increase the
need for training because the new work environment will no longer 'naturally'
provide step-like sequences of learning. The loss of the natural ladder
of successively more complex jobs in the work environment has clear implications
for training. The gap to move a worker from A to B is now so much larger,
and involves moving workers to jobs that they have no comprehension of"
[27].
In response to these developments, training systems must
focus on producing multi-skilled employees. The goal of multi-skilling
is to increase employees' flexibility and productivity and thereby allow
both the individual employee and the corporation as a whole to adapt more
quickly to organizational and market changes. For example, a multi-skilled
production worker can fix a broken piece of machinery rather than calling
in someone else to fix it, reducing downtime and increasing productivity
[28]. In today's workplace, however, it is difficult to accurately forecast
all the skills an employee must possess in order to be efficient and effective.
The speed with which employees' roles and responsibilities change requires
a training system that can adapt to these changes fluidly, and deliver
training information that is directly related to the performance of the
employee's present tasks. Effective multi-skilling can only occur as the
product of sustained incremental training interventions that are directly
linked to each individual employee's current needs.
Another essential goal of effective training is the upgrading
of employees' interpersonal skills. These 'soft skills' are increasingly
in demand as effective collaboration and teamwork have become critical
elements to organizational efficiency. The primary reason for loss of employment
has been cited as a lack of these 'soft skills' rather than an inability
to perform any specific skill [29]. Interpersonal skill development is
one of the many areas in which interactive multimedia with high-quality
video footage has had demonstrated success [30].
1.4.2 Changes in the
workforce
Studies have also noted changes in the nature of the workforce:
-
Along with a smaller, post-baby-boom pool of entrants into
the workforce, greater numbers of workers are experiencing increasing job
shifting throughout their lifetime [31].
-
Training systems will therefore have to accommodate "a more
diverse population of trainees than they have dealt with in the past...many
of those entering the workforce, as well as many already in it, will not
have the prerequisite skills for the new jobs that they will be required
to do" [32].
The key trend associated with these changes in the workforce
is that employees will increasingly be life-long learners. The traditional
sequence of formal school-based education followed by a predetermined career
path is no longer the norm. In its place, adults will iterate cycles of
learning and applying skills in response to workplace demands, and continuously
develop personal knowledge bases for the duration of their professional
lives. Employers must support this essential life-long learning process
with a flexible and powerful training system, as employees will not only
increasingly rely on it, they will also increasingly demand it.
1.5 Conclusion: The core
requirements of a successful training system
Limitations on the effectiveness of conventional training
solutions stem from three entrenched beliefs that are no longer in line
with current work environments [33]:
-
"The best training takes place through person-to-person contact."
-
"Training is event based and largely driven by classroom
training."
-
"The primary outputs of a training department are workshops
and training programs."
These three beliefs must be replaced by three new tenets
that reflect the changed demands on, and of, training:
-
The best training delivers knowledge and skills whenever
and wherever they are needed at the lowest cost.
-
Training should be a continuous process, closely associated
with on-the-job performance.
-
Training departments must focus on producing organization-wide
systems that enable users to both obtain specific information quickly and
support more general skill development.
In order to realize these goals, a successful training system
must:
-
Deliver instruction in the most efficient and effective
format for learning and retention. The training system must deliver
multimodal, interactive instruction in a low-stress, individualized learning
environment.
-
Enable Day-One performance by all employees. The training
system must be able to address Just-in-Time training needs, in order to
eliminate lengthy time periods between the occurrence of a performance
problem and the training solution.
-
Institutionalize corporate 'best practice'. The training
system must ensure that employees have continuous access to instruction
and information that enables them to optimize their performances in a manner
consistent with the rest of the organization and its quality standards.
-
Produce multi-skilled employees with strong generalizable
problem-solving and interpersonal skills. The training system must
be able to develop these critical competencies that are required by all
employees in the workplace through interactive instruction that provides
an opportunity to practice these skills in a low-risk environment.
-
Accommodate a heterogeneous employee population. The
training system must support the delivery of instruction that is tailored
to each individual's needs, abilities, and learning style in order to maximize
comprehension and retention.
-
Integrate training information seamlessly into the work
environment. The training system must be an integral part of the work
environment in order to be flexible and responsive to employees' actual
workplace needs, as well as incorporating the most up-to-date information
available within the organization.
-
Maximize Return On Investment, and provide a means to
evaluate the ROI. The training system must utilize the most cost-efficient
training methods available, and produce data that can be used to evaluate
the effectiveness of training and its impact on the bottom line.
2. The Benefits of Networked
Interactive Multimedia Training
2.1 Overview
Networked interactive multimedia training systems utilize
an advanced delivery platform for training information that harnesses the
combined power of three technologies:
Networks: Computer local area networks (LAN) and
wide area networks (WAN) allow the delivery of training and other information
types directly to the desktop workstation. By merging the training environment
with the work environment, the training system can address such organizational
needs as Just-in-Time training, as well as substantially reduce the costs
associated with the delivery of conventional types of training material.
Interactivity: Interactive training software dynamically
reacts to the trainee's actions, allowing for instructional delivery that
is tailored to and optimized for each individual user. It also provides
opportunities for practicing acquired skills within the training system,
which is critical for retention and transfer of skills from training to
the workplace.
Multimedia: Instead of restricting instruction
to primarily unimodal delivery (i.e., textbooks, lectures) with the occasional
audiovisual aid, multimedia-capable training systems can utilize text,
graphics, animation, audio, and high-quality video. This allows the training
content to be delivered in the most effective mode or combination of modes
for comprehending and retaining that type of content. The different learning
styles of trainees can also be accommodated for by using multimodal delivery.
These separate technologies have been developed independent
of one another over the past few decades. Basic forms of interactivity
have been a part of computer-based training since the advent of drill-and-practice
systems in the 1960's. Extensive academic and commercial research and development
has resulted in modern interactive software products, eliminating the wasted
time and other inefficiencies of generic, unresponsive instruction. Multimedia-capable
computer systems are largely the product of the personal computer revolution
that took place during the 1980's, and now both office and home computers
can support audiovisual content. Multimedia has itself evolved to include
high-quality video, which enables it to increase its scope to training
in interpersonal skills as well as complex technical instruction. In the
early 1990's, interactive multimedia training gained widespread acceptance,
despite the limitations of CD-ROMs and other storage media - such as the
need to either physically deliver the CD-ROM to the employee requesting
the training, or else purchase one copy of the software for every employee
who may need access to it. This constrained the impact interactive multimedia
training could have within an organization, as the number of potential
beneficiaries was necessarily limited by the number of copies of the instructional
product and their accessibility. Although computer networks have long been
an intrinsic component of modern corporate information systems, bandwidth
limitations related to the network transmission of multimedia data have
until recently prevented the delivery of interactive multimedia training
to every desktop in an organization. As cost-effective hardware solutions
now exist for interactive multimedia training delivered through existing
LANs and WANs, the merger of these powerful technologies has become a reality,
allowing for the examination of networked interactive multimedia training
systems as a present-day solution.
This section of the white paper addresses the critical
advantages of networked interactive multimedia as a training delivery platform.
2.2 Why is networked
interactive multimedia training the optimal training solution?
2.2.1 Advantages for
learners
Whenever an employee requires training in order to complete
a task or upgrade skills, there are three categories of factors that determine
the effectiveness and efficiency with which a training delivery platform
can satisfy the training requirement. These categories are access to training,
instructional format, and quality of learning.
Access to training refers to the ease with which
an employee can access the exact information required. This involves both
selecting the appropriate training content and receiving/retrieving it.
Networked interactive multimedia training provides critical
advantages for employee access to both Just-in-Time and conventional training
through:
-
Enabling task-driven selection of training content:
network delivery of interactive software allows for non-linear access to
the exact instruction required to solve a performance problem caused by
skill deficiencies.
-
Eliminating extraneous content: non-linear access
also eliminates the wasted time and cognitive load associated with instruction
that is not relevant to the employee's job or task.
-
Offering continuous availability: employees do not
have to wait for critical training to be made available; the training content
resides permanently on a server that can be accessed at any time.
-
Allowing flexible scheduling: as access to training
content is not dependent on other individuals in the organization, employees
can integrate training time efficiently into their personal work schedules
and make productive use of 'idle time'.
-
Requiring zero travel time: employees do not have
to travel to a classroom, training center, or computer lab to receive instruction;
training content can be delivered directly to the desktop.
-
Allowing access to a wide range of training: all of
an organization's training material is archived on the network server,
eliminating the possibility that a required course is 'no longer offered'.
-
Providing a reference tool after initial exposure to instruction:
after receiving instruction, employees can access the training material
during task performance to refine their understanding.
Instructional format refers to the framework within
which the training is delivered. The components of this framework are the
interactions between the employee, the training content, and the learning
environment.
Networked interactive multimedia training systems enable
an optimal instructional format through:
-
Creating a non-threatening learning environment: individualized
learning at the desktop removes the stress associated with learning in
a group of peers in a classroom setting.
-
Allowing self-pacing by the learner: interactive training
is under the control of the individual learner, allowing advanced learners
to quickly progress through the content and providing novice learners with
the ability to receive additional instruction in problematic concepts.
-
Accommodating for a variety of learning styles: interactive
multimedia training allows learners to select the type of instructional
delivery that is most compatible with their learning styles.
-
Delivering multimodal instructional material: multimedia
training uses multimodal instructional content to harness a 'whole-brain'
approach to learning, and allows for an optimal match between media mix
and instruction.
-
Providing interactivity: interactive training catalyzes
active involvement with, rather than passive reception of, instruction,
improving comprehension, retention, and transfer of knowledge.
-
Providing immediate individualized feedback: interactive
training utilizes test instruments that can provide feedback to the user
concerning their progress and comprehension level, enabling more effective
self-monitoring of learning.
-
Increasing motivation: interactive multimedia training
creates a rich sensory learning environment that responds dynamically to
each individual learner, which increases motivational levels.
-
Decreasing training time: as the content and delivery
of training information is optimized for each individual trainee, substantial
reductions in training time are realized, and employees attain skill gains
more quickly.
Quality of learning refers to the results of the training
as determined by the increased performance of the employee in the workplace.
Training must be understood, remembered, and applied by the employee in
order to have a significant impact on performance.
Networked interactive multimedia training systems ensure
quality training through:
-
Enabling mastery learning: instead of being constrained
to the pace of a group of learners, trainees can learn at their own optimal
pace as determined by their abilities and the complexity of the content.
This allows for all employees to master skills prior to completing instruction.
-
Increasing retention: interactive multimedia training
has a substantially higher retention rate than classroom instruction, raising
the probability that new skills and knowledge will be remembered and employed
in the workplace.
-
Decreasing training cycle time: training content that
is delivered over a network to the workplace reduces the time between requesting
training, receiving it, and utilizing it, eliminating the problems associated
with separate learning and performance environments.
-
Empowering the learner: training content that is readily
available on a network empowers and motivates learners to continually upgrade
skills and improve overall performance and productivity.
2.2.2 Advantages for
the training department
A critical advantage of networked interactive multimedia
training systems is that they allow an organization without an established
training department to implement a turnkey solution that will satisfy the
training requirements of their employees. For those organizations with
formal training systems already in place, networked interactive multimedia
training offers important benefits for training departments which impact
all three phases of the training process:
-
Creation of instructional material: trainers can now
cost-effectively utilize realistic examples, practice exercises, and simulations
through high-quality interactive video; as well, short-notice changes to
training content can be quickly made by adding, deleting, and merging existing
content digitally. Training content never has to be 'burned' permanently
on CD-ROM or videotape; delivery over a network allows all employees to
have constant access to the most up-to-date training information available.
-
Delivery of instructional material: networked training
systems support geographically-dispersed trainees and sporadic, uneven
user loads.
-
Monitoring training's effectiveness and efficiency:
networked training systems centralize testing and content access data,
providing valuable information for administration, evaluation, and revision
of training system performance.
Another key benefit is realized by the training department
when the training system automates the delivery of instructional material
and the collection of usage and performance data:
-
Addressing other critical aspects of training's corporate
role: networked interactive multimedia training systems allow trainers
to focus more attention on improving an organization's productivity by
freeing them from the necessity of delivering training content through
classroom instruction.
2.2.3 Advantages for
management
The advantages of networked interactive multimedia training
systems discussed above are articulated from the perspectives of the trainers
and trainees using the system. These advantages combine to create the following
benefits at the managerial level:
-
Cost-effectiveness: networked interactive multimedia
training systems provide more effective training at a lower cost than other
training systems.
-
Increased productivity: as interactive multimedia
training enhances learning and retention, and network delivery aligns the
training content with the demands of the workplace, higher levels of productivity
are attained through multi-skilled employees.
-
Improved quality control: all employees receive the
same quality of instruction with content that is consistent and not subject
to human error during instructional delivery.
-
Integration of training information with corporate information
assets: training information becomes a part of the corporate information
system, and is not inaccessible or at-risk as a result of employee absenteeism
or turnover, or during periods of organizational restructuring.
-
Efficiency and effectiveness of evaluation: the data
that is automatically collected and archived by the delivery system on
such parameters as test results and frequency of content access can be
used to more accurately link training to performance gains in the organization;
the costs of fulfilling audit and legal requirements associated with training
(such as professional certification and OSHA regulatory compliance) are
substantially reduced through automated and centralized training data collection
as well.
-
Improved internal relations: investing in the development
of existing human resources promotes employee morale.
Furthermore, a networked interactive multimedia system offers
benefits beyond those that are directly related to training:
-
A powerful internal communications tool: along with
training content, multimedia-capable LANs and WANs can be utilized for
disseminating other critical information throughout an organization, such
as technical diagrams or safety policy changes.
-
A powerful external communications tool: a computer
network that is capable of delivering high-quality video can be utilized
as support for such organizational requirements as sales presentations
to visiting clients.
-
An enhanced corporate image: the existence of a formal
training and development program has become a definite advantage in recruitment,
especially at the university level. Entry-level candidates are increasingly
asking prospective employers about employee development practices. Demonstrated
commitment to employee development can positively affect the image of a
company as a preferred employer.
2.3 Summary: The benefits
of networked interactive multimedia training
2.3.1 For trainees:
-
Enables task-driven selection of training content
-
Eliminates extraneous content unconnected to task demands.
-
Offers continuous availability and access to instruction.
-
Allows flexible scheduling of training sessions.
-
Requires no travel time.
-
Allows access to a wide range of training.
-
Provides a reference tool for use after initial exposure
to instruction.
-
Creates a non-threatening learning environment.
-
Allows self-pacing by the learner.
-
Accommodates for a variety of learning styles.
-
Delivers multimodal instructional material.
-
Provides interactivity.
-
Provides immediate individualized feedback.
-
Increases motivation.
-
Decreases training time.
-
Enables mastery learning.
-
Increases retention.
-
Decreases training cycle time.
-
Empowers the learner.
2.3.2 For trainers:
-
Allows for more efficient and flexible creation of instructional
material.
-
Enables more cost-effective and timely delivery of instructional
material.
-
Increases the ability to monitor training's effectiveness
and efficiency
-
Provides an opportunity to address other critical aspects
of training's corporate role rather than devote a majority of time to stand-up
instruction.
2.3.3 For management:
-
Achieves optimal cost-effectiveness.
-
Increases productivity.
-
Improves quality control.
-
Integrates training information with corporate information
assets.
-
Increases efficiency and effectiveness of evaluation.
-
Improves internal relations.
-
Can function as a powerful internal and external communications
tool.
-
Enhances the corporate image.
3. The Benefits from
an ROI Perspective
3.1 Overview
An unfortunately all-too-common management attitude towards
training expenditures positions training as a necessary expense that is
rarely evaluated in terms of Return on Investment (ROI). This has largely
been due to the complexity associated with gathering and analyzing data
on training costs and procedures, and linking this data with metrics that
accurately reflect changes in workplace performance caused by training.
Despite this complexity, increased competition in the marketplace and reduced
operating budgets have created a necessity for greater accountability in
training systems. As well, the critical importance of training in maintaining
and refining a corporation's ability to respond to market changes requires
a flexible and adaptive training system whose efficiency can no longer
be assumed. Effective evaluation has been cited by training departments
and management alike as the essential development that must occur to ensure
and refine training's positive impact on profitability [34].
To rectify this accountability gap, the necessary first
step is to accurately assess the costs associated with training systems.
Assessment must comprehensively include not only readily apparent direct
costs, such as training department staff and delivery systems, but also
the hidden costs that arise from such factors as lost productivity during
classroom training.
The second step is to integrate assessment of training's
organizational benefits within the formal training system. Networked interactive
multimedia training systems, by automatically gathering data on such factors
as training time per employee, performance on interactive simulations and
testing instruments, and types of training content accessed, allow management
and training departments to optimize the training system to the requirements
of their organization. This data can also be linked to productivity and
performance metrics in order to evaluate training's impact. By assessing
this impact, the benefits of training can be determined and integrated
into a complete cost-benefit analysis that brings the accountability of
training in line with other aspects of corporate operations.
This section of the white paper addresses the benefits
of networked interactive multimedia training from a ROI perspective.
3.2 Why is a new approach
to cost-benefit analysis necessary?
From its survey analyses of U.S. businesses, the Business
Research Group has determined that interactive multimedia training, although
highly successful as a corporate training technology, has heretofore been
limited primarily to isolated desktops or small work groups, with a concomitant
restriction on the scope of its effectiveness: "most of the corporate workforce
doesn't have access to it because of cost and the enormous bandwidth needed
to distribute multimedia applications over networks" [35]. With recent
developments in server technology, the limitations of cost and bandwidth
have been greatly reduced, and this creates a need to readdress ROI within
this context. In order to accurately assess the ROI for training systems
in general and networked interactive multimedia training in particular,
shortcomings in common approaches to training cost-benefit analysis must
first be resolved. This necessity was noted in a Conference Board of Canada
report [36] which stated that "the benefits to be gained from using new
technology in training, such as lower costs, individualized instruction,
more timely delivery and more convenient scheduling are too powerful to
ignore. However, it does appear that at the present time, current methods
for budgeting may actually be hindering the complete exploitation of new
technology in training."
For example, many training ROI models presume fixed cost
categories (such as student costs, instructor costs, and the costs of training
facilities), and different training systems are evaluated and compared
on the basis of these categories. Advances in technology, however, can
substantially alter the validity of the categorization itself. Conventional
training is delivered primarily through seminars which often last for several
days, whereas in the case of networked interactive multimedia training
systems, much of the training will occur in short Just-in-Time sessions
of a few minutes in response to a current performance problem. As no dedicated
classrooms, training centers, or travel expenditures are required for networked
systems that deliver instruction straight to employees' workplaces, several
cost categories are also no longer relevant. Furthermore, a video-capable
network server that functions over the corporation's current LAN or WAN
adds substantial functionality to the existing communication infrastructure,
positively benefiting aspects of the corporation's operations that are
not related to employee training. An organization that does not evolve
its ROI model to incorporate these developments runs the risk of inaccurate
assessments.
3.3 How can the cost
of training be measured and analyzed?
The most obvious and easily tracked costs associated with
training are the funds paid for a specific training program or system.
Leaders in training ROI evaluation have established, however, that these
expenses rarely equal more than 10% of training's total cost. Instead,
"the major costs of training activities relate to people's time - to salary
costs for people conducting or participating in a specific training program"
[37]. A comprehensive training ROI model must at minimum encompass each
of these four areas: trainee costs, instructional delivery costs, instructional
content costs, and management costs.
3.3.1 Trainee costs
Trainee costs (costs associated with employees as trainees/learners)
are often overlooked when calculating the cost of training, yet they often
represent more than 80% of the cost of a training program [38]. Changing
training variables such as course length has a direct impact on these costs,
and therefore will significantly lower the cost of the overall training
system.
Trainee cost factors are generally calculated by summing
the employees' salaries during training with lost opportunity costs. Lost
opportunity costs (the value of the reduced productivity and/or time lost
due to the employee's absence from the workplace) are complex to measure
and analyze, but some heuristics have been developed. If temporary personnel
are utilized to replace absent employees, the salary of the temporary personnel
multiplied by a factor representing their lower efficiency (generally a
value between 1.2 and 1.5) can be incorporated into the costing model here.
If replacement is handled internally, the effect of the organizational
disruption on productivity should be estimated for all the individuals
involved. If there is no replacement of absent employees, then the missing
productivity of the individual can be calculated by estimating the individual's
impact on the organization's annual gross revenue factored by the period
of time consumed by training. This approach is especially effective for
sales personnel, as their impact on gross revenue is relatively easy to
calculate from extant sales data.
Networked interactive multimedia training minimizes trainee
cost factors through:
-
Reducing time to content mastery: interactive multimedia
training generally requires half the time of conventional training methods
for equivalent instruction, substantially reducing productivity losses
arising from an employee being 'in training' as opposed to performing work
tasks. This has been a factor in training ROI at Andersen Consulting (60-76%
less time was needed for training; the interactive multimedia training
system is expected to have a $10m ROI) [39], CSX Transportation (50% less
time; the training system is expected to pay for itself in 2 years, with
a viable operational lifespan of 5 to 6 years) [40], and the Los Angeles
Department of Water and Power (50% less time; the training system is expected
to pay for itself in 1 year) [41].
-
Eliminating the necessity of travel or retrieval:
network-delivered training comes to the employee instead of the employee
going to the training, as is necessary with instruction delivered through
classrooms and training centers. Lost time due to travel is consequently
eliminated. Trainees also do not lose time or increase overhead costs due
to the necessity of finding and retrieving instructional media in the form
of print-based documents, videotapes, and CD-ROMs.
-
Reducing training cycle time: networked interactive
multimedia training content resides on a central server that can be accessed
at any time, and enables the employee to select and receive only the required
training to solve performance problems as they arise. This Just-in-Time
training eliminates the wasted time associated with waiting for an entire
course to be offered in order to receive one critical piece of information.
3.3.2 Instructional delivery
costs
Instructional delivery costs arise from the nature of the
delivery system used to bring trainees and instructional content together
during training. In the case of classroom instruction, these costs include
instructors' salaries, classroom facility costs, and travel expenses. Training
systems that utilize instructional media (including print-based documents,
videotapes, CD-ROMs, and networked systems) rather than instructors incur
costs from the implementation, usage, and maintenance of the delivery system.
Many delivery systems also have hidden costs associated with the delivery
process, such as the usage of consumable materials by trainees during training.
Networked interactive multimedia training systems minimize
instructional delivery cost factors through:
-
Eliminating the necessity of using traditional approaches:
rather than being constrained to a limited array of time- and resource-expensive
delivery systems, such as group-based classrooms and one-to-one tutoring,
trainers can effectively deliver instruction to variable trainee populations
using the most effective media mix for comprehension and retention. For
example, the FAA is now delivering pilot training at a distance through
three-hour modules in a networked system rather than through the traditional
method of local seminars [42]. The lengthy seminars, where large amounts
of information were delivered over a few days, led to poor comprehension
and retention but were utilized due to constraints imposed by trainees'
having to travel to training sites.
-
Freeing trainers from lecturing: the time consumed
by stand-up lecturing can be redirected to assessing and refining the training
system's impact on corporate operations - an essential requirement in the
high-performance workplace. At Nynex, face-to-face lecturing is now considered
as a 'last resort' for instructional delivery [43].
-
Reducing consumable materials: trainees do not have
to 'walk away' from classroom training with print-based retention aids;
training content is continuously available at the desktop as a future reference
tool.
-
Utilizing the existing desktop workstation and LAN or
WAN: with interactive multimedia training delivered to the trainee's
desktop, training facility costs are eliminated completely. Although the
network may need modification, there are no additional overhead costs associated
with maintaining the network beyond those already part of the corporation's
MIS budget. Furthermore, networked delivery does not require the trainees'
desktop computers to be upgraded with CD-ROM drives, sound cards, or extra
memory, allowing for further cost reductions [44]. An upgrade for MPEG
video is the only requirement.
-
Increasing delivery system reliability: servers that
utilize RAID technology have high levels of reliability, insuring that
training is always available. Information is managed centrally as part
of the organization's IT group, unlike CD-ROMs which can become lost or
stolen and must be installed each time they are used by a new user. CD-ROM-based
training systems can also be prone to breakdown as their internal mechanisms
are exposed to airborne contaminants [45].
-
Disassociating delivery costs from the number of users:
network distribution of digital training content means that instructional
delivery costs are unaffected by the number of trainees using the system;
cost per student ratios decrease with increased usage. With other instructional
media, such as print-based documents, videotapes, and CD-ROMs, the number
of users is limited by the number of available and accessible copies, causing
cost per student ratios to remain high and creating the need to ration
training between employees [46].
-
Eliminating travel costs: as training is delivered
to the trainees' workplaces, there are no travel costs associated with
trainees or trainers.
3.3.3 Instructional content
costs
Instructional content cost factors include the costs to internally
develop or acquire the instructional material used in the training system,
as well as the cost to maintain and revise this content, amortized over
the expected lifespan of the course.
Networked interactive multimedia training systems minimize
instructional content cost factors through:
-
Allowing the seamless integration of third-party training
content into the system: instructional developers can reduce production
and personnel costs by utilizing existing interactive multimedia training
software produced by third-party developers.
-
Allowing repurposing of content: digital training
content can be indexed and combined in numerous ways, allowing trainers
to develop customized training material from existing courseware in order
to respond to changes in organizational needs. Courseware can also be optimized
over time as evaluation data is gathered and analyzed. The U.S. Department
of Defense's research on training systems has linked the reusability of
digital instructional material to reduced training costs [47].
-
Allowing digital revision of content: the modular
nature of digital training materials significantly extends the lifespan
of courseware and enables cost-efficient revisions and modifications to
be made incrementally and economically.
3.3.4 Management costs
Management cost factors are the costs associated with gathering
data on the training system and evaluating its effects on the organization's
productivity, as well as managing the fulfillment of regulatory and legal
requirements associated with training such as mandatory safety certification.
Networked interactive multimedia training systems minimize
management cost factors through:
-
Automating data gathering processes for evaluation: the
usage of interactive training software generates valuable data on trainee
performance, which can be automatically gathered and collated through the
network. This reduces the high cost of conventional data gathering techniques
necessary for evaluation.
-
Automating data gathering for compliance with regulatory
and legal requirements: automated data gathering and collation through
a network server significantly reduces the overhead costs of managing mandatory
employee certification in work tasks regulated by external organizations.
Owens Corning has stated that it can have new training material online
in its distance education system within 2 hours of changes in OSHA requirements
[48].
3.4 How can the benefits
of training be measured and analyzed?
The benefits of training must be formally evaluated in order
to link them to cost factors within a comprehensive cost-benefit analysis.
The difficulty in establishing this link is primarily the result of difficulties
in collecting training process data and deriving workplace performance
metrics that training can potentially impact. As networked interactive
training automates and formalizes the collection of training process data,
a new opportunity exists for statistically analyzing this data against
a broad range of performance metrics to quantitatively explore interrelationships.
For example, differences between various departments or teams within an
organization can be examined from the perspective of amount and type of
training consumed, as well as performance on simulations and testing instruments.
This data can then be categorized and compared with changes in productivity
and performance at the levels of the individual employee, the work group,
the department, and the organization as a whole. Management can then use
these analyses to guide decision-making on training and related issues
with much greater confidence than was possible within the conventional,
intuition-based paradigm.
Performance metrics that have been identified in the professional
and academic literature [49,50] as being relevant for analytical evaluation
with training interventions include measures based on:
-
Decreases in time to find information and 'idle time'.
-
Increases in number of job tasks completed
-
Decreases in error rates and revision/rectification time
-
Decreases in amount of waste produced
-
Decreases in work backlog
-
Decreases in customer complaints and increases in compliments
-
Decreases in frequency of equipment malfunction
-
Decreases in number of equipment repairs by external personnel
-
Decreases in inventory shortages
-
Increases in corporate output (units produced, cases processed,
etc.)
-
Increases in sales
-
Decreases in safety violations
-
Decreases in injuries in the workplace
-
Decreases in absenteeism and tardiness
-
Decreases in turnover rate
-
Increases in quality of performance appraisals
The Gartner Group has conducted studies which show that the
average savings of companies adopting interactive multimedia training is
64% [51].These training systems have experienced successful implementations
in contexts ranging from management training to entry-level skills in sales
and customer service, as well as specific technical skills. For example,
Shell International, Federal Express, and Pacific Bell have all linked
their interactive multimedia training systems to reduced accident rates
for their drivers [52]. Increases in sales have been identified for salespeople
at Apple [53] as well as smaller organizations [54]. The U.S. Air Force
has stated that the troubleshooting accuracy of technicians trained using
interactive multimedia increased by 90%; over a three to five year period,
the Air Force expects a 20 to 25-fold ROI [55]. As more accurate evaluation
of training systems becomes the norm due to automated data collection and
analysis through computer networks, the number of success stories will
continue to rise.
3.5 Conclusion: Evaluating
the ROI on a networked interactive multimedia training system
One of the key benefits of a comprehensive training ROI model
is the ability to derive a checklist of essential networked interactive
multimedia training system components that must be present in order to
obtain maximal ROI. This allows organizations to quickly identify whether
any given system will satisfy their present and future needs. Questions
that should be asked of the training system vendor include:
-
Can the training system be integrated into my existing LAN/WAN
as a regular server?
Hardware that cannot utilize an organization's
existing LAN/WAN dramatically increases both initial and overhead costs.
-
Can I utilize high-quality video through the system?
High-quality video is essential if the training
system is to be able to positively impact interpersonal and complex technical
skill development.
-
Will the system be able to handle the user load of my organization?
For the training system to have a powerful effect
on an organization, it must be able to support variable user loads without
negatively impacting the speed of information delivery; this is especially
critical if the system is integrated into an existing LAN/WAN, as slowdowns
may affect the entire network.
-
What is the maximum number of concurrent users that can follow
the same course?
In many situations it is often necessary to train
as many employees as possible in a short period of time. With some training
systems, it becomes very expensive to provide support for more than 25
simultaneous users and course content must be replicated on the server
to meet the needs of additional students.
-
What kind of scaleability will the system provide to support
future expansion needs?
A system that is initially installed to service
a single training center with 25 users may eventually be required to provide
desktop training on demand throughout an organization. The system should
scale in a linear fashion and demonstrate superior price/performance ratios
as more users are added.
-
Is off-the-shelf interactive training material available
that meets my needs?
A bundled software/hardware solution minimizes
the time between purchase of the training system and return on investment.
-
What level of tracking and administration is provided by
the training system?
The training system should include software controls
for all administration functions in addition to providing detailed user
statistics. Optional billing support should also be available. All tracking
and administration controls should be implemented through a single user
interface that works seamlessly with existing desktop operating systems
to minimize learning time and avoid compatibility issues.
4. References
[1] McAteer, P. F. (1994). Harnessing the power of technology.
Training & Development, August.
[2] Gephart, M. A. (1995). The road to high performance.
Training & Development, June.
[3] Dove, L. (1996). PC as teacher now a reality. Computing
Canada, 21(20).
[4] McKean, K. J. (1995). What is this thing called Accelerated
Learning? Training & Development, June.
[5] Dove, L. (1996). PC as teacher now a reality. Computing
Canada, 21(20).
[6] Jughes, J. R. (1996). Acting as tutor and mentor.
Computing Canada, 21(20).
[7] Garcia, Y. (1995). Multimedia in the future: A
study using multimedia in business communication. Unpublished thesis,
Executive MBA program, Concordia University.
[8] Jackson, M. A. (1995). Try multimedia for full effect.
Computing Canada, 21(20).
[9] McKean, K. J. (1995). What is this thing called Accelerated
Learning? Training & Development, June.
[10] Garcia, Y. (1995). Multimedia in the future: A
study using multimedia in business communication. Unpublished thesis,
Executive MBA program, Concordia University.
[11] Grace, T. (1995). SHL puts training to the test:
integrator's study shows advantage goes to computer-based versus classroom
education. Computer Reseller News, 646.
[12] Jackson, M. A. (1995). Try multimedia for full effect.
Computing Canada, 21(20).
[13] Jughes, J. R. (1996). Acting as tutor and mentor.
Computing Canada, 21(20).
[14] Reich, R. (1990). Training in business. Harvard
Business Review, January/February.
[15] Geis, G. (1991). As Training Moves Toward The
Next Decade. Ontario Training Corporation.
[16] Conference Board of Canada (1990). Training and
development 1990: Expenditures and policies (Report 67-91).
[17] Conference Board of Canada. (1991). Training and
Development.
[18] Ichniowski, C. (1995). Human resource management
systems and the performance of U.S. manufacturing businesses. National
Bureau of Economics Research working paper 3449.
[19] The Globe & Mail Report On Business (1990). The
Top 1000 Companies, July 1990, p.86
[20] Gherson, D. J., & Moore, C. A. (1987). The role
of training in implementing strategic change. In L.S. May, C. A. Moore,
& S. J. Zammit (eds.), Evaluating Business And Industry Training.
Kluwar Academic Press.
[21] National Society for Performance and Instruction.
(1995). News and Notes.
[22] Carnevale, A. P., & Schulz, E. R. (1990). Return
on investment: Accounting for training. Training & Development,
July.
[23] Goldstein, I. L. (1989). Critical training issues:
Past, present, and future. In I.L. Goldstein (ed.), Training And Development
In Organizations. Jossey-Bass.
[24] Muszynski, L., & Wolfe, D. A. (1989). New technology
and training: Lessons from abroad. Canadian Public Policy, 15(3).
[25] Geis, G. (1991). As Training Moves Toward The
Next Decade. Ontario Training Corporation.
[26] Doherty, K., & Allen, S. (1995). Electronic Support
Online Help System. Paper presented at the 17th Annual Conference On
Interactive Systems For Training, Education, And Job Performance Improvement.
[27] Geis, G. (1991). As Training Moves Toward The
Next Decade. Ontario Training Corporation.
[28] Conference Board of Canada. (1991). Training and
Development.
[29] Conference Board of Canada (1990). Training and
development 1990: Expenditures and policies (Report 67-91).
[30] Bainbridge, S. V. (1995). You can't teach soft skills
on a computer...can you? Paper presented at the 17th Annual Conference
On Interactive Systems For Training, Education, And Job Performance Improvement.
[31] Rothstein, F. R. (1989). Continuing To Work: Issues
In Training And Employment. National Association of Counties.
[32] Geis, G. (1991). As Training Moves Toward The
Next Decade. Ontario Training Corporation.
[33] Jughes, J. R. (1996). Acting as tutor and mentor.
Computing Canada, 21(20).
[34] Conference Board of Canada. (1991). Training and
Development.
[35] Johnson, I. (1994). How fast is multimedia really
catching on? Computer Dealer News, December 28.
[36] Conference Board of Canada. (1991). Training and
Development.
[37] Head, G. E. (1985). Training Cost Analysis.
Marlin Press.
[38] Head, G. E. (1985). Training Cost Analysis.
Marlin Press.
[39] Jerram, P. (1994). Who's using multimedia? NewMedia,
October.
[40] Trisram, C. (1995). Stream on: Video Servers. NewMedia,
April.
[41] Piazza, D. (1995). L.A. Department of Water and Power:
Power to the people. Multimedia Today, July-September.
[42] Filipczak, B. (1995). Putting the learning into distance
learning. Training, October.
[43] Howard, B. (1995). Taking multimedia into production.
Paper presented at the 17th Annual Conference On Interactive Systems
For Training, Education, And Job-Performance Improvement.
[44] Piazza, D. (1995). Language labs serve up LAN video.
Multimedia Today, July -September.
[45] Piazza, D. (1995). Language labs serve up LAN video.
Multimedia Today, July -September.
[46] Miller, R. (1990). Learning benefits of interactive
technologies. Videodisc Monitor, February.
[47] Garcia, Y. (1995). Multimedia in the future: A
study using multimedia in business communication. Unpublished thesis,
Executive MBA program, Concordia University.
[48] Filipczak, B. (1995). Putting the learning into distance
learning. Training, October.
[49] Chichelli, J., & McMahon, C. (1995). Evaluating
EPSS - Getting to the bottom line. Paper presented at the 17th Annual
Conference On Interactive Systems For Training, Education, And Job Performance
Improvement.
[50] Carnevale, A. P., & Schulz, E. R. (1990). Return
on investment: Accounting for training. Training & Development,
July.
[51] Garcia, Y. (1995). Multimedia in the future: A
study using multimedia in business communication. Unpublished thesis,
Executive MBA program, Concordia University.
[52] Garcia, Y. (1995). Multimedia in the future: A
study using multimedia in business communication. Unpublished thesis,
Executive MBA program, Concordia University.
[53] Waldman, D. (1995). Interactive multimedia: Measuring
the ROI. Paper presented at the 17th Annual Conference On Interactive
Systems For Training, Education, And Job Performance Improvement.
[54] Bainbridge, S. V. (1995). You can't teach soft skills
on a computer...can you? Paper presented at the 17th Annual Conference
On Interactive Systems For Training, Education, And Job Performance Improvement.
[55] Jerram, P. (1994). Who's using multimedia? NewMedia,
October.
This white paper was researched and prepared in conjunction with
Paul Cholmsky and Robert Gordon of The
Article 19 Group.
| |
Copyright © Alex Informatique
SA France May 1998. All rights reserved.
|