| The
most important findings from the University of Pittsburgh's evaluation
of InfoLink focus on lessons of particular relevance to other institutions
operating education, training, and workforce development programs
serving teens, whether technology-oriented or not. Below is a summary
the major lessons identified from the evaluation.
To
read the full report, click here.
Adobe Acrobat Reader software is required.
Lesson
#1: Even when giving away training of significant value, recruiting
and retention are a challenge.
Program staff consistently identified student retention as the greatest
challenge the program faced, and they invested heavily in developing
close relationships with students and working to solve whatever
problems were making it difficult for them to continue in the program.
This approach recognized that many of the reasons students were
dropping out had less to do with the design and implementation of
the program itself than with other issues in the students’
lives. However, analysis of the reasons students dropped out suggests
that the real leverage for InfoLink and other programs seeking to
keep their slots filled may be less in trying to hang on to all
students who self-select into the program than in recruiting a pool
of applicants that is both bigger and better suited to the program.
Top
Lesson #2: Having an impact on low income
teens isn’t cheap: operating programs at full capacity is
key.
The InfoLink program had an ambitious mission: impacting the long-term
educational and occupational attainment of low-income students who
were on the wrong side of the digital divide and faced multiple
educational and social disadvantages. Appropriately, they developed
a resource-intensive model based on challenging opportunities and
the personalized support needed to meet them. Inevitably, the cost
per student to achieve these kinds of broad outcomes with this population
was going to be high. However, because many of the costs of running
the program were fixed (instructor salaries, software, etc.), running
the program under capacity drastically increased the cost per student,
making the economics of the program much more unfavorable. There
was strong consensus among program stakeholders that the key to
making the cost per student more sustainable was not scaling back
the program’s infrastructure but moving more students through
it.
Top
Lesson #3: Students need to bring significant
commitment to the table to succeed in a program like this.
Given the intense time commitment required and the challenging nature
of the work, it is not surprising that the model only works well
for certain kinds of students. How to find those students is more
difficult. One strategy that InfoLink used was “self-selection
through the communication of rigorous expectations.” Staff
made a point of emphasizing to students how much time and effort
would be required, assuming that this would deter those looking
for a less challenging experience from even applying. The high dropout
rates, however, suggest that self-selection alone is insufficient
to bring in students who have what it takes to complete such a rigorous
program. Some program insiders felt that student’ grades were
the best available proxy for their ability to manage their time
and willingness to work hard. Others, however, cautioned that some
of the best candidates for the program may have low grades because
high school classes have failed to challenge and motivate their
latent talents.
Top
Lesson #4: It is possible to take students
with minimal technology familiarity and give them baseline comfort
with a wide range of applications in a short period of time.
Data indicated that InfoLink succeeded in taking students who came
in with generally low levels of familiarity with technology and
in a very short period of time making them comfortable with a broad
range of sophisticated applications.
Programs
wishing to replicate these successes should note two important points
about the InfoLink curriculum:
- It was not static: The curriculum was kept fresh by adding the
latest cutting-edge technologies.
- It was not didactic (or “academic” in the negative
sense): To every extent possible, skills were taught in context
through the use of group projects, student presentations, and other
hands-on learning experiences.
Top
Lesson #5: Programs should be clear on the
tradeoff they are making between breadth of exposure and depth of
skills.
Although the breadth of software comfort gains across so many applications
documented above is impressive, it raises an obvious question about
the depth of skills acquired and the ability of students to apply
these skills in the workplace. In fact, The InfoLink program made
a deliberate tradeoff in favor of breadth of exposure over depth
of skills. They were always very clear that this program was something
more (but also something less) than the standard software training
available through many sources. Rather than immediate workplace
application, their aim was to build students’ general technological
literacy and their confidence and aspirations for further education
and career. By taking a “literacy” approach, InfoLink
aimed to help students understand the basic logic and functionality
of each software covered, the range of professional situations in
which it could be used, and achieve enough hands-on familiarity
with it to enable them to keep on learning on the job or on their
own.
Top
Lesson #6: Outcomes such as awareness, aspirations,
and confidence are at least as important as technical skills in
terms of the program’s impact.
Clearly, the outcomes that InfoLink targeted (and achieved) with
its technology curriculum are better conceptualized as “literacy”
than “skills.” It is possible to go even further and
state that the entire software curriculum was largely instrumental,
a vehicle for achieving other outcomes such as increased awareness
of and orientation to technical careers and increased educational
and occupational aspirations and confidence. Certainly, having real
software skills and seeing how those are valued in the workplace
provided concrete grounding for the informational and aspirational
outcomes the program targeted. But ultimately, program stakeholders
believe, it is these less concrete outcomes that are more important,
because they change how students see themselves and their futures.
This opens the door not just to specific skills-based slots in today’s
economy but to a career trajectory into tomorrow’s.
Top
Lesson #7: Individual relationships matter.
At-risk teens often move anonymously through vast high schools with
no particular guidance or support to help them map their pathways
to the future. Literature on programs that are effective in connecting
vulnerable teens with higher education and/or a productive, meaningful
career shows that diverse programs have one thing in common: individualized
support through caring relationships.
The
InfoLink program built and nurtured such relationships through a
variety of mechanisms. In the classroom, we used a “professional
training model” based on Masters-level programs at the host
university. The core value of this model is taking students seriously-
providing frequent opportunities for collaborative and self-directed
work and respecting their ideas. Simple things like being on a first
name basis with college professors can make a world of difference
to students who may never have been treated with this kind of respect
and high expectations. To make this responsive model work, the program
invested in having a high ratio of adults to students in the classroom,
allowing for a great deal of hands-on learning and individualized
feedback. Beyond the classroom, the program invested in supporting
and counseling students through the program’s rigorous demands.
Although the program maintained a strict policy of releasing students
who had three unexcused absences, they worked hard to provide the
individualized support that would keep students from getting to
that point, following up on issues ranging from family obligations
to transportation difficulties.
Top
Lesson #8: The commitment of individual program
“champions” must be supported by administrative capacity.
Like many programs for low income youth, InfoLink would never have
succeeded without the commitment of individuals who were true believers
in the program’s mission, willing to work long hours to make
the vision a reality. As is often the case, such commitment was
deeply personal, not necessarily rewarded by professional or institutional
incentives. Although the energy of such program champions is vital
in creating and sustaining a program, in the long run it must be
buttressed by more workaday administrative capacity, the kind needed
to process applications, prepare mailings, enter data, make phone
calls to check on absent students, and the like. In the absence
of such capacity, the commitment of program leaders is likely to
be pulled in too many directions and subject to burn-out.
Top
Lesson
#9: In preparing students for the high tech workplace, it is critical
to balance hard and soft skills.
Although InfoLink was always envisioned as more than a technology
skills training course, over the years the program’s model
shifted to incorporate more and more “soft skills” such
as interviewing techniques, appropriate workplace attire, and taking
feedback constructively. The shift was driven by two factors. First,
early experiences with internships made it clear that even students
who were very well prepared technically would not succeed without
these soft skills. Second, program staff became more aware of the
deficits that students coming into the program had in these areas.
One stakeholder reflected that they found it was well worth spending
time “being explicit about things you might just take for
granted they would understand, like what is an acceptable reason
to be late for work and what is not.” Many of the students
in the program simply did not know adults who worked in the professional
world and therefore lacked the kind of role modeling of behavior
and attitudes that kids growing up in middle-class neighborhoods
often have.
Top
Lesson
#10: Making internships a learning experience requires careful planning
and ongoing follow-up.
When InfoLink stakeholders were asked in the program’s final
year to reflect on the greatest, most persistent challenges the
program faced, recruiting internship hosts was second only to student
recruiting/retention. InfoLink learned through experience that finding
internship slots is only the beginning of the challenge. If internships
are to be structured as true learning experiences, giving students
the opportunity to apply the advanced skills learned in the program
rather than just perform routine office tasks, hosts need upfront
guidance and ongoing support. InfoLink had limited success in getting
hosts to attend workshops or training about how to structure an
internship effectively, and had to rely on more informal, opportunistic
guidance. Post hoc, however, intern hosts had high levels of agreement
on a survey item about “need(ing) to do more upfront planning”
before hosting an intern again.
Top
Lesson
#11: Intern hosts come to dispense charity, stay because of organizational
benefits.
When initially approached about hosting an intern, particularly
a low-income teenager, many organizations were wary. Most ultimately
agree to do it out of a sense of obligation to the community. At
the end of the experience, however, most were surprised that students
had brought real skills to the table and contributed to meaningful
organizational tasks. As one stakeholder put it, “once they
see what the interns can do, they realize it is the deal of the
century and they almost always want to participate again.”
In fact, when asked to rate the outcomes of the internship, hosts
gave higher scores to their own organizational learning (specifically
about how to develop young workers) than to either the technical
or “workplace skills” learning of the students over
the course of the internship. Thus, data from the InfoLink evaluation
suggest that for programs facing difficulty recruiting intern hosts,
testimonials from past hosts- both about the contributions students
are able to make and what the host can learn- may be a powerful
tool.
Top
Lesson
#12: High schools are difficult institutions for external programs
to influence systematically.
InfoLink’s original design included the idea of “high
school partnerships.” Through working with the same schools
over a number of years, the program aimed to build up a presence
in the schools that would both help with recruiting students and
ultimately impact the culture and curriculum of the school as a
way to extend the influence of the program beyond those students
participating in the intensive training.
It became clear early on in the evaluation that of all the program’s
components, the high school partnership model was achieving the
least traction. While student application data suggest some small
recruiting benefit of students having heard about the program through
alumni at their high schools, broader impacts on high schools failed
to materialize. On the contrary, the program faced numerous logistical
and communications difficulties in trying to run the recruiting
and the after school spring component of the program through the
high schools.
Top
Lesson #13: There are pros and cons to basing
workforce development programs at a university.
In terms of replication, and particularly scaling up to impact more
students, the importance of the university location to achieving
the program’s impacts has been one of the biggest question
marks about InfoLink’s model. Being university based certainly
had tangible benefits, specifically access to well-equipped computer
labs, the most advanced software, and instructors who were expert
in using it. Stakeholders argued that the intangible benefits were
just as important. For low income high school students, being taken
seriously by college professors, challenged with the most rigorous
expectations, and simply seeing that they can fit in and function
in the collegiate environment were all important contributors to
the program’s goal of building confidence and raising educational
and occupational aspirations.
These
benefits did not come without costs, however. Several stakeholders
wondered whether the program’s high cost per student could
be lowered while maintaining most of the benefit of the curriculum
by relying on less expensive instructors. Another major downside
of the university location was that it kept the program locked into
a summer-intensive model (since that is when labs and instructors
were available) when there may have been a number of advantages
to a more year-round model.
Top
Lesson #14: Long-term outcomes focused on
improved life chances are difficult to measure.
The long-term outcomes stated in InfoLink’s mission (improved
educational and occupational attainment) are, to the say the least,
ambitious for what was essentially a summer enrichment program,
albeit an unusually rigorous and intense one. The InfoLink experience
suggests that similar programs should be modest in their expectations
about producing and measuring these kinds of long-term outcomes.
There are a variety of reasons for this:
| (1)
The program was a relatively small intervention in the course
of a student’s life, with no mechanism for long or even
medium-term follow-up and reinforcement. |
| (2)
The long-term outcomes are not only far down the causal chain
from the intervention, they are quite temporally distant too.
These kinds of outcomes would not be expected to show up for
years or even decades after students leave the program, introducing
obvious logistical challenges for research. |
(3)
Finally, on a very practical note, low income teens are a
highly mobile population and it only gets harder to stay in
touch with them the longer they have been out of the program.
|
Top
Lesson
#15: The program’s most significant long-term outcome was
changing how students saw their own future.
Despite all of the cautions in Lesson #13 about producing and measuring
long-term outcomes, it is possible to say a few things about how
InfoLink affects its participants beyond the kinds of skills and
attitudes measured in the immediate post-program survey. When they
were asked to reflect on the program’s most significant impact
on students, most stakeholders pointed to the dramatic changes in
students confidence and expectations even over the short period
they were involved in the program. In different ways, everyone interviewed
in the final year of the program made the point that InfoLink helps
students see themselves differently.
Top
|
