Management and Evaluation of a pan-Canadian Graduate Training Program in Health Informatics

Marilynne Hebert and Francis Lau

VOL. 24, No. 1, 107 - 122

Abstract

Eight Canadian universities partnered to establish a Collaborative Health Informatics PhD/Postdoc Strategic Training Program (CHPSTP). The 6-year goal was to increase research capacity in health informatics in Canada. Three cohorts of 20 trainees participated in the training, which included online Research Learning Experiences, annual face-to-face workshops and conferences. Different management models were explored to more closely align funding, user engagement and deliverables. Evaluation included measures of success at program, faculty and student levels. Data collection templates facilitated reporting and program management. Linkages between user engagement and funding should be built into the grant proposal.

Résumé

Huit universités canadiennes se sont unies en partenariat pour mettre sur pied le Collaborative Health Informatics PhD/Postdoc Strategic Training Program (CHPSTP). L’objectif sur six ans était d’accroître la capacité de recherche en informatique de la santé au Canada. Trois cohortes de 20 étudiants ont participé à la formation, laquelle comprenait des expériences de recherche en ligne, des ateliers annuels en face-à-face et des conférences. Différents modèles de gestion ont été explorés afin d’aligner le plus possible le financement, l’engagement de l’utilisateur et les livrables. L’évaluation comprenait des mesures de la réussite au programme, au niveau des enseignants et des étudiants. Des grilles de collecte de données ont facilité le suivi et la gestion du programme. Des liens entre l’engagement de l’utilisateur et le financement devraient faire partie intégrante de la demande de subvention.

Background

Health Informatics (HI), the use of computers and communications technology in health care delivery, administration and education has been steadily increasing over the past 30 years. Given the growing number of HI applications such as electronic health records and clinical decision support systems, the demand for expertise in planning for implementation and evaluation of these systems has also expanded. HI graduate training has been identified as a high priority need in Canada, particularly since it has been difficult to achieve a critical mass for a program of study in HI (School of Health Information Science, 2001).
In the fall of 2002, eight Canadian university teams (Victoria, Calgary, Waterloo, McMaster, Toronto, McGill, Sherbrooke and Dalhousie) partnered to establish a Collaborative Health Informatics PhD/Postdoc Strategic Training Program (CHPSTP). This was made possible through a successful funding application to the Canadian Institutes of Health Research (CIHR) and Michael Smith Foundation for Medical Research (MSFMR) (Author, Author, 2002). The 6-year program was intended to increase research capacity in health informatics (HI) in Canada through:

• jointly offering 4-5 intensive “Research Learning Experiences” (RLEs) to trainees not available in local programs;
• strengthening collaboration among partner institutions in HI education and research;
• building a virtual community engaged in HI research, education and practice; and
• assuming a leadership/facilitation role as an academic HI community in Canada.

Program Goals and Objectives

The Program Goal was to create a shared resource of the knowledge and expertise in the eight HI research teams and through this to provide a wide range of HI research training opportunities to enhance the health system in ways not feasible within any single institution. Together the different perspectives and experiences of key mentors and stakeholders from across Canada could potentially take transdisciplinary HI research to a level well beyond what could be achieved locally.

The curriculum and delivery methods were developed to overcome the barriers in access, diversity and depth faced by local programs. The use of e-learning and virtual presence technologies enhanced effectiveness of teaching, learning and collaboration, which previously had been done mostly in traditional face-to-face settings. Collaboration with stakeholder organizations such as health-care facilities and government agencies through the use of physical and virtual collaboratories were intended to ensure the relevance and accelerated pace of HI research.

The Program Objectives were to:

1. increase the number of researchers at PhD and postdoctoral levels in Canada;
2. establish a critical mass of HI researchers otherwise not achievable in local programs;
3. build an effective collaborative research network to bridge the current and next generation of HI researchers;
4. foster a culture of policy responsive researchers;
5. explore new frontiers of HI research and
6. address the growing demand for HI leadership in health within Canada.

Program Delivery was planned in three iterations, with each cohort of trainees studying together for approximately 2 years. The CIHR funding rules for this grant stipulated that 70% of the $1.2 million was to be spent on trainee stipends. The funding was allocated annually to the Principal Investigator (PI) located at the University of Victoria, and then was distributed to the partner institutions.

Of course what was planned and what unfolded changed over the three iterations. Program management, design and evaluation plus lessons learned are highlighted for each iteration of the program. These contributed to the overall lessons for management of large e-learning projects, in particular important issues around user engagement.

Program Iteration 1—Health Themes and Health Informatics Domains (January 2003-December 2004)

A large gap existed between receiving program funding and accepting students into the program! Prior to accepting the first cohort of 20 trainees a program infrastructure was established. A part-time Program Director was hired to develop the processes and documentation required. The Program Director and one mentor from each university oversaw operational activities through a Program Management Committee (PMC). An Evaluator was hired part-time to assess how well the program met its objectives and CIHR reporting requirements. Responsibility for distributing the funds remained with the PI University of Victoria. Thirty percent of the annual funds were allocated to operational costs and the remaining 70% were divided equally among the 8 partners.

As part of the accountability for receiving government funding as well as to provide overall strategic direction, an external Program Advisory Committee (PAC) was established in January 2003. PAC members were identified among future employers of the graduates, students, HI industry, and academia. They met three times a year to provide external feedback on program design, implementation and evaluation.

Program Design

The first cohort of 20 trainees participated from January 2003-December 2004. There were no shortages of trainees and partners were invited to enroll additional trainees depending on local funding availability. PhD/Postdoc trainees had to be registered in one of the existing local graduate programs prior to participating in CHPSTP. In addition, while participating in these RLEs, trainees were also expected to enroll in required courses within their local programs. Training components included online Research Learning Experience (RLE) modules with virtual classes, annual on-campus workshops and face-to-face meetings at HI related conferences.

HI competencies, identified through two nationally funded projects (School of Health Information Science, 2001; Covvey, Zitner & Bernstein, 2001) provided the basis for designing the RLEs. The RLEs also combined a unique delivery concept with shared collective expertise. Four RLEs were offered—Frontiers of HI Research, HI Virtual Community, HI Knowledge Transfer and HI Onsite Experience (Table 1).

Table 1. Summary of Research Learning Experiences

Given the researchers' expertise, the program focus was narrowed to include six health domainsÑcancer research, healthy aging, public/population health, health policy and services, and cardiovascular and respiratory health. Selected crosscutting HI themes included knowledge management, intelligent health systems, telehealth and tele-learning, organizational informatics, e-health applications, and e-research applications. These themes were woven throughout the RLEs based on faculty and student interest. This demonstrates the breadth of topics that the trainees were exposed to as opposed to one or two research interests at the local university.

Program Evaluation

The goals of the program and funder were partially fulfilled. A community of HI researchers was successfully developed with increasingly more formalized communication. A University of Calgary Master's student studied the development of this community of practice as part of her thesis work (Palacios, 2005). The funder's goals, to increase additional student funding and publications were also starting to be achieved.

Program evaluations were conducted regularly through-out this iteration including post-course evaluation surveys for RLEs and face-to-face workshops. The possibility of offering trainees a certificate of completion was explored but rejected by the funding body. Each individual training program was free to provide certificates. The challenge was determining how much of the program had to be completed to receive a certificate—attend 80% of the web seminars and produce a manuscript? Or attend 75% and produce two manuscripts? This highlighted the challenge of determining what was considered adequate participation. Only 9/22 trainees submitted their progress reports for the overall CHPSTP Progress Report: April 2002-March 2004. However the list of deliverables for these 9 trainees was impressive with multiple conference presentations and publications.

The CIHR also developed a “Performance Measurement and Evaluation Framework” that focused on increased opportunities for health research training and trainee deliverables that would not have been possible without this funding, e.g., publications and successful funding applications.

Iteration 1—Lessons Learned:

• a broad range of faculty and trainee interests created challenges in sustainability
• allocating funding on an equitable basis was not effective in guaranteeing participation or tangible outputs
• cohesion could be enhanced by introducing smaller groups who were interested in common research areas
• the Program Advisory Committee, while active, was not effective in setting the program's strategic direction.

Program Iteration 2: Cohort Model
(January 2005-August 2006)

Program Management

Program Management was assumed by the PI as the Program Director with the support of a program coordinator from University of Victoria. A Program Management Committee (PMC) remained in place with one mentor from each institution to oversee all research training activities. The PMC held monthly virtual meetings to review progress. The Evaluator continued in her appointment to assess whether the program met the objectives and CIHR reporting requirements.

A re-constituted Program Advisory Committee (PAC) with 6 members nominated by the PMC plus two non-voting members provided input to overall direction. The PAC met twice a year to review achievements and next steps. Each institution had to submit an annual performance report summarizing its activities for the year in order to receive further funding. UVic then submitted an aggregate report to CIHR each year.

Funding Support

Each partner institution was provided with $28k each year to support all CHPSTP related activities. This funding was intended to cover at least one trainee including his/her stipend and travel, as well as the administrative/travel/communication expenses for the mentors. Each partner site was required to submit a one-page plan to outline its projected expenditures for the year. Continued funding to each partner institution was contingent on satisfactory performance at the end of each fiscal year. In addition, each partner had to expend its entire stipend each year before receiving further allocation since CIHR discouraged carryover of funds.

Each mini-cohort was also allocated $7,500 per year to support additional meetings, and the respective coordinators had to provide a 1-page plan to outline the budget and activities for the 20 months ahead. The annual workshop was covered through central funding of $30K administered by University of Victoria. Another $7,500 was allocated to evaluation of the training program.

Participation and Level of Commitment

In addition to a change in funding support which now required a plan for expected expenditures, the type of participation and level of commitment required in order to take part in this program were made more explicit for the second iteration. These are briefly outlined below:

• The training program is made up of mentors, trainees, staff and collaborators.
• There are three categories of trainees based on whether they are fully or partially funded through this program, or affiliates funded elsewhere. Different stipend amounts are provided depending on the type of trainee and whether he/she is full or part time in the program. (CIHR stipends are approximately $20k for PhD trainees and approximately $40k for fellows including benefits per year assuming full-time status. Practitioners are provided with up to a $25k stipend for buyout time over 20 months.)
• Each trainee has to be nominated by a mentor from his/her home institution. The mentor who is responsible for the trainee has to participate in the program at the same time. All mentors and trainees have to enroll in one of the three mini-cohorts.
• Additional meetings will be scheduled around HI related conferences, such as e-health, AMIA and MedInfo conferences. All trainees and mentors will be encouraged to attend these events.

To ensure the success of this program, every participant was expected to commit time and effort to take part in the activities and produce the outputs required. A summary of the commitments and expectations for all participants was developed and circulated, including the activities, deliverables and timelines for each role.

Program Design

The training program was intended to be closely aligned with the research areas of the mentors so their work could be linked to the trainees and across partner institutions. Given the breadth of research interests, three thematic areas based on the CIHR health domains and HI research themes of partners were suggested in order to focus this cohort: primary care informatics (PCI); e-health policy and evaluation (e-health); and knowledge management (KM). These three themes were only intended to provide a high-level categorization of relevant research areas to encourage focused collaboration among the mentors, yet were still considered sufficiently broad to allow diverse HI research and development to be undertaken by the trainees.

The basic structure of the training program consisted, again, of online Research Learning Experience (RLE) modules with regular virtual classes, annual on-campus workshops and face-to-face meetings at HI related conferences. Three mini-cohorts were established and coordinated by two mentors from two partner institutions. These coordinators were responsible for organizing and facilitating specific research training activities for each mini-cohort. They were to finalize the research training schedule in Fall 2004. With the help of their mentors, trainees were to submit a research training plan in January 2005 to outline their goals, activities, deliverables and timelines. A sample training plan was provided.

University of Victoria was responsible for organizing the online sessions for the RLEs, while the mini-cohort coordinators led the sessions. In addition, University of Victoria coordinated and organized the annual on-campus workshop, which was hosted at one of the partner institutions in the spring.

Evaluation: Tangible Outputs

To meet the CIHR reporting requirements, tangible outputs were needed. Specifically, trainees were required to produce tangible outputs within their selected theme, which included a peer-reviewed publication and one other deliverable such as a research proposal, conference paper, peer-reviewed journal paper, research report, business plan or case study. Each trainee was to submit a one-page training plan when starting the program, then a one-page progress report at the end of year one , and a one-page summary report in year six after completion, to describe their accomplishments and how they had benefited from the program.

Iteration 2—Lessons Learned:

• dividing into cohorts under broad umbrellas of interest was intended to assist in bringing focus to the research areas. This was a realistic approach given any one researcher would not be an expert in all HI areas. The trainees were still exposed to the range of HI research through the face-to-face meetings and on-line presentations.
• local leadership for each of the mini-cohorts was inconsistent
• rather than depending on the “good will” of participants who were busy in their own institutions, leadership required taking a more explicit stance as to what was required and aligning the funding with deliverables.

Program Iteration 3: Knowledge Translation (KT) for Best Practices in HI

Program Management

The final phase of this initiative was intended to engage mentors and current as well as former trainees and fellows in this program in knowledge translation (KT) to foster best practices in HI to improve the Canadian healthcare system. This KT cohort model has three parts—monthly real-time virtual seminars, moderated issues-driven asynchronous discussion forums, and supporting trainees/fellows to present papers at health IT conferences. These activities were intended to provide the foundation towards establishing a virtual CHPSTP community that links HI research with best practices.

Program Design

Eight seminars were delivered monthly from September 2006 to May 2007 as a series of virtual classes via a real-time Web conferencing tool called Elluminate®. Each of the eight university partners was responsible for one of these sessions, which was presented by the mentors, trainees, alumni or guest speakers from that institution. These sessions were promoted widely to HI practice communities such as COACH, Canadian Society for Telehealth and other health professional associations.

Four asynchronous discussion forums were held from September 2006 to May 2007 using the online forum tool in MOODLE®. These online forums were moderated by designated mentors to discuss specific issues for a fixed time period of usually two to three weeks each. Everyone was encouraged to participate in these forums by posting comments, which were to be summarized by the group at the end as CHPSTP position papers.

To enhance linkages between HI research and best practices, trainees and fellows were encouraged to submit papers to health IT conferences. The mentors were to work with trainees/fellows on formulating ideas, writing a paper and providing feedback for the paper submission. Trainees/fellows who had their papers accepted could qualify for financial support of up to $1,500 to attend the conference (until the funding pool was exhausted). Examples of such conferences included the International Symposium on Health Information Management Research Conference held by Dalhousie, the ImproveIT Workshop by University of Toronto, the Canadian Society for Telehealth Conference by University of Calgary, e-Health Conference by COACH-Infoway and the Spring AMIA conference.

Trainees/fellows were encouraged to work closely with their mentors to establish a training plan with specific objectives, tasks, timeline and deliverables. Similarly, alumni interested in working with mentors and/or peers were encouraged to put forth specific research plans and proposals in order to be included as part of the CHPSTP research training activities.

Evaluation: Tangible Outputs

Through specific KT activities this cohort was intended to produce tangible deliverables in the form of established best practices in HI that can have measurable impacts and improvements in the healthcare system in Canada.

Iteration 3—Lessons Learned:

• it was easier for trainees and mentors to participate with specific deliverables being targeted
• funding was tied to production, i.e., papers accepted for conference presentation, yet did not lead to any more or less papers produced by trainees/fellows.

Discussion

End-user engagement (both trainees/fellows and mentors) was a critical factor in managing this e-learning project. During the first program iteration, participation was expected to occur as a result of the collegial nature of a multi-institutional grant. Mentors at each site were not paid but were expected to participate based on the fact they were listed as co-investigators on the funding application. Trainees were funded to participate and for the most part did so. For the few who did not fully participate in the RLEs or attend face-to-face meetings, other than gentle persuasion and reminders, there was no leverage to change the funding allocation or encourage increased participation.

By the second program iteration (year three) the amount of time and effort required to develop, implement and sustain the program became more evident. Creating data collection templates facilitated reporting and managing the program. Data were collected to complete project reports to funding agencies as well as to plan/manage continued program development.

A number of tensions emerged including one between developing an academic research program which faculty were comfortable with versus developing and supporting a research training program. A second tension existed for trainees who were mandated to finish their required course credits at their home institutions and yet wanted to participate in additional research training. Both of these tensions interfered with promoting user engagement.

Management was by exerting a positive influence to encourage participation. This was not successful among participants with competing interests and for whom the project did not resonate.
The final iteration (years five and six) attempted to bring a closer connection between the academic and practice communities. The context had changed as more HI programs had developed across the country and needed faculty. The CHPSTP program had contributed to preparing PhDs for this increased number of jobs.

Conclusion

Evaluation activities included measures of success at program as well as individual faculty and student levels. Feedback from specific learning exercises such as face-to-face workshops, was useful for planning for future activities but less helpful for program management. However trainees/fellows and mentors strongly agreed that the annual face-to-face meeting should be held earlier in the year because this boosted user engagement when they made personal connections with participants.

Overall, the lessons learned contribute to strategies for successful management and user engagement of future multi-university e-learning projects. Comparisons of expected evidence of training program success and actual outcomes were used to develop Management Strategies for consideration in the next e-learning program to ensure user engagement (Table 2).

Table 2. Recommended Changes in Management Strategies.

These strategies include:

• making the partner agreement tighter;
• making curriculum/credit a requirement;
• having closer ties to industry;
• having each partner bring projects to the table;
• make each partner responsible for one major activity each year as well as one aspect of the program.

These strategies may not be acceptable to potential partners and trainees who have insisted on doing their own research and are too busy with coursework to do these projects. They should then find their own funding and not rely on this program. User engagement requires partners who are willing to work together on team projects side by side with the industry and the health professionals to solve real healthcare problems.

Acknowledgements

This work is supported through grants from the Canadian Institutes of Health Research (CIHR) and the Michael Smith Foundation for Health Research (MSFHR).

References

Covvey, H.D., Zitner, D., & Bernstein, R. (Eds). (2001). Pointing the way: Competencies and curricula in health informatics, Version 1.0, March 27.

Author, Author. (2003). Innovations in health informatics education: Delivering a nationwide PhD/Postdoc Health Informatics Training Program. IMIA WG 13 2003 Conference: Teach Globally, Learn Locally. Portland, Oregon; April 23-25.

Palacios, M. (2005). Virtual collaboration in health informatics research. Master’s Thesis. University of Calgary, Calgary, Alberta.

School of Health Information Science, University of Victoria. (2002, August 20). A pan-Canadian Health Informatics Education and Change Management Strategy, Companion Report: HI Competency Requirements: Current Supply and Issues of Canadian HI Programs.

Marilynne Hebert is an associate professor in the University of Calgary's Department of Community Health Sciences. E-Mail: hebert@ucalgary.ca

Francis Lau is an associate professor an eHealth Chair in the University of Victoria's School of Health Information Science. E-Mail: fylau@uvic.ca