Research | Open | Open Peer Review | Published:
Factors affecting the performance of maternal health care providers in Armenia
Human Resources for Healthvolume 2, Article number: 8 (2004)
Over the last five years, international development organizations began to modify and adapt the conventional Performance Improvement Model for use in low-resource settings. This model outlines the five key factors believed to influence performance outcomes: job expectations, performance feedback, environment and tools, motivation and incentives, and knowledge and skills. Each of these factors should be supplied by the organization in which the provider works, and thus, organizational support is considered as an overarching element for analysis. Little research, domestically or internationally, has been conducted on the actual effects of each of the factors on performance outcomes and most PI practitioners assume that all the factors are needed in order for performance to improve. This study presents a unique exploration of how the factors, individually as well as in combination, affect the performance of primary reproductive health providers (nurse-midwives) in two regions of Armenia.
Two hundred and eighty-five nurses and midwives were observed conducting real or simulated antenatal and postpartum/neonatal care services and interviewed about the presence or absence of the performance factors within their work environment. Results were analyzed to compare average performance with the existence or absence of the factors; then, multiple regression analysis was conducted with the merged datasets to obtain the best models of "predictors" of performance within each clinical service.
Baseline results revealed that performance was sub-standard in several areas and several performance factors were deficient or nonexistent. The multivariate analysis showed that (a) training in the use of the clinic tools; and (b) receiving recognition from the employer or the client/community, are factors strongly associated with performance, followed by (c) receiving performance feedback in postpartum care. Other – extraneous – variables such as the facility type (antenatal care) and whether observation was on simulated vs. real patients (postpartum care) also had a role in observed performance.
This study concludes that the antenatal and postpartum care performance of health providers in Armenia is strongly associated with having the practical knowledge and skills to use everyday tools of the trade and with receiving recognition for their work, as well as having performance feedback. The paper recognized several limitations and expects further studies will illuminate this important topic further.
Training and service delivery organizations have tried for decades to improve the quality and access of healthcare services in developing countries by providing training to health care providers. The assumption has been that the gap in providers' performance is attributable to inadequate knowledge and skills – and therefore, training is what is called for. Millions of training dollars later, healthcare indicators are little improved and providers are still in need of support .
Realizing that training is oftentimes not the only solution, organizations such as IntraHealth International have searched for a key to improving outcomes at the most basic point of contact between the provider and the client. IntraHealth analyzed both domestic research in Human Performance Technology (HPT), as well as the current research on the systems influencing family planning provider performance within developing countries themselves. A line of research on family planning provider performance has focused on capturing a holistic view of the entire family planning program in a given region or country, where data is collected from experts in the area . Although this research has offered evidence of overall program effects on population-level outcomes, it is of limited use in understanding specific conditions affecting the performance of providers at the facility level.
HPT experts in domestic countries, on the other hand, have long established that optimal worker performance is predicated on supporting the worker in a variety of areas, including: (1) clear job expectations; (2) timely performance feedback; (3) adequate environment and tools; (4) internal motivation and/or external incentives; (5) knowledge and skills; (6) capacity to do the job. IntraHealth/PRIME II adapted this list of factors to the international development field, subtracting 'capacity' and adding 'organizational support' to address inadequacies in the other five areas, such as supervision to deliver feedback and clarify expectations for providers. With these premises at hand, organizations have recently developed and utilized the Performance Improvement (PI) approach to improve provider performance and project interventions in the developing world [3–8].
Although the logic of looking at workers' overall environment when analyzing productivity has been widely documented , there is little empirical research exploring the relationships between worker performance and the performance factors, and even less in developing or newly-independent countries. One of a few domestic studies was carried out by Hwang using a mail survey among employees and supervisors at the Office of the Inspector General in the Florida Department of Children and Families. Using an elaborate framework of five components of performance (quantity, quality, efficiency, problem-solving capacity and adaptability) and seven performance factors (performance specification, capacities, knowledge and skills, job/task design, incentives, feedback and resources and tools), he finds that in the employees' questionnaire, feedback is related to performance, in particular the efficiency dimension of performance. Though there are relationships between the factors and performance, in the questionnaires filled out by supervisors, results are less definite. Among the limitations of the study is that its cross-sectional design precluded making cause-effect inferences .
This study uses a simple framework: it assumes that human performance is facilitated and/or hindered by the 'performance factors' listed above and hypothesizes that not all factors are equal in their effects on performance. The objective of the study is to determine which factor(s) have a higher association with provider performance (See study framework in Figure 1). In order to control for extraneous factors, all providers are of similar cadre (i.e. nurse-midwives) and working in the same institution (MOH). Also, the type of establishment is limited to the primary/intermediate level of care (which is the PRIME project mandate), namely polyclinics, health centers and posts. Although human performance is comprised of both behavior and its accomplishments , for this study only the behavioral component of performance is investigated and will be determined through observation of health worker's realization of job tasks.
The study was conducted in Armenia as an extension to a baseline assessment of provider performance to inform the implementation of a USAID-funded project to improve maternal and neonatal health. It constituted a facility-based survey drawing on a sample of nurses and midwives working at three types of service delivery points: polyclinics, health centers and primary posts or FAPs, in the regions of Lori and Shirak. Consenting health care providers and clients were observed while providing two kinds of reproductive health services (prenatal and postpartum care); trained clinical data collectors recorded the completion of clinical and non-clinical tasks using a checklist derived from the MEASURE Evaluation's Quick Investigation of Quality (QIQ) tool . Because it was not always possible to observe a real client visit (the total fertility rate in Armenia is a low 1.7 live births/woman – DHS 2000) , providers were sometimes asked to simulate a situation with a second interviewer standing in for the client (analyses will take this into account). After the provider was observed, she was interviewed about her work environment and the other performance factors.
Using the StatCalc feature of Epi Info 6.04, the desired sample size was estimated to be 300 based on a total provider population of 3000; an expected frequency of 40% (average performance for clinical skills); a worst acceptable error of 15%; a confidence level of 95%; and a 10% margin for loss of data. Given the high numbers required, all nurse-midwives working at outpatient facilities in both Lori and Shirak regions were selected for interview. The final number of providers observed and interviewed was 285.
A total of 11 two-person teams, an observer (physician) and an interviewer (non-clinician), collected data in the field. The clinician carried out the observations while the non-clinicians administered the performance factors questionnaires. Data collectors participated in a weeklong training on proper methods of completing questionnaires and checklists. Study instruments had been reviewed by local clinical experts, translated into Armenian, and pre-tested twice for consistency and comprehensibility.
Field work took place simultaneously in Lori and Shirak regions. An average of two providers was observed/ interviewed each day per data collector team spending a total of 13 working days to complete observations/interviews during August 2002. Some providers, especially at the FAP level, were visited multiple times to complete the interviews, as they were not found at their post. Supervisors ensured correct application and completion of instruments and a field coordinator organized all logistics and performed quality control of the field work.
The average performance score for each provider was compared to the dichotomous answers received from the performance factor questionnaire. The resulting significant factors were entered into a stepwise multiple regression model to discover which factor(s) would become better overall predictors of performance. Some background variables, such as clinic type, provider's age and years working at the facility were added to the model.
All respondents but one were female and 82% were married at the time of the interview. With a median age of 42, they had worked in reproductive health on average for 20 years, with 14 years on average at the interview facility. These variables are dichotomized for the bivariate analysis for ease of interpretation, but were run as originally (continuous) in the multivariate analysis.
Prenatal care skills
The prenatal care checklist included 42 tasks to perform during each client visit and all providers were expected to fulfill them. Performance was substandard across the board (an average score of 14.2, or 38% of the total possible score) with acutely low scores in several triage and clinical areas such as "Washes hands with soap and water and dries them" and "Takes temperature" and counseling areas such as "Informs woman of positive and side effects of medicines during pregnancy" (see Table 1). Though there was a slight difference in performance scores between real (14.9) and simulated (13.5) client-provider interactions, such difference did not reach statistical significance (p = 0.07).
Postpartum care skills
The postpartum skills checklist included 37 tasks that nurse-midwives had to perform during the client visit. Again, performance was substandard, although slightly higher than the prenatal care results (average score of 16.4, or 51% of the total possible score). Again, there were weak clinical areas, such as examining the skin and conjunctivae as well as for swelling and varicosity in legs. Low scores are also seen in preventive areas such as orienting the woman on sexuality, follow up services and contraception (See Table 2). Unfortunately, only 23% of the 278 observations were done in real patients and the performance observed here was significantly higher (score of 19.2) than with simulated observations (score of 15.7).
The performance factors: Are they present or absent?
Clear job expectations
Almost 7 of 10 providers lacked job description; when asked how they knew what to do for their jobs, 69% answered "through oral explanation from the supervisor or other person." When providers were asked whether standards for their performance had been set (i.e. they were told how they should do their job), 78% responded affirmatively; in further questioning, 37% stated having guidelines, 35% other written material, and 21% had protocols.
Motivation and incentives
Asked whether providers received bonuses or raises for good performance, 92% of nurses/midwives responded negatively. Asked what types of non-monetary incentives they received, providers working in prenatal care (clinic-based) responded: "verbal recognition from supervisors" (44.3%), followed by "training courses" (21.3%), and "free/reduced medicine" (14.6%). Providers offering postpartum care (community-based) responded "verbal recognition from clients or the community" (36.3%), closely followed by "respect in the community" (31%), "in-kind products" (e.g. pack of coffee, chocolate bar) (19 %), and "services in return" (e.g. cutting firewood, farming, etc.) (11.4 %).
Feedback on performance
A large majority of nurses/midwives (95%) stated they had received "feedback about [their] job performance." When probed on the characteristics of the feedback, nearly all answers considered it work-related (99.6 %), related to standards and not to behavior (93.6%), immediate and frequent enough to help remember what they did (92.8%), selective and specific (92.5%), and positive and constructive (93.2%). When asked for an example of the feedback, however, nearly two-thirds cited occasions when a supervisor or a doctor had praised them for some good deed, indicating that what was considered to be feedback could also be considered verbal recognition. When asked from whom they received the "feedback", 600 answers listed a range of sources, from clients and colleagues, to the mayor's office.
When asked if they had received performance reviews from supervisors, 64% said they had; in describing the reviews, more than 80% said they were oral and informal exercises.
Supervision visits occur frequently, according to respondents: 94% of providers had received a supervisory visit during the previous 6 months. Visits often lasted over 2 hours, with physicians providing services such as specialty consultations or vaccinations. Though supervision was often and quite long, 86% of providers characterized supervisors' tasks as administrative (e.g. "checks forms, vaccinations") and 24% as clinical ("sees patients and works in the clinic"). In less than 10% of responses, the supervisor was said to have provided feedback on performance. This finding is perhaps explained by administrative visits by nurses and occasional clinical consultations by visiting physicians.
Environment (tools and equipment) and work organization
Asked if their workplace was adequate, 76% said "yes", although only 40% said it was "comfortable". Nearly 60% of providers said they did not have the tools "to do the job well." When asked what equipment was needed, First Aid medicines (13.5%), surgical instruments (10.5%) and scales (9.4%) were listed, averaging 2.8 items mentioned per individual.
As a follow-up question on the availability of equipment and tools, providers were asked if they had been trained in the use of the clinic tools – nearly 75% said they had been.
The last area explored was whether the provider was satisfied with the way the work was organized. Most respondents (75%) answered "yes" to this question.
Knowledge and skills
Forty percent of providers had not received any training in reproductive health. Of those trained, about one third had attended a course in 2002, with the majority attending in 2000. Most providers (82%) believed they had the necessary knowledge and skills to do their job and 97% of them claimed to be able to apply what they had learned to their work.
Relationships between performance factors and actual performance
Among the 19 variables representing all the performance factors, 11 were found to have a significant relationship to performance (see Table 3). The most significant are:
Having a job description
Receiving non-monetary incentives, from the employer and from the community; receiving bonuses or raises, opportunities for promotion and disincentives are found not to be related to performance.
Being satisfied with the organization of the work;
Having the necessary equipment, instruments and supplies was only critical to performance for prenatal care.
Three of four variables in knowledge and skills were found to be significant with prenatal and postpartum care performance: if the provider believes [he/she] has the necessary skills to do the job, whether they have received training in RH and having been trained in the use of clinic tools appear significantly related to all scores.
Receiving performance reviews
Appropriate feedback does not reach significance by close margin (p = 0.047 and p = 0.05) while the more direct question on whether the provider receives feedback about his/her performance was not found to be influential.
Whether the provider received supervision in the last 6 months was related to performance only in the area of prenatal care. The kind of supervision, on the other hand, produced some small differences only in the area of prenatal care, which did not reach statistical significance.
Variables significantly associated with performance at the bivariate level plus available background characteristics were selected as independent variables in multiple regression analyses of performance (the dependent variable) – see Table 2. Analyses were done separately for prenatal and postpartum care performance.
Prenatal care and performance
Eleven of the 21 performance factor variables had a significant relationship with prenatal care performance and were included in the linear regression analysis, as well as the four background variables of importance (age, years working in the facility, type of worker and facility type), despite the lack of association in the first two. Results appear in Table 4.
The table shows that 3 of the 15 variables entered into the regression equation became best predictors of performance in this clinical area. These are, in order of importance (based on the standardized -Beta coefficients), the type of clinic the provider works in, whether the provider [has] been trained in using the tools and whether the provider receives non-monetary incentives from the employer. The adjusted R Square for the model is 0.09.
Postpartum care and performance
There were nine significant performance factors related to postpartum care performance at the bivariate level. These and the four background variables (despite not showing associations) were included in a stepwise multiple regression as before. Type of observation was also included because of its significant relationship with performance. Table 5 presents the results obtained.
This time, the regression analysis yielded four predictor variables. Using the Beta coefficient as a measure of relative order, [has] been trained in using the tools is the strongest predictor for postpartum care, followed by whether the scenario was real vs. simulated. A third variable of significance, having received performance reviews, is followed closely by receiving non-monetary incentives from the community. The adjusted R Square for this model is 0.15.
The first important finding of the study is the statistical significance of relationships between performance and the performance factors. On the one-to-one relationships, 11 of the 21 variables tested had a significant effect on performance and more importantly, 7 of the 11 variables were the same for both clinical areas. They are:
having a job description,
having had performance reviews,
receiving non-monetary incentives (from the employer),
receiving non-monetary incentives (from the community),
Knowledge and skills
having been trained in the use of tools [for the job],
believing to have the necessary skills for the job, and
Having received training in RH.
A second key finding is that out of bivariate and multivariate analyses the performance factors seem to have more weight on performance than critical background conditions such as age of the worker or years working in the facility (see Table 6).
Additionally, it is clear that for prenatal care, the type of facility in which the provider works is definitely associated with performance. This same variable does not exert any influence over performance in postpartum care. This seems to be largely due to the fact that nurses and midwives who work at the higher polyclinic level provide more clinical services than those at FAPs. They also receive better support in supervision and facility maintenance. The same does not hold true for postpartum care. Most providers working in FAPs contact and refer pregnant women to higher level clinics for service. Training in the use of clinic tools and equipment and verbal recognition by employer or community are predictors for both areas of performance – quite a finding for the study. For postpartum care performance, having performance reviews also predicts performance. The type of observation (real vs. simulated) also affected the outcome variable for this clinical area.
Although further studies of this nature are needed to confirm the relative importance of factors, in this study it is interesting to find that a number of aspects in the providers' work environment (as perceived by them) do correlate with observed performance. The multivariate analysis further determines a subset of factors and variables that seem more critical to performance. For example, results seem to indicate is not so much the theoretical knowledge but the practical application of skills in the use of everyday clinic tools that associates with improved performance. This finding seems to reinforce the need for 'essential learning' approaches for improved performance that several training organizations, including IntraHealth, are developing for use in developing countries.
Another predictor of importance is receiving non-monetary incentives by either providers' employers or the community for whom they work. In a context of economic hardship affecting fair and prompt payment of salaries to health workers such as in Armenia, non-monetary incentives in the form of recognition, in-kind contributions, community respect and assistance with services can become powerful motivators to enhance performance. The difference between employer's and community incentives between prenatal and postpartum care seems to indicate where these services occur mostly, prenatal care facilities level and postpartum care at the community level.
For postpartum care, having a performance review from a supervisor as a predictor of performance suggests that this mechanism is most important when people work in less structured environments, often working away from the clinic or the office.
This study is largely of exploratory nature. As mentioned, there have been no empiric studies that we know of combining perceived performance factors and observed performance in the field of FP/RH in low-resource settings. Hence, interpretation of findings needs to be done with caution. Several caveats to the study should be mentioned.
First, the cross-sectional nature of the study, and corresponding associations found, do not imply causal relationships between the factors and provider performance. Though questions related to performance factors were asked with a retrospective connotation (e.g. "in the past 6 months, have you received any supervisory visit?"), memory recall and rationalization may curtail providers' recollection of factors affecting their performance. The multivariate analyses has been carried out by merging data from two different exercises and data collectors, thus arguably adding variation (and a degree of error) to the usual limitations.
Another aspect worth mentioning is the inherent challenges in the application of the factors questionnaire by interviewers. Though pre-tested and improved through pilot testing, several concepts with heavy Western connotations (e.g. what providers answered as "feedback" seems to have been other types of interaction).
We have also used a "proxy" of performance, through the observation of skills. There are other components of performance (e.g. accomplishments) not included in the study. The low R square rates obtained implies our model has captured only a small percentage of the total variation, suggesting that there might be several other factors associated with performance or perhaps better instruments/methods to ascertain it.
In addition, there is an added bias associated with the "Hawthorne effect" in both direct observation and "hypothetical client" approaches to measuring provider performance. Although these approaches are widely recognized as having inherent limitations, both are considered among the best methods for assessing a provider's clinical behavior in a truly technical and systematic way [14, 15]. One such study, however, directly stated that the data collected through direct observation using the QIQ tool, which was utilized in the study, is as reliable as alternative methods for data collection, such as client exit interviews and "mystery clients" . For our purposes, however, where we were assessing a provider's technical performance, direct observation was key for obtaining reliable data.
Furthermore, there is a perceived loss of complexity within the provider-client interaction when a "hypothetical client" is used in lieu of a normal client, as was used in some of the data collection. Case simulation might not replicate entirely the quality and completeness of the normal exchange occurring during direct observation of a provider with a real client, thus producing lower performance scores and possibly affecting the appearance of performance factors in unknown ways . However, once again, "hypothetical clients" have been widely recognized as a reliable and systematic way to assess technical competency when other alternatives are not viable due to low case load for years . In our case, though unavoidable, differences did appear in our measures of performance for postpartum care between real and hypothetical scenarios.
In all, this study has demonstrated the feasibility of the model. As applied, performance was associated mostly with workers' perceptions of having the practical knowledge of tools existing in the facility and of being acknowledged by employers or clients, two important factors to have in mind when considering quality and productivity of work. Further refinement of observation and interview tools are needed. Replication of this study (planned for Nigeria and Bolivia) should shed light about the relative importance of factors in other contexts. Their results can assist development efforts involving human resources by focusing on priority areas, and save valuable financial resources.
Health center or prime post
Human performance technology
Ross J, Stover J: The family planning program effort index: 1999 cycle. International Family Planning Perspectives. 2001, The Alan Guttmacher Institute, 27 (3): [http://www.agi-usa.org/pubs/journals/2711901.html]
Busquet M: USAID's vision: getting results through performance improvement. Speech delivered at International Trade Center; Washington, DC. 12 December 1999
Luoma M, Voltero L: Performance Improvement, Stages, Steps and Tools. 2002, Chapel Hill, NC: Intrah
Performance improvement. [http://www.jhpiego.org/global/pi.htm]
de la Peza L: Lessons from the field: Strengthening supervision using a performance improvement approach. Management Sciences for Health. Presentation at session. 6021, Atlanta, GA, USA, October 21–25, 2001, .2, Performance Improvement – a Common Model Focusing on Health Results, 129th Annual Meeting and Exposition, APHA
Ashton J: Using performance improvement to increase vaccination rates in the Guatemalan Highlands. University Research Corporation, presentation at session. 6021, Atlanta, GA, USA, October 21–25, 2001, .2, Performance Improvement – a Common Model Focusing on Health Results, 129th Annual Meeting and Exposition, APHA
Mielke E, Beattie K, Kaniauskene A, Adrience D: Using COPE® and performance improvement for community involvement. EngenderHealth, presentation at session. 6021, Atlanta, GA, USA, October 21–25, 2001, .2, Performance Improvement – a Common Model Focusing on Health Results, 129th Annual Meeting and Exposition, APHA
Lande R: Performance improvement. Population Reports, J: 52, Baltimore, The Johns Hopkins Bloomberg School of Public Health, Population Information Program, Spring. 2002
Rummler G, Brache A: Improving Performance: How to Manage the White Space on the Organizational Chart. 1995, San Francisco, CA: Jossey-Bass Inc
Hwang S: An examination of the relationships among major performance factors and performance. PhD dissertation. 2002, Tallahassee: The Florida State University College of Education
Fort A: Measuring provider performance: summary of a technical meeting. PRIME Best Practices Series. 2002, Chapel Hill, NC
Measure Evaluation: Quick Investigation of Quality (QIQ). A user's guide to monitoring quality of care in family planning. Manual Series No. 2. University of North Carolina at Chapel Hill: Carolina Population Center. 2002
National Statistical Service [Armenia], Ministry of Health [Armenia] and ORC Macro: Armenia DHS 2000. 2001, Calverton, Maryland
Simmons R, Elias C: The Study of Client-Provider Interactions: A Review of Methodological Issues. Stud Fam Plann. 1994, 25: 1-17.
RamaRao S, Mohanam R: The quality of family planning programs: concepts, measurements, interventions, and effects. Stud Fam Plann. 2003, 34: 227-248. 10.1111/j.1728-4465.2003.00227.x.
Bessinger RE, Bertrand JT: Monitoring quality of care in family planning programs: a comparison of observations and client exit interviews. International Family Planning Perspectives. 2001, The Alan Guttmacher Institute, 27: 2-[http://www.agi-usa.org/pubs/journals/2706301.html]
Leon F, Espinoza V, Espinoza A, Meza B: Expectancy of being observed and quality of care offered. Unpublished manuscript. 2003
Woodward CA, MacConvey GA, Neufeld V, Norman OR, Walsh A: Measurement of physician performance by standardized patients. Refining techniques for undetected entry in physicians' offices. Med Care. 1985, 8: 1019-1027.
Study undertaken through support by USAID G/PHN, under the terms of PRIME II Cooperative Agreement No. HRN-A-00-99-00022-00. The views are solely of the authors. They wish to acknowledge Hayk Gyuzalyan, Rebecca Kohler, Leah Levin, Marc Luoma, Sona Oksuzyan, Marcel Vekemans, Jenna Boley and other PRIME, USAID and Armenia officers for their invaluable help during field work and manuscript writing.
AF designed the study, analyzed and interpreted the data and wrote a first draft of results. LV provided the Performance Improvement framework and definitions, wrote and edited substantial sections of the paper and contributed to interpretations and conclusions. Both authors read and approved the final manuscript.