1. Introduction
In 1992, the Economics Department at Mary Washington College (MWC) set out to define the skills and abilities it wanted and expected graduates of the program to have. As a logical follow up to the resulting inventory, the Department began to discuss when, where and how in the program, students should obtain these competencies. It became clear that many of these specific elements were taught on an ad hoc basis by some instructors in courses where the elements were deemed appropriate. Thus, some competencies were introduced more than once, while others were not taught at all. The Department decided to address this problem by developing a sophomore level introduction to research methods course (ECON 300). The purposes of this course were: 1. to ensure that all majors be trained in these skills at an early level, 2. to allow faculty of upper level courses to be able to count on students having those skills, and to require their use, and 3. to free up time in those courses for content related material, time which might previously have been spent teaching those skills.
The specific skills of ECON 300 include the use of mathematical methods in economic analysis, computer skills in economics, how to do a literature search in economics, how to find and collect economic data, how to use statistical analysis to test economic models, and how to present the results of such a study orally and in writing. The theme linking these various elements is the preparation of an economics research project, to be done in a series of stages, each stage reflecting one or more of the specific competencies of the course. For more details of the course, see the syllabus at http://departments.mwc.edu/econ/www/ray300.htm .
The course has been taught three times: during the Fall semesters in 1996, 1997, and 1998, the first two times by Steve Greenlaw, and most recently by Margaret Ray.
2. Problems Inherent in the Course
An introductory research methods course taught at the sophomore level suffers from a number of "congenital" problems.
1. More than most courses, this one suffers from a tension between depth and breadth. By construction, it is a survey course, where students will only be introduced to a range of skills. Mastery of the skills will come after one or more subsequent courses (For example, ECON 301, ECON 361 or ECON 462 are each designed to teach one of the topics covered in ECON 300.) At the same time, students need to be able to learn enough about each skill in this course to enable them to complete the research project. Thus, the course needs to do more than simply "cover" the material.
As a consequence, students are likely to be less than fully satisfied with the course, and some will always wish they had taken other, more advanced courses first. One indication of this problem came from a survey given after the first year which asked whether too little time, too much time, or the right amount of time was spent on each topic covered in the course. Of the 13 topics covered, students indicated that too little time was spent on five topics, about the right amount was spent on eight topics, and too much topic was spent on no topics. While it's true that students could get more out of a research methods course (e.g. a senior capstone experience) after having taken the others, they are missing the point of this course. The real question is to what extent does completion of this course enhance the learning in subsequent ones.
2. The product of this course is a research paper; but since the tools need to be taught before the paper can be produced, there will always be a tendency to have a lot of assignments due in the second half of the course, perhaps without adequate time between subsequent ones.
One means of dealing with this is carefully choose the order of the topics to be covered. Another is to omit material which doesn't directly contribute to completing an applied research project. For example, material like the philosophy of science might be better taught in a senior level research course. Another is to combine assignments where possible to more widely space them.
3. Students at the sophomore or even junior level, have difficulty grasping the notion of selecting an appropriate theoretical model to analyze an economic issue or problem. Perhaps this is because few students at this level of their education have reached a high enough level of cognitive development to handle abstractions like this. This was especially apparent in 1996, when one assignment asked them to describe their theoretical model and many students simply couldn't do it.
Perhaps this is asking too much for sophomores. One alternative might be to ask students to simply replicate a published study. The downside of this is the lack of studies done with relatively simply empirical analyses. Another alternative, which was used successfully in 1998, was to limit research topics to either: an aggregate consumption function, or a micro demand function. This seemed to work well. For one thing, we can be reasonably sure that have learned these theories. Perhaps we also need to spend more time explaining how to read and understand published research, emphasizing how analysts more often adapt an existing model than create a truly original one.
3. What are the Costs & Benefits of the Course
We have been able to get a sense of the costs and benefits of the course from four sources of evidence: the College's student course evaluation instrument (SIR-II), an in-house survey designed specifically to assess the outcomes of this course, student written comments about the course, and an econometric analysis designed to determine if taking the course significantly improved performance in subsequent courses.
i. SIR-II Scores
This course regularly generates low course evaluations, at least the first time an instructor teaches it. This result is likely due to several factors. Students find the course extremely challenging and demanding. Many complain that the expectations are too high. This reflects:
1. the genuinely heavy workload;
2. the genuinely challenging material; and
3. the fact that students cannot master all the topics in this course, as they feel they can in a content-oriented course. This makes them feel the course or the instructor is unfair. Why don't students generally react the same way in a principles course? The most likely reason is that the final exam of a principles course does not require them to use what they have learned in the way that completing a serious economics research project does in ECON 300.
The low course evaluation scores in and of themselves say little about the effectiveness of the course. Remember that the SIR-II is given roughly 3/4 the way through the term or during the most intense time of the course, when the research project has not yet come together. In later surveys, discussed below, students report that they learned a great deal from the course; they find the course extremely rewarding and useful in subsequent courses; they find they can do the work if they put in the effort and do all the assignments (which build on each other), i.e. keep up with the work, and can deal with the discomfort of 3. above.
As student expectations of the course load become more well known, one should expect the evaluations to improve-they did the second time Greenlaw taught the course. At the same time, it is likely that the workload and difficulty of the course will continue to bias the evaluations downward.
ii. Outcomes Assessment Survey
In ECON 300, students responded to a survey at the beginning and end of the course. Each survey question was tied to a specific item in the Department's Inventory of Skills and Abilities. The average response for each question pre- and post-course is reported in Table One for the first offering of the course and Table Two for the second offering of the course. Tables Three and Four report the demographic characteristics of the first and second classes. The results show a significant increase in the students' reported proficiency for all but three skills in the first year, and all but two skills in the second year. In the first year, students did not report a significant increase in their ability to use a word processor or to participate in projects or group discussions. This is likely because their pre-course responses in these areas were already quite high. In the second year, students reported a significant increase in their proficiency using word processors, but still not in group discussions or group projects. The different results in year two may have come about because of the use of a new version of word processors, installed on the campus network prior to the start of classes.
iii. Student Written Comments
After each of the first two times ECON 300 was offered students were asked to provide detailed comments about the course. The list of questions is shown in Table Five. For the post-1996 comments, students indicated that they didn't feel that the two principles courses provided enough background for the course. Some students said they thought that ECON 303 or ECON 304, or ECON 361 should be prerequisites for the course. A majority of the students said that the course was much more difficult and required much more work than their other courses. Many students felt the course lacked structure. One frequent comment was that the course should concentrate on a few topics, rather than breezing through many. On the other hand, students overwhelmingly found that the best aspect of the course is that they learned to write economic research papers. A majority of students also indicated that the course increased their interest in taking other economics courses, and majoring in economics (of those not already declared).
For the post-1997 comments, students identified the ability to write an economic research paper as the most important thing they learned. Several students suggested that ECON 301, Mathematical Economics would be a good prerequisite for the course. The students concluded that while demanding, the hard work paid off in the end. As one student said, "it's tough, but not impossible if you stay on top of the assignments."
iv. Econometric Analysis
ECON 462, Economic Forecasting is taught the final semester of the senior year as a capstone course. Students conduct a major econometric research project for the class. Econometrics is a prerequisite for Forecasting.
ECON 300, while not a prerequisite, must be taken before Forecasting due to scheduling. This makes forecasting a good place to test the effectiveness of the ECON 300 preparation. All economics majors must take Forecasting. Since the ECON 300 course was offered as an elective for the first two offerings, some students completed the course prior to taking Forecasting and some did not. Thus, it is possible to look at whether, holding demographic and academic characteristics constant, the course helped students to improve their performance in Forecasting.
Data on student demographic and academic characteristics were collected at the beginning of Forecasting classes. Student grade data was added after the semester. The survey questions asked are included in the Appendix.
The model used to test the long-tem effectiveness of the ECON 300 course is similar to the ordered probit model used by Ray and Grimes (1992). It hypothesizes student performance, measured by the final grade in Forecasting, is a function of demographic characteristics, initial knowledge and whether students completed ECON 300. The hypothesized determinants of student performance reflect he accepted findings of previous empirical studies in the economic education literature (Siegfried and Fels, 1979; Siegfried and Walstad, 1990). Specifically,
Student Performance = f( initial knowledge, demographic characteristics, treatment )
The definitions, means and standard deviations of variables used in the estimation are included in Table 6. Initial knowledge is measured by the student's overall GPA and the number of economics courses they have completed. These measures account for previous student success in courses as well as the extent of their previous exposure to economics courses. Race and gender are included as demographic characteristics and ECON 300 is a dummy variable representing whether the student completed the Introduction to Economic Analysis course.
The estimated results are reported in Table 7. The pseudo R**2 (the equivalent of an R**2 for a probit analysis) indicates that the equation explained 36% of the variation in student performance. This is consistent with the results of other probit equations estimating student performance (Ray and Grimes, 1992). The estimated coefficients of all variable except race were statistically significant. The results indicate that male students received higher grades than female students, holding other factors constant. This is consistent with the findings of other economic education research that find gender differences in performance in economics classes in favor of men (Siegfried, 1979; Ferber, 1990). Higher GPAs and taking more economics courses were found to significantly improve performance in the Forecasting course. Most important for this study, students who completed ECON 300, received significantly higher grades in the Forecasting class, holding all other things constant.
4. Conclusions:
We conclude that ECON 300 is an effective course, which has achieved its objectives for several reasons. Faculty in other courses no longer need to teach the skills covered in ECON 300 (e.g. use of excel, use of descriptive statistics), thus freeing up time to use those skills, and to cover more advanced material. The fact that students complain so much about the difficulty and workload of this course suggests that this is something our students need. Finally, the regression evidence clearly suggests that students who complete ECON 300 learn more in subsequent courses. We need to keep teaching and improving it.