Assessing Undergraduate Education
in USU's Wildland Resources Department
Undergraduate Education Assessment Plan
The Wildland Resources (Wild) Department at USU offers four undergraduate majors in Conservation and Restoration Ecology (CREC), Forest Ecology and Management (FEMA), Rangeland Ecology and Management (REMA), and Wildlife Ecology and Management (WEMA). Our goal in educating students is for them to become professionals that excel at the work that society needs them to do. We determine how well we achieve this goal by breaking down each of our majors into five similar but unique learning objectives (LOs), then we estimate how well our graduates meet those objectives. We feel that, if a student can accomplish these objectives at a high enough level by the time they graduate, they will be ready for the next step(s) in their careers, whether those are getting a professional position, going to graduate school, or being an ecologically fluent citizen. We also look at data from a number of other sources that can indicate success, including IDEA Student Course Ratings, Graduating Senior Survey, USU and departmental data on employment and continuing education of graduates, professional program accreditation review results, ongoing collection of enrollment, performance, and retention data, and feedback from natural resources employers. Each of these will be presented and discussed below.
Measuring Student Progress on LOs Using Selected Course Grades
Our LOs are real and relevant because we incorporate elements of them into Wild course syllabi and into course objectives specified in the IDEA Student Ratings program (www.usu.edu/aaa/idea.cfm). For key non-WILD courses we examine syllabi and determine which LOs they address. In brief the LOs are (click here for detailed LOs):
LO1 – Knowledge of biology and ecology in relation to your major.
LO2 – Competence in collecting and analyzing data.
LO3 – Understanding of the social context in which NR management is conducted.
LO4 – Ability to communicate.
LO5 – Ability to apply what is learned to manage, conserve, and restore natural resources.
How We Calculate LO Attainment
To estimate our graduates’ level of attainment for each LO we calculate a weighted mean “GPA” of final numeric grades for many of the courses they took while they were here, then we assign each person to an achievement category for each LO based on that GPA. The categories are Achieves Mastery (AM; GPA > 3.333), Achieves Proficiency (AP; GPA 2.667 to 3.333), Approaching Proficiency (ApP; GPA 2 to 2.666), and Lacks Proficiency (LP; GPA <2). The courses we include and their weighting are decided separately for each major and LO by Wild faculty who teach in these majors. The table of courses that were used for each LO and major, along with their weightings, are listed in our Course Table.
Some course grades are used without weighting while others are weighted (multiplied by a weighting factor between 0.10 and 0.75) if we determined that their contribution should be split between two or three courses. An example of how we come up with a score and how the weighting is done follows. A year after a student graduates we contact the USU AAA (Analysis, Assessment, and Accreditation) office and request the final grades for each individual graduate for all of the courses that we include in our calculations. We get a spreadsheet with a row for each graduate and columns for each course that we use to calculate five LO attainment scores for four majors. Many of the cells are blank because different majors and LOs require different courses.
For some LO/major combinations the calculation is simple, involving one course grade, while for others it is much more complex, with 1 to 13 course grades involved (see Course Table). For example, attainment of LO3 (Understanding of the social context in which natural resource management is conducted) for FEMA majors, uses the final grades for three courses they are required to take, APEC 3012 (Introduction to Natural Resource and Regional Economics), ENVS 3300 (Fundamentals of Recreation Resources Management), and WILD 5700 (Forest Assessment and Management). Its formula is SUM(G16, P16, (AN16 x 0.1))/BL16, with G16 being student 16’s final grade for APEC 3012, P16 is their final grade for ENVS 3300, and AN16 is their final grade for WILD 5700, which only gets a 0.1 weighting. If student 16 gets a grade of 4.0 for APEC 3012, 3.33 for ENVS 3300, and 4.0 for WILD 5700 then 4.0, 3.33, and 4.0 x 0.1 are added together, for a sum of 7.73, which is then divided by N = 2.1 resulting in a score of 3.681 or a rating category of AM. The divisor is N in the formula for a mean, and it only includes courses that a student actually took and has a grade for. It also corrects for course weighting.
When the weighting system was developed it was decided that for a given major, a course could be divided into as many as three separate weights that are distributed over that major’s five LOs, and those weights had to add up to 1. In the FEMA example above, the WILD 5700 grade is multiplied by 0.1 for LO3, 0.3 for LO4, and 0.6 for LO5.
How We Use LO-based Attainment Data
Results of our LO-based assessments are assembled annually by the department head with the assistance of the Wild Curriculum Committee. In addition, the department head and the committee provide a meaningful synthesis of the LO attainment results and recommendations to the WILD faculty at the annual retreat. The Wild Curriculum Committee is tasked with keeping this assessment current as degree program requirements change, and assuring that the learning objectives and assessment metrics are providing useful and accurate information.
The links to the LO attainment results are here:
Other Assessment Approaches
IDEA Student Ratings
The IDEA course evaluation system adopted by USU has been useful to us in pointing out courses and teaching methods that may need to be modified. IDEA focuses on student assessment of instructors and course material, rather than faculty assessment of students, but administrators and faculty pay close attention to IDEA scores. Moreover, the IDEA system allows instructors to place greater weight on certain survey questions by selecting the learning objective categories that are most emphasized in a given course. Although not a one-to-one link to our program learning objectives, the IDEA instructor ratings provide a source of existing data to track program performance. IDEA scores for specific instructors and courses are provided by the USU AAA office, and summaries are provided on our Outcomes Data page.
IDEA summaries for the Wildland Resources Department are available here:
• Spring 2013, Fall 2013
• Spring 2014, Fall 2014
• Spring 2015, Fall 2015
• Spring 2016, Fall 2016
• Spring 2017, Fall 2017
• Spring 2018, Fall 2018
• Spring 2019
Graduating Senior Survey
The department head meets informally with graduating seniors at an end-of-semester luncheon. During the luncheon, the students and department head discuss high and low points of their educational experiences at USU. In the past, students attending the luncheon filled out an anonymous survey to self-assess their accomplishment of the 26 historical learning objectives. We will continue to post results from these surveys on the Outcomes Data page; however, beginning in fall 2016 students will self-assess their accomplishment of the new, contemporary learning objectives using the same scale used by the faculty. Because graduating senior participation in the exit survey has been low in the past, ranging from 34% to 69%, we are evaluating strategies to increase the response rate. Results from the student self-assessment survey are posted here using our old learning objectives:
and here using our contemporary learning objectives:
Employment & Continuing Education of Graduates
The Wild Department conducted an employment survey in 2008. USU Career Services also conducted an employment and continuing education telephone survey including graduates from all USU colleges and departments:
We feel that these or similar data are very important because they provide an indication of how well we are recruiting and preparing our students to become working natural resource professionals. We are exploring how frequently these university-wide surveys will be conducted in the future, and also to find out what other information is being gathered (if any) and how we can access it. Such data can be used to alter our recruitments strategies, placement into specific degree programs, and modification of the degree programs themselves in order to keep up with employment and continuing education demands.
Professional Society Accreditation
We feel that an important part of our program assessment is the periodic external assessment by professional accrediting bodies, the Society of American Foresters (SAF) for our FEMA major and the Society for Range Management (SRM) for our REMA major. The WEMA and CREC degrees do not have associated accrediting bodies. Though SAF and SRM accredit those two majors, the entire Wild Department, and to some extent the entire college and university, participate in these assessments of what we are capable of and what we are accomplishing. If successful the SAF and SRM accreditations are valid for 10 years. Our last SRM site visit was in 2018 when we were fully accredited for 10 years, so our next visit should be in 2028. Our last SAF site visit was in 2010 when we were fully accredited for 10 years, so our next visit will be in 2020.
The SAF accreditation report in 2010 mentioned that faculty are actively engaged in program planning and assessment through annual retreats and regular meetings during the academic year. Accreditors commented that outcomes assessment developed by forestry faculty is used to guide curriculum development. Assessment results had been cross-referenced with SAF competency areas, and this approach was considered particularly noteworthy, along with our adaptive management model of program improvement.
Our last couple of SAF and SRM program accreditation reports are available below:
Volume I: A Self-Evaluation Report
Volume II: Appendices
Committee on Accreditation Summary Findings and Action
SAF Accreditation Process Overview
Volume I: A Self-Evaluation Report
Volume 1: A Self-Evaluation Report
SRM Accreditation Process Overview
Ongoing Enrollment, Performance, and Retention Data
The Director of the Quinney College of Natural Resources Academic Services Center maintains and compiles annual data, tracking student enrollment, performance, and retention. These data are used to anticipate instructional and tutorial needs for future semesters, and to make decisions about course prerequisites and recruitment strategies. These data are then provided to the QCNR Dean and Associate Dean, Department Heads, and WILD Curriculum Committee, and is synthesized and discussed annually at the faculty retreat.
Feedback from Natural Resources Employers
Faculty have many opportunities to interact with federal, state and local agencies, non-governmental organizations, and private entities. These interactions are frequently in the context of research, but may also involve placing students in internships or providing recommendations for seasonal/entry-level employment. While these interactions are opportunistic and not a source of quantitative data, they are a valuable source of feedback on the quality and preparation of our students.
Analyses of program assessment outcomes data are shared with faculty annually at our Department Retreat, and discussed during subsequent Departmental meetings. Based on these analyses and discussions, suggested changes to courses, curricula and programs are identified. In large part, the details of any such changes are handled by the Departmental Undergraduate Curriculum Committee, which meets at least once per semester. Changes made as a result of trends in the program assessment outcomes data are then documented and highlighted in this section of our website.
Example – 2017; Establishment of Scientific Writing Center: In response to faculty and employer concerns that natural resource students too often have poor communication skills and difficulty finding courses designated as communication intensive (CI), we obtained CI designation for WILD 5700 Forest Assessment and Management, and are encouraging our faculty and students to make use of the new Science Writing Center, which was initiated in Fall 2017. Additionally, the WILD Curriculum Committee is actively exploring other ways to increase the communications component of our programs without creating excessive additional degree requirements.
Example – 2016-17; Increasing Future Emphasis on Technical Writing in English 2010: An important source of information, though it is unavoidably biased, is the students themselves when the Department Head talks with them at a luncheon just before graduation. Though many are quiet, the students readily speak about problems they encountered. An example from last year, is their expression of a perceived need to get more instruction and experience with technical writing while at USU. We are not sure where this perception is coming from; however, upon reflection on our current curriculum, and also after visiting with the English Department Head about what is, and what is not, included in the English courses that our students are currently required to take, we have begun discussions with the English Department about designating a section (or two) of English 2010 to better meet these needs.
Example – 2016-2017; Updating Degree Program Names: Based on a) enrollment data suggesting that students often transfer into our programs from other majors and b) communication with graduating seniors, incoming freshmen, and potential employers, we determined that the “traditional” degree names for wildlife, forestry and range majors were not conveying the integrated and ecologically-based nature of our programs. This seemed to be contributing to a delay in students finding our programs until they had been at USU for at least a year. The faculty were also concerned that we were missing recruitment opportunities because of this misconception. As a result, in 2017 we changed the names of our majors from Wildlife Science, Forestry, and Rangeland Resources to (Wildlife, Forestry, Range) Ecology and Management, along with Conservation and Restoration Ecology, of which the name did not change.
Example – 2016; Increasing Rangeland Ecology and Management (REMA) Major Flexibility to Increase Enrollments: In response to data showing persistently low enrollment in our REMA major, one of our faculty, Dr. Eric Thacker, talked to a faculty member at Snow College in Ephraim, UT, Dr. Chad Dewey, about having students take a portion of their range courses at Snow College. These students would transfer to USU for their senior year having taken a mix of courses that include a) Snow College courses with content coordinated with us such that credits transfer readily and meet our requirements, b) USU broadcast courses that students take before coming to the USU Logan campus, and finally c) courses taken on the USU Logan campus during their senior year. The need for this was supported with our enrollment data for RLR indicating that, although the numbers are increasing, they are still lower than our other major programs. We discussed this idea at our August 2016 retreat and agreed to initiate this program with Snow College, beginning Fall 2016.
Example – 2015 and 2016; Changes in Commons Course Prerequisites: Based on student retention data obtained from the USU AAA office, we determined that while enrollments in our undergraduate programs are increasing, especially for our Wildlife Science major, an unacceptably low proportion of our incoming majors graduate (48%). This means we are busy teaching students who will not complete their degree with us, which indirectly negatively affects other students through high student:teacher ratios. Partly as a result of this, and also because students are avoiding some classes until too late in their programs, we added prerequisites to some classes to force students to take some critical classes earlier (e.g. calculus), and raised the level of what counts as having satisfied a prerequisite in other cases. For the latter, we realized that requiring students to take and pass Biology 1610 for example, often was not meeting ours or the students’ needs, since passing requires only a “D” final grade. We therefore raised the requirement to meet the prerequisite to a C- for several courses. This has already moderated our enrollment growth, and should force students to face challenging, required courses earlier so that if they aren’t going to make it to graduation, it will be recognized sooner. This will also allow students to consider alternative educational and career options. Based on our Fall 2016 enrollment data for our commons entry course, WILD 2400, it appears that this may have worked. After several prerequisite changes in the past year, enrollment in WILD 2400 went from 58 to 44. Note that decreasing enrollments overall was not our goal in making these changes, but rather to get students into the correct major for them. Though we feel like our wildlife enrollments had gotten higher than the job market could support, we anticipate that some of those students move into less math and science intensive majors within QCNR, like Environmental Studies and Recreation Resources in the ENVS Department. This appears to be happening.
Example – 2015; Modifying WILD 2000, Natural Resources Professional Orientation: Based on enrollment data and communication with graduating seniors, we determined that incoming students were often drawn to our wildlife degree program, but often had misconceptions about our other degree programs in our Department and College. To facilitate improved awareness of the breadth of careers and degree options among our incoming students, we changed our orientation course to provide exposure to these options. Instead of having separate Department-specific orientation courses, we combined this course across Departments, with cross-listings (WILD 2000, ENVS 2000, WATS 2000). The course now provides an orientation to all QCNR undergraduate programs. We anticipate that this will reduce the amount of degree-switching by students, and therefore may reduce times to completion.
Example – 2015; Offering a Wildlife Minor for Fisheries Majors: Based on student feedback, over-subscription to our wildlife major, and under-subscription to (but interest in) the Fisheries and Aquatic Sciences (FAAS) major in QCNR’s WATS Department, we created a Wildlife minor for Fisheries and Aquatic Sciences students. This minor was designed to allow students to focus on fisheries and aquatic sciences while also earning a wildlife credential. Several WEMA students have switched majors to FAAS as a result of this change.
Example – 2014; Undergraduate Curriculum Revision: Periodic revision of our undergraduate curriculum is an example of a decision we have made based on assessment outcomes data. We extensively revised our undergraduate curricula in 2014, changed course requirements, and added several courses to our commons curriculum, while removing others. The Curriculum Committee that designed this revision took into account student survey data and other information (e.g. cancellation of courses offered by other Departments). They also used knowledge gained from interviews with agency personnel who hire our students, and knowledge of who is being hired in what jobs. Specific changes included:
- Addition of WILD 2400 Wildland Resource Techniques; in response to recommendations from employers that our students need more hands-on skills.
- Addition of WILD 3820 Forest Plants and WILD 3830, Range Plant Taxonomy and Function; in response to the loss of plant-related courses in the Biology Department and the US Office of Personnel Management Wildlife Biology Series (0486) requirement for “at least 9 semester hours in the field of botany and related plant science”, we added these courses. WEMA, FEMA, and CREC students are required to take one of these courses, REMA students are required to take both courses.
- Abandonment of the single WILD-wide capstone course, and the establishment of degree-specific capstone courses, in response to low IDEA ratings for the unified capstone course and negative feedback from faculty and graduating seniors.
Example – 2014; Change in GIS Requirement: A specific example of a decision to change our curriculum involves requiring all undergraduate majors to take a GIS class before they graduate. The Forest Ecology and Management and Rangeland Ecology and Management majors had required GIS up to 2014, but the Wildlife Ecology and Management and Conservation and Restoration Ecology majors did not. We made the change to require GIS for all undergraduate majors because of our changed understanding of what students are expected to be able to do when they get summer and permanent positions, and also what we were requiring in later classes that depended on students having GIS skills.
Example – 2010; Increased Emphasis on Timber Harvesting in Forestry Major: After going through professional society accreditations, we were provided with programmatic recommendations and deficiencies. When the Department met with the SAF accreditation team in 2010, they identified a need for our Forest Ecology and Management students to learn more about forest harvest operations. A course on this topic used to exist, but due to changes in the expertise of our faculty and other priorities in our curriculum, that subject was no longer being addressed. We partially addressed the SAF requirement by having the forestry Extension team take students on the annual timber harvest tour that is held in the fall, and also participate in other programs that Extension puts on, like webinars on the subject.