Product analysis
Overview
Product analysis is the systematic examination of instructional products, such as student created objects, portfolios, assignments, or writings, to assess the effectiveness of the course or instruction. Responses to exam or quiz items may also be considered a type of instructional product. Use product analysis to gain insight, assess changes in instructional practice, or measure the effects of instruction.
Suggested uses of product analysis:
- Determining whether you are communicating learning objectives
- Determining whether assignments are teaching learning objectives
- Identifying skills, concepts, and resources students use when creating a product
- Assessing strengths and weaknesses of assignments or products
- Identifying possible changes to student products
- Measuring the effects of an instructional change or innovation as part of a quasi-experimental design
Limitations of product analysis:
- Not suitable for assessing individual student performance
- May not reveal reasons for poor quality work or failure to meet learning objectives
- May be difficult to clearly define criteria
- Analysis is complex and time-consuming
Resource requirements
Product analysis requires minimal resources, although training and experience in content analysis is helpful. The collection, analysis, and assessment of products require a medium to high time commitment. [more]
Plan your product analysis
STEP 1. Describe the context
Include the age, majors, educational background, motivation level, and skill levels of students. Also consider central goals of the course, your ability to implement changes, and how the instructional setting impacts your course. A worksheet is available to help you document your instructional context.
STEP 2. Identify stakeholder needs and develop central questions
Identify what is most essential for students, your needs, and any organizational priorities that impact your course. Central questions, informed by these needs, specify what you want to learn from an assessment. For example, "Are students effectively using online technology in my course?" A worksheet is available to help you identify stakeholder needs and develop central questions.
STEP 3. Determine the purpose of the product analysis
Establish clear, focused goals that specify which areas of learning or instruction you will evaluate and which products you will examine. The analysis should include questions about the instructional purpose of the products, how you and students are using them, and how they are contributing to learning. For formative evaluation, product analysis helps you assess students' learning and address misunderstandings, ambiguities, or deficiencies. For summative evaluation, product analysis helps you assess the quality of instruction or learning after the course is over. A worksheet is available to help you develop and refine your study’s purposes.
STEP 4. Determine how you will use results
How you intend to use results should also guide the focus of your product analysis. If analyzing a particular product will not guide course or program content or instruction, choose a different product or consider another assessment method. A worksheet is available to help exemplify how to use results after determining the purpose of a study.
STEP 5. Develop product analysis criteria
To decrease bias, establish clear criteria before you analyze products and define ratings such as "excellent," "meets criteria," and "needs improvement." If some criteria are valued more than others, specify their relative importance or weight. Refer to the teaching assessment planning process for additional help in developing your criteria.
Additional information
Patton, M.Q. (1990). Qualitative evaluation and research methods, 2nd ed. Newbury Park, CA: Sage.
Sweet, D. (1993) Student Portfolios: Classroom Uses. U. S. Department of Education Consumer Guide, Number 8, Archived material. Retrieved July 13, 2006 from the Office of Educational Research and Improvement (OERI) of the U.S. Department of Education Web site: http://www.ed.gov/pubs/OR/ConsumerGuides/index.html
