Product analysis
Overview
Product analysis is the systematic examination of instructional products, such as student or client created objects, portfolios, assignments, or writings, to assess the effectiveness of program activities or services.
Suggested uses of product analysis:
- Determining whether program staff are communicating learning objectives
- Determining whether program activities or services are teaching program objectives
- Identifying skills, concepts, and resources students/clients use when creating a product
- Identifying possible changes to student/client products
- Measuring the effects of a program change or innovation as part of a quasi-experimental design.
Limitations of product analysis:
- Not suitable for assessing individual student/client performance
- May not reveal reasons for poor quality work or failure to meet learning objectives
- May be difficult to clearly define criteria
- Analysis is complex and time-consuming
Resource requirements
Product analysis requires minimal resources, although training and experience in content analysis is helpful. The collection, analysis, and assessment of products require a medium to high time commitment. [more]
Plan your product analysis
STEP 1. Describe the context
Include the age, majors, educational background, motivation level, and skill levels of students/clients. Also consider central goals of the program, your ability to implement changes, and how the organizational context impacts your program. A worksheet is available to help you document your program context.
STEP 2. Identify stakeholder needs and develop central questions
Identify what is most essential for clients, program staff, and any organizational priorities that impact your program. Central questions, informed by these needs, specify what you want to learn from an evaluation. A worksheet is available to help you identify stakeholder needs and develop central questions.
STEP 3. Determine the purpose of the product analysis
Because it is too cumbersome to examine every aspect of a program at once, start with clear goals about what you would like to learn and narrow your focus. Whether you analyze products to immediately make program changes or as part of an overall evaluation will determine your focus and what documents to review. A worksheet is available to help you develop and refine your study’s purposes.
STEP 4. Determine how you will use results
How you intend to use results should also guide the focus of your product analysis. If analyzing a particular product will not guide program content or operations, choose a different product or consider another data gathering method. A worksheet is available to help exemplify how to use results after determining the purpose of a study.
STEP 5. Develop product analysis criteria
Establish clear criteria before you analyze products. How deeply you analyze products depends on your central question(s). Make sure you establish clear criteria for ratings such as "none," "little," "medium," or "extensive" and concretely define the relative importance of different criteria.
Additional information
Patton, M.Q. (1990). Qualitative evaluation and research methods, 2nd ed. Newbury Park, CA: Sage.
Sweet, D. (1993) Student Portfolios: Classroom Uses. U. S. Department of Education Consumer Guide, Number 8, Archived material. Retrieved July 13, 2006 from the Office of Educational Research and Improvement (OERI) of the U.S. Department of Education Web site: http://www.ed.gov/pubs/OR/ConsumerGuides/index.html
