TekBots™ Evaluation Project

Introduction
The purpose of this evaluation is to understand how the use of the TekBots™ robot in the ECE program at OSU affects how students develop depth of knowledge, breadth of knowledge, community, innovation, and skills in troubleshooting. The TekBots robot is being developed at ECE with funding provided by Tektronix. Each entering freshman purchases a kit of parts that eventually becomes the robot. During various classes, students purchase additional kits that are used to expand the function of the robot. The goal is that after four years students will have a robot that is fully and remotely controllable through the Internet.

The robot serves as a platform for learning. The vision is that the robot is a common experience that can be used as a reference in lectures on electronics. In addition, the robot promotes development of a community of students and instructors, trouble shooting abilities, and innovation as each student works on it during lab sections. This platform for learning is more than just a robot. It involves integration between actual robot, laboratory instruction, and lecture. Perhaps the greatest effort for incorporating the robot into instruction takes place in the lab sections. Old labs have been rewritten and new ones have been developed that revolve around the electronics of the robot. In addition, the robot has provided a vehicle to create challenge problems in the lab.

The purpose of this evaluation, then, is to look at the effectiveness of the learning platform that includes lab activities and actual robot. This initial study will focus on the areas of depth of knowledge, community, and innovation. Breadth of knowledge and troubleshooting were eliminated from this preliminary study because the operational definitions involved aspects of the TekBots curriculum that have not yet been developed.

This evaluation uses a survey and exam scores as data sources. Data will be collected from students enrolled in the ECE 271 course. This course was chosen for several reasons. First, the TekBot will be integrated into the lab for the course for the first time in the spring of 2002 so a baseline can be established. Second, the course is required for all freshmen in ECE as well as computer science majors. The ECE majors are all required to take the lab associated with the course but the computer science majors are not required to do so. This sets up a good research opportunity for a causal comparative study where the computer science students serve as a comparison group.

Development of Evaluation Design
This evaluation will focus on the three areas previously mentioned; depth of knowledge, community, and innovation. Each construct was drawn from documents produced by the faculty of the School of EECS. Operational definitions for the constructs were also obtained from School documents and discussions with EECS faculty. Depth of knowledge refers to the students' ability to apply mathematics, science and engineering to solve engineering problems. It also includes the ability to conduct experiments, analyze data, and interpret data. Community refers to the students' ability to work as a team, mentor other students, feel comfortable working with mentors, and develop leadership skills. In this case, mentors refer specifically to upper class and graduate ECE students who often serve as TA's for the labs. Innovation refers to the students' ability to be creative and customize the robots. Innovation should also produce excitement in students that is related to discoveries they have made.

Innovation and community will be evaluated on pre and post surveys administered to the students in the ECE 271 class. The survey (see attached) consists of three parts. Part 1 is only on the pre survey. It consists of questions about demographics. The questions were chosen to separate CS students who are not taking the lab from ECE students who are taking the lab. Some demographic questions are included to check for correlations with other factors that are not directly connected to the laboratory instruction. For example, students who have worked in engineering jobs may have higher scores on innovation because of their previous job experience rather than their participation in the TekBots curriculum.

Part 2 of the survey focuses on community. There are thirteen Likert scale statements related to leadership and mentoring. Students are asked to report on the frequency that each statement has occurred over the previous term. Questions 8, 10, 12, 14, 16, and 18 on the pre survey are all related to mentoring. The questions were modified from a questionnaire on mentoring developed for beginning teachers (Reiman & Edelfelt, 1991). Questions 9, 11, 13, 15, 17, 19, and 20 on the post survey are all related to leadership. Some of these questions were modified from questions in surveys listed in Building Leadership Capacity in Schools (Lambert, 1998). The post survey questions are identical, but the numbering system is different because of the questions relating to demographic information. Subtracting seven from each number on the pre survey can find the equivalent numbered questions on the post survey. Although all the questions on mentoring and leadership had a basis in surveys meant for teachers, they were reviewed by a group of 8 students and faculty from ECE. Questions were then modified based on the feedback so that they addressed issues specific to engineering and the TekBots project.

Part 3 of the survey focuses on innovation. It is evident from the written materials about innovation created by ECE and from discussions with faculty, the construct of innovation was different from the general construct of creativity that appears frequently in the literature. More specifically, ECE faculty consider innovation to be something that is specific to context so that it may be evident in one setting for an individual and not in another setting. This is consistent with the Kirton Adaptor-Innovator theory (Kirton, 1989). Kirton developed an instrument to measure innovation and has applied that instrument to a variety of groups including engineers. Unfortunately, the instrument is not available for use by this evaluation project. Sprecher (1963) attempted to develop an operational definition of innovation and creativity in engineers. The work did not result in an instrument for measuring innovation, but it did provide a method for refining an operational definition for innovation. The process used by Sprecher was used for this project with the result being a set of questions chosen that allow participants in the evaluation to self-report on innovation.

The development of part 3 of the full survey requires some more detailed description of the method used to develop some level of construct validity. A list of 72 statements in 12 categories describing behaviors that could be related to innovation was derived from the work by Sprecher (1963). The modifications mostly involved creating a more gender-neutral language. Eight ECE faculty rated each statement as to whether or not it was an important characteristic of creative and innovative engineers. The scores were compiled and questions in five of the original 12 categories were used in the final evaluation survey. The categories of novel ideas, many solutions, likes problems, technical competence, and valuable answers each are represented in the evaluation survey by four questions. The table below identifies the questions associated with each category on the post survey. The post survey questions are identical, but the numbering system is different because of the questions relating to demographic information. Subtracting seven from each number on the pre survey can find the equivalent numbered questions on the post survey.

Participants use a Likert scale to respond to each statement on the survey of innovation. In this case they are asked to consider how others would describe their behavior over the previous term. The five-point scale ranges from strongly agree to strongly disagree.

The third construct being measured is depth of knowledge. This measurement will come from scores on exams given in the ECE 271 lecture. Currently the School of EECS records composite scores broken out by course learning objective. This will allow some level of content validity. There will be one change in the recording process. For the purpose of this evaluation, the scores will be recorded by major. This way, composite scores for ECE majors who take the lab section can be compared to those of computer science majors who do not take the lab.

Data Collection
Surveys will be administered to all 300 students enrolled in the ECE 271 course during the spring term of 2002. The pre survey will be administered during the first week of class prior to any lab meetings that would involve working with the robot. The post survey will be administered toward the end of the term after the labs have been completed. Responses will be digitized for future analysis. Students will use an identifying number on their response form to allow analysis at the level of the individual student.

Proposed Data Analysis
The purpose of the evaluation design is to compare the measurements of depth, community, and innovation between two groups of students. One group, the treatment group, participates in the lab for ECE 271. The comparison group does not. A t-test will be used to compare demographic data between the two groups to determine if the only significant difference is related to enrollment in the lab. A paired t-test will be used to compare responses on the survey for community and innovation. A t-test will also be used to compare scores on exams between students who take the lab section and students who do not. If the TekBots program has a significant impact on depth, community, and innovation there should be a significant difference in the scores.

Time constraints have limited the possibilities to gather reliability data on the survey and test scores prior to actual data collection. The pre survey data will be analyzed using coefficient alpha to determine the reliability of the instrument since it is using Likert scales. In addition a factor analysis will be applied to the community and innovation sections to determine which items most clearly define separate categories. This data will be used to determine what items are most appropriate to use in the final analysis of the post survey.

IRB Approval
An application for Institutional Review Board approval has been submitted to the Research Office at Oregon State University. Attached to this report is a copy of that application with surveys and informed consent documents.

References

Kirton, M. J. (1989). A theory of cognitive style. In M. Kirton (Ed.), Adaptors and innovators: Styles of creativity and problem solving. (pp.1-33). London: Routledge.

Lambert, L. (1998). Building leadership capacity in schools. Alexandria, VA: Association of Supervision and Curriculum Development.

Reiman, A. J. & Edelfelt, R. A. (1991). The opinions of mentors and beginning teachers. Research Report: North Carolina State University. (ERIC Document Reproduction Service No. ED329519).

Sprecher, T. B. (1963). A proposal for identifying the meaning of creativity. In C. W. Taylor & F. Barron (Eds.), Scientific creativity: Its recognition and development. (pp. 77-88). New York: John Wiley & Sons.