Surveys as a Method to Describe Community-based Organizations

by Ted Calvin

During the past year, I have been a part of a research group whose charter was to evaluate the efficacy of community technology centers in the Austin area. This group of community-based organizations is a diverse collection, to say the least. Our pool of eleven centers ranged from a multiple-site non-profit that serves several hundred participants to a church-based program operated by one man that serves a much smaller number of clients from one neighborhood. Some of the centers had begun programs within the past year, whereas others had been established for over a decade.

Our attempts to measure and describe these programs began with an intense effort to get to know the staff and participants of each place on a personal basis. Then we embarked on a series of surveys to gather information about the group in a standardized manner for analysis and presentation. We achieved commendable success in the former enndeaor, and mixed results in the latter. I believe one of the most useful pieces of information that was actually generated by the survey project was a better understanding of what can and cannot be captured effectively by a survey.

The Theory

We had five basic hypotheses that we wanted to examine, all of which tried to relate the success of each center to some quality of the center. The hypotheses are the following:

1. CTC's that are integral parts of the community foster empowerment because the participants are comfortable with and involved in the programs.
2. A CTC with comprehensive programming will be more effective at developing community competence.
3. CTC's that provide structured learning opportunities are more effective at developing community competence.
4. CTC's that are sustainable foster empowerment because acquisition and maintenance of diverse funding sources, physical resources, skilled activists and technological expertise contribute to the collective development and capacity of participants, staff and the community at large.
5. CTC's that build social capital are more effective at developing community competence.

The hypotheses included a few relevant corollaries, and each hypothesis was broken down into several research questions whose answers would either support or disprove the hypothesis. From the research questions, actual survey questions were generated to be administered to the people involved in each CTC (staff, clients, or executive directors). The following are a couple of sample research questions associated with hypothesis 5, the social capital hypothesis:

1. As a result of their increased social networking at CTC's, do participants have increased physical opportunities?
2. Are there physical multiple partnerships between organizations?

Interesting to note is that very few of the research questions probe the actual effectiveness of the center. The social capital questions does the best job, with several questions related to the actual development of community competence and social capital. The majority of the research questions instead focus on inputs and outputs, with little space devoted to outcomes. Lacking is a working definition of 'effective,' either with respect to an individual working hypothesis or on the scale of the center or community as a whole.

Because of this skew towards inputs, the research questions set the stage for surveys that are mostly descriptive. We, the researchers have created a list of qualities that we believe an ideal CTC would exhibit. The surveys allow us to look out into the world to see if centers do in fact fit our model. Were our sample size large enough and diverse enough, perhaps it would be possible to later label each center as 'effective' or 'ineffective,' or perhaps grade them against one another. As it is, we have ten centers that are all successful in different ways. Consequently, our surveys allow us to discuss practices that succesful CTC's in Austin use, and others that are not in use. Actually connecting cause and effect is a task left for future research.

The Surveys

The first survey used was a list of questions to be used as a guide for conversations with the executive director of each site. This survey contained 31questions, most of which required a free-formed, text answer. A typical question is the following:

What kind of feedback mechanisms does the center use? (For example: suggestion box, staff-participant liason, participant advisory council)

and the answer:

The center relies on parent-teacher conferences to monitor the development of students.

No attempt was made to perform any sort of quantitative analysis with these responses. This executive director-level survey was an extension of the getting-to-konw-you approach employed before the surveys.

The second survey was a 36-question survey that was sent to staff members at each center. Each question in the staff survey was a multiple-choice or yes/no answer[1]. The number of respondents for the staff survey was substantially larger than for the executive director survey. To facilitate data analysis, all of the surveys from each site were aggregated to form a single, composite survey for that site. Some of the questions were very simple, such as, "Are community members on the board of directors?" Others were quite complicated:

7.) Which of the following software do you have available?

The data for all centers was presented together for inspection. Again, no quantitative analysis was attempted with this data set. Whereas there had been no question as to how the first survey should be analyzed, the second set of surveys was in sort of a no-man's-land. The results were multiple-choice or numerical, but the diversity of centers and the varying number of respondents at each made any sort of statistical anaylsis problematic. Furthermore, many of the questions were of the trivia type, where asking multiple respondents the same question served no obvious purpose.

All of the difficulties encountered in the second round of surveys also attended the third round. The third survey asks questions of a similar format and content as the second, but was administered to program participants. Rather than attempt any formal data analysis on these results, the data was returned to the prose format. All of the surveys from each CTC were summarized in a short CTC profile that highlighted results of interest.

Results

The CTC profiles capture interesting material from each center and make for a nice read, but what of the original research intent of the surveys? I believe that we got some really good information out of the survey effort, but we also have a lot of room for improvement. The following are what I consider to be the high points and the low points of the surveys. They are the types of inquiry that worked well, and others that did not.

High points

The executive director survey is excellent at getting to know the person who runs the show and their views on the organization. It is an ideal way to get statistic-type information, such as number of years in operation, funding sources, reason for choosing current location. The best thing that resulted from the E.D. surveys was seeing each director's philosophy for leading the organization, and comparing with different kinds of CTCs and directors with different backgrounds. Also, the E.D. is good person to ask about vision for the future-- where the CTC is headed in the next few years, what challenges and opportunities await them, and how they would like to see the center grow.

The staff surveys are a good way to see how well networked individual staff members are, and how aware they are of the community they serve. To the extent that we asked them the same questions, the staff survey shows the degree to which the E.D. and the staff are all on the same page as far as the mission of the center, what the cneter offers, and where it is headed. Some things make sense to quantify, to enable comparisons across centers: number of other CTC's staff interact with, how long they've been with center (we didn't ask that one). Other questions get at qualitative data about the center, such as staff involvement in participants' lives, overall satisfaction with the center, and needs for improvement. These answers should not be quantified, because trying to attach a number to such an answer often renders it meaningless.

The participant surveys actually hit enough people that it would make sense to gather statistical data about centers at this level. Questions that get at the participant's reltaionship to the center are good-- do you live in walking distance, how long have you been coming here, how involved are you in a specific activity at the center, has the center broadened your horizons in this way? In particular, questions that measure the change in a participant's ability to do something, such as stay in touch using email or find useful information on the web, are good because they measure increases in competence caused by the center using a before-after format. Asking about participant satisfaction is good, but like staff surveys, should be kept qualitative since numbers (0-5 scale) may not compare well across centers and communities.

Notes

1. Except for the last question, which asked if there was anything the staff member would like to add.