Annex IX: Surveys

Consider the creation and distribution of a questionnaire on local priorities. The questionnaire on local priorities asks residents to identify their most significant issues, challenges and needs. Local governments can deploy various types of surveys to collect data on citizen priorities, each of which is explored in further detail below. 

Illustrative example: Moldova local priorities questionnaire

Take the following example of a questionnaire, which is translated from one used by an actual mayor and her team in Moldova: 

This questionnaire is straightforward and easy to understand. It aims for the respondent to place a check next to one of five preselected options, allowing residents to tell city hall more about why they selected a particular option. In Moldova, the mayor and her team went door to door in their community, aiming to hand the survey to residents from each neighborhood; they returned after some time to collect the survey. The questionnaire also captured both quantitative and qualitative data from community members. 

• Quantitative: Residents were asked to rate a preset list of local issues in order of priority.
• Qualitative: Residents were able to define the most important negative effects or challenges associated with these issues. Structured as an open-ended short-answer question, residents could provide additional detail on why they rated issues as they did.

When it comes to analyzing the survey results, the mayor and her team simply counted the number of checks for each issue; this will help them determine which issue or issues are the most important to residents. Once those issues have been identified – let’s say, for example, roads earned 50 percent, and water supply scored 30 percent of responses – the mayor and her team can look back through the “roads” pile of questionnaires to read what people wrote about why roads are a particular issue. There are likely a number of subordinate challenges making roads an issue – poor snow clearing, prominent potholes or degraded paving – the mayor and her team may need to conduct further consultations. This could include discussion with the local transit department to see what maintenance is planned and may involve public meetings in various neighborhoods to discuss the conditions of their roads.

Consider the quote from a Moldovan mayor using this survey-based approach. Overall, the survey serves as a guide for further action, directing the mayor and her team toward a general priority that requires follow-up and attention and guiding their efforts throughout the year. Ultimately, some action should be taken, such as funding for repaving a particularly troublesome stretch of main street. In addition, the mayor can use the contact information provided through the survey to update residents with the survey results, send invitations to public meetings to discuss roads or other top priorities, and send summary letters of actions taken or planned on these priorities.

There are multiple types of surveys, differentiated by their goals and the methods used to get responses, and the Moldova questionnaire is an example of a “concerns survey.” 

Before issuing a questionnaire, it will be worthwhile testing the questionnaire on a small number of residents (perhaps 10 individuals). This can help to identify any potential issues with the questionnaire: unclear wording, a confusing ranking system, or missing priorities or options that should be included. Additional recommendations include: 

• Wording the questionnaire in accessible and locally specific language
• Including valid contact information and be receptive to residents’ questions
• Ensuring compliance in data collection information and methods with any confidentiality and privacy legal obligations (e.g., GDPR)

When to conduct a survey 

Analysts and pollsters rely on surveys for three reasons: (1) direct, often quantifiable, answers to questions; (2) anonymity; and (3) randomness. Note that often, these pollsters and analysts have funding, technical training and capacity, and human resources (paid enumerators and interviewers) to help them collect, analyze, and present data. But even if you don’t have those resources, you can design simple surveys to serve your purposes.

You should conduct a survey when: 

• You want to learn about community needs or identify top priorities.
  • Relatedly, you want to hear about needs you didn’t know existed (affecting people or groups outside your immediate social circle or neighborhood).
• You want objective feedback.
  • You want respondents to be more honest and open than you might get through existing or public feedback channels (due to privacy concerns, fears of reprisal, embarrassment or social stigma, or even a simple fear of public speaking).
• You want a reliable and systematic way to document or quantify community needs and priorities.
  • This proof of needs can be used to apply, lobby or advocate for funding (from government ministries, international organizations or other sources).
• You plan to take actions or set an agenda based on the responses.
  • You can point to the survey results as the basis for taking action. Results can be further used to drum up support for your plan of action (should you face resistance from opposing stakeholders) and aim to get more people involved in your effort.

Other reasons to conduct a survey: 

• You’re involving community members early on and helping them start thinking about pressing issues in the community. This can be helpful if you want greater public engagement down the road, such as in the dialogue and monitoring stages.
• Surveys reach community members directly, meaning that you do not have to rely on professionals, service providers or others with vested interests in the issues.

Writing your questions

This guide assumes limited funds, time and technical resources or know-how (such as how to conduct rigorous statistical analyses). 

Strengths of a simple survey

Consider replicating the strengths of the simple survey issued by the Moldovan mayor above. 

Strength 1: It is short: 

There are two questions: What issue is most important to you? How would you recommend solving this issue?  

Note on framing: After asking “what” issue is most important, it is better to ask “how” that person would fix it (rather than “why” it bothers them). Asking “how” puts the respondent in a solution-oriented frame of mind (which is where your team should be already) and offers ideas that you might find useful. Asking why may simply be asking the obvious. For example, it’s clear why a pothole or unemployment is a problem. What may be less clear is the solution: take the pothole. Perhaps the pothole is regularly filled, but oversized trucks quickly degrade the road again somewhere else; the solution would be weigh stations or redirecting oversized traffic to roads.

Strength 2: Responses contribute to scoring or ranking: 

Each checkmark indicates the issue is a problem. The scoring system is also simple: Add up all of the checkmarks, and the issue with the most checks is the top priority, and so on.

While there are instructions to respond with only one checkmark, respondents can theoretically still select more than one. In that case, your team would need to decide how to score those responses – would these responses have equal or lesser weight in the overall count? For example, if they selected two issues (say “water” and “local roads”), would the sums for each of those issues rise by one (+1) or by one-half (+0.5)? There are advantages to both: If each gets a vote, you have a clear count of the exact number of residents who said, “This is a top priority for me.” If instead you split the votes, you ensure that each resident’s opinion is counted equally (picking one issue gives that response more “weight” compared to someone who picked multiple issues).

Strength 3: The survey is flexible and has an open-ended option: 

Respondents can reject the choices provided and give their own top priority. Respondents also have space to write why a particular issue bothers them the most (and perhaps how it should be solved). These comments are more difficult and time-consuming to enumerate and analyze (even if the space provided is small). But these “open-ended” elements are not so open-ended – nearly all comments will generally relate to one of the four provided issues and can more easily be categorized and tracked. The exception, of course, is if someone writes in their own issue (although, as mentioned, this has advantages).

Strength 4: The survey provides information to the respondent: 

The respondent learns the name of their representative and how they can be contacted by phone or email. The survey also shares (although with minimal detail) the next step: We [the government] will respond to your answers. 

Strength 5: The survey asks for contact information: 

The respondent can volunteer their contact information; it is not required. If the respondent selected a particular issue and wrote a specific enough complaint in the provided space, a member of the feedback loop team may follow up with them to learn more about the problem and fix it. Otherwise, their contact information will be used to provide general process updates (results of the survey, invitations to public meetings and progress reports). 

If possible, use a contact management system that the municipality already has to keep track of this information and leverage it for future communications. Note, however, that having this information may pose a legal concern if it is protected, personally identifiable information – consult your data governance and legal framework to ensure you have the proper permissions to handle this data. 

More advice for writing questions

In addition to the above, consider the following when writing your survey questions: 

Keep in mind your objectives (and keep your objectives specific)

Your objectives should guide your question design. What do you want to know? How can you frame your questions to get that data? 

Note that the Moldovan example offers a small subset of general issue areas: water, roads, playgrounds and cleanliness. The municipality has authority over these services, but the municipality has other responsibilities – why were these four selected? While the survey allows you to write in your own response, this initial list is a form of choice architecture. Choice architecture is the idea that “presenting choices in different ways will impact decision making” about those choices.

By listing these issues, city hall is encouraging respondents to pick one of them. You can also take advantage of choice architecture in your survey, but you should not abuse it. For instance, if you want a ranking of priorities, do not simply list your campaign priorities, forcing the results of the survey to inevitably result in the endorsement of one or more of your priorities. Instead, like the Moldovan example, create the option for a write-in/other answer, but also direct responses to predetermined categories. Focus on areas your team thinks are likely priorities and those within the control or remit of your local authority. 

Ensure that your objectives are specific enough: You are not trying to discover every resident’s issue related to the local authority administration. The survey should be focused on its goal: from a preselected list, rank specific priorities, and identify key problems. 

Keep in mind your audience

If you hope that anyone and everyone in your community responds to the survey, it must be clear and straightforward. A resident should not require specific or technical knowledge to answer the survey. 

Be clear and test your survey 

Your survey may seem clear to you and your team members, but you must test it on someone who – like your average resident – has never seen it before and may only have a vague understanding of what it will be used for. They will help you understand which questions are unclear, whether they are confused by ranking systems, and if you need to make edits before printing several hundred more.

Case study: In one partner city, a survey asked residents to select their top five issues from a list (of more than five) on a scale of one to five. A score of one meant it was most important to them, and five least. Some residents reported being confused by the numbering system and responding instead with the reverse (5 points being more important, 1 point being less). Had the municipal team tested the survey on a select number of residents, they might have identified the issue in time to redesign the survey. It would have been better to instead offer a predetermined scale, where each issue is scored between 1 and 5 (see concerns survey below) or to allow residents only to pick one issue as a priority. 

A warning about open-ended questions and knowing your limits

As discussed above, responses to the Moldovan example survey will be extremely easy to analyze. Check marks can be added up. The write-in sections are small, making it manageable to read and track a few hundred – some residents may go off script, attaching pages of text about a specific issue (maybe a dangerous intersection) that they are passionate about. But most will use the survey as directed. 

Case study: One partner city included an open-ended “any other feedback” question at the end of their survey. They received feedback amounting to more than 1,000 additional issues through those written comments. The city’s feedback team did not have the time or skills to sift through those responses and categorize them; a skilled data analyst had to be hired, but the comments were only usable in a general sense. The city could not individually respond to these comments. The key lesson is to use open-ended questions extremely cautiously. If you expect to receive hard-copy responses, open-ended comments will be prohibitively burdensome and complicated for the city to analyze and respond to effectively. Instead, leave only a small space for written topics on preselected issues. Save open-ended questions for public meetings and dialogue sessions, where additional detail can be gathered and individuals can be followed up with on a case-by-case basis.

Suggestion about framing: Open-ended questions create the assumption that the government WILL respond to the issue. Governments that are unprepared to respond to each open-ended response should change the framing or wording. For example, asking respondents for a comment (or similar word/phrase that does not imply a response) eases this concern.

Self-administered vs. interviewer-administered survey

Finally, note that the Moldova survey is an example of a self-administered survey, meaning that the person responding to it is also filling it out. Conducting a self-administered survey assumes that your respondents can read and write in the language of the survey. If your community has a low literacy rate, however, or is linguistically diverse, you may need a different approach in which interviewers (who should be trained to reduce bias and not influence a respondent) survey respondents; this can be done in person or over the phone.

Types of surveys

The following box describes sampling, its relevance to your data collection, and the kinds of surveys that may be useful to you as you seek to understand community priorities. 

Resource considerations and constraints

Conducting a survey presents a logistic challenge: How will you get your questionnaire to respondents, and how will your respondents get the survey back to you?

You have, generally, two options: a written survey or an online survey. There are other methodologies, but we are excluding in-person and phone interviews (as they pose practical challenges, and interviewers must be trained and remunerated). Instead, we are looking for a survey that respondents can respond to in their own time and with privacy.

Consider the following when deciding which survey design and distribution method to use:

• Cost of materials
  • Printing costs
  • Postage (if mailing)
  • Survey design software (if conducting an online survey and not using Google Forms)
• Staff time required for distributing surveys
  • How many staff hours will be needed for door-to-door canvassing, staffing booths if convenience sampling, or filling and addressing envelopes? If you select door-to-door canvassing, how far will you need to travel: will you need a ferry, bike or car to reach your sample, or can you walk?
• Staff time required for following up and collecting surveys
  • If door-to-door canvassing, will you have staff return to each door and follow up on or collect surveys, or can residents drop the survey off somewhere?
  • If you use drop boxes or mail surveys, will staff need to periodically check mailboxes or drop boxes?
  • If you do not receive enough responses, will you have staff send reminders via mail, expand convenience sampling efforts, or conduct other outreach to encourage survey return?
• Staff time required for analysis of surveys
  • Once you collect a sufficient number of surveys (based on your goal sample size), you can begin analysis. If your survey was issued online, transferring responses into a single tracker is very easy – Google Forms will automatically create a Google Sheet of the responses.
  • For hard-copy surveys, you will need to enumerate the responses – i.e., manually count with pen and paper, or type the responses manually into an online format (Google Sheets will work well here and is free). Depending on the complexity of your survey, this process can be time-consuming and a little repetitive, but accuracy is important. If this is your process, think ahead: Surveys with open-ended questions and responses will take significantly longer to type than simply marking a “1” in a column for each response.
• The cost in time and effort for the respondent
  • You should also consider the cost for the respondent – how long will it take to complete the survey, and how difficult or easy is it for the respondent to return it once they are done? You should aim to make it as easy as possible to return the survey – there should be no monetary cost and as minimal time or convenience cost as possible.
    • Online surveys are easiest to return – simply click submit, but you do need an internet connection, etc.
    • Mail-in surveys should come with an addressed and stamped return envelope so that the respondent can simply mail it back. Paying for this postage may incur a cost for the municipality.
    • Surveys acquired by door-to-door canvassing, convenience sampling, or otherwise handed out or picked up may be returned via mail (if postage and an envelope are provided) or in a drop box. Depending on the size of your community, drop boxes should be conveniently located, secured against weather (and vandalism/mischief, perhaps within view of someone or the public), and easily accessible (e.g., in a first-floor lobby).

Real-life example: In Moldova, feedback loop teams repeated their door-to-door canvassing routes and returned in person after two weeks had passed to collect the survey; Moldovan municipal leaders consider this step an important part of demonstrating accountability and interest in the responses. In addition, picking up a survey in person makes it clear to the resident that their response has been received and will be counted, compared with clicking “submit” on an online survey or dropping a response in the mail or a drop box. That said, Moldovan communities are usually rather small in geography and population, making a repeat of the canvassing approach slightly easier. In Slovakia, city officials initially made efforts to pick up surveys but had to do so on their own time and often missed residents who were not at home; to compensate, they opted to create a drop box in city hall, which helped to address the issue.

Door-to-door canvassing

One common request to NDI by elected officials participating in feedback loop processes featuring door-to-door canvassing is for a script. Even those who publicly campaigned and won elections to public office get nervous about talking face-to-face with their neighbors. In reality, there is nothing to be stressed about. People love being asked their opinion, meeting new people can be fun and interesting, and it gives people a chance to meet their elected officials and public servants. In addition, canvassing is a chance to observe: getting outside and walking around your community is a great way to observe issues you’re asking about in practice, get a sense of quality of life, and see streets and neighborhoods you might not have visited before.

Door-to-door canvassing tips:

1. Have a script (see below).

You do not need to memorize the script and should avoid reading it when the door opens. Instead, view the script as the key pieces of information you think the resident should know about why you are standing in front of them: who you are; why you are giving them a survey; what is in the survey; how to return it/hand it in; and what will happen next.

2. Be prepared for questions.

Your team should generate frequently asked questions (FAQs) and answers about the preselected issues on the survey so that volunteers or other team members can refer to them during canvassing. This can include responses to expected questions: “Why only these issues, not X?” “Who will see this survey?” Your team members should be prepared to answer questions about how data – such as contact information – will be stored and kept secure.

If you are a volunteer and the question has a subjective response (or you don’t know the answer), it is likely best to direct the resident to contact city hall using the contact on the survey. If you are the elected official and don’t know the answer, you can take down the resident’s contact information and follow up with them later.

3. Have a plan for outreach and map your route.

If you are trying to reach a certain number of residents, have a sampling plan, or otherwise have a large area to cover, you should have an accurate map of the area and plan your route with your team members. This way, you can avoid knocking on the same doors and break up the outreach into manageable metrics and timelines. Is it realistic to reach 10 to 15 doors an hour? How many days of outreach would it take you to reach 100 residents? Use the tools of the municipality (such as census data) to your advantage. For example, do you know which homes or buildings are empty and can be skipped?

As part of your plan, consider when people are most likely to be home. If you are canvassing at noon on a Wednesday, who is most likely to answer the door? If you canvass on a Saturday morning instead, would more people be home? Consider work schedules, times of prayer and worship, care duties, and other social activities and reasons that would have people out and about.

4. Keep things moving but be polite.

Related to the above, canvassing can be time-consuming, but in order for the survey to be representative and useful, you will need to reach a large number of residents – or as many as possible. You should aim to be courteous and respectful, but you have a job to do, and efficiency is important. At times, an excited resident (perhaps someone you know) may want to keep you talking or have a debate, and you’ll need to find a way to politely move on. Having a script can be useful in these situations. If you spend more than 10 or 15 minutes at every house, you will spend significantly more time than your schedule allows collecting responses.

5. Have a plan for when all family members are not home.

Depending on your survey’s goals and sampling (or its lack of sampling), you may want responses from everyone in the household (or at least everyone older than 18 years old, for example). If only one member is home, give them the required number of surveys so that all eligible respondents can participate in the survey if they choose.

Some respondents may suggest that they (a married couple, for example) can respond to a single questionnaire together. Politely ask that even if they agree, they must respond to separate questionnaires; the rule “one person, one questionnaire” is important because the priorities will be ranked based on the number of responses.

6. Be flexible and don’t be afraid to ask for help.

Your initial outreach plan may not work. Bad weather may cause issues. Be flexible and consider alternatives. Canvassing is low cost and has benefits, but it is time-consuming. For various reasons, it may not be the most effective choice for your particular geography or community. You know your community best – innovate and adapt this approach to what feels comfortable for you and your team. In addition, don’t be afraid to ask for help. One person can knock on hundreds of doors (as one city councilor in Vrable, Slovakia, did, reaching every door in his district), but it’s a huge undertaking. Ask for help – get support from volunteers, team members, civil society organizations and others. Just be sure that if you do recruit others, you provide them with FAQs (see above) and offer guidance and support.

7. Don’t forget the practicalities: Hydrate and wear comfortable clothes and shoes.

Think ahead! Test your survey

The next section will deal more in depth with the analysis of survey results. However, you should consider this process when planning the kind of survey you will use and the distribution method.

As noted above, you should test your survey questionnaire’s design on a small group of individuals. Your testers can be a small group – at least four or five individuals – and can be people you have reasonably easy access to: colleagues in the municipal administration, your friends or family members. These individuals should have as much knowledge of the survey as the average respondent and should not have had a role in designing the survey. This way, they can give feedback from a fresh perspective. Consider asking your testers the following questions:

• Was the survey too long?
• Are the questions clear? Was it clear what was being asked?
• Is anything important missing? Should information be added on how to return the survey, what the survey will be used for, etc.?
• Are there any sensitivities or concerns the tester thinks should be accounted for? Should some questions not be asked or be asked differently?

Think ahead! Plan enumeration and scoring methods

Now that you have a final survey questionnaire, you should think about the data you will receive. Skip ahead to the following section, which discusses how to analyze feedback from the different survey questions. Understanding what your analysis might look like – and how it will (or will not) answer your team’s initial question – will help you answer the key question at the moment: Should I use a survey? If so, what kind?

What kind of data do I want to collect? Will learning this data be worth the effort (of your team and your respondents)?

• Will I find that data meaningful? Meaningful data teaches you something or helps you to make a choice. Ideally, your data will help you and your team plan a local strategy – which issues to address first and some ideas about how to address them.
• Will I be able to analyze that data quickly and efficiently? As mentioned above, open-ended questions can be more trouble than the value they provide. If you are concerned about time, resources and being able to respond effectively, use your surveys to collect quantitative data only.
• Will I be able to act on my findings? Related to the idea that data is meaningful, your data should point toward concrete action, even if that concrete action is the need to learn more about one issue in particular.

To get started in answering these questions, gather your team around your draft survey, and imagine or guess what the results of 100 responses would be. If 70 percent of respondents pick an issue, what will that mean for your team? Will you have learned something new or just confirmed anecdotal evidence or assumptions?

• More data: Would it be useful to know that 70 percent of respondents picked an issue, but also useful to know something else in addition, such as the gender or ages of the respondents? By useful, we mean that this data point would impact the action you would take. If that additional data would matter, you should add in a demographic question asking respondents to identify their gender.

Planning ahead by considering these questions, testing your survey questions and evaluating survey length BEFORE you issue the survey to the public will help you design an accessible survey that yields useful information. Planning ahead by considering these questions, testing your survey questions and evaluating survey length BEFORE you issue the survey to the public will help you design an accessible survey that yields useful information.