Utilizing ML & NLP to Minimize Bias in Survey Design Assignment | Get Paper Help

Need to make some tweaks to Order # 315720077.

• Change heading of the first section to just “Introduction”
• Make the introduction a bit more thorough. Keep in mind that some people read just the intro and the conclusion. Also, add a brief paragraph outlining the paper, so the reader has some further expectations.
• Although this paper needs to be accessible to the students in our class, I would not use the text-book (or magazine, or typical blog) style of writing. Try to rewrite some of those sections to a more research paper style.
• Make sure you add references when you talk about non-trivial stuff
• You discuss how AI , ML and Deep Learning can help overcome the bias in creating the survey but I wonder if you have any insight on how these can alternatively proliferate bias?
• There is an opportunity for you to tie the specific AI techniques back to how they improve the previously noted common mistakes – making a stronger connection between the two sections.
• Try to address how you envision this could technology could be applied. Do you think companies like Survey Monkey/Qualtrics would want to implement this as a value to the customer, or perhaps it could be developed for in-house use at market research firms/consultancies, etc.? I would be curious to hear about potential applications, perhaps a sentence or two in the conclusion?
• Need to address the issue of bias in surveys and questionnaires and how your chosen algorithms would mitigate some of that bias.
• Proofread and review grammar

 

Utilizing Machine Learning and Natural Language Processing

to Minimize Bias in Survey Design

 

 

Edwin Amirian

HF760

 

 

Abstract

Surveys have been used by researchers for centuries in order to make statistical inferences about the characteristics, behaviors, and opinions across various populations. Market research surveys, user experience surveys, public opinion polls, human resource surveys, and censuses are all examples of quantitative research that use survey methodology to answer questions about a population. While surveys provide researchers with a cost-effective and convenient method to gain high representativeness of a population when compared to other research methods, it does not come without its disadvantages. One major drawback is that the questions used in the survey can be susceptible to biases and common mistakes that can creep into the results and affect the quality of the survey’s responses and conclusions. This paper will examine the causes and effects of these various forms of bias and illustrate how an intelligent user interface with the help of machine learning and natural language processing can aid researchers in identifying and correcting such biases while building their surveys.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

1. Introduction to the Topic

Some studies require access to unique forms of primary data, that the only way to collect them is through bespoke surveys. In such cases, researchers are required to come up with a systematic way to design surveys. There are many guidelines that a researcher can follow when designing a survey. The Pew Research Center (2020), for example, has published a set of guidelines that researchers can follow in designing survey instruments. The most common mistakes tend to be committed in the part where the researchers have to formulate the survey questions. Some researchers, for example, tend to make the mistake of creating questions that are leading and obvious (Pew Research Center, 2020). This, according to previously published studies, can have a significantly negative impact on the reliability and validity of the survey instrument, and by extension, that of the study’s entire results and findings as well (Brace, 2018; Willis, 2004; Abdul-Kader & Woods, 2015; Olsson, 2009). The objective of this paper was to examine the possible uses of Artificial Intelligence (AI), Machine Learning (ML), Deep Learning (DL), and Natural Language Processing (NLP) on the overall process of survey design.

2. Cognitive Response Process for Surveys

Whenever researchers attempt to analyze the data that they have collected from human participants, assumptions are often made about the mental processes that lead to the generation of those data. With the use of surveys as a primary data gathering instrument, for example, researchers often make the implicit assumption that each question would be answered by the target respondents based on their optimal cognitive response process (Brown, Kass, & Mitra, 2004; Schwarz & Strack, Context effects in attitude surveys: Applying cognitive theory to social research, 1990).

There are numerous steps that are involved in a typical survey data collection process. It usually starts from the Comprehension phase. This is where the respondents start to understand the question, what it entails, based on the literal interpretation of the words. The second phase is the Retrieval phase. This is where the respondents start to identify the type of information that they would most likely need to come up with an appropriate answer to the survey question. The third phase is the Judgement phase. This is where the respondents try to assess the completeness and appropriateness of the information that they have decided to retrieve from their memory, experience, among other sources, to answer the survey question. The fourth and last phase is the Response phase. This is where the respondent finalizes the information that he retrieved and assessed, after judging its appropriateness and completeness.

The quality, reliability, and validity of the data generated from survey-based methods of conducting research greatly depend on the degree to which the respondents complete the aforementioned steps of answering a survey question (Sudman, Bradburn, & Schwarz, 1996; Peterson, 2000). Despite this, researchers should not attempt to directly control the overall process that respondents follow when undergoing this process (of answering surveys) (Tourangeau, 2018). The reality, however, is that there are indeed some actions that can be taken by the researchers (either consciously or unconsciously) that can have a direct impact on the survey participants’ responses, examples of which will be provided in the succeeding sections.

3. Types of Respondent Burden

There are different types of respondent burden. Generally, they can be categorized into two types, namely: 1) Extrinsic Burden, and 2) Intrinsic Burden. Extrinsic Burdens refer to those that affect the respondent’s time and attention, independent of the actual survey process. These usually include external factors that have little to nothing to do with the actual questions that are indicated in the survey instrument. Because of the externality of these factors, the researcher and the survey designer usually have no control over such burdens. This, however, does not change the fact that they can and will have an impact on the quality, reliability, and validity of the data that will be captured.

Intrinsic Burdens, on the other hand, refer to factors that are inherent to the survey instrument and the questions that were constructed and designed to illicit and collect the intended responses from the participants. Survey questions are often intended to be clear and simple. It is, however, not uncommon for this not to be the case. Survey respondents who are already overtaxed with extrinsic factors may find it difficult to come up with conscientious responses if they also have to contend with intrinsic burdens. Intrinsic respondent burdens are often those whose cause can be traced back to the poor overall design of the questionnaire.  Some of the most common examples include the use of leading words, phrases, and questions; double-barreled questions; the use of absolutes; failing to ask one question at a time; failing to give mutually exclusive choices; failing to ask direct questions; and the use of imbalanced scales (Wenemark, Persson, Brage, Svensson, & Kristenson, 2011; Krosnick & Presser, 2009; Wenemark, The respondents’ perspective in health-related surveys, 2010).

4. Designing Questions to Maximize Data Quality and Reduce Respondent Burden

In order to get the highest quality, most valid and reliable responses from the survey participants possible, survey designers and researchers have to minimize the occurrence, degree, and impacts of intrinsic respondent burdens (Guin, Baker, Mechling, & Ruyle, 2012; Galesic & Bosnjak, 2009; Wiese & Schwartz, n.d.). In a paper that was featured in the journal American Psychologist, for example, Schwarz (1999) argued that one of the common flaws of survey-based studies is the inherent difficulty of avoiding leading questions, and other bias-inducing ways of collecting data. Schwarz (1999) noted that by using leading questions in surveys, for example, there is a real tendency for those questions to dramatically shape the respondents’ answers. This is why in most survey design guidelines that are available today, it is recommended that the survey questions be as neutral as they can possibly be (Lavrakas, 2008; Miller & Willis, 2016; Vannette, 2014).

Below is a table that summarizes the common mistakes (Intrinsic Burdens) that survey designers and researchers commit. In order to more clearly see the differences between the different common mistakes, a hypothetical study that aims to examine the effectiveness of using Grammarly as an editorial tool will be used in the process of giving examples of bad and good questions.

Name	Explanation/What is it?	Example of a bad question	Example of a good question
Leading questions/words	These are questions or words that illicit a predetermined set of responses	You were able to improve your editorial skills after using the product, correct?	How were your overall editorial skills affected as a result of using the product?
Loaded Questions	These are questions that contain an unjustified assumption.	Since it is your first time using an editorial software product (assumption), rate the level of difficulty you experienced in using it?	How can you describe the level of difficulty you experienced in using the product?
Double Barreled Questions	These are questions that usually touch more than one issue, but allows or requires only one answer.	How often and how much time have you spent learning how to use the product?	How much time have you spent learning how to use the product?
Using Absolutes	These are questions that contain absolute terms such as “Always”, “Never”, among others.	You have never encountered any problems in using the product for your editorial needs, correct?	How frequently did you experience problems in using the product for your editorial needs?
Failing to ask one question at a time	These are questions that contain more than one thought, but only require one answer, similar in a lot of ways to double-barreled questions.	How expensive and how valuable do you think the product is as an editorial companion?	How valuable do you think the product is as an editorial companion?
Failing to give mutually exclusive choices	Providing choices to a question that cannot be adequately differentiated by the respondent.	Rate the product in terms of its ease of use: Choices: Easy to Use, Not Easy to Use, Slightly Easy to Use	Rate the product in terms of its ease of use: Choices: Easy to Use, Neither Easy nor Hard to Use, Hard to Use
Failing to ask direct questions	Including terms and phrases that are not related to the concept being examined in the survey question	Judging by the price of the product, how would you describe your overall experience in using it in terms of portability?	How would you describe your overall experience in using it in terms of portability?
Not balancing scales carefully	The use of scale-based choices that are not calibrated.	Do you agree with the following statement? The editorial functions included in the product are complete. Choices: Strongly Agree, Agree, Neutral, Slightly Agree, Strongly Disagree	Do you agree with the following statement? The editorial functions included in the product are complete. Choices: Strongly Agree, Agree, Neutral, Slightly Disagree, Disagree

 

5. Introduction to Artificial Intelligence

Artificial Intelligence (AI) is a computer science discipline that focuses on the type of intelligence demonstrated by machines, most notably, computers. AI stands in contrast to natural intelligence, or the one type of intelligence displayed by animals and humans. While there are still a lot of unknowns surrounding AI, e.g., what their actual use cases are, there are already numerous theories that are circulating about the scopes and boundaries of their abilities. There is, for example, a theory that suggests that AI can be used to improve the way how people do their jobs. This is especially true in fields and disciplines that involve and or require a certain degree of data processing and simulation. In this paper, the researcher sought for evidence from previously published studies that show how artificial intelligence can be used to minimize bias in the process of survey designs.

Machine Learning (ML) is a subset or a sub-discipline of artificial intelligence. Unlike AI which is more focused on how certain human cognitive functions can be mimicked by machines, Machine Learning is more focused on how computer algorithms and processes can be developed in such a way that continuous and automatic improvements in the way how the machine (i.e., computer) operates would be possible. This is often done to improve the quality, efficiency, and other aspects (sometimes simultaneously) of a process or experience. In this case, the exact process that may be improved through the use of machine learning is the process of designing surveys.

There are numerous ways how machine learning can be used to improve the outcomes of survey design processes. A good example would be how a data set consisting of past surveys with and without issues can be used to detect possible errors on how some surveys (within the set) were committed. In most cases, this would be done by developing a machine learning model or algorithm that uses the past surveys that have been certified (often by a human) to have no errors. The machine learning algorithm can then be used to perform tests and analyses that are aimed at detecting the errors that have been committed in the survey instruments that have a faulty design.

Deep Learning (DL) is an even more specific subset of artificial intelligence. In most computer science literature, DL is considered as a direct subset of Machine Learning. Basically, there are different AI specialties that fall under ML, including but not limited to dimensionality reduction, meta-learning, and topic modeling. Deep Learning puts emphasis on the development of artificial neural networks with varying levels of supervision (e.g., supervised, unsupervised, or semi-supervised). Deep Learning’s use cases often include audio recognition, machine translation, computer-aided visual input recognition and processing, speech recognition, natural language processing, and social network filtering.

Natural Language Processing (NLP) is one of the areas where Deep Learning, and in general, AI, can be used. NLP refers to a discipline that combines some of the core elements of computer science, linguistics, and artificial intelligence. The goal of NLP is to understand and, at some point, optimize the way how humans interact with machines (i.e., computers) using natural languages. Computers are traditionally operated using binary language. While this has definitely worked well so far, the use of binary language to allow machines to communicate with humans (and vice versa) indeed comes with a variety of limitations. One of such limitations is the fact that computers would only be able to understand and therefore analyze a limited variety of data; this also means that only individuals with the appropriate knowledge and skill to write and understand the appropriate computer codes would be able to execute the commands on the machines in question. The use of NLP technologies basically removes this limitation, as they would, in theory, enable machines to understand and analyze data that are expressed in natural language.

The process of building language models for survey design processes usually fall under two categories. The first category is a rule-based NLP model; the second one is a statistical NLP model. The decision on which between the two language models to choose usually depends on the objectives of the survey or any NLP project that the model would be used for.

Rule-based language models tend to rely on the application of rules to certain texts. This can be done by programming an algorithm that enables the machine to recognize which parts of a database of words are nouns, and which ones are verbs. This can be done through the execution of a Python code. From there, insights can be generated from the database of words that a data scientist can then process further.

Statistical language models, on the other hand, rely on the identification and examination of probabilities with the alternatives encountered in the course of analyzing an utterance or a text and accepts the most probable outcome as the correct one (Mitkov, 2005).

6. Leveraging AI in Survey Design

Bias will always be a part of a survey design process. This is true not just for the survey design process, in general, but for all other aspects of data gathering, and data analysis as well. The reality is that bias cannot be completely eliminated, even with the use of increasingly precise methods to address them, such as AI, ML, and NLP. The best way to approach bias in designing surveys and eventually in the interpretation of their results is to minimize it. In a study that was conducted by Mehrabi et al. (2019), they examined different real-world applications of machine learning where bias seemed to be evident. Their analysis led them to conclude that different AI and ML applications carry with them varying levels of bias. An example of an unfair (i.e., biased) ML and NLP algorithm, for example, is one that skews the results toward or against a certain group of people, or in the case of NLP, towards or against a certain group of words (i.e., phrases).

In order to address these problems, ML and NLP professionals should start the development of a standardized way of evaluating and identifying bias. This, according to Mehrabi et al. (2019) is a necessary step to make sure that the biases in ML and, by extension, NLP technologies are adequately assessed. It is, however, understandable to have this kind of problem, because ML and to a greater extent, NLP, are still in the early stages of their (respective) development and lifecycle. Mehrabi et al. (2019) suggested an algorithmic bias detection system that can capture the different types of bias, discrimination, and unfairness in data models would be necessary to truly address this issue. Some of the examples of bias that they covered in their study included (but were not limited to) historical bias, representation bias, measurement bias, aggregation bias, evaluation bias, the Simpson’s Paradox, presentation bias, ranking bias, user-interaction bias, temporal bias, direct discrimination, indirect discrimination, statistical discrimination, and explainable discrimination. These unfairness, discrimination, and bias types were actually demonstrated in the problems that were identified in the Designing Tables index, presented in Section 4.

Deep Learning can be used to create multiple layers of data processing that would enable researchers to progressively extract more complex data from an otherwise simple raw input. This raw input, within the context of survey designs, may come in the form of actual responses from participants, or a collection of correctly designed and incorrectly designed survey instruments. Deep Learning, in such cases, would allow the data analyst to generate unique insights form an otherwise simple and raw collection of data such as responses, and actual survey designs. A perfect case study that demonstrates ML and DL’s ability to perform these functions would be Grammarly.

The objective of the DL algorithms that Grammarly used in their system was to mimic the way how a human would perform the task of checking the contents (i.e., input data) submitted by the users, for grammatical and other editorial mistakes. This means that Grammarly’s system should indeed be capable of detecting good and bad samples from a large collection of input data. Grammarly has so far revealed that it has implemented this process by teaching its DL algorithms how to perform the intended tasks, the way they ought to be performed, by focusing on repetitions. Repetitions in this case refer to the process of feeding its DL system large volumes of data, and then essentially showing it how to analyze those data (i.e., detecting grammatical errors and editorial mistakes), with emphasis on the use of a proprietary outline of rigidly predefined steps (Toncic, 2020; Grammarly, 2018).

Aside from actual feedback from the developers of the algorithms (telling the DL system what is right and wrong through the rigidly predefined steps that were mentioned earlier), Grammarly also factors in the feedbacks coming from its actual users in the future iterations of its continuously evolving algorithm. Grammarly collects user-generated responses to its system-generated reports. Whenever users hit the ignore button in their control panels, for example, it gives the Grammarly DL system’s developers the signal that the aspects of the algorithm that are behind the user suggestions in question should be revised, to make them organically more accurate.

In the same way, evidences that show that the users are actually following the system’s editorial suggestions give the developers the signal that the algorithms they have been using are performing as intended. This means that the DL system that Grammarly is using is also reliant on the use of positive and negative feedbacks from various sources, to continuously improve its effectiveness.

7. Conclusions

The solutions that have so far been presented in this paper are so far appropriate for the problems at hand. The previously published studies that have been reviewed so far show that biases, discrimination, and unfairness in the process of designing surveys that affect the way how data are eventually collected and processed can only be truly mitigated through the use of Machine Learning and Natural Language Processing Technologies. Grammarly serves as a good case study of what a survey design system that incorporates the core elements of ML and NLP should look like. Researchers who are trying to design surveys that incorporate such elements, however, must remember that just because those elements worked for one use case does not automatically mean that the same elements would yield the same outcomes once they actually start to incorporate them in their survey design project. It is always best to design a survey based on contextual and circumstantial factors. That said, future researchers should consider conducting further studies that are aimed at coming up with a standardized taxonomy or classification of the different biases, discrimination, and unfairness that come with the development and use of AI, ML, DL, and NLP-based systems.

 

 

 

 

References

 

Abdul-Kader, S., & Woods, J. (2015). Survey on chatbot design techniques in speech conversation systems. International Journal of Advanced Computer Science and Applications.

Brace, I. (2018). Questionnaire design: How to plan, structure and write survey material for effective market research. Kogan Page Publishers.

Brown, E., Kass, R., & Mitra, P. (2004). Multiple neural spike train data analysis: state-of-the-art and future challenges. Nature Neuroscience, 456-461.

Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 349-360.

Grammarly. (2018). How We Use AI to Enhance Your Writing. https://www.grammarly.com/blog/how-grammarly-uses-ai/.

Guin, T., Baker, R., Mechling, J., & Ruyle, E. (2012). Myths and realities of respondent engagement in online surveys. International Journal of Market Research, 613-633.

Krosnick, J., & Presser, S. (2009). Question and Questionnaire Design. 1-81.

Lavrakas, P. (2008). Cognitive Aspects of Survey Methodology (CASM). Sage Research Methods.

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019). A Survey on Bias and Fairness in Machine Learning. 1-31.

Miller, K., & Willis, G. (2016). Cognitive Models of Answering Processes. The Sage Handbook of Survey Methodology.

Mitkov, R. (2005). The Oxford Handbook of Computational Linguistics. Oxford University Press.

Olsson, F. (2009). A Literature Survey of Active Machine Learning in the Context of Natural Language Processing.

Peterson, R. (2000). Constructing Effective Questionnaires. Thousand Oaks, CA: Sage.

Pew Research Center. (2020). Questionnaire Design. PRC, https://www.pewresearch.org/methods/u-s-survey-research/questionnaire-design/.

Schwarz, N. (1999). Self-Reports: How Questions Shape the Answers. American Psychologist, 93-105.

Schwarz, N., & Strack, F. (1990). Context effects in attitude surveys: Applying cognitive theory to social research. European Review of Social Psychology, 31-50.

Sudman, S., Bradburn, N., & Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. Jossey-Bass.

Toncic, J. (2020). Preparing Tomorrow’s Leaders for Yesterday: AI and Standardized Grammar Assessment. New Jersey English Journal.

Tourangeau, R. (2018). The survey response process from a cognitive viewpoint. Quality Assurance in Education.

Vannette, D. (2014). Questionnaire Design: Theory and Best Practices. Stanford University Institute for Research in the Social Sciences.

Wenemark, M. (2010). The respondents’ perspective in health-related surveys. Linkoping University Medical Dissertations, 1-101.

Wenemark, M., Persson, A., Brage, H., Svensson, T., & Kristenson, M. (2011). Applying Motivation Theory to Achieve Increased Response Rates, Respondent Satisfaction and Data Quality. Journal of Official Statistics, 393-414.

Wiese, C., & Schwartz, L. (n.d.). Need for respondent-centered survey design.

Willis, G. (2004). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage Publications.