This is Maxwell's chapter on validity. There are two questions: what threats to validity do I see in my research, and what strategies might I use to assess whether these threats are serious?
1. Potential Threats:
I think that my bias could be a threat to the validity of my study. I have very strong opinions about what I want to find and establish with my research. I believe that effective teachers in a EFL setting use certain strategies, and I want to document those strategies. Perhaps by believing those strategies exist, I will create a self-fulfilling prophecy and find them.
Another threat might be the limitations of my setting. I am semi-consciously expecting to find a phenomenon that is generalizable to international English as a medium of instruction settings everywhere, but so far I have only observed it in a couple of settings and am tentatively planning to study it in one setting. There is a threat that I will overgeneralize beyond my setting and overlook the ways that my setting is unique from other EMI settings--especially in other cultures.
There is the threat of reflexivity--that my interviewed subjects will try to give me what I want to here. That I might ask leading questions. That my familiarity with my interview subjects will cause them to neglect to mention pertinent facts (because they think I know them already).
2. Ways to Assess Seriousness of Threats
I think the best way to guard against bias is to make it explicit. Maxwell states that the researcher cannot hope to not influence the material they gather or what kind they get. All researchers are looking for specific answers, so if I make my bias explicit, I can also check to see if there are other interpretations to my findings. In the same way, I have to be careful about not over-generalizing. Perhaps, I can find people to interview who are not in my immediate setting, but equally legitimate would be to be careful not to apply my findings beyond my own setting. For the third threat, it may be a good idea to go and interview someone who does not know me or my context and find out what kind of answers they give me, and then go back and cross check these answers with the people I interview at Sias.
Maxwell mentions several strategies that I think are already incorporated into my research design. Intensive, long-term involvement: I am already deeply involved in this context and have a long history to inform my research question and the data I gather. Rich data: I think there is no limit to the amount of data I could collect on this topic. I could interview teachers, I could interview students, I could observe classrooms, I could video tape and record. I could collect sample assignments and student responses. Probably its a question of what I have time to do in the parameters of this dissertation.
Respondent validation: this is something I could implement fairly easily. I know that most of the teachers I interview will want to see my findings. I'm not so sure about intervention, however. I think I want to collect more information before I suggest changes to the way people teach. I may do second interviews, though, where I ask if our first interview influenced the way the teacher thought about their teaching methods. I hope that I will find a variety of answers to my questions about the best methods--because I know their is variation in our faculty, and so I am actually hoping to get opinions that disagree with my expected hypothesis. I think they will make the data richer and uncover assumptions that need to be addressed.
Triangulation would be, within my setting, choosing a diverse population (new teachers, old teachers, teachers who teach large lectures, teachers who teach break out sessions, MBA students, professors with Ph.D.s etc.). Triangulation outside my setting would be finding teachers in other schools here in China, in other countries in Asia, in countries in Europe and the Middle East and in Africa. There is a lot of potential for adding a quantitative element: conducting surveys of students here at Sias or in other EMI programs is the first one that comes to mind.
The question I think I will have to settle for my proposal is what is the scope of my study and what are the most serious threats, since I don't think I have time to implement all of Maxwell's suggestions. My strongest inclination is to explicitly limit myself to this setting. I think there is enough data here to illustrate significant findings that can be explored further in other settings at a later date.
Monday, May 16, 2016
Saturday, May 14, 2016
Reflection on Observation
The observation was an interesting process. I think I do a lot of observing subconsciously, so when I sat down to write, it was hard for me to decide what to focus on first. For example, I have spent a lot of time in China watching the customs of people from the corner of my eye and I have spent the same amount of time forming theories about why they do things the way they do. It's a unique experience when you don't know the language because it forces you to focus on cues--what people wear, the expressions on their face, their body language--but after you've watched them for a while, you forget how they struck you on first impression. You forget that you used to think it was strange that that boy is carrying his girlfriend's purse, or that those schoolgirls are holding hands...
So it was really kind of hard to decide what to write down.
One of the comments that you made was that I should have spent a little more time interpreting what I saw. I wondered about that. The thing is, I have a lot of speculations that I could write down about the things I see, but usually if I get off speculating, I miss what is there, or at least I have the potential to miss it. I think as part of this exercise, I was trying to stay present and not evaluate. I think for me those two modes, observing versus interpreting, are very distinct and I would have liked a little more structure in the assignment as to how much of each I should be doing (for example 60% observation; 40% interpretation).
What do you think? Would it have been acceptable for me to fall back on my years of experience observing Chinese people and start with themes--for example: "Chinese people have a collective culture. Let me draw all of the aspects of this scene that illustrate the collective nature of their society..." Would that have been a legitimate approach? I often explain my insights to people who are here visiting or who are new to China. To what extent does my historical observations count as legitimate data in a qualitative study?
So it was really kind of hard to decide what to write down.
One of the comments that you made was that I should have spent a little more time interpreting what I saw. I wondered about that. The thing is, I have a lot of speculations that I could write down about the things I see, but usually if I get off speculating, I miss what is there, or at least I have the potential to miss it. I think as part of this exercise, I was trying to stay present and not evaluate. I think for me those two modes, observing versus interpreting, are very distinct and I would have liked a little more structure in the assignment as to how much of each I should be doing (for example 60% observation; 40% interpretation).
What do you think? Would it have been acceptable for me to fall back on my years of experience observing Chinese people and start with themes--for example: "Chinese people have a collective culture. Let me draw all of the aspects of this scene that illustrate the collective nature of their society..." Would that have been a legitimate approach? I often explain my insights to people who are here visiting or who are new to China. To what extent does my historical observations count as legitimate data in a qualitative study?
Reflection on Interview Assignment
I enjoyed working on my interview assignment, although I found it to be quite the extended process. I chose to interview my colleague, in part because she fit into the population that I wanted to examine for my dissertation, but also because I had helped her a little with her research recently, and so I thought she would be more interested in helping me (reciprocity). I may be asked to help more in the future, but that is fine with me. I need to get better at helping my colleagues with their research if I'm hoping for them to help me with mine.
It turns out that Mary doesn't really like to speculate and expand on a topic. She wanted me to be very specific in the way I asked questions. This was a little frustrating for me because I wanted to drop a word or sentence and get her stream of thought. On one hand, maybe I should have explained that to her at first (if I was even aware of it). On the other hand, I don't know if maybe that isn't a personality thing. I really like learning about personalities, and Myers Briggs personality typing has the iNtuitive versus Sensor category. Sensors are very practical and down to earth. They focus on the here and now and don't spend to much time speculating on abstract ideas or the future. iNtuitives, by contrast, spend a lot of time making abstract connections and creating paradigms as well as speculating on the future. Most people in higher education are on the iNtuitive side of things, but I think Mary tends to think more concretely. I don't really have a way to compensate for that except that maybe I just need to be ready to take a different approach if my interviewee has a different way of thinking about these things.
Anyway, it was an interesting experience.
By the way, I chose not to transcribe my interviews because I come from a linguistic background and I'm very aware of how much "noise" there is in a recording. I had to transcribe all of the small messy details of a five minute recording as a final project in one of my classes. It is a fascinating process, but it also demonstrated to me how bad I am at transcribing. I noticed that Rev does kind of a sloppy job, though. I'm not sure if that reflects their quality or my expectations. Maybe I just try too hard to get it right, and that is why I think I can't transcribe... I don't know.
It turns out that Mary doesn't really like to speculate and expand on a topic. She wanted me to be very specific in the way I asked questions. This was a little frustrating for me because I wanted to drop a word or sentence and get her stream of thought. On one hand, maybe I should have explained that to her at first (if I was even aware of it). On the other hand, I don't know if maybe that isn't a personality thing. I really like learning about personalities, and Myers Briggs personality typing has the iNtuitive versus Sensor category. Sensors are very practical and down to earth. They focus on the here and now and don't spend to much time speculating on abstract ideas or the future. iNtuitives, by contrast, spend a lot of time making abstract connections and creating paradigms as well as speculating on the future. Most people in higher education are on the iNtuitive side of things, but I think Mary tends to think more concretely. I don't really have a way to compensate for that except that maybe I just need to be ready to take a different approach if my interviewee has a different way of thinking about these things.
Anyway, it was an interesting experience.
By the way, I chose not to transcribe my interviews because I come from a linguistic background and I'm very aware of how much "noise" there is in a recording. I had to transcribe all of the small messy details of a five minute recording as a final project in one of my classes. It is a fascinating process, but it also demonstrated to me how bad I am at transcribing. I noticed that Rev does kind of a sloppy job, though. I'm not sure if that reflects their quality or my expectations. Maybe I just try too hard to get it right, and that is why I think I can't transcribe... I don't know.
Article Critique
The qualitative study that I read was "Positive Aspects of International Student Transitions: A Qualitative Inquiry" by Lisa Moores and Natalee Popadiuk (2011).
This article focused on the positive critical incidents reported by seven international students at a Canadian university. The researchers interviewed each student seven times (semi-structured). They used a technique called the Critical Incident Technique (CIT) and combined it with aspects of positive psychology. The authors state that the unique gap that their study fills in the literature is that it focuses on the positive aspects of international students' experiences rather than problematizing them. The researchers categorized the critical incidents reported into eight groups: Growth and/or change, Social support/building relationships with peers, learning to navigate the host culture, enjoyable activities outside of schoolwork, previous experience/preparation, supportive faculty and staff, persevering through hard times, and sense of belonging. The researchers further refined these categories into four themes: connecting with others, maintaining a foundation, embracing the process, and discovering strengths within.
I thought these researchers had an interesting and insightful approach to examining international students' experiences. The focus on the positive aspects of their experience does seem to be a unique approach to gaining understanding and insight. They did a thorough job of describing how they established the credibility of their methods, citing Guba and Lincoln's techniques which they used to test their own methods.
Perhaps because of the page limitations put on a journal article, the authors left me with some questions. For example, the article starts with statistics regarding foreign student populations in the U.S. but this study was conducted in Canada. I didn't figure that out until I was about halfway through the article. I think that authors could have explicitly acknowledged this and reflected on whether or not patterns and experiences for international students vary in Canada as opposed to the U.S. Perhaps they don't vary, but I would have liked some kind of explanation. Qualitative research is about focusing on singularity--not trying to create broad conclusions. The authors could have shed further light on their context by discussing the fact that it was Canadian.
Further, in the introduction, the authors mention, in passing, the applications of their research to college counselors--an application that they expand on in their implications. Since the article was published in the Journal of College Student Development (which is feasibly read by non-counselors), I would have liked some explanation as to how coming from a counseling perspective shaped this study--as opposed to if it had come from some other branch of student services.
Also, just because of my own recent experience in conducting interviews, I would have liked a more in-depth account of the seven interviews--what kind of questions the authors asked and whether they had any difficulties eliciting information (maybe they didn't). It just seems that these kind of interviews would require a certain understanding of researcher expectations (cultural norms) that the international students may not have had. The authors do mention briefly that all interviews were conducted in the students' second language which may have interfered with the material elicited.
Along these lines, the authors also mention that perhaps the students felt constrained by the request for only positive incidents (although some negative incidents were mentioned). I can see how this may have limited the authors' results. I think they could have cast their net wider and collected both positive and negative incidents and then interpreted them through the positive psychology lens, but I'm not sure if that approach would have served their aims. The focus solely on the positive aspects of the students' experiences does seem limiting to me, however. I suppose I just believe that positive incidents inform negative incidents and vice versa. It would have been nice to read a discussion about this at least.
This article focused on the positive critical incidents reported by seven international students at a Canadian university. The researchers interviewed each student seven times (semi-structured). They used a technique called the Critical Incident Technique (CIT) and combined it with aspects of positive psychology. The authors state that the unique gap that their study fills in the literature is that it focuses on the positive aspects of international students' experiences rather than problematizing them. The researchers categorized the critical incidents reported into eight groups: Growth and/or change, Social support/building relationships with peers, learning to navigate the host culture, enjoyable activities outside of schoolwork, previous experience/preparation, supportive faculty and staff, persevering through hard times, and sense of belonging. The researchers further refined these categories into four themes: connecting with others, maintaining a foundation, embracing the process, and discovering strengths within.
I thought these researchers had an interesting and insightful approach to examining international students' experiences. The focus on the positive aspects of their experience does seem to be a unique approach to gaining understanding and insight. They did a thorough job of describing how they established the credibility of their methods, citing Guba and Lincoln's techniques which they used to test their own methods.
Perhaps because of the page limitations put on a journal article, the authors left me with some questions. For example, the article starts with statistics regarding foreign student populations in the U.S. but this study was conducted in Canada. I didn't figure that out until I was about halfway through the article. I think that authors could have explicitly acknowledged this and reflected on whether or not patterns and experiences for international students vary in Canada as opposed to the U.S. Perhaps they don't vary, but I would have liked some kind of explanation. Qualitative research is about focusing on singularity--not trying to create broad conclusions. The authors could have shed further light on their context by discussing the fact that it was Canadian.
Further, in the introduction, the authors mention, in passing, the applications of their research to college counselors--an application that they expand on in their implications. Since the article was published in the Journal of College Student Development (which is feasibly read by non-counselors), I would have liked some explanation as to how coming from a counseling perspective shaped this study--as opposed to if it had come from some other branch of student services.
Also, just because of my own recent experience in conducting interviews, I would have liked a more in-depth account of the seven interviews--what kind of questions the authors asked and whether they had any difficulties eliciting information (maybe they didn't). It just seems that these kind of interviews would require a certain understanding of researcher expectations (cultural norms) that the international students may not have had. The authors do mention briefly that all interviews were conducted in the students' second language which may have interfered with the material elicited.
Along these lines, the authors also mention that perhaps the students felt constrained by the request for only positive incidents (although some negative incidents were mentioned). I can see how this may have limited the authors' results. I think they could have cast their net wider and collected both positive and negative incidents and then interpreted them through the positive psychology lens, but I'm not sure if that approach would have served their aims. The focus solely on the positive aspects of the students' experiences does seem limiting to me, however. I suppose I just believe that positive incidents inform negative incidents and vice versa. It would have been nice to read a discussion about this at least.
Subscribe to:
Posts (Atom)