Academics Studying Nike Web Document 

http://business.nmsu.edu/~dboje/nike.html 

TITLE:  ISSUES OF VALIDITY AND RELIABILITY FOR Academics Studying Nike

BY David Boje

Date: August 12, 2000, updated September, 2000

VALIDITY - in quantitative research is a technical thing, but in qualitative research it has to do with the rigor of the description and the credibility of the explanation (See Janesick in HQR-Handbook of Qualitative Research, p. 216). Validity is also a matter of providing good audit trails in the research documentation (HQR, p. 230). The problem for Nike and Price Waterhouse Cooper is that the audit trails are for the most part kept secret and only the summaries are put out for public consumption. And participants (workers) do not get much chance to review the PWC or FLA findings. Nike has moved to Global Alliance as a way to involve workers in focus groups (Note: as of this writing the results are not yet available on GA).  

Internal Validity -- do findings or inferences correctly map the phenomenon (Denzin & Lincoln Handbook of Qualitative Research, SAGE 1994: 100). Does the research get at the substance of the story to be told? For example subcontractor monitoring depends upon what Guba & Lincoln (1994; 106 HQR) call the "discovery dimension of inquiry." But discovering the underlying story is not so easy as checking a few observation boxes by a monitor.  In terms of Nike, the question is the extent to which discovery gets cut off by (a) pre-conceived item templates that miss the main stuff; (b) announced visits that allow time to set up a masquerade inspection; (c) fear of reprisal of the workers so they say what subcontractors expect; (d) cultural distance of researcher and workers; (e) neglecting the dynamics of the lived world of experience of workers and subcontractors (i.e. subs have little negotiating room with Nike; workers work by quota systems that are stretched to make up subcontractor costs, such as fines for breaking sewing needles). Unless these issues are dealt with in the study, internal validity is suspect. Some argue that having workers participate in the research will help (action research that starts with the workers of Nike doing the research themselves has been done by Ballinger's most recent wage study -- see Kincheloe & McLaren in HQR, p. 150 for more on worker participative designs). Since most of the workers are female, the work of feminist researchers applies. Feminist qualitative research assumes that researcher and participant co-create the data (Oelsen in HQR, p. 166). If workers lives and situations are the focus of the research then feminists stresses not exploiting them as "subjects." Rather the focus is on empowering working women via participation in the research process. Internal validity is suspect when workers do not have a VOICE in the research process.  The danger to internal validity is just asking questions does not get at how respondents' accounts have been already mediated before workers come to the interview (this is the claim of many NGOs). For example, is there internal validity if Price Waterhouse Coopers, Fair Labor Association and Global Alliance monitors are assumed by workers to work for Nike and/or the subcontractor? 

Audit Trail "Careful documentation of the conceptual development of the project should leave an adequate amount of evidence that interested parties can reconstruct the process by which the investigators reached their conclusions" (Morse in HQR, p. 230). Morse suggest six types of documentation: raw data, data reduction/analysis products, data reconstruction or synthesis products, data notes, materials related to intentions, and instrument development info. The problem for Nike is Price Waterhouse Cooper does not want their audit trail on the WWW.  But, without disclosure their reports lack credibility and in the end fail the test of internal validity. 

External Validity -- the degree to which findings are credible and can be generalized to other settings similar to the one in which the study occurred. Nike bases credibility on the use of the expert (and sometimes the celebrity) while for the NGOs it is the believability of the story that counts. Is the story revealed or constructed in one factory applicable to another? The external validity problem in Nike-sponsored tours of factories is we suspect the findings in (model) subcontract factories are not generalizable to all subcontract factories (Denzin & Lincoln Handbook of Qualitative Research, SAGE 1994: 100). Alternatively, Nike suspects the external validity of "gotcha" research which is only about problem-finding. The problem with Nike and some 600 subcontractors and 600,000 workers is Nike-world is not a uni-causal universe. Therefore Kincheloe & McLaren (HQR, 151) argue that traditional positivistic notions of external validity are too simplistic. They prefer Lather's (1991) construct of "catalytic validity.

Catalytic Validity -- which points to the degree to which research moves those it studies to understand the world and the way it is shaped in order for them to transform it (HQR, p. 152).  This is a more rigorous test of validity, and one that will help resolve the problems that recur between Nike, subs, and workers. Does the research lead people to better understand the story they are scripting and how to restory? 

Partial Truth - The idea of "partial truth: is at the heart of the "new ethnography" (Harper in HQR p. 407). Wolcott also argues that traditional ways of thinking about generalizability are inadequate (HQR, p. 216 in Janesick). He objects to aggregating the numbers since it hides the meaning and interpretation of individual cases. And issues of Code compliance are at the individual level. 

Tamara - In Tamara we are multiple storytellers in different rooms, rarely in the same room, and therefore immersed in fragmented and wandering discourse. The Tamara perspective argues that we do not all experience the same story trail and even when we see the same segment of story, we experience and enact it from different view points (See Tamara explanation and Disney example). It is precisely this understanding of the situations of Nike, NGOs, workers, and activists that David Barry concluded was the Nike research scene (See Toronto Time and Nike session). We are embedded in different story spaces and contexts. The stories are polysemous and polyphonic, so hard to agree upon. The researcher story is only one of many interacting stores in the storied context of Nike and its 600 (plus) subcontractors. The stories attributed to workers take place in the context of a larger than life story of globalization and the relationship between 1st and 3rd world economies.  

Construct Validity  Construct validity refers to the degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based. (Source press here for more). Again without audit trails and field notes as well as worker participation in the study construct validity is hard to achieve. 

Conclusion validity asks is there a relationship between the program and the observed outcome? (Source press here for more). Here there are serious problems. If we look at Nike's relationship with Price Waterhouse Coopers, the Fair Labor Association and Global Alliance, we must assume that those relationships partially produce the observed outcomes.  These are paid contract relationships, not independent audits.  IN addition, the contractor and contractee work at co-constructing a mutually agreed protocol as well as disclosed times of visit, and sustaining an expert's distance from workers at that sites. 

Interpretative Validity - Altheide & Johnson (HQR, 1994) argue for  interpretative validity, but not in the universalistic laws of interpretation of positivist science.  The focus is instead on "definitions of the situation" through deep immersion into the life world so as to learn the native point of view. For the qualitative researcher this would mean becoming the worker, taking a job in a Chinese subcontracting factory and dealing with quotas, fines for breaking a sewing needle and fines for daring to talk to a co-worker.  Instead of acontrol over objectivity or bias, the approach is to become the other (as much as possible to go native). In this way knowledge claims come from a grounded and experienced understanding of what it is like to work in a Nike so-called sweatshop.  It includes copying with issues of resistance and subordination i the Nike social world of work that are assumed to escape other modes of validity. I understand that just such ethnographic investigations are occurring secretively in China. The self-reflexive ethnographies will soon be written (in what Van Maanen, 1988 calls the impressionist tale). Another aspect of interpretative validity is how the text communicates with the reader. How the text demands an answerability from the reading of the tales of sweatshop life.  It is these tales and this postmodern storytelling that is lacking in much of the reporting of universalized stories and aggregated statistics of many current studies. As Lyotard argued in the Postmodern Condition "Let a thousand blossoms bloom" or let a thousand stories tell the tale of thousands of workers. 

RELIABILITY - extent to which findings are replicable by another researcher.

Test/Retest - the more conservative method to estimate reliability.  (See press here for more). In qualitative research observing people at different times (day, night) and in different seasons (winter, summer or when work is slack or excessive) helps to enhance test/retest reliability (see HQR, p. 381-2). But how reliable is research of sweatshops by those who have never worked in one?

Internal consistency estimates reliability by grouping questions in a questionnaire that measure the same concept.  (Source press here for more). Here triangulation is mentioned (HQR, p. 382). But the technology of consistency is a shallow substitute for an experiential narrative of human experience, or the proliferation of multiple stories that get at the heart of sweatshop experience. And what is the internal consistency by which the corporation is able to engage in a textual production of high coherence but no validity at all for the voices of the workers are silent in those aggregated narratives? Rather than consistency, what is important from a qualitative perspective is the number of different tellings, the many biases in the many stories of lived experience.  I seek her a narrative production that accounts for the workers' telling of their many stories. An expert account by an MBA accountant will not do. The sweatshop is a primordial beast where consistency is not really the issue. We are after the internal consistency of Asian women's stories not the consistency of a bunch of accountant auditors. And it is an internal consistency to be worked out in dialogue and in acts of rhizomatic and situated reliability, not in abstractions by auditors and monitors. And as Denzin (HQR, p. 511) argues "these stories should be connected to larger institutional, group, and cultural contexts, including written texts and other systems of discourse..."  That would be the intertextuality of internal/external consistency, knowledge written into a network of texts. 

OBJECTIVITY - findings are free from bias. Habermas and Marcuse argue that rigor-fixated research puts the "how" ahead of the "what's important" (see Kincheloe & McLaren in HQR, p. 151 for more on this point). The technology of the research becomes more important than issues of validity as well as objectivity. At issue is the idea that researchers are experts or that Nike is able to enact an "imperialist gaze" that colonizes the workers as subjects (HQR, p. 152). The question here, for example, is how is it that workers come to know the implications of Nike Code of Conduct, beyond (in China) a packet that is tied about their necks (in some locations). There is a kind of ethnocentrism in assuming that the packets will be interpretable by the workers. Nike's code is a cultural text that needs to be viewed from the outside and the insider (from etic and emic). Objectivity is realized (more or less) over time as various social movements, as well as Nike's staff put pressure on subs to make Codes meaningful.  Bias is for many qualitative researchers a concept particular to logicopositivist work (HQR, p. 165). Every researcher brings their biases into the research setting. Every interview and observation is a disturbance of the field of study.  It is through reflexivity that the qualitative inquirer controls for bias. In addition, trust is required for inquirers to communicate with workers. And there is a significant concern about objectivity when research is conducted on the company ground. There is by most credible accounts a situation of surveillance (at work) that can not be taken for granted. Janesick argues that "becoming immersed in a study requires passion: passion for people, passion for communication, and passion for understanding people" (HQR, p. 217). Bias in positivistic epistemology is the elimination of both passion and subjectivity. But for many critical qualitative researchers, the idea is to enage in "critical subjectivity" and avoid "naive subjectivity."

Critical Subjectivity - means we do not suppress our primary subjective experience. We accept that knowing is from a perspective and we therefore control bias be articulating that perspective (see Reason in HQR, p. 327). Again this is an issue of being more self reflective rather than assuming or presuming there is no bias here.  

Naive Subjectivity - the problem with walking tours of factories is that naive subjectivity takes over. Inquirers are not able to check their bias by getting quotes from workers and managers, and from bringing them into meaningful participation (see Adler & Adler in HQR, p. 381). The use of diverse teams (diverse in age and perspective) can help, but is not a substitute for spending quality time with the workforce. 

Critical Bias - As Denzin (HQR, p. 512) argues it is "the immediate, local, personal, emotional biases of many [that] lead them to tell stories that work outward from the self to society." And we could work with workers to narrate their own Nike lives or to unnarrate stories of Nike that no longer speak to them. 

In sum, issues of validity and reliability are complex when it comes to Nike and Academics Studying Nike.  Rather than voyeuristic factory tours, it is time to do some studies with some solid research methodology. Ones that deal with the kinds of issues just raised. At issue is the relationship between power and knowledge, the power of the corporation and the knowledge it manufacturers.  It is all about poly-storying (Tamara), in the many stories, and in the authoring practices of corporations who speak on behalf of their stakeholders. Right now the storytelling like the research is expert-driven rather than a collaboration of dramatists and narrators.  For Nike and 600,000 workers the new story of factory life is not yet in place. It remains to be narrated in valid and reliable ways. - David M. Boje

 

 

To the issue of research methodology, here is list of suggestions.

  1. We need to interview workers away from the gazing eye of Nike officials and subcontractor managers. Otherwise what happened to Lap Nguyen will happen to our interviewees. 
  2. We need academic studies that are independent of Nike corporate sponsorship (such as the Dartmouth Amos Tuck Wage studies). Yet, to get into a Nike Factory requires Nike's permission. We therefore need Nike to endorse studies where factories are sampled by the researcher and not by Nike.
  3. No more Nike factory tour studies. We saw the results of Nike tour studies with Andrew Young and with the Kahle et. al study. Nike continues to promote factory tour studies.
  4. We need Academy level task forces of academics in accounting, management, marketing, political science and other disciplines to do joint field work on Nike subcontract factories.
  5. Related to #4 is the idea that only be combining an Accounting and an Organization theorist in the same study team can we start to make sense of Nike's self-monitoring and monitor methodologies used by Price Waterhouse Coopers, Global Alliance, and Fair Labor Association.  
  6. Get to Know Research Methods since both Nike and its critics each accuse the other of invalid and unreliable research methods. To get beyond the polemics requires some basic understanding of Research Methods: Visit RMD page. Syllabi;  

"Nike sweatshops is a research methods issue. We need better research designs that address issues of validity, reliability, and epistemology.  It is a research ethics issue."