The Effect of 9/11 on Business Materials Evaluation

Abstract

Business English courses often require the course designer to customise materials to the specific needs of the learners. This article describes and critically views this process for the materials employed with groups of students from Islamic countries preparing to study Business-related courses at U.K. universities before and after the terrorist attacks of 9^th September, 2001. At that time, the potential religious and political sensitivities of teaching Muslim students in the U.K. quickly became heightened and were reflected in changes in syllabus design. This design, or re-design, process of the syllabus involved revision of the existing materials evaluation checklist and the embracement of more regular student feedback, the latter of which, importantly, introduced the notion of ‘negotiating’ the syllabus content. It is proposed in this article that such an on-going evaluation process is necessary for such Business English programs in a time when external events influence student lives and, therefore, needs.

1. Introduction

This study will look at the means by which the evaluation of customised Business English materials designed for groups of Muslim students changed in light of the terrorist attacks in New York on September 11^th, 2001 (hereafter referred to as 9/11).

Before looking at the evaluation issues themselves, I shall briefly describe the structure of the English language program and the background of the students. Following this contextualisation of the setting and students, I will show how the college teachers and students themselves reacted to the events of 9/11 and created student feedback sessions to discuss the existing materials. Then I will provide a detailed analysis of how the pre-9/1 evaluation criteria were formulated from existing evaluation models and, subsequently, how key adaptations were made to those evaluation criteria in the immediate aftermath of that event. Finally, some conclusions for materials evaluation will be drawn.

1. 1 English language program and student background

All the students in question enrolled on the English language preparation course at a small college in the U.K. in 2001 wished to embark upon bachelor or master courses in business-related fields in U.K. universities commencing either Autumn 2001 or 2002. Their English language preparation course was organised by the staff in the English for Academic Purposes (EAP) section of an English as a Foreign language department at the college. The EAP program was approved by several U.K. universities as a sufficient preparation course for direct entry to specific business-related Bachelor and master programs. Although not exclusively geared to students from Islamic countries, those enrolled during 9/11 were predominantly from Turkey, Morocco, Syria, Libya, Saudi Arabia and Guinea and all of the Islamic faith. The preparation course was therefore designed with consideration of the demands of future university study, as well as the individual students’ specific linguistic and future course content needs in mind, in this sense being ‘customised’ to each student group (between ten to twenty) as far as was possible.

The syllabus took the form of a set framework of subjects including academic writing and research skills, lecture listening and note-taking, seminar interaction, presentation skills, and some General English lessons to enable students how to cope with everyday life. Supplementary test preparation (GMAT and IELTS) was available if required by each university to which the students wished to apply. Individual lessons constituting each syllabus component could vary according to the student level, needs and preferences from a pre-course test and needs analysis. In the months before 9/11, the materials chosen in light of the test and needs analysis were evaluated by the teaching staff in a formative and summative manner, i.e. at the beginning and at the end of each term (December for the Autumn term, March for the Spring term, June for the Summer term and September for the Intensive Summer course). Minor adjustments were occasionally made mid-course if teachers perceived the students as finding the materials linguistically or content-wise unsuitable. That evaluation was, however, in essence teacher-directed.

The events of 9/11 quickly revealed flaws in the materials evaluation criteria and the process in which it was conducted. On the personal level, the shock among students at the college manifested itself in a pre-occupation with the terrorist attacks, the potential effects on the world economy and, perhaps not to be overlooked, the students’ own feelings of security studying in the U.K. Reports came in of some being verbally abused by local people whilst shopping, some veiled female students being shouted at in the street and a general feeling of being ‘unsafe’ outside the college campus or dormitories. As a consequence, college teachers and staff took on the role of offering pastoral care to the students at a level unprecedented in their careers. Some students even wished to return home and reported that their interest in studying at university had been lost.

1.2 Reacting to student concerns

The extra teacher to student contact in the immediate aftermath of 9/11 led to not only an increase in the teachers’ awareness about student feelings of insecurity, but also about a greater need as to how to redirect their attention back to the English preparation studies. Some students naturally moved on to universities for the October term, however, those remaining at the college and those joining them on the program tended to show a lack of interest in some materials used in the pre 9/11 period. The syllabus framework itself could not be altered due to its university approval, however, as a group of teachers, it was decided to review the materials themselves within that framework for the October term.

One point which became clear was the necessity to change the timing of the evaluation from not simply its formative and summative timing, but to one which was, at the start, on a weekly basis. The previous evaluation checklist itself now required supplementation with new criteria which placed more emphasis on issues immediately ‘relevant’ to the students’ worlds, in this case, the Islamic background of the students. It was thought that the ‘voice’ of students on Islamic issues previously expressed in pastoral care, in corridors between lessons and in telephone calls at night could best be brought into the classroom and integrated into the syllabus itself. In brief, the syllabus as a framework was seen as providing the bridge between student real-life concerns and their academic development.

Furthermore, the weekly evaluation of the materials initiated from October required the extra dimension of more regular student feedback about what had been taught and what was upcoming on the syllabus for the following week. This was the introduction of the notion of a partially ‘negotiated’ syllabus (Nation 2000) between the students and teachers, focusing on the day-to-day materials used to constitute the syllabus rather than the university-approved framework itself. The challenge was clearly to listen to student feedback on the relevance of the pre-9/11 materials to Islamic issues, source them and implement them into the existing program whilst still meeting the academic needs within the framework. This entailed much cutting of lesson materials which had previously been deemed as well-researched and relevant to previous students’ needs.

1.3 Student feedback

The student feedback sessions, although requested by students and approved by all teaching staff, showed perhaps some growing pains due to the fundamental idea of allowing students to criticise materials. It took time for some students to dare to give negative feedback on what had been taught, and even longer to offer suggestions as to preferred themes for upcoming lessons. Some teachers raised the issue of to what extent such ‘autonomy’ was a culturally accepted strategy by which course material should be determined. This is echoed by Sinclair (1997) who warns about its wholesale import into language programs with students unaccustomed to giving potentially face-challenging feedback to those in authority. These concerns slowly dissipated after a few weeks when key students from each national group started to become convinced that this approach was beginning to show its benefits in the more motivating nature of the materials.

Materials started to take on a distinct Islamic content orientation in content. Due to the extra media coverage at that time, reading texts and video recording were relatively abundant compared to the pre-9/11 period. This led to key components of the program to include seminar discussions based on the effect of globalisation on Islamic nations, wall presentations on FDI (Foreign Direct Investment) into their respective countries and regions, and recorded video programs on the reaction to the 9/11 attacks. One BBC recording on the influence of Islam upon the history of commerce, though linguistically challenging, resulted in stimulating analysis and debate. In brief, the world knowledge of the students, their ‘schemata’, had been activated and put to use inside the framework of the program.

However, materials evaluation itself needed to continue and the criteria revised. I turn now to how the evaluation criteria had been formulated in the pre-9/11 period and how it was revised.

2. Evaluation criteria

The pre-9/11 period, as previously described, had involved a basic formative, occasional mid-term and summative evaluation pattern in assessing the chosen materials for the EAP program. That system had been considered as working effectively since 1996. As seen, though, its effectiveness to react more dynamically to critical incidents surrounding the lives of the students required radical revision during September 2001.

The evaluation of materials focused around a set of criteria formulated from five existing models. I will consider firstly those models in turn and then explain how an amalgamated checklist was created. The models chosen were of varying detail, depth and perspective and were popular reference points for evaluation in the field of English as a Foreign Language at the time of the original compilation of the evaluation checklist.

2.1 Models

The first model is that of Morrow (1977) who advises four simple and direct criteria to be considered :

1. “What is my material about ?” 2. “Why was my material produced ?”

3. “Who was my material produced for ?” 4. “How was my material produced ?” i.e. is the language in an appropriate “mode” (written, spoken e.t.c.)?

The second model is more detailed and student-centred in nature and emanates from the work of Sheldon (1988, pp. 241-245) who presents “common-core factors”: the “rationale” factor considers the needs analysis, examining all student background information (level, learning preferences and culture). “Layout/graphics” addresses appearance and instructions for self-study. “Selection/grading” considers the depth to which the language is taught. “Physical characteristics” look at the space on the material for note-taking. “Sufficiency” asks the teacher to consider the quantity of material. “Cultural bias” assesses its suitability for students’ backgrounds and expectations. “Stimulus/practice/revision” asks if the material is “interactive”, allowing the students to use the language enough for eventual storage. “Flexibility” fundamentally considers whether the material is demanding too much of the teacher for preparation and the students for homework. Finally, “guidance” investigates the clarity and depth of the instructions. Of note in this model is the inclusion of “cultural bias” which was clearly, on reflection, overlooked in the pre 9/11 materials design process.

The third model is that of Breen and Candlin (1987, pp. 13-26) who provide a more structured model in terms of its procedure in two phases, looking firstly at the materials and secondly focusing on the learner. Phase 1 considers :

I. “What the aims and content of the materials are.”

II. “What they require learners to do.”

III. “What they require you, as a teacher, to do.”

IV.“What function they have as a classroom resource.”

Phase 2 then investigates :

a) “learner needs;”

b) “learner approaches to language learning;”

c) “the teaching/learning processes in your classroom.”

Phase 1’s criteria offer extra dimensions and depth over Morrow (1977) and Sheldon (1988). They consider what is potentially “missing” in the material and “..the mental operations or steps which the learner has to undertake to be successful” (p15). Of particular interest though is their evaluation of how the teacher makes use of student input in to the materials (p17), a point previously not fully exploited in the pre 9/11 period.

Phase 2 uniquely asks the teacher to question whether the materials can cater for the students’ “long-term goals”, suggesting that guidance should be offered somehow through the materials as to post-course study. Although Sheldon (1988, p. 243) considers the learners’ background learning preferences, Breen and Candlin (1987,p. 24) go further, advocating :

“..we need to be sure that the materials we use are open enough to accommodate both our own and our learners’ preferences on appropriate procedures.”

On reflection, this important student-centred criteria was perhaps neglected, perhaps since the stress in the model was placed upon procedural preferences, i.e. methodology, rather than materials content itself.

For a fourth model, Cunningsworth (1984,15-20) considers more linguistic factors of the materials in terms of “language content”: “form” (phonology, vocabulary, grammar, discourse features), “functions” (e.g. obligations), “appropriateness” (i.e. where the language is suited to the situation), “varieties” (i.e. dialect, style, register and the medium of communication) and “language skills” (receptive, productive, integrated or translations). These were all valid criteria and were thought to complement the more macro features of Morrow (1977), the detail and focus on the learner of Sheldon (1988), and the procedural considerations of Breen and Candlin (1987).

What I sought at that time was a framework into which all these considerations could be integrated. For this purpose, I adopted the model specifically designed for Business English materials evaluation by Ellis and Johnson (1994, pp. 128-129). They ask six easily-worded questions which encourage reflection by the materials designer covering the learner, “language objectives”, “topic areas”, “methodologies”, “the role of the material” and lastly the appearance. In this process of integration of criteria from various models, it was decided not to include too much detail which made it easy to “fail” the material (Swales 1980). As a consequence of this user-friendly consideration, despite the fact that all existing criteria presented valid areas for reflection, a simplified amalgamation was sought. The resulted in the following set of criteria to be selected.

2.2 The Pre-9/11 Evaluation list

The pre-9/11 list, formulated in 1996, operated for roughly five years. Its amalgamation with selected criteria from other models took the form of six reflective questions from Ellis and Johnson (1994), as follows:

What kind of learner is the material aimed at?

This considers work area, work experience, language level, cultural background, educational background, age, and learning style, aligning itself broadly with Morrow’s (1977) third criteria and Sheldon’s (1988) needs analyses and student background criteria.

What are the main language objectives?

Such language objectives include skills, functions, structures, vocabulary and is extended to include Cunningworth’s (1984, pp. 15-20) “language content” criteria.

What are the main topic areas covered?

Topic areas include the following questions: Does the material introduce the subject ? Does it use the topic area as a contextual background? Is the topic content high in credibility ? (Ellis and Johnson 1994, p. 128). As extra considerations, Morrow (1977) warns of materials selection based on topic alone as the type of language students need to use may not be included in those materials. Also, it is difficult to evaluate “credibility” for pre-experience learners until they actually graduate and enter the business world. In this light, the materials designer must predict the students’ future needs rather than meet the present ones.

What is the main methodological approach?

This is supplemented by Breen and Candlin’s (1987) Phase 2 “learner approaches to language learning” and “the teaching/learning processes in your classroom” to consider learners’ assumptions about how they are to be taught.

What is the role of the material?

The role of the materials studied “to practise language” and “to check or test knowledge” of the theme and language is considered here (Ellis and Johnson 1994, p. 128). This evaluation point, similar to Breen and Candlin’s (1987) Phase 1, investigates whether the materials were created as a linguistic resource, a topic-based quiz or debating tool for the students.

Is the material attractive?

This criteria was supplemented by the idea of lay-out and note-taking space (Sheldon 1988).

2.3 Post-9/11 Supplements

In consideration of the aftermath of 9/11 and the quickly changed needs of the students, the existing checklist was seen as lacking in certain areas. Firstly, the purpose of evaluation was questioned and who actually should be involved in its process. Although Rea-Dickins and Germaine (1992, p. 67) advocate that “the ELT practitioners” alone should conduct the evaluation, in contrast, up to even ten possible evaluators are referred to by Low (1987) as an ideal body working together. This was perhaps impractical in our circumstances, but nevertheless influential upon the rationale for including the voice of all students and all teachers in a more democratic decision-making process.

The second supplementary area to the existing checklist addressed the contentious area of authenticity. In the literature, Johnson’s (1993) support of the use of authentic materials is contrasted with Morrow (1977), who criticises their use on the grounds that they are in effect rendered immediately unauthentic when brought in to the ‘unreal’ setting of the classroom. Further to this, Robinson (1980, pp. 35-36) adds to the debate by pointing out that “authentic” materials can lack “relevance” to the students’ needs. “Relevant” materials may in fact be altered to these needs and do not have to be “authentic”. Among staff and students at our college, however, the desire to experience authentic texts and video recordings overshadowed these arguments against authenticity. As previously stated, this at times resulted in text choice above the linguistic level of some students, yet appeared to be compensated by the more stimulating content and perception of its relevance to the students’ backgrounds.

The next supplement looked at to what extent the process of updating the materials should continue. In this regard, Potts (1985, pp. 19-20) looks at the curriculum in terms of two broad categories, “objective” curricula which are based “in advance” of the course on structural/functional criteria, and “communicative” curricula which are less prescriptive. It was decided that the ‘objective’ sense to this argument took the form of the framework of the program itself, i.e. the fact that lecture note-taking had to be taught, because it was required of universities which had approved the program. The ‘less prescriptive’ aspect was applied to the materials making up the framework as it they were open to student negotiation.

The final supplementary area to the checklist concerned the area of cultural hegemony. It was discussed among teachers as to what extent the existing materials addressed, or enabled, Islamic students to refer to their local cultural settings in topics related to business. The concept of cultural hegemony, that is, in this case, of an anglo-centric stance to topics introduced on the course needed to be addressed by the realisation that the Islamic world had previously been greatly underrepresented in materials choice, hence giving perhaps the impression of it being intentionally marginalised by the course designers. It was then decided to review all future materials according to this criteria.

3. Conclusions for future materials evaluation

This paper has looked at the impact which the 9/11 has had on the materials evaluation criteria of an English preparation course at college level. It is seen that the ability to evaluate materials in times of relative non-conflict may tend to be deficient when compared to the times of conflict deeply affecting the lives of the students who use those materials. The upheavals in student life caused by the events of 9/11 at this particular college required more than extra teacher to student pastoral care, but major changes in the way that Business English programs addressed students’ real-life needs. The decision to democratise the materials evaluation process did raise issues of the suitability of giving so much ‘autonomy’ to some students so quickly, yet the increased perception that the new materials possessed more relevance to their new circumstances seemed to counter-act that initial cynicism. The materials evaluation ‘list’ itself was revised to fundamentally address not only the materials content in terms of its authenticity and potential cultural hegemony embedded in them, but also who evaluates them and how often. Revisions, in brief, took account of content sensitivity and procedure whilst still meeting the requirements set out in the framework itself.

Post-script

In the few years following the readjustments made to the materials evaluation criteria on this Business English preparation program, the teaching team involved has moved on, as have I myself. The students themselves who were at the college during the times of changes to their lives are now studying at various universities. Perhaps the final question remains as to what the participants, both teachers and students, in that re-evaluation process in Autumn 2001 have learned from that experience and taken with them into their present study and work environments. Finally, it is hoped that they are involved in materials evaluation which embraces all voices in the educational process and is an on-going one, continually improving the quality, relevance and cultural sensitivity of materials for the Business English classroom.

4. References

Breen, M. P. and Candlin, C. N. (1987). Which materials ? A consumer’s and designer’s guide. In L.E. Sheldon (Ed.) ELT Textbooks and materials: Problems in Evaluation and Development ELT Document 126 (pp. 13-28) London: Modern English Publications.

Cunningsworth, A. (1984). Evaluating and Selecting EFL Teaching Materials Heinemann Educational Books.

Ellis, M. and Johnson, C. (1994). Teaching Business English Oxford University Press.

Johnson, C. (1993) Business English. Language Teaching 26, 201-209.

Morrow, K. (1977). Authentic texts and ESP. In S. Holden (Ed.) English for Specific Purposes (pp. 13-17) London: Modern English Publications.

Low, G. (1987). The need for a multi-perspective approach to the evaluation of foreign language teaching materials. Evaluation and Research in Education. 1/1,19-29.

Nation, P. (2000). Designing and improving a language course. Forum. Vol. 30/4, pp. 2 – 11.

Potts, P. J. (1985). The Role of Evaluation in a Communicative Curriculum, some consequences for Materials Design. In J. C. Alderson (Ed.) Lancaster Practical Papers in English Language Education. Vol. 6. Evaluation (pp.17 – 44). London: Pergamon.

Rea-Dickins, P. and Germaine, K. (1992). Evaluation. Oxford: Oxford University Press.

Robinson, P. (1980). ESP (English for Specific Purposes): the present position. London: Pergamon.

Sheldon, L. E. (1988). Evaluating ELT textbooks and materials. ELT Journal 42/4, 237-246.

Sinclair, B. (1997). Learner Autonomy: The Cross Cultural Question. IATEFL Issues. Downloaded 21/05/00 from http://www.iatefl.org/in97.html.

Swales, J. (1980). The Textbook Problem. ESP Journal Vol.1/1, pp. 11-23.

Ugg Boots