Assessment in ESP

A few weeks ago, I was asked an interesting question by an ESP teacher called Rosa in Algeria. The question related to an ESP course Rosa was running as part of her research for her doctorate. She'd conducted a careful needs analysis and designed her course based around those needs. But then the course had stopped unexpectedly before its scheduled end, which meant that there was no way of assessing if the course aims had been achieved. The question was: is the research still valid.

Well, let me start of by admitting that I’ve never been involved in the academic side of ESP. My experience of ESP courses has always been either in-company (where the employer pays) or with mixed groups of professionals and pre-experience students at the British Council (where the students themselves or, occasionally, their parents, pay).

The reason I bring up the grubby subject of who pays here is that it has a big impact on assessment. The person paying for the course has a large say in what the aims of the course should be, and therefore what constitutes a successful course. Whenever I ran an in-company ESP course, I (or my colleagues in sales) had to justify it to the customer in that company. In other words, in my experience, assessment is a service for the employer (usually represented by the HR department) and/or for the student (and his/her parents). The student is also paying in another way – in terms of time invested in attending the course and self-study.

Now, of course, when we get to the public university sector, we have a different customer: the taxpayer. Is the taxpayer getting good value for money out of the education they are paying for? It’s an interesting question, and I sometimes wonder if teachers in the public sector realise they’re doing a service for me as a taxpayer. Of course, individual taxpayers aren’t in a position to check the effectiveness of the courses they pay for, so responsibility is delegated to the government, the universities and ultimately the English teachers themselves, who are expected to provide evidence that they’re actually teaching and that the students are actually studying and even … learning something useful.

OK, so let’s look at Rosa's question first, and then we’ll step back to look at some broader issues. First of all, the problem with courses stopping before they’ve finished is, unfortunately, very common. If assessment is an important part of your course, don’t leave it all to the end. You can actually get a lot of assessment done during the course – not just in formal tests, but also by assessing role-plays, listening activities, written work, etc. You can conduct assessment as part of your regular teaching, to check how much they’ve learnt during the lesson, but perhaps it’s more useful to assess in a later lesson, to check how much they remember. For example, if one of your course aims is to teach your nursing students to conduct a patient admission, you could teach it and practise it in a role-play in one lesson and then repeat the role-play a few weeks later, in controlled conditions (so you can grade it properly) – perhaps without warning the students that there’s going to be a test.

It’s important to think carefully about your criteria for assessment – should everything be based on your course aims? A good way of planning aim-based assessment is by writing your course aims in the form ‘By the end of the course, students will be able to …’. This will then form the basis of your assessment: can they do it or can’t they. Ideally, break each aim down into several sub-aims, so you can give a detailed and objective assessment of how well each student can do things, rather than just an impression mark or whether they can do it or not.

But this then raises the next question. Have they learnt this ability during the course, or could they already do it pretty well before the course? In other words, have they actually improved? To assess this, you’ll need to do some benchmarking at the beginning of the course to identify the starting point.

Of course, not all assessment needs to be aims-based. You can also assess their general level of skills – reading, writing, listening and speaking. For this, I’d recommend using professionally created assessment materials, such as practice tests for Cambridge exams (FCE, Advanced, BEC, etc.). This obviously won’t be connected with the topics you’ve studied in your course, but it can still be valuable to check for and measure improvement in, say, general listening comprehension skills.

In other words, assessment is a complicated business, and to do it properly, you’ll need to do lots of it. But then … will you have any time left to actually teach the poor students? Too much assessment can demotivating for learners and a huge drain on your time both in the classroom and away from it (marking!!!).

This was something that used to bother me when I was teaching at the British Council. We had a wonderfully sophisticated assessment system, with all sorts of assessment events scheduled throughout the semester, which generated a page full of statistics that could be combined into scores on a range of skills for each student. The problem was, there wasn’t really time both to complete the assessment regime and to teach, so some teachers (dare I admit I was one of them?) invented some of the figures in order to prioritise teaching.

You see, I guess I need to come clean about one of ESP’s guilty secrets. We claim to be very sophisticated with our detailed needs analyses and carefully designed courses, but we’ve actually got very little control over what our students take away from the course. For example, you may be doing an exercise with the aim of developing their reading subskills, but what they’ll actually get out of it is some new vocabulary (which you didn’t even notice) or a deeper understanding of some grammar rule, based on the way it’s used in the text. In another lesson, your garbled explanation of a grammar point may fail to teach your students much about the grammar point in question, but they’ll become aware of some nice expressions and idioms for giving explanations (or for apologising for failing to explain something). In yet another lesson, while you are arguing with your students over why answer X from the listening exercise is right, and answer Y is wrong, you may be helping their negotiation skills more than their listening skills. (Note that this only works if the class is conducted in English).

(This is connected with a concept I call the leaky pipeline, which I’ll have to explain in a separate post, along with the related concept of obliquity – achieving great things by not trying too hard to achieve them.)

That’s not to say that our needs analysis and course design is a waste of time – far from it. We need it as a starting point for our teaching, and it needs to appear relevant, interesting and useful in order to motivate our students to engage with it. We also need to include a wide range of language and skills work in our courses – they’ll certainly benefit from it, but perhaps not in exactly the way that we planned. A strong syllabus also provides a focus for their study: a list of good words or useful phrases to learn, for example, is surely beneficial. But we’re kidding ourselves if we think this is all they’ll get out of our lessons, or even the most important thing.

So what does this mean for assessment? Well, on the one hand, it means assessment is less important than we make it out to be. But on the other hand, assessment can be extremely useful in motivating students to learn. Customers (by which I mean parents, employers, taxpayers and others who pay for the courses) also have a right to expect some measurable results from the course. And we mustn't forget that not all teachers are as competent or conscientious as those who read this blog - there are lazy, incompetent teachers out there (apparently), and assessment is perhaps the only way of keeping them on their toes.

So we definitely should assess, both formally (in mid-course and end-of-course tests – including writing, role-plays, etc.) and informally (during the course). But we also shouldn’t take the results too seriously. By the end of the course, a good teacher should know which students are good and hard-working and which are clueless or lazy simply because you have spent time getting to know them and their English (including plenty of time hearing them speak and reading their writing). The formal assessment should simply confirm what you already know.

Does this answer Rosa’s question? Not really. If I were her professor assessing her research, I guess I’d just discuss with her if she thought she’d achieved her aims. I wouldn’t worry too much about missing end-of-course tests. But perhaps that’s why I’m not a professor. I guess ultimately it comes down to each university’s policy on what counts as valid research, and my common-sense approach doesn’t really have much bearing on individual universities’ policies!

So sorry, Rosa, for not really answering your question - although you've given me lots to think about. Perhaps some readers of the blog can add their opinions.

Related posts:
Exams: Financial English ... if such a thing exists
Needs: English for Nursing
Syllabus design: ESP consultancy, Cyprus

English for nursing

I was sitting in the staffroom between lessons this evening and one of the teachers asked the room, 'What's the name of that thing nurses wear on their heads?' No-one seemed to know, and then someone suggested, "Ask Jeremy - he's got two books on English for Nursing".

Well, yes. But there weren't any hats in either book, as far as I remember. Not a priority. As far as I'm concerned, the thing nurses wear on their heads is called a nurse hat. Perhaps a kind reader could fill me in on the proper name, although I'll admit now that I'm not really that worried about not knowing.

The point is this: what do nurses (or any other sets of ESP learners, for that matter) need English for? To explain the various parts of their uniform? Or to deal professionally and symathetically with patients in crises or high-emotion situations and with other medical professionals in situations where accuracy may make the difference between life and death?

In our new book, Cambridge English for Nursing Pre-Intermediate, we teach nurses and nurse trainees how to speak with patients who have suffered embarrassing situations (like incontinence). Just stop for a second and think how you would help someone maintain their dignity in that situation ... and then try doing that in a foreign language ... at pre-int level.

We also teach them how to reassure patients who are about to undergo unpleasant operations, such as having a tube inserted through their nose into their stomach. Again, stop and think for a second how you'd deal with that.

We've got a unit on communicating with terminally ill patients.

Last week I had a phone meeting with Virginia Allum, one of the authors of our nursing books (together with Patricia McGarr). Virginia's a hugely experienced nurse and nurse educator (as well as being a great English teacher and writer). She told me that every single dialogue in both books was based on real situations she'd been through as a nurse. Incontinence, tubes up noses, dying patients, everything. It was quite moving hearing her talking about her experiences, and how absolutely important language skills are - for native speakers as much as for foreign language speakers.

Sometimes they were great experiences where she'd done everything right. Others were based on failures, where she'd later analysed what went wrong and what she should have done.

It's easy to lose sight of the fact that we're talking about real-life situations here. This isn't just about teaching people to talk about their holiday palns or to use future perfect instead of future continuous or whatever. It's about making a real difference to the lives of our students and, in turn, to the lives of the people they'll deal with in English.

As I've said before, the most important person in the classroom may not actually be in the classroom. It may be a patient with a tube up their nose or worse.

I hope users of the books go on to use the language and techniques from them. If, as a result of this book, a patient is treated with extra dignity and tact, is reassured when scared, is listened to when they need to talk ... well, for me, that's what it's all about.

Related posts:

• Free online teacher's notes
• Wordlists in ESP
• BESIG conference 2009 part 2

Two approaches to ESP course design

Another brief posting ...

There are basically two types of ESP courses, which we might call English-through and English-for. 

English-through means teaching English through the lens of an ESP field. The aim of the course is to bump your students up to a higher level of global language proficiency (e.g. from CEF level B2 to C1). That means teaching all the grammar, vocab and pronunciation that all other language learners have to study. And making sure your students understand the language structures at that level and can use them as well as others of the same level. It also means working on the four skills - to improve reading speed and listening comprehension, spoken confidence and written style. All that sort of thing.

In other words, it's just like any other English course. The only difference is that everything is done in the context of the ESP field. So you teach present perfect through examples from that field and practise it with a field-relevant role-play, or whatever. You work on their reading skills by giving them increasingly challenging things to do with texts about their field. The ESP field exists in the the course primarily as a means of keeping the course interesting and relevant. If you work in finance, for example, you might get more out of a report-writing task on the causes of the credit crunch than on the pros and cons of fox hunting. Or whatever.

English-for is different. This type of course focuses specifically on the language and skills that are directly relevant to your students' present and future work situations. It's all needs-based. Crucially, it ignores the non-essential language or skills and focuses exclusively on the target language. So if Nurse X never has to write reports for work in English, it doesn't need to be in his/her course. If Engineer Y only ever has to read and write technical English, and never needs to speak, why worry about his/her fluency or pronunciation?

In many ways, English-for is more short-term. It's about giving the students the language they need right now to do their job. Longer-term things, like what they'll need English for in 5 years, is not a priority.

Now, of course in real life, we tend to mix and match - I can't imagine many courses fit the extremes as I've portrayed them. But I think it's important to plan, right from the start, what sort of ESP course you're creating, (mainly) English-through or (mainly) English-for. Which would be more useful for your students right now and in the long run?

English-through courses are quite easy to create. You basically get your syllabus - created by you, the language expert - and find materials to fit it. OK, that's not exactly a piece of cake, but it's doable.

English-for courses are much more challenging for the course designer. You need to get a really deep knowledge of your students' field and somehow find out what language and skills they will need in their jobs. You can find out a lot by asking them, but very often they themselves don't know what they need until it's too late. Very tricky.

(That's one of the big reasons, by the way, why the books in my series, Cambridge English for ..., focus much more on this tricky side - to save teachers the hellish job (or at least reduce it) of finding out for themselves what language people need in particular professions. But I didn't plan this post as an advert for the series, so I'll stop going on about it!)

Anyway, I've got my terminology now, so I'll probably use those labels in other posts too. I'm sure I'm not the first person to come up with the distinction, but I wonder if anyone's used the labels before ...

Related posts:

• Grammar syllabuses for ESP
• A wish list for ESP course design
• Needs, lacks and wants

Needs, lacks and wants

I've been thinking a lot this week and last about student needs. You see I've just taken over a couple of groups of lawyers for a month, while their regular teacher is doing the dreaded CELT-YL qualification. And the book we're using ... English File Upper Intermediate.

Don't get me wrong, it's a fine book. I like the grammar presentations, and the topics are interesting. But last week, after a quick get-to-know-you, I found myself asking the lawyers "What do you know about chimpanzees? Have you ever seen one in a zoo? Can they think?"

Hmmm ...

The weird thing was, though, that I was the only one who seemed distressed by the silliness of asking lawyers to talk about chimps. They were quite happy, and answered all the questions fully and intelligently. It turns out that these students, as well as their bosses (who had arranged the contract with my employer, the British Council), had specifically requested a general English course. I was actually not allowed to touch on legal English. Fair enough ...

And this reminded me of the famous trio of things to consider during a needs analysis: not just NEEDS (what they have to do in their jobs in English) but also their LACKS (what they're currently unable to do) and their WANTS (what will motivate them to study during and between classes). I'd become too focused on NEEDS, and overlooked their WANTS. I also discovered an important LACK during my second lesson with them: serious holes in their understanding of Present Perfect Simple and Continuous (hardly surprisingly, I suppose, given that they're Upper Ints). And that was something they WANTed to overcome.

So I decided to swallow my ESP pride and give them what they lack and what they want.

I've got another group of lawyers who are slightly more complicated. At the start of the course, they and their bosses agreed that they needed to learn about legal English, so we got them copies of International Legal English, and I started working my way through unit 1. But it turns out that what they need and what they want are actually very different. They spend all day dealing with legal problems in English and Polish, and by 5.30pm they're sick of legal English. When I take the book out of my bag they roll their eyes and beg for interesting articles from the internet. Hmmm ...

Here I have a delicate balancing act, because ultimately it's their boss who is the customer, and my brief is to teach them legal English. But I also need to motivate and engage them. So I try to hide little nuggets of legal English in 'internet articles' (such as the one I talked about the other week here). Very tricky.

And what of LACKS? The best example of this was a few years ago when I was working on a course called 'English for European Patent Attorneys' (you see - my imaginative course-naming system doesn't get any better over the years), which I wrote for the British Council in partnership with the European Patent Office (EPO) in Munich. The aim was to have a course for patent attorneys in countries such as Spain, Poland and Italy, which had large populations but whose languages are not official EPO languages.

I was flown to Munich for a couple of days' intense training, including sitting in on some amazing hearings about disputed patents. I learnt about the language needed to apply for a patent, to challenge someone else's patent, to take part in face-to-face hearings, and all the rest of it. Needs, needs, needs. I was even given a big pile of excellent materials developed by the EPO's in-house English teacher.

Anyway, to cut a long story short, I came back to Warsaw and started writing and then teaching my wonderful course. It went pretty well, and the patent attorneys were very happy (not least to have a teacher who had more than a very basic idea of what they did). But it was pretty obvious that they knew all the stuff in the course already. This was their job, what they did every day. It was all new to me, and of course they were interested in the materials, but I was focused too much on their NEEDS and not enough on their LACKS.

So what did they lack? What they were desperate for was the language to write to an important client who hadn't paid for work done several months earlier. It turns out that this is extremely common in the world of patents - it's really quite shocking how often this happens. So that was one of the things I added to the course: some model letters with focus on the ability to be firm but polite.

That's of course only one example, but it taught me an important lesson: focus on what they can't do, not on what they do every day.

By the way, last year I wrote up my lesson on delicate-but-firm emails and put it on Professional English Online. You can download it here. I've used patent attorneys as the context, but of course the language is suitable for all sorts of professionals. I've used that lesson with most of my students, no matter what their background is.

Over to you: how do you determine your students' needs, lacks and wants? Are they all equally important? Do students really know what they want? (Or what they need or lack, for that matter.)