Saturday, 30 July 2011

Assessment in ESP

A few weeks ago, I was asked an interesting question by an ESP teacher called Rosa in Algeria. The question related to an ESP course Rosa was running as part of her research for her doctorate. She'd conducted a careful needs analysis and designed her course based around those needs. But then the course had stopped unexpectedly before its scheduled end, which meant that there was no way of assessing if the course aims had been achieved. The question was: is the research still valid.

Well, let me start of by admitting that I’ve never been involved in the academic side of ESP. My experience of ESP courses has always been either in-company (where the employer pays) or with mixed groups of professionals and pre-experience students at the British Council (where the students themselves or, occasionally, their parents, pay).

The reason I bring up the grubby subject of who pays here is that it has a big impact on assessment. The person paying for the course has a large say in what the aims of the course should be, and therefore what constitutes a successful course. Whenever I ran an in-company ESP course, I (or my colleagues in sales) had to justify it to the customer in that company. In other words, in my experience, assessment is a service for the employer (usually represented by the HR department) and/or for the student (and his/her parents). The student is also paying in another way – in terms of time invested in attending the course and self-study.

Now, of course, when we get to the public university sector, we have a different customer: the taxpayer. Is the taxpayer getting good value for money out of the education they are paying for? It’s an interesting question, and I sometimes wonder if teachers in the public sector realise they’re doing a service for me as a taxpayer. Of course, individual taxpayers aren’t in a position to check the effectiveness of the courses they pay for, so responsibility is delegated to the government, the universities and ultimately the English teachers themselves, who are expected to provide evidence that they’re actually teaching and that the students are actually studying and even … learning something useful.

OK, so let’s look at Rosa's question first, and then we’ll step back to look at some broader issues. First of all, the problem with courses stopping before they’ve finished is, unfortunately, very common. If assessment is an important part of your course, don’t leave it all to the end. You can actually get a lot of assessment done during the course – not just in formal tests, but also by assessing role-plays, listening activities, written work, etc. You can conduct assessment as part of your regular teaching, to check how much they’ve learnt during the lesson, but perhaps it’s more useful to assess in a later lesson, to check how much they remember. For example, if one of your course aims is to teach your nursing students to conduct a patient admission, you could teach it and practise it in a role-play in one lesson and then repeat the role-play a few weeks later, in controlled conditions (so you can grade it properly) – perhaps without warning the students that there’s going to be a test.

It’s important to think carefully about your criteria for assessment – should everything be based on your course aims? A good way of planning aim-based assessment is by writing your course aims in the form ‘By the end of the course, students will be able to …’. This will then form the basis of your assessment: can they do it or can’t they. Ideally, break each aim down into several sub-aims, so you can give a detailed and objective assessment of how well each student can do things, rather than just an impression mark or whether they can do it or not.

But this then raises the next question. Have they learnt this ability during the course, or could they already do it pretty well before the course? In other words, have they actually improved? To assess this, you’ll need to do some benchmarking at the beginning of the course to identify the starting point.

Of course, not all assessment needs to be aims-based. You can also assess their general level of skills – reading, writing, listening and speaking. For this, I’d recommend using professionally created assessment materials, such as practice tests for Cambridge exams (FCE, Advanced, BEC, etc.). This obviously won’t be connected with the topics you’ve studied in your course, but it can still be valuable to check for and measure improvement in, say, general listening comprehension skills.

In other words, assessment is a complicated business, and to do it properly, you’ll need to do lots of it. But then … will you have any time left to actually teach the poor students? Too much assessment can demotivating for learners and a huge drain on your time both in the classroom and away from it (marking!!!).

This was something that used to bother me when I was teaching at the British Council. We had a wonderfully sophisticated assessment system, with all sorts of assessment events scheduled throughout the semester, which generated a page full of statistics that could be combined into scores on a range of skills for each student. The problem was, there wasn’t really time both to complete the assessment regime and to teach, so some teachers (dare I admit I was one of them?) invented some of the figures in order to prioritise teaching.

You see, I guess I need to come clean about one of ESP’s guilty secrets. We claim to be very sophisticated with our detailed needs analyses and carefully designed courses, but we’ve actually got very little control over what our students take away from the course. For example, you may be doing an exercise with the aim of developing their reading subskills, but what they’ll actually get out of it is some new vocabulary (which you didn’t even notice) or a deeper understanding of some grammar rule, based on the way it’s used in the text. In another lesson, your garbled explanation of a grammar point may fail to teach your students much about the grammar point in question, but they’ll become aware of some nice expressions and idioms for giving explanations (or for apologising for failing to explain something). In yet another lesson, while you are arguing with your students over why answer X from the listening exercise is right, and answer Y is wrong, you may be helping their negotiation skills more than their listening skills. (Note that this only works if the class is conducted in English).

(This is connected with a concept I call the leaky pipeline, which I’ll have to explain in a separate post, along with the related concept of obliquity – achieving great things by not trying too hard to achieve them.)

That’s not to say that our needs analysis and course design is a waste of time – far from it. We need it as a starting point for our teaching, and it needs to appear relevant, interesting and useful in order to motivate our students to engage with it. We also need to include a wide range of language and skills work in our courses – they’ll certainly benefit from it, but perhaps not in exactly the way that we planned. A strong syllabus also provides a focus for their study: a list of good words or useful phrases to learn, for example, is surely beneficial. But we’re kidding ourselves if we think this is all they’ll get out of our lessons, or even the most important thing.

So what does this mean for assessment? Well, on the one hand, it means assessment is less important than we make it out to be. But on the other hand, assessment can be extremely useful in motivating students to learn. Customers (by which I mean parents, employers, taxpayers and others who pay for the courses) also have a right to expect some measurable results from the course. And we mustn't forget that not all teachers are as competent or conscientious as those who read this blog - there are lazy, incompetent teachers out there (apparently), and assessment is perhaps the only way of keeping them on their toes.

So we definitely should assess, both formally (in mid-course and end-of-course tests – including writing, role-plays, etc.) and informally (during the course). But we also shouldn’t take the results too seriously. By the end of the course, a good teacher should know which students are good and hard-working and which are clueless or lazy simply because you have spent time getting to know them and their English (including plenty of time hearing them speak and reading their writing). The formal assessment should simply confirm what you already know.

Does this answer Rosa’s question? Not really. If I were her professor assessing her research, I guess I’d just discuss with her if she thought she’d achieved her aims. I wouldn’t worry too much about missing end-of-course tests. But perhaps that’s why I’m not a professor. I guess ultimately it comes down to each university’s policy on what counts as valid research, and my common-sense approach doesn’t really have much bearing on individual universities’ policies!

So sorry, Rosa, for not really answering your question - although you've given me lots to think about. Perhaps some readers of the blog can add their opinions.

Related posts:
Exams: Financial English ... if such a thing exists
Needs: English for Nursing
Syllabus design: ESP consultancy, Cyprus

Wednesday, 6 July 2011

Where was I...?

It’s been a long time since I’ve written for this blog, or at least anything more than the briefest of posts. So it’s about time I talked about what I’ve been up to. It’s been such a crazy couple of years that it’s difficult to know where to start, but perhaps two years is a good time to go back. That’s roughly when my life went from being seriously busy to unbelievably crazy. It’s also when I started seriously neglecting this poor blog.

This time two years ago, summer 2009, I was in a kind of ESP heaven. I was teaching loads of legal English, and getting pretty good at it. I was finishing work on two books for my series, Cambridge English for Nursing Pre-Intermediate and Cambridge English for Marketing.

In addition to my role as series editor, which was pretty much complete for those two books by summer 2009, I also had some additional work on both books. I edited the free online teacher’s notes for Nursing Pre-Int and wrote a series of grammar worksheets, one for each unit of the book. You can find both the teacher’s notes and the grammar worksheets here: …

For our Marketing book, I wrote the online teacher’s notes myself. It seems like a little job to write online teacher’s notes, but it’s a good couple of months’ work, just as hard as writing a full teacher’s book. In fact, the only difference is that printed teacher’s books generate more money – the workload’s the same.

I was also getting into technical English in a big way: I did a lot of work on an award-winning webcourse called e-Xplore Technical English, an online course developed by the HTWK University of Leipzig. (It won its awards before my involvement, I hasten to add).

The course already existed and was very good, but my job was to dramatically extend of the materials without adding new content. For example, for every reading or listening text, where there were, say, 5 comprehension questions, I wrote another 20. This meant that the computer could select 5 from the bank of 25 (using clever algorithms), so that every person taking the course had a slightly different set of questions, which eliminated the possibility of cheating (which had been a big problem earlier). I also did my usual editorial stuff on the course, fixing things that I didn’t like and suggesting improvements and extensions. Good fun and very satisfying, but a lot of work. But I learnt a lot about technical English on the way, as well as the benefits and drawbacks of e-learning.

Oh, and I was also doing my regular work for Professional English Online: 2 activities (lesson plans) per month, 2 financial English jargonbusters per month, and a quote of the week every week. So I can add financial English to the legal, medical, technical and marketing English I was working on at the same time. And of course I was also teaching at the British Council.

So that’s our starting point, late summer 2009: life was seriously busy, but not yet unbelievably crazy. This poor blog was a bit neglected, but not yet abandoned.

That’s when I got not one but two dream job offers, neither of which I could possibly turn down.

The first one came as a result of a combination of sheer luck and hard work – the usual combination in this business. A year or so earlier, I’d received an email from one of the Grammar editors at Cambridge University Press: she’d noticed my name on the International Legal English teacher’s book, and wondered if I was the same Jeremy Day that used to work with her in Krakow, about 10 years earlier. And of course I am. She remembered that I’d been into grammar in a big way back then, and had always written worksheets and done training sessions for other teachers, so she gave me the chance to do some odd jobs for the Grammar team at CUP. Brilliant – it goes to show that you can never predict which people from your present life will turn out to be useful contacts in the future.

Anyway, those odd jobs included lots of reviewing and evaluation work on Grammar for Business – a very nice and useful book, by the way. And I managed to get a mention in the acknowledgements, which was nice.

I was then asked to write a series of revision units and end-of-unit tests for level 1 (elementary to pre-intermediate) of a new three-level grammar series for teenagers. It was a great break but a huge amount of work: there were 14 revision units and over 70 tests to be written – a substantial fraction of the whole book, in fact.
Anyway, to cut a long story short, I slogged away at that for many months, and finally, about three years later, in early 2011, the book finally emerged: Active Grammar Level 1. Series editor: Penny Ur, one of my ELT heroes. Anyway, I got a small mention in the thanks pages of both books.

But here’s where it gets really interesting: as a result of my hard work on Active Grammar Level 1, I was invited to submit a sample unit to be a co-author of Level 3 of the same series, the advanced level. This was, as I say, towards the end of summer 2009 – the dream job I’d been waiting for all my career. (Yes, it’s true. Much as I love ESP, I’m at heart a grammar guy, and I’ve been obsessed with grammar since the day I started teaching).

So, there I was, September 2009, waiting to see if my sample unit for that project would be approved. Around the same time, I gave a presentation for the British Council at the IATEFL Poland conference, on ‘My favourite grammar structures’. At the end of that presentation, I was approached by one of the editors from Pearson Longman, who asked me if I’d ever thought of writing a book! Again, to cut a long story short, I was invited to submit a sample unit for a general English course for teenagers preparing for their school-leaving exams (e.g. the Polish matura exam). Actually, they were looking for someone to co-author the second edition – the first edition has been a best-seller here in Poland and elsewhere for several years. How could I turn down a chance like that?

Amazingly (and to cut two long and stressful stories short), my sample units for both books were accepted, so I found myself co-authoring a major new grammar book for Cambridge and a major coursebook for Pearson. As I described in an earlier post, writing for two publishers at the same time is never easy: you can’t turn round to one and say ‘sorry, I can’t meet your deadline because I’ve got a more important deadline on my other book’. You’ve just got to work harder than ever before, and sacrifice other parts of your life. That’s around the time I really stopped writing for this blog.

Again, that grammar book has just come out: Active Grammar Level 3. It was a fantastic experience and I learned a huge amount. What I like best about Active Grammar is that it’s a CLIL-based course, which means that you learn about all sorts of things (geology, arts, chemistry – but always in a fun way) at the same time as you’re studying grammar. I certainly learnt lot and probably enjoyed writing it more than any other book I've been involved with.

The Pearson coursebook, New Success Upper Intermediate, was very slightly easier, because it was the second edition I was working on, so we could follow the general plan of the original edition. But it’s still plenty of work to keep you busy for a year – my editor warned me at the beginning to make sure I had a clear schedule (!) to work on it. Anyway, it’s out next year. I’m also really proud of the work I did on it – again, some really interesting topics and I was able to be a lot more creative than with my other books. Also, writing for teenagers is very different from writing for adults, so as usual, I learnt a huge amount along the way.

But that’s not all. My role as Series Editor can’t be switched on and off depending on what other projects I’m working on, so that rolled on at the same time. We commissioned the next two books in the series, Cambridge English for Scientists and Cambridge English for Human Resources, around the same time (late 2009), so for well over a year I was working on four big writing projects at the same time, plus all my other little projects. Anyway, Cambridge English for Scientists came out a couple of months ago, and it’s looking really good. I think that one deserves a separate blog post, which will come soon.

Cambridge English for Human Resources came out a couple of weeks ago, and I’m also really proud of it. Again, I promise to blog about this properly soon.

The downside of working on those four books (Active Grammar 3, New Success Upper Int, Cambridge English for Scientists and Cambridge English for HR) is that they should all generate income for me in the coming years … wonderful, but not much use to me as I try to feed my family and pay my mortgage now. So in addition to all of these, I also needed to take on plenty of other writing jobs, especially as the work on the four big ones was coming to an end about a year ago – summer 2010.

So what did I take on? Bizarrely, I received offers to write or co-write four teacher’s books, all around the same time (a year ago) and all offers I couldn’t resist. The first was Dynamic Presentations, written by another of my ELT heroes, Mark Powell.

Again, my trainer’s notes are online, so no nice book to put in pride of place on my shelf, but still a lot of work and a great opportunity to be involved with such an excellent and important book. The book came out late last year, in time for the BESIG conference (where I was delighted to find myself sharing a taxi with Mark Powell himself – see my blog post here for a report on that conference).

The second teacher’s book was Flightpath, a new course for pilots and air traffic control officers (ATCOs).

Aviation English had been one of the big gaps in my ESP portfolio – I’ve done something on all the other big ESP fields (except maybe IT English), so this was my opportunity to plug that gap. I was invited to co-author the teacher’s book, together with Philip Shawcross, the author of the Student’s book and the president of ICAEA (the International Civil Aviation English Association, ), which was very reassuring. He provided the expertise, while I asked all the silly questions (and got wonderfully detailed answers) and made sure it worked in terms of methodology. As with everything else, I’ll have to come back to this topic later. The books are out around September 2011.

The third teacher’s book was the new edition of International Legal English.

Having written the teacher’s book for the first edition, I wasn’t going to let someone else re-write my masterpiece (!), so I had no choice but to take that on too. That’s out later this year too. 

Finally, there’s the online teacher’s notes for Cambridge English for Human Resources, which I co-wrote with George Sandford, the author of the Student’s book. As with my work on the Marketing teacher’s book, it doesn’t look like much work when you just have a URL to show for it, but it was very hard and time-consuming. (Actually, I haven't even got a URL or image for the teacher's notes - I guess they'll be up on the resource site in a few days).

Oh, one other thing. I also ended up writing two sets of worksheets for the new edition of … [sorry, but I guess that’s still top secret for the time being].

Is that all? Well, I also co-wrote a short handbook, An Introduction to Teaching English for Specific Purposes, with Mark Krzanowski, the co-ordinator of IATEFL’s ESP SIG and a very important person in the world of ESP. It was only a little handbook, so not much work (for a change), but it’s still nice to have that on my CV. You can download the handbook for free here.

In the meantime, I also left my job at the British Council, and have now actually stopped teaching. As I mentioned briefly before, I’m now working for English360, which I think has got to be the future of ESP (and possibly the whole ELT industry).

One project there that has dominated my time with English360 recently has been the trainer’s notes for the TKT (Teaching Knowledge Test) course, which I was asked to write as part of the deal to put the TKT course online the platform. It’s been a huge project for me, and extremely time-consuming, but I’m very proud of it. I’ve just finished off the trainer's notes this week, so they'll appear on the website very soon I guess. That'll be the last of ten books that I was working on at the same time for most of the last 12 months.

Another book I’ve finished off this week (today, in fact) is the trainer’s notes for Communicating Across Cultures, an innovative new course by Bob Dignen. The course is part of Cambridge’s new Business Skills series (along with Dynamic Presentations), and will be really useful for anyone who needs to work in an intercultural environment. (The book will be out soon, and hopefully I'll have a URL and image for the trainer's notes in a couple of months).

Of course there are more books on the way: I’ve started working on one more already, with several more on the horizon, including some really exciting ones. But I can’t say more. I’ve probably already said too much anyway. So I’ll end now with a summary of the past two years, more for me than for you: I can’t believe I’ve done all of this in two years (publication/completion dates in brackets):

As editor / series editor:
(2009) Cambridge English for Nursing Pre-intermediate
(2009) Cambridge English for Nursing Pre-intermediate (Teacher’s Notes)
(2009) Cambridge English for Marketing
(2011) Cambridge English for Scientists
(2011) Cambridge English for Human Resources

As teacher’s book author / co-author:
(2009) Cambridge English for Marketing (Teacher’s Notes)
(2010) Dynamic Presentations (Trainer’s Notes)
(2011) Flightpath (Teacher’s Book – co-author)
(2011) International Legal English 2nd edition (Teacher’s Book)
(2011) Cambridge English for Human Resources (Teacher’s Notes – co-author)
(2011) The TKT Course (Trainer’s Notes)
(2011) Communicating Across Cultures (Trainer’s Notes)

As consultant / writer of supplementary materials
(2009) Cambridge English for Nursing Pre-Intermediate (Grammar Worksheets)
(2009) e-Xplore Technical English
(2009/10) Professional English Online (Activities and Jargonbusters)
(2011) Active Grammar 1 (Review Units and Tests)
(2012) XXX (sorry – still top secret)
(2012) XXX (sorry – still top secret)

As co-author:
(2011) Active Grammar 3
(2012) New Matura Success Upper Intermediate 2nd edition
(2011) An Introduction to Teaching English for Specific Purposes

Sooooo … that’s where I’ve been for the last two years. I hope that explains my absence from the blogosphere for so long. (I could also add that over the last two years my work has taken me on about four tours of Poland, plus Germany, the UK, Austria, Czech Republic, Slovakia, Cyprus, Switzerland, Bosnia and Serbia). I promise I’ll come back and blog about everything properly as soon as things calm down … if they ever do.

Jeremy Day, July 2011

Related posts:
Back from BESIG 2010 
All change 
Good to be back