Page 507 - Handbooks of Applied Linguistics Communication Competence Language and Communication Problems Practical Solutions

Intercultural competence and assessment: perspectives from the INCA Project 485

tings reinforced the wide range of different constraints under which a generic
assessment tool would have to operate. While some assessees were permitted to
take a day off for participation in the pilots, some were not. Some employers in-
3
sisted on “on-the-job” assessment. INCA On-line offered the opportunity for a
number of the assessment tests to be undertaken whenever the assessee wished,
yet the pilots confirmed that a mix of computer-based and other forms of assess-
ment produced the most balanced results. The interaction-oriented exercises
(role-plays and a group exercise) were also necessary to provide the assessor
with behavioural data as well as theoretical responses.
In the summer of 2003, more than 50 assessees from five countries (Austria,
Czech Republic, Germany, Netherlands, UK) participated in the pilot exercises
to evaluate the assessment tests. The process also offered feedback on the INCA
grid and the appropriateness of the INCA Portfolio. Alongside the tests for in-
tercultural competences, the INCA project team created a manual for use by as-
sessors as well as a separate, less detailed manual for assessees. Assessors for
the pilot exercise were trained by INCA team members and gave their feedback
on the manuals.
The on-line tests types were tested both online and, where resources did not
permit, using ‘pencil and paper’. Tests in the form of role-plays were also con-
ducted. While the assessment process, tests and assessment guidelines were
standardized, the tests were localized to the testing environment. Care was
4
taken to ensure that the tests and assessment formats were sufficiently generi-
cally applicable to be appropriate for each country. While some of the tests dealt
with a specific culture (e.g., one test set in China), some focused on non-culture-
specific knowledge or behaviour (cf. Matsumoto et al. 2003).
Modifications were made to all aspects of the INCA products as a conse-
quence of the pilot exercises. The pilot exercises offered a limited amount of
empirical data against which to validate the INCA grid, although validation did
not prove conclusive at this stage.
Managing the multinational administration of a multi-faceted assessment
process proved a very challenging endeavour! At the outset of the INCA project
the intention was to develop an assessment tool generically applicable for all
participating countries, localized as necessary. In the pilot phase, the different
exercises were approached in a different way in the various pilot groups, which
supported the principle of localization. So, while the exercises can be under-
taken in all countries, a localization to the prevailing context is required. Further
localization is necessary to ensure that the process and tools are appropriate to
local or company requirements.
The pilots revealed “national” preferences for different kinds of tools, no
doubt a consequence of the richness of the national heritage concerning edu-
cation, training, and assessment expectations and current practices. Whereas
UK apprentices responded well to the open-ended questions associated with the