A series of studies into Cultural Intelligence with the ultimate goal of developing a better Cultural Intelligence measurement instrument.

Project 1:

BCIQ (CQ instrument for business settings)

Research team: Ilan Alon, Michele Boulanger, Judith Meyers, Vas Taras

Status: Initial paper introducing BCIQ published in Cross-Cultural Management: International Journal in 2015; research continues in an attempt to further refine the instrument.

Project 2:

CQ-O (instrument that relies on objective, rather than subjective measurement of CQ)

Rather than asking respondents to assess their own cultural intelligence, the instrument relies on measurement of frequencies and occurrences of actual observable behaviors, skills, and knowledge that could be used as markers of cultural intelligence and effectiveness in cross-cultural settings.

Lead researcher: Vas Taras

Status: Initial testing of the concept in progress

Working version of the paper on CQ-O

The Development and Validation of

The Quasi-Observational Cultural Intelligence (QO-CQ) Instrument

(A Working Paper)

Abstract

As cross-cultural interactions are becoming commonplace, cultural intelligence becomes critical to career and organizational success. Accordingly, measuring cultural intelligence is a very important task. Despite a variety of available CQ measures, most of them suffer from a series serious limitations. This interactive session submission presents a new Cultural Intelligence instrument that is drastically different from all earlier CQ measures. While existing instrument almost exclusively rely on subjective self-evaluation and self-reporting, this new instrument relies on observation data. Using an analogy with IQ tests, the existing CQ measures do not directly assess one’s cultural intelligence with questions that have correct and wrong answers (e.g., 2+2=? in IQ tests), but instead ask the test takers to subjectively assess their own cultural intelligence (e.g., On a scale from 1 (I have no clue) to 5(I know very well), how well do you know the answer to the question “2+2=?”). The QO-CQ instrument uses the quasi-observation approach to resolve the problem.

The Problem

The Importance of Cultural Intelligence

The process of globalization can be traced all the way back to the great exploration and colonization times of the 15-17^th centuries. The pace of globalization picked-up with the advent of the telegraph and steam engine. The spread of the Internet starting in the 90s further accelerated the process. Finally, the arrival of the social media and various crowdsourcing, crowdfunding, sharing, and trading platforms accelerated globalization even further.

If several decades ago cross-cultural interactions were still a rarity, a prerogative of expatriates and top managers in MNEs, today cultural diversity is an inherent part of any workplace. Certainly in the so-called “developed” countries, but also increasingly so in the so-called “developing” countries, interactive with people from other cultures is an inevitable reality. Even if a particular person does not have to make international trips, there is a very good chance there are immigrant or visa workers at the company, or the company’s clients, suppliers or partners come from different cultures, not to mention person cross-cultural interactions during international travel, or with friends and colleagues from other countries.

Furthermore, the ubiquity of virtual communication technologies all but ensures that even lower-rank employees working on routine tasks have to collaborate online, if only occasionally. Since virtual communication tools are not limited by the geographic separation, such collaboration tend to increasingly cross national borders and cultures. As Crowne puts it (2008), even though “…some workers may never work outside their country of citizenship, many will interact with customers, clients, suppliers, and co-workers who are themselves outside their home country” (p. 396).

An ability to function effectively in these cross-cultural situations emerges as a critical skill required for organizational and personal success. Ability to recognize and navigate cultural cues, communicate and negotiate across cultures, avoid conflicts and achieve desirable outcomes when dealing with people of diverse cultural backgrounds can determine who gets a job, promotion, or contract and who completes the project faster, at better quality level, and to a greater satisfaction of all involved parties.

The various competencies, skills, and behaviors necessary to effectively function in cross-cultural contexts are normally referred to as “cultural intelligence” or the Cultural Intelligence Quotient (CQ). The CQ is defined as a system of interacting knowledge, adaptive skills, and a repertoire of leadership behavior that make one effective in different intercultural situations and allow to adapt to, select, and shape the cultural aspects of their environment (Thomas et al., 2008). Simply put, the greater one’s cultural intelligence, the more likely one is able to effectively manage culturally diverse settings (Ang et al., 2007). This can include both international situations, requiring cross-border leadership effectiveness (Alon and Higgins, 2005; Rockstuhl et al., 2011) as well as domestic situations. CQ appears in literature under different names. Often, it is referred to as “cultural competencies”, “cultural aptitude”, “cultural awareness”, “global mindset”, and other related terms.

CQ in International Business

The importance of the CQ in international business and management has been well established. It has been shown to affect work and academic performance and judgment, work- and academic-related attitudes, psychological adjustments, sociological adjustments, work and academic performance, project grades, peer evaluations, and managerial assessments (for reviews see Gabrenya et al., 2011, Ang et al., 2007).

Measuring CQ

As cultural intelligence becoming ever more critical for individual careers and organizational performance, CQ measurement becomes an important task. As Lord Kelvin put it in 1901,

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.”

Although the CQ is a relatively new construct in international business research, a number of instruments have already been developed to measure it. The existing CQ measures have been criticized for problems with construct conceptualization of their underlying models (Thomas et al., 2008), subjective perception and self-reporting nature, limited scope, and limited predictive and discriminant validity (Gabrenya et al., 2011)

The instrument offered by Ang and colleagues (2007) is probably the most popular publicly available one. I have been able to catalogue about two dozen other CQ measures, some commercial, some academic. The review of the instruments reveals a number of problems that limit these instrument’s validity, and often reliability. The biggest limitation of the existing CQ instruments is their reliance on subjective self-evaluation. Let me explain this problem by comparing a cognitive abilities test (IQ) to its cultural intelligence counterpart (CQ). IQ tests directly measure one’s intelligence. They are comprised of items that have correct and wrong answers and the respondent either knows the answer (indicates high IQ) or does not (indicates low IQ). For example, the question could be (and I simply here), “2+2=?” If the respondent answers “4”, she/he is awarded a point. If she does not know the answer, she gets no credit for the question.

CQ measures, in contrast, do not directly assess one’s skills or abilities. Instead, they ask the respondent to subjectively evaluate and report her/his skills or abilities. The question may ask the respondent to, for example, report how well she/he is familiar with customs and traditions of different countries, or how easily one can adjust to new cultural environments. If IQ test questions followed the same subjective self-report format, they would not ask “2+2=?”, but instead would be phrased like this,

On a scale from 1 (I have no clue) to 5(I know very well), how well do you know the answer to the question “2+2=?”

Clearly, one’s belief that one knows the answer to the question very well does not necessarily mean it is the case. For example, just because one believes her or his knowledge of the customs and traditions of different countries is very high (for example, 5 on a 5-point Likert scale) does not mean that this is an accurate representation. All kinds of factor can sway this score away from the truth. One may not have a good reference point. Different people may benchmark “5” differently. How much is “very well”? The answer may be affected by cross-cultural response style differences. For example, people from some cultures have been shown to prefer middle points on Likert-type scales, while others tend to choose the extremes (Harzing, 2006; Johnson, Kulesa, Cho, & Shavitt, 2005; Smith, 2004; Smith & Fischer, 2008). Thus, my “4” may not be your “4”. Finally, CQ tests scores are prone to manipulation. One may simply lie to appear more culturally skilled and knowledgeable.

The Present Study

The present study attempts to address this limitation by developing a CQ instrument that relies on observation of behaviors that are manifestations of cultural skills, abilities, and motivations, rather than on subjective assessment of such skills, abilities, and motivations. Admittedly, it is practically impossible to actually observe how the test takers behave in different cross-cultural situation. However, the present instrument tries to approach it as closely as possible by asking the respondents to recall a particular cross-cultural event for priming reasons, and then to describe their exact behaviors in those situations. The answer options also provide specific measurable anchors (e.g., on a recent trip to a new country, leaned “1-2 new words”, “3-5 new words”, “5-10 new words”, etc.). This is why I refer to this instrument as “quasi-observational” CQ.

While such approach, clearly, still remains open to manipulation and socially-desirable responses, it effectively addresses the problem of subjective judgement, ambiguity arising from different benchmarking and cross-cultural response style differences. There will be no question where the respondent would say, as it often happens with traditional CQ measures, “Oh gosh, I don’t even know how well I …. Let me think.” All questions refer to specific measurable behaviors and outcomes and one does not have to guess how much is “well” or “poor.” Even socially desirable responses could be minimized using this approach as most of the answer can be verified (e.g., the question asking how many new words one learned on a recent trip overseas could be followed up with a request to list those words, if the tests, such as prospective employer, wanted to verify the accuracy of the responses).

Development of Quasi-Observational CQ (QO-CQ) Instrument

The Approach

The Quasi-Observational CQ instrument presented here relies just on that: quasi observations. The test starts with a question whether or not the respondent has ever travelled internationally, and if so, a request to recall the last few trips to new countries (a different version of the instrument for those who haven’t travelled internationally is available and detailed further below):

What were the last 2-3 new countries that you visited as a tourist or for work (short-term trips, not immigration or long-term education)?”

Then the respondent is presented with a series of priming questions to refresh the respondent’s memory about the trip, such as:

What year(s) did you go there?

Did you travel for business (work) or pleasure (tourism)?

How long did you spend in those countries on average?

Then, the respondent is presented with a series of 25 questions that ask the respondent to describe her/his specific behaviors as she/he was preparing for the trip, during the trip, after the trip, as well as a few questions about one’s views, and lastly a series of questions that test the respondent’s hard knowledge about world’s cultures. Each question is followed by a series of answer options designed to represent a specific number or frequency of actions, observations, interactions, facts, or other points of interest. The complete list of items is provided in Appendix 1.

An alternative “domestic” version of the QO-CQ instrument was developed for people who have never travelled internationally. Just like the “international” version of the instrument, this one starts with a series of priming questions, such as:

Do you often interact with people from other countries, in person or online?

What are the 1-2 nationalities/countries with which you interact most often?

How many people from those countries have you met (know their names and some personal information)?

How often do you interact with people of that nationality?

The rest of the instrument is very similar to the international version, just with the wording adjusted for domestic cross-cultural interaction situations. To sort the respondents into the “international” vs. “domestic” version, the respondents are first asked if they ever traveled to other countries:

Yes, I have visited other countries (other than the country where I was raised)

No, I have never been outside the country where I was raised

Dimensionality

Prior literature has explored the conceptualization and dimensionality of CQ in depth. It is not the goal of the present study to challenge these premises. My goal is to approach an existing CQ construct, as described in the literature, and offer a more objective, valid, and reliable way to measure this construct along multiple earlier described dimensions.

The multi-faceted nature of CQ has been long recognized. The CQ dimensions originally proposed by Ang et al (2007) appear to be have survived the test of time. Their variations are found in virtually all more recent instruments. These are the four dimensions in Ang et al’s original CQ instrument:

Metacognitive CQ focuses on higher order thinking and involves mental processes used to understand cultural knowledge. It has been shown to predict situational cultural judgments, decision-making, and performance of tasks (Ang et al., 2006; Ang et al., 2007).

Cognitive CQ focuses on knowledge of norms, practices, and conventions in different cultures acquired from education and personal experiences (Ang and Van Dyne, 2008), and knowledge of basic frameworks for cultural values, such as those offered by Hofstede (2004).

Motivational CQ is the capability to direct attention and energy toward learning about and functioning in situations that are characterized by cultural differences (Ang and Van Dyne, 2008a).

Behavioral CQ is the capability to exhibit appropriate verbal and nonverbal actions when interacting with people from different cultures (Ang and Van Dyne, 2008), including using culturally appropriate words, tones, and gestures (Gudykunst and Ting-Toomey, 1988).

Other models of CQ offered somewhat different dimensions of the construct, although parallels are obvious. The Center for Leadership and Cultural Intelligence[1] (CLCI) model includes: CQ Strategy, CQ Knowledge, CQ Drive, and CQ Action, which roughly correspond to the Ang et al’s (2007) dimensions of “Meta-cognitive”, “Cognitive”, “Motivational”, and “Behavioral.”

The models by and Van der Zee and Van Oudenhoven (2000) are somewhat different from those by Ang and Van Dyne (2008), but they still contain factors that roughly correspond to the motivational, cognitive, and behavioral components. Specifically, the Matsumoto’s Intercultural Adjustment Potential Scale (ICAPS) is also based on a four-factor model comprised of emotional regulation, flexibility, openness, and critical thinking Matsumoto et al. (2001). A related instrument by Van der Zee and Van Oudenhoven’s Multicultural Personality Questionnaire (2000) assesses cultural empathy, open-mindedness, emotional stability, flexibility, and social initiative.

Likewise, the Kozai Group has developed two CQ models for academic use. Their Global Competencies Inventory (GCI) and Intercultural Effectiveness Scale (IES) are based on three dimensions that also include motivational and cognitive components, namely: Continuous Learning (cultural self-awareness and continuous exploration and the learning of cultures); Interpersonal Engagement (a global mindset and interest in cross-cultural interactions), and Hardiness (a positive cultural view and emotional intelligence).

Lastly, the Thunderbird’s Global Mindset Inventory (Thunderbird Najafi Global Mindset Institute, 2015) is another proprietary model/instrument developed based on a sample of actual business expatriates. Its three dimensions have notably different name, but still can be related to Ang et al’s (2007) dimensions of CQ motivation, knowledge, and behaviors. They include: Intellectual Capital (global business savvy, cognitive complexity, and cosmopolitan outlook), Psychological Capital (a passion for diversity, a quest for adventure, and self-assurance), and Social Capital (intercultural empathy, interpersonal impact, and diplomacy).

QO-CQ Dimensionality

When developing QO-CQ, I was trying to stick with the dimensionality described in earlier literature. However, because the focus of the instrument is on behaviors in specific cross-cultural situations, I had to update the list of the dimensions so that they follow the natural progression of preparation for, behavior and experiences in, and reflections after cross-cultural interactions.

Accordingly, the following dimensions are included in QO-CQ:

Preparation (3 items): Behaviors and skills related to preparation to international interaction. Here is a sample item included in this dimension: “In the weeks before your trips, how much time did you spend preparing and learning about the cultures and traditions of those new countries?”

Seeking (4 items): Behaviors related to seeking opportunities and engaging in cross-cultural interactions. A sample item here would be:, “When visiting those new countries, how many new local people did you meet and befriend (asked for their name, what they do, what they like, etc.)?”

Adaptation (4 items): Behaviors and skills related to adapting one’s communication and interaction style to make it more suitable for the specific cross-cultural situation. Here is a sample item: “As you were interacting with the locals, did you adjust your communication style?”

Learning (3 items): Ability to spot cultural differences and learn from them to be better prepared and better adjust next time. Here is a sample item: “When you were visiting those new countries, did you notice many differences between the local and your own cultures?”

Culture Shock (3 items): The respondents ability to adapt to the cultural differences and function stress-free in the new culture. A sample item would include: “When visiting those countries, were there moments when you missed your home and wanted to leave back?”

Knowledge (25 randomly selected items from a larger question bank): Finally, the last batch of test questions is comprised of a constantly updated bank of about 100 questions about cultures and countries of het world. Since the test is taken online, the system randomly presents 25 question from the larger question bank. All knowledge questions have only two answer options: “True” and “False” which makes the test more efficient. With a total of 25 question, a random response would give 12-13 correct answers, or 50%. In the global sample, the percent of correct answers varied between 75 and 80%, with hardly any respondent getting all questions correctly, and scores below 55 being also very rare. Here are a few sample questions, “4 is considered a lucky number in China” (T/F), or “Japanese negotiators are very expressive and emotional” (T/F), or “In India, shaking your head from side to side is a sign of disagreement and confrontation.”

Validation

Sample

The QO-CQ instrument was developed in five waves. Approximately 3,500 MBA and undergraduate business students participating in an international business consulting project completed the various versions of the test in each wave. In each round, poorly fitting questions were dropped, new ones were added, and some questions were revised until the final list of questions presented here was has been selected.

The X-Culture consulting project (www.X-Culture.org) was used to collect the data. X-Culture is a large-scale international experiential learning project that involves over 3,500 MBA and business students from 100 universities from 40 countries on six continents every semester. The students are placed in global virtual teams of about seven, each student coming from a different country. Working with people from around the globe and dealing with cultural differences, time-zone dispersion, and global communication challenges, the teams complete a consulting project for a multi-national company.

This provides a context that is very similar to the real workplace, particularly with respect to the GVT environment. First, the cross-cultural international settings were very real. The study participants worked in international virtual teams, each composed of about seven people with 5.2 different countries represented on each team (sometimes two team members were from the same country while the rest of the team member each came from a different country). The geographic and time-zone dispersion, cultural and language differences were real. 183 international teams took part in the study. Finding a large number of international work teams like this is simply impossible in the workplace. At most, an organization would have a few dozen international teams, and usually fewer than a dozen, whose performance could be observed to validate a cultural intelligence instrument with respect to behavior and performance in cross-cultural settings.

Second, the study task and environment were designed to resemble the corporate world as closely as possible. The team member interacted daily during 8-9 weeks, which is a typical project length in the corporate world.

Once the students enrolled in the course that participated in the project, they were required to take part in the project. The team assignment was random and students had no choice over the countries represented on their teams. This is similar to how it works in the corporate world: accepting a job offer is voluntary, but once in a job, one has little choice as to what projects to work on and with whom.

The project involved development of a solution to real-life business challenges presented by real-life companies. The task involved market research, market entry plan development, and product design. The project was supervised by instructors with rich business consulting experience and managed as a regular business consulting project.

Just like in the corporate world, the teams were given significant autonomy in terms of the extent and type of communication methods. However, all participants were introduced to and were encouraged to use free collaboration tools, such as email, voice and video conferencing tools (e.g., Skype), document and collaboration platforms (e.g., Google Docs and Dropbox), and social media (e.g., Facebook and Google +), similar to what is commonly used in a corporate environment.

The stakes were very high and the project was effectively a temporary employment for the client organization. First, the project accounted for 20 to 50% of the course grade. A failure on the project usually meant a failure in the course, with all resulting negative effects on future career prospects. The members of the best teams were invited project participants symposiums held once a year. Most attended received travel stipends. Additionally, organizations offered post-market commission, as well as prospects of internships and job offers. So from every angle, the project settings and work design were not different from those in organizations and the threat that the findings of the present study would not generalize to the corporate employee population is extremely small.

Most important, the advantages of the large international sample from the X-Culture project probably greatly outweighed the possible disadvantages due to marginally younger sample demographics. For example, the sample was comprised of participants from 31 countries by the current country of residence and 81 countries by the country of origin. One of the big concerns with instruments designed for international audiences is the instrument’s generalizability across cultures. Using a large international X-Culture sample allowed me to confirm that the QO-CQ psychometric properties meet the standards across various national subsamples.

Internal Reliability and Factor Structure

Presented in this working paper are just the basic assessments of the QO-CQ factor structure and psychometric properties. So far, the results are excellent, particularly given the brevity of the instrument (only 3-4 items per dimension).

The final version of QO-CQ was validated using a sample of 3,502 MBA and business students who took part in X-Culture in 2016. Of those, 701 individuals never travelled internationally and took the QO-CQ_D (domestic) version of the instrument, while the remaining 2,801 took the QO-CQ_I“international” version.

The internal reliabilities (Cronbach’s alpha) were very similar in both version all around or above the commonly accepted (although often criticized) cutoff point of 0.70.

Dimension	QO-CQ_I	QO-CQ_D
Preparation	0.71	0.73
Seeking	0.82	0.80
Adjustment	0.77	0.75
Learning	0.72	0.74
Culture Shock	0.69	0.68

The exploratory factor analysis strongly supported the hypothesized 5-factor model. With one minor exception, all items loaded heavily on their respective factors, and none loaded on multiple factors. Since the results are virtually identical for the “domestic” and “international” versions of the instrument, provided here are the results for the “international” version only.

——————————————————————————-

Variable | Factor1 Factor2 Factor3 Factor4 Factor5 | Uniqueness

————-+————————————————–+————–

p1 | 0.0912 0.4300* 0.2558 0.3357 -0.0022 | 0.6286

p2 | 0.1974 0.7058* 0.1532 0.1295 -0.0405 | 0.4210

p3 | 0.3435 0.6924* 0.1173 0.1022 -0.0303 | 0.3775

s1 | 0.4901* 0.3694 0.2027 0.2950 -0.0168 | 0.4950

s2 | 0.6026* 0.2013 0.1388 0.0170 0.1122 | 0.5642

s3 | 0.7626* 0.2035 0.1313 0.1152 -0.0366 | 0.3451

s4 | 0.5809* 0.2077 0.0944 0.2011 -0.0711 | 0.5650

a1 | 0.4112 0.2328 0.3639* 0.1097 0.0455 | 0.6302

a2 | 0.2469 0.2067 0.5606* 0.1243 0.0295 | 0.5657

a3 | 0.2546 0.2695 0.4757* 0.1713 0.0354 | 0.6057

a4 | 0.1287 0.2165 0.5164* 0.2842 -0.1646 | 0.5620

l1 | 0.2940 0.1030 0.4040 0.3283* -0.1307 | 0.6149

l2 | 0.3940 0.2335 0.2699 0.5239* -0.0933 | 0.4342

l3 | 0.2040 0.3380 0.2275 0.4928* 0.0371 | 0.5481

sh1 | 0.0208 -0.1275 -0.0585 -0.0552 0.5780* | 0.6427

sh2 | -0.0454 0.0211 -0.0024 -0.0575 0.6097* | 0.6225

sk3 | 0.0076 0.0248 -0.0334 0.1426 0.4088* | 0.8107

——————————————————————————-

As can be seen, these tests provide the support for the basic psychometric quality standards of the test. As I am continuing testing the instrument, additional results will be available soon, by the time of the paper presentation at the AIB-SE conference, should it be accepted.

Future Research

Presented here were just the initial results on the psychometric properties of QO-CQ. Additional tests will include:

Confirmatory factor analysis to confirm the dimensionality of the instrument.
Validation against other CQ measures. The same study participants have also completed the Ang et al’s (2007) instrument, as well as the recently published BCIQ instrument (Alon et al., 2016).
Criterion validity against a series of theoretically relevant criteria, such as the extent of the prior international experience, multi-cultural identity, and the like. The project participants completed a number of other surveys that detail their demographics and cultural background, and if QO-CQ is indeed measuring CQ, many of these personal characteristics and constructs are expected to correlate with QO-CQ.
Predictive validity. The project also involved an extensive measurement of performance in cross-cultural situations, including peer evaluations, conflicts in cross-cultural settings, leadership in GVTs, and many more. Ultimately, the CQ instruments are supposed to predict one’s ability to effectively function in cross-cultural situations, such as when working in a GVT. Thus, QO-CQ will be subjected to these tests to establish its predictive validity, including as compared to that of other popular instruments such as Ang et al’s CQ measure.

References

Alon, I., Boulanger, M., Meyers, J., Taras, V., Tung, R., & Ralston, D. (2016). The development and validation of the Business Cultural Intelligence Quotient. Cross Cultural & Strategic Management, 23(1), 78-100.

Ang, S., Van Dyne, L., Koh, C., Ng, K. Y., Templer, K. J., Tay, C., & Chandrasekar, N. A. (2007). Cultural intelligence: Its measurement and effects on cultural judgment and decision making, cultural adaptation and task performance. Management and Organization Review, 3(3), 335-371.

Harzing, A.-W. (2006). Response styles in cross-national survey research: A 26-country study. International Journal of Cross Cultural Management, 6(2), 243-264. doi:

Johnson, T., Kulesa, P., Cho, Y. I., & Shavitt, S. (2005). The relationship between culture and response styles: Evidence from 19 countries. Journal of Cross-Cultural Psychology, 36, 264-277. doi: 2061

Smith, P. B. (2004). In search of acquiescent response bias. Journal of Cross-Cultural Psychology, 35(1), 50-61. doi:

Smith, P. B., & Fischer, R. (2008). Acquiescence, extreme response bias and culture: A multilevel analysis. Multilevel analysis of individuals and cultures, 285-314.

Thomas, D. C., Elron, E., Stahl, G., Ekelund, B. Z., Ravlin, E. C., Cerdin, J.-L., . . . Aycan, Z. (2008). Cultural intelligence domain and assessment. International Journal of Cross Cultural Management, 8(2), 123-143.

[1] The CLCI instrument is proprietary; users must receive certification to administer the measure and to interpret the results.

Development of a better way to measure Cultural Intelligence

The Problem