Key Digested Message

The coronavirus pandemic has highlighted a broad set of challenges that faces the discipline of psychometrics. How should psychometrics specialists (test developers and users) respond to the challenges of the digital age?

While innovation is imperative, we must ensure that fundamental principles of best practice in the science of assessment are maintained and applied to the new methodologies of testing. This article explores key issues in this focal challenge.

Introduction and context

The primary motivation for this article has been my experiences of the recent past years in which I have encountered new technologies in assessment for both talent acquisition and talent management. These include symposia at EAWOP and DOP conferences where new approaches such as game-based assessments have attracted strong opinion and debate about how such techniques should be evaluated. Then, in my practice with Aston Business Assessments (ABA), and as an academic, I have been frequently contacted by start-up companies in the assessment space, who are thinking innovatively, and in sometimes, radically about how data can be used to draw conclusions about psychological attributes. They have challenged my own thinking about the nature of psychological measurement in the digital age. Finally, it is my view that we have a strong ethical obligation to ensure we stay in the wider professional conversation about psychological assessment as the digital economy grows. Recent debates about data privacy, and revelations about the potential role of profiling in shaping public opinion for example, only serve to underline why psychologists need to adapt and to guide practice, attending to issues of assessment effectiveness and ethics.

The full article available on our downloads page, and originally published in the BPS Assessment & Development Matters Journal, explores three substantive issues to illustrate challenges of assessment in the digital age. In this post, I focus on two:

  1. The need to conceptualise assessments flexibly, as modular kits rather than fixed instruments.
  2. Ensuring the relevance of testing for the new kinds of jobs that the digital economy enables.

  1. Designing Assessment for Flexible Application

Digital innovation now means that development time has been substantially reduced. For example, platforms such as MTurk and Prolific provide access to trial data in days, enabling fast design of questionnaires and surveys. Online presentation of materials also means there is no reason for psychometrics to be conceptualised as ‘ready-assembled’. Rather, individual components of tests can be compiled in modules for construction into various forms to meet the needs of different contexts.

For example, the Trait Personality Inventory (ABA, 2011) has been conceptualised as a flexible taxonomy, which has a standard form, but which has also been configured in multiple ways for different clients and assessment needs.

Designing assessments as modular kits will enable psychometric developers to be more responsive to methods of digital service creation. Digital start-ups grow organically, testing products with users, incorporating elements that users want and need. Insistence on the part of psychologists that our assessment technologies are, fixed and non-configurable will prevent our participation in the development of new HR analytic technology.

  1. Jobs in the Digital Economy

The shift to remote working and the rise of the gig economy demonstrates how digital platforms also impact the environments in which psychometrics are applied. For instance, how would we position the role of psychometrics as making an impact on people and businesses that operate in gig-economy? Who is the primary user of psychometric data, the worker (in order to self-assess into gig-economy careers) or the company that engages them (as an indicator of the potential of the worker)? I have yet to encounter a published peer-reviewed study on the criterion effects of individual differences on performance in these kinds of work environments, and so the evidence base from which to work in answering these questions is limited.

Potential routes to move into new work contexts are illustrated; however, by looking at another emergent area of digital entrepreneurship, internet start-ups. In these micro-businesses, success factors include the composition of the team of founders. In this context, it is highly probable that personality composition also makes an impact, given what we know about the benefits of diversity when tasks are complex (Guillaume, Dawson, Otaye-Ebede, Woods & West, 2017). Psychometrics could have a key potential role in determining the potential success and failure of digital micro-business start-ups if methods and tools are developed and applied in ways that enable them to be accessed and understood.

Summary and Conclusion

The highlighted areas in this article are designed to illustrate and reflect how the digital economy might impact on the practice of psychometrics and assessment. The full article also considers the technical challenges faced in evaluating the reliability and validity of new forms of digital assessments.

These reflections represent one way of considering the issues, others may likely take different and alternative views. However, it is clear is that psychometric test developers, test users and psychologists must engage in critical reflection and debate to ensure that our approaches remain innovative and relevant in the digital age.

The author

Professor Stephen Woods is a Professor of Work and Organisational Psychology at the University of Surrey and the founder of Aston Business Assessments (ABA).

References

Aston Business Assessments Ltd (2011). Trait: Manual and User Guide. Birmingham UK; https://www.astonassessments.com
Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality psychology in Europe, 7(1), 7-28.
Guillaume, Y. R., Dawson, J. F., Otaye‐Ebede, L., Woods, S. A., & West, M. A. (2017). Harnessing demographic differences in organizations: What moderates the effects of workplace diversity?. Journal of Organizational Behavior, 2(38), 276-303.
Selenko, E., Berkers, H., Carter, A., Woods, S.A., Otto, K., Urbach, T., & De Witte, H. (2018). On the dynamics of work identity in atypical employment: Setting out a research agenda. European Journal of Work and Organizational Psychology.
Woods, S. A., & Anderson, N. R. (2016). Toward a periodic table of personality: Mapping personality scales between the five-factor model and the circumplex model. The Journal of Applied Psychology, 101(4), 582-604.
Woods, S. A., & West, M. A. (2014). The psychology of work and organizations 2e. Cengage Learning EMEA.