Characterizing the evolution of tasks within occupations

AI has started to transform occupations, with the potential to make both labor and machines more efficient and productive. To better understand this transformation, our team at the MIT-IBM Watson AI Lab presented a new study at the AAAI Conference on AI, Ethics, and Society on “Learning Occupational Task-Shares Dynamics for the Future of Work” that shows how to predict changes in the economy’s demand for different tasks.¹ As AI continues to evolve and tasks transform, the next decade is expected to see significant changes in task requirements across occupations.

Our team is studying how task demands of different occupations have changed over the last decade by leveraging a large dataset of online job postings. To understand how the occupations are evolving, we dive deeper into how tasks within them are shifting. The job postings provide information about the prevalence of tasks within each occupation. This is used to generate time-series data of tasks counts for each occupation. This provides a measure of employer demand over time for workers who can perform specific tasks. We additionally incorporate wages and employment shares data from the Bureau of Labor Statistics (BLS), which publishes annual statistics of the average wages and number of employees for 964 occupations. We normalize the task demand time-series data by the share of workers employed in each occupation to derive the unique task-shares dynamics data for each task-occupation pair. The changes in the occupations during that period are further characterized by the evolution of the task-shares within each occupation. This allows us to document trends in occupations and tasks as well as occupational wage terciles (low, medium, high).

To study how AI and related technologies are impacting the labor market at the initial phase of adoption, this study focuses on the Information Technology (IT) task cluster family to look at specific task clusters. Figure A illustrates the evolution of task-shares of selected clusters within the IT task family across high, mid and low (HML) wage occupations. Although task clusters for “SQL Databases and Programming,” “Java,” and “JavaScript & jQuery” have the highest shares in high and mid wage occupations, their demand is steadily declining. In contrast, even though the “Artificial Intelligence” and “Big Data” task clusters have low task-shares in the high wage occupations, their demand increased at a very high rate during 2010-2017. Task clusters like “Scripting Languages” (includes Python) and “Cloud Solutions” gained task-shares in high wage occupations, whereas, most IT task clusters lost task-shares in low wage occupations. This evolution of IT task demands confirms the industry trends towards developing AI-based products and services in the cloud requiring workers to perform AI, Big Data, Scripting Languages, and Cloud Solutions based tasks, while focusing less on traditional software products and services that require workers to perform SQL, Java, and Data Management oriented tasks.

We use the temporal aspects of the task-shares data to learn the dynamics of tasks and occupations, and then quantitatively predict the task-shares for the near future with confidence bounds. We train an autoregressive integrated moving average (ARIMA) model on this data to learn the dynamics of the task-shares of different task cluster families across High-Mid-Low wage occupations over its first 72 months (2010-2015). We then use this ARIMA model to make one-month-ahead predictions of the task-shares. The paper reports less than 5% mean absolute percentage error (MAPE) of predictions. Figure B shows the task-share forecasts (black lines) with 95% confidence intervals (grey areas) to compare against the true task-shares (dotted lines) for a few selected task cluster families across High (red line), Mid (green line), and Low (blue line) wage occupations. The accuracy of the task-share predictions is a clear indicator of the benefit of developing robust and more accurate forecasting models to characterize the evolution of occupations and their tasks. Such predictive capabilities on the labor market might help the workers reskill themselves, corporations retrain their employees, or new graduates to learn the skills to be able to execute the tasks of the future.

This empirical research sheds new light on the transformation of work by characterizing occupations in terms of task-shares dynamics. There are still many open questions remaining in the study. Today, we know the changes AI and related technologies will bring to the labor market are relatively small, but real. To prepare for continued adoption and advancements in technologies, the development of accurate, comprehensive and robust predictive models are crucial to provide guidance to educational institutions, workers, employers, and new graduates on skills and tasks of the future.

Learn more about:

Future of Work: We’re researching how businesses prepare for the post-COVID workplace. We’re working on tools to improve recruitment, skills assessments, and remote worker productivity.