Cognitive Horizons Network

A consortium of the world’s leading universities committed to working with IBM to accelerate development of important technologies needed to fulfill the promise of cognitive computing.

Explore the partnerships

Publications

About the Cognitive Horizons Network

IBM Researchers, world-class faculty, and top graduate students work together on a series of advanced research projects and experiments designed to accelerate the application of artificial intelligence, machine learning, natural language processing and related technologies. Projects are designed to apply the technologies to some of the world’s most enduring challenges, ranging from disease and the environment to transportation and education. The Network addresses the entire cognitive computing stack from analyzing the unstructured and structured data required to train these systems to building the new computing infrastructures needed to optimize the new data-intensive workloads of a truly digital world.

Massachusetts Institute of Technology

Video comprehension

Explore

Massachusetts Institute of Technology

Laboratory for Brain-inspired Multi-Media Machine Comprehension (BM3C)

Goal

Developing machines that can emulate the human ability to understand inputs from multiple video streams and predict potential future events in real-time.

Technologies

Short activities recognition (in 3 seconds videos), visual Q&A, generative models for simulated worlds.

Research team

Josh Tenenbaum, Josh McDermott, and Danny Gutfreund (IBM), Andrei Barbu, Pouya Bashivan, Maddie Cusimano, Yen-Ling Kuo, Jonathan Malmaud, Mathew Monfort, Candace Ross, Guy Ben-Yossef, Sue Felshin, David Mayo, James Traer

Bold text denotes principal investigator

Learn more

Download poster

Press release

@ IBM Research


Rensselaer Polytechnic Institute

Cognitive environments

Explore

Rensselaer Polytechnic Institute

Cognitive and Immersive Systems Lab (CISL)

Goal

Building environments that explore and advance natural, collaborative problem-solving among groups of people and machines, with the goal of improving how people work together to make decisions.

Technologies

Multimodal interaction and dialog, Spatial context, Spherical microphone array, Group dynamics understanding, Situations reasoning and planning, Collaborative decision making, Narrative/Analogy generation, Data visualization and sonification, Multimodal storytelling.

Research team

Selmer Bringsjord (RPI), Jonas Braasch (RPI), Hui Su (IBM), Kevin Blissett, Zev Battad, Samuel Chabot, Gyanendra Sharma, Kang Wang, Matt Peveler, Robert (Boning) Dong, Devavrat Jivani, Atriya Sen, Rui Zhao, Wennan Zhu, Robert Rouhani, Ryan Weaver, Anitra Chowdhury, Rahul Divekar, Corey Robinson, Rose Clare Pisacano, Anna Stephenson, Huang Zou, Yueqing (Eve) Dai, Guangyuan Li, Lilit Balagyozyan, Hongyang (Lion) Lin, Shuyue (Chelsea) Zheng, Yihao Zhu, Haoming Li, Lisheng Ren, Ziyi Song

Bold text denotes principal investigator

Learn more

University website

Publications


University of Illinois Urbana-Champaign

Optimized systems

Explore

University of Illinois Urbana-Champaign

Center for Cognitive Computing Systems Research (C3SR)

Goal

Developing the stack of applications, accelerators, software, hardware and networking needed to support larger and more complex cognitive workloads.

Technologies

Cognitive application builder, creative experiential learning advisor, cognitive workload acceleration, new system architectures for cognitive systems

Research team

Wen-mei Hwu (UIUC), Jinjun Xiong (IBM)Mohamed El-Hadedy, Haizi Yu, Carl Pearson, Xinheng Liu, Xiaofan Zhang, Raymond Yeh, Abdul Dakkak, Cheng Li, Nguyen Mac, Ashutosh Khar, David Min, Tarek Sakakini , Hongyu Gong, Xiou Ge, Omer Anjum, Ketan Hemant H Date

Bold text denotes principal investigator

Learn more

University website

Press release


University of Michigan

Conversational technologies

Explore

University of Michigan

Sapphire Project

Goal

Developing technology to allow people to interact more naturally and effectively with computers through text or speech dialogues.

Technologies

Discourse disambiguation, structured disentanglement, NL queries over relational & knowledge bases, emotion detection, dialog action & next utterance prediction, multiparty conversation dynamic management, response generation.

Research team

Satinder Singh (Michigan), David Nahamoo (IBM), Lazaros Polymenakos (IBM), Jonathan Kummerfeld, Mason Hill, Stephanie O'Keefe, Janarthanan Rajendran, Lajanugen Logeswaren, Rui Zhang, Sai Gouravajhala, Falk Pollok, Charlie Welch, Zakaria Aldeneh, Harmanpreet Kaur, Lucy Jiang , Janarthanan Rajendran , Matthew Skach , Ram Kannan, Vignesh Athreya, Fei Li

Bold text denotes principal investigator

Learn more

University website


Université de Montréal

Deep learning

Explore

Université de Montréal

Montreal Institute for Learning Algorithms (MILA)

Goal

Developing next-generation deep learning algorithms and techniques to help computers improve their understanding and interpretation of language, speech, and vision.

Learn more

Deep learning, generative models, adversarial learning, reinforcement learning, natural language processing, dialogue, speech recognition, hardware acceleration of inference with deep models.

Publications

Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation
Proc. AAAI, 2017
I. V. Serban, T. Klinger, G. Tesauro, K. Talamadupula, B. Zhou, Y. Bengio, and A. Courville

A Structured Self-Attentive Sentence Embedding
Proc. International Conference on Learning Representations
I. V. Serban, T. Klinger, G. Tesauro, K. Talamadupula, B. Zhou, Y. Bengio, and A. Courville

Pointing the Unknown Words
Proc. ACL, 2016
Ç. Gülçehre, S. Ahn, R. Nallapati, B. Zhou, and Y. Bengio

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Proc. SIGNLL Conference on Computational Natural Language Learning (CoNLL), 2016
R. Nallapati, B. Zhou, C. dos Santos, Ç. Gülçehre, and B. Xiang

Invariant Representations for Noisy Speech Recognition
NIPS Workshop on End-to-End Speech Recognition, 2016
D. Serdyuk, K. Audhkhasi, P. Brakel, B. Ramabhadran, S. Thomas, and Y. Bengio

Oracle performance for visual captioning
NIPS Workshop on End-to-End Speech Recognition, 2016
L. Yao, N. Ballas, K. Cho, J. R. Smith, and Y. Bengio

Empirical performance upper bounds for image and video captioning
International Conference on Learning Representations (ICLR), 2016
L. Yao, N. Ballas, K. Cho, J. R. Smith, and Y. Bengio

Research team

Yoshua Bengio (Montreal), Brian Kingsbury (IBM), Anirudh Goyal , Jose Sotelo, Philemon Brakel , Sungjin Ahn , Dzmitry Bahdanau , Xavier Bouthillier , Junyoung Chung , Akram Erraqabi , Kyle Kastner , César Laurent , Ahmed Touati , Ying Zhang, Aristide Baratin , Tom Bosc , Tong Che, Laurent Dinh , Taesup Kim, Samuel Laférière , Dmitriy Serdyuk, Olexa Bilaniuk, Matthieu Courbariaux, Rosemary Ke

Bold text denotes principal investigator

Learn more

Blog

University website


University of Maryland at Baltimore County

Cognitive Cybersecurity

Explore

University of Maryland at Baltimore County

Accelerated Cognitive Cybersecurity Lab (ACCL)

Goal

Applying cognitive computing to cybersecurity via analytics and machine learning, while also exploring specialized computer power optimized for these new intensive computing workloads.

Learn more

Advanced Named Entity Recognition (NER), Relation Extraction (RE), Event Extraction (EE) and text summarization tools to support effective text analysis for cybersecurity

Research team

Anupam Joshi (UMBC), JR Rao (IBM), Sudip Mittal, Ankur Padia, Arpita Roy, Sandeep Narayanan, Priyanka Ranade, Ashwinikumar Ganesan, Askhay Peshavve, Pooja Parmeswarappa, Kaushik Veluswamy

Bold text denotes principal investigator

Learn more

University website

Press release


Rensselaer Polytechnic Institute

Health and wellness

Explore

Rensselaer Polytechnic Institute

Health Empowerment by Analytics, Learning, and Semantics (HEALS)

Goal

Advancing the understanding of chronic condition prevention in the pre-disease stage by focusing on the identification of actionable personal-risk determinants via a coupling of the knowledge of lifestyles risks and data-driven health causal and correlational factors.

Learn more

Multi-model semantics graph integration, NLP of health forum, Personal health data wrangling, Automatic behavior phenotypes identification and analysis, Watson powered conversation agents, Engagement Apps.

Research team

Jim Hendler (RPI), Ching-Hua Chen (IBM), Henry Chang (IBM), Yu Chen, Sabbir Rashid, Jonathan Harris, Dylan Elliott, Oshani Seneviratne, Ryan Sherman

Bold text denotes principal investigator

Learn more

Press release


Publications

CVPR
Massachusetts Institute of Technology

Moments: A Large-Scale Database for Video Understanding

Mathew Monfort , Tom Yan, Bolei Zhou, Dan Gutfreund, Quanfu Fan, Carl Vondrick, Aude Oliva

View poster

XRDS
Rensselaer Polytechnic Institute

The cognitive and immersive situations room

Hui Su

Read paper

IEEE Conference on Computer Vision and Pattern Recognition, 2017
Rensselaer Polytechnic Institute

Simultaneous Facial Landmark Detection, Pose and Deformation Estimation under Facial Occlusion

Yue Wu, Guo Chao, and Qiang Ji

Read paper

5th Annual Conference on Advances in Cognitive Systems (ACS), Troy, NY
Rensselaer Polytechnic Institute

Establishing Social Presence with a Voice-Only Chatbot

Rahul Divekar & Mei Si

IEEE Conference on Computer Vision and Pattern Recognition, 2017
Rensselaer Polytechnic Institute

Simultaneous Facial Landmark Detection, Pose and Deformation Estimation under Facial Occlusion

Yue Wu, Guo Chao, and Qiang Ji

In Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci), London, UK
Rensselaer Polytechnic Institute

A Data-Driven Approach for Making Analogies

Craig Carlson & Mei Si

In Proceedings of the IJCAI-17
Rensselaer Polytechnic Institute

Improving Group Decision-Making by Artificial Intelligence

Lirong Xia

In Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-17)
Rensselaer Polytechnic Institute

Optimal Decision Making with CP-nets and PCP-nets

Sujoy Sikdar, Sibel Adali, and Lirong Xia

31st AAAI Conference on Artificial Intelligence (AAAI-17)
Rensselaer Polytechnic Institute

Mechanism Design for Multi-Type Housing Markets

Sujoy Sikdar, Sibel Adali, and Lirong Xia

The Journal of the Acoustical Society of America 141, 3512 (2017)
Rensselaer Polytechnic Institute

High-density data sonification of stock market information in an immersive virtual environment

Samuel Chabot and Jonas Braasch

Read paper

The Journal of the Acoustical Society of America 141, 3465 (2017)
Rensselaer Polytechnic Institute

Using binaural and spherical microphone arrays to assess the quality of synthetic spatial sound fields

Jonas Braasch, Nikhil Deshpande, Jonathan Mathews, and Samuel Chabot

Read paper

The Journal of the Acoustical Society of America 141, 3726 (2017)
Rensselaer Polytechnic Institute

Real-time localization and tracking of musical instruments with spherical microphone arrays

Jonathan Mathews and Jonas Braasch

Read paper

The Journal of the Acoustical Society of America 141, 3896 (2017)
Rensselaer Polytechnic Institute

Using visual cues to perceptually extract sonified data in collaborative, immersive big-data display systems

Wendy Lee, Samuel Chabot, and Jonas Braasch

Read paper

Intelligent Virtual Agent 2016, Los Angeles
Rensselaer Polytechnic Institute

Using Multiple Storylines For Presenting Large Information Networks

Zev Battad, Mei Si

International Conference on Interactive Digital Storytelling 2016, Los Angeles
Rensselaer Polytechnic Institute

Presenting Large Data with Anchor Points

Mei Si, Zev Battad and Craig Carlson

ACM Interactive Surfaces and Spaces 2016
Rensselaer Polytechnic Institute

Interactions in a Human-Scale Immersive Environment: the CRAIVE-Lab

Gyanendra Sharma, Jonas Braasch, Rich Radke

Immersive Education Summit 2016, Padova, Italy
Rensselaer Polytechnic Institute

Toward the Cognitive Classroom: Mathematical Physics

Atriya Sen, Selmer Bringsjord, Rich Radke, Matt Peveler

Soc. Am. 140, 3450 (2016)
Rensselaer Polytechnic Institute

A method for real-time, multiple-source localization using spherical harmonic decomposition

Jonathan Mathews and Jonas Braasch, J. Acoust

Read paper

Soc. Am. 140, 3292 (2016)
Rensselaer Polytechnic Institute

Spatialized sound reproduction for telematic music performances in an immersive virtual environment

Samuel Chabot, J. Acoust

Read paper

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016
Rensselaer Polytechnic Institute

Facial Expression Intensity Estimation Using Ordinal Information

Rui Zhao, Quan Gan, Shangfei Wang and Qiang Ji

Read paper

ACM Symposium on Eye Tracking Research & Applications, 2016
Rensselaer Polytechnic Institute

Deep eye fixation map learning for calibration-free eye gaze tracking

Kang Wang, Shen Wang and Qiang Ji

Read paper

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016
Rensselaer Polytechnic Institute

Constrained Joint Cascade Regression Framework for Simultaneous Facial Action Unit Recognition and Facial Landmark Detection

Yue Wu and Qiang Ji

Read paper

European Conference on Computer Vision Workshop (ECCV-W), 2016
Rensselaer Polytechnic Institute

Shape Augmented Regression for 3D Face Alignment

Chao Gou, Yue Wu, Fei-Yue Wang, and Qiang Ji

Intelligent Virtual Agent 2016, Los Angeles
Rensselaer Polytechnic Institute

Using Multiple Storylines For Presenting Large Information Networks

Zev Battad, Mei Si

Read paper

International Conference on Interactive Digital Storytelling 2016, Los Angeles
Rensselaer Polytechnic Institute

Presenting Large Data with Anchor Points

Mei Si, Zev Battad and Craig Carlson

International Conference on Pattern Recognition (ICPR), 2016
Rensselaer Polytechnic Institute

Real Time Eye Gaze Tracking with Kinect

Kang Wang and Qiang Ji

Thirty-first Annual Conference on Neural Information Processing Systems (NIPS)
University of Illinois Urbana-Champaign

Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts

R. Yeh, J. Xiong, M. Doh, W. Hwu, A. Schwing

Thirty-first Annual Conference on Neural Information Processing Systems (NIPS)
University of Illinois Urbana-Champaign

Probabilistic Rule Realization and Selection

H. Yu, T. Li, L. Varshney

IEEE international conference on Rebooting Computing (ICRC)
University of Illinois Urbana-Champaign

Rebooting the Data Access Hierarchy of Computing Systems

W. Hwu, I. Hajj, S. Gonzalo, C. Pearson, N. Kim, D. Chen, J. Xiong, Z. Sura

IEEE High Performance Extreme Computing Conference (HPEC), Graph Challenge
University of Illinois Urbana-Champaign

Collaborative (CPU + GPU) Algorithms for Triangle Counting and Truss Decomposition on the Minsky Architecture

K. Date, K. Feng, R. Nagi, J. Xiong, N. Kim, W. Hwu

Conference on Empirical Methods in Natural Language Processing (EMNLP)
University of Illinois Urbana-Champaign

A Joint Model for Semantic Sequences: Frames, Entities, Sentiments

S. Chaturvedi, H. Peng, and D. Roth

27th International Conference on Field-Programmable Logic and Applications
University of Illinois Urbana-Champaign

High-Performance Video Content Recognition with Long-term Recurrent Convolutional Network for FPGA

X. Zhang et al.

SIGKDD
University of Illinois Urbana-Champaign,

A half-day Workshop on Creative Assistants - Augmenting Creativity using Machine Learning

A. Sankaran, K. Sankaranarayanan, P. Agrawal, L. R. Varshney, and K. R. Varshney etc.

SIGKDD
University of Illinois Urbana-Champaign

A half-day Workshop on Data-Driven Discovery

Y. Ding, J. Evans, S. Spangler, L. R. Varshney, and D. Wang

KDD Workshop on Data-Driven Discover
University of Illinois Urbana-Champaign

A Coupon-Collector Model of Machine-Aided Discovery

A. Vempaty, L. R. Varshney, and P. K. Varshney

Third IEEE Smart World Congress (IEEE SWC)
University of Illinois Urbana-Champaign

Effective Object Detection from Traffic Camera Videos

H. Shi, Z. Liu, Y. Fan, X. Wang, T. Huang

Conference on Natural Language Learning (CoNLL)
University of Illinois Urbana-Champaign

A Joint Model for Semantic Sequences: Frames, Entities, Sentiments

H. Peng, S. Chaturvedi and D. Roth

ACL
University of Illinois Urbana-Champaign

MORSE: Semantic-ally Drive-n MORpheme SEgment-er

T. Sakakini, S. Bhat, P. Viswanath

Keynote for the International Symposium on Low Power Electronics and Design (ISLPED)
University of Illinois Urbana-Champaign

Architecture and Software for Emerging Low-Power Systems

W. Hwu, J. Xiong, N. Kim, D. Chen, I. Hajj, A. Dakkak, L. Chang, S. Garcia and C. Pearson

CVPR
University of Illinois Urbana-Champaign

Semantic Image Impainting with Deep Generative Models

R. Yeh, C. Chen, T. Lim, A. Schwing, M. Hasegawa-Johnson, M. Do

IEEE Micro Magazine
University of Illinois Urbana-Champaign

Heterogeneous Computing Meets Near-memory Acceleration and High-level Synthesis in the Post-Moore Era

N. Kim, D. Chen, J. Xiong, and W. Hwu

Invited Talk, ShanghaiTech Workshop on Emerging Devices, Circuits and Systems
University of Illinois Urbana-Champaign

Cognitive Computing on Heterogeneous Hardware Systems for the AI Revolution

Deming Chen

Plenary talk for the International Association for World Englishes (IAWE)
University of Illinois Urbana-Champaign

Crowdsensing, Crowdsourcing and Creativity

L. Varshney

10th International Conference on Educational Data Mining
University of Illinois Urbana-Champaign

Learner Affect Through the Looking Glass: characterization and Detection of Confusion in Online Courses

Z. Zeng, S. Chaturved, S. Bhat

International Symposium on Information Theory
University of Illinois Urbana-Champaign

Towards Optimal Quantization of Neural Networks

A. Chatterjee, L. Varshney

Poster for the 8th International Conference on Computational Creativity (ICCC)
University of Illinois Urbana-Champaign

Creating Experiential Learning Activities

X. Ge, J. Xiong, L. Varshney

Presented at NetSci: International School and Conference on Network Science
University of Illinois Urbana-Champaign

The Eurekometric Connectome: Discovering Unexplored Areas of Neuroscience Research

M. Jere, R. K. Raman, and L. R. Varshney

arXiv: 1705.03487
University of Illinois Urbana-Champaign

Sukiyaki in French style: A novel system for transformation of dietary patterns

M. Kazama, M. Sugimoto, C. Hosokawa, K. Matsushima, L. R. Varshney, Y. Ishikawa

IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM)
University of Illinois Urbana-Champaign

Efficient GPGPU Computing with Cross-Core Resource Sharing and Core Reconfiguration

Dhar and D. Chen

5th International Conference on Learning Representations (ICLR)
University of Illinois Urbana-Champaign

Towards Deep Interpretability (MUS-ROVER II): Learning Hierarchical Representations of Tonal Music

H. Yu and L. R. Varshney

UCI Department of Computer Science Distinguished Lecture Series
University of Illinois Urbana-Champaign

Innovative Applications and Technology Pivots – A Perfect Storm in Computing

W. W. Hwu, J. Xiong, A. Dakkak, and C. Pearson

31st AAAI Conference
University of Illinois Urbana-Champaign

Geometry of Compositionality

H. Gong, S. Bhat, P. Viswanath

2017 Information Theory and its Applications Workshop (ITA)
University of Illinois Urbana-Champaign

Illum Information

R. K. Raman, H. Yu, and L. R. Varshney

INFORMS Annual Meeting
University of Illinois Urbana-Champaign

Multiattribute Preference Models for Computational Creativity,

D. Bhattacharjya and L. R. Varshney

IEEE Big Data, 2016
University of Maryland at Baltimore County

Inferring Relations in Knowledge Graphs with Tensor Decompositions

Padia, Kalpakis, Finin

Int. Semantic Web Conference
University of Maryland at Baltimore County

Cleaning Noisy Knowledge Graphs

Padia

IEEE Transactions on Affective Computing
University of Michigan

ISLA: Temporal Segmentation and Labeling for Audio-Visual Emotion Recognition

Yelin Kim and Emily Provost

Interspeech
University of Michigan

Capturing Long-term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition

Soheil Khorram, Zakaria Aldeneh, Dimitrios Dimitriadis, Melvin McInnis, Emily Mower Provost

Interspeech
University of Michigan

Progressive Neural Networks for Transfer Learning in Emotion Recognition

John Gideon, Soheil Khorram, Zakaria Aldeneh, Dimitrios Dimitriadis, Emily Mower Provost

Interspeech
University of Michigan

Discretized Continuous Speech Emotion Recognition with Multi-Task Deep Recurrent Neural Network

Duc Le, Zakaria Aldeneh, Emily Mower Provost

Read paper

Col.ing
University of Michigan

Targeted Sentiment to Understand Student Comments

Charles Welch and Rada Mihalcea

Read paper

ICLR Conference Track
University of Michigan

Reasoning in Dialogue Management

Xiaoxiao Guo, Tim Klinger, Clemens Rosenbaum, Joseph P. Bigus, Murray Campbell, Ban Kawas, Kartik Talamadupula, Gerald Tesauro, and Satinder Singh

Read paper

ACL
University of Michigan

Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection

Youxuan Jiang, Jonathan K. Kummerfeld and Walter S. Lasecki

Read paper

AAAI
Université de Montréal

Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation

I. V. Serban, T. Klinger, G. Tesauro, K. Talamadupula, B. Zhou, Y. Bengio, and A. Courville

Read paper

International Conference on Learning Representations
Université de Montréal

A Structured Self-Attentive Sentence Embedding

Z. Lin, M. Feng, C. N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio

Read paper

ACL
Université de Montréal

Pointing the Unknown Words

Ç. Gülçehre, S. Ahn, R. Nallapati, B. Zhou, and Y. Bengio

Read paper

SIGNLL Conference on Computational Natural Language Learning (CoNLL)
Université de Montréal

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

R. Nallapati, B. Zhou, C. dos Santos, Ç. Gülçehre, and B. Xiang

Read paper

NIPS Workshop on End-to-End Speech Recognition
Université de Montréal

Invariant Representations for Noisy Speech Recognition

D. Serdyuk, K. Audhkhasi, P. Brakel, B. Ramabhadran, S. Thomas, and Y. Bengio

Read paper

Proc. British Machine Vision Conference (BMVC)
Université de Montréal

Oracle performance for visual captioning

L. Yao, N. Ballas, K. Cho, J. R. Smith, and Y. Bengio

Read paper

International Conference on Learning Representations (ICLR)
Université de Montréal

Empirical performance upper bounds for image and video captioning

L. Yao, N. Ballas, K. Cho, J. R. Smith, and Y. Bengio

Learn more about AI research @ IBM