Vision & Language 

Model the relationship between images and text through captioning and multimodal dialog systems

• Self-critical Sequence Training for Image Captioning (CVPR 2017 Oral, top entry in MS COCO captioning 2017)

• Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts (NIPS2017)      blogpost

• 'Dialog-based Interactive Image Retrieval           Demo video

• Improved Image Captioning with Adversarial Semantic Alignment

• Crafting Adversarial Examples for Neural Image Captioning (ACL 2018)

Multimodal Video Analysis

Extract  and interpret salient moments in video based on audio/visual analytics 

• Automatic Curation of Sports Highlights using Multimodal Excitement Features

        Demo Videos   Golf Masters  Wimbledon   US Open

        Press coverage  NY Times  Fortune  NBC News  ZNET CNET

• Harnessing A.I. for Augmenting Creativity: Movie Trailer Creation (ACM MM 2017 Best Brave New Ideas Award) 

• Moments in Time Dataset blogpost

• Prior-Less Multi-Face Tracking in Unconstrained Videos (CVPR 2018)

Efficient Visual Analysis

Computer Vision operated on neuromorphic CMOS integrated circuits. Efficient architectures for large scale learning

• A Low Power, Fully Event-Based Gesture Recognition System (CVPR 2017)
• Application of TrueNorth (Science 2014)
• Blockdrop: Dynamic Inference Paths in Residual Networks (CVPR 2018)
• Training ImageNet 22K in 50 minutes   blogpost
• AI Everywhere with IBM Watson and Apple Core ML

Object Detection

Fast and accurate object localization

• Geometry-aware Traffic Flow Analysis by Detection and Tracking (Oral at AI city challenge @CVPR 2018)
• MS-CNN: A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection (top entry in Kitti 2016)

Applications

The power of computer vision to produce real world solutions

• Watson cloud based APIs
  • Watson Visual Recognition
  • IBM Watson & Apple Core ML

• Medical Imaging
  • Melanoma recognition in dermoscopy images
    • Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images
    • Segmentation of both Diseased and Healthy Skin from Clinical Photographs in a Primary Care Setting
    • Skin Lesion Analysis Toward Melanoma Detection: A Challenge at ISBI 2017
    • Deep Learning Ensembles for Melanoma Recognition in Dermoscopy Images 
  • Cognitive computing and radiology

• Safety and Security

• • IBM Smart Surveillance System

• Assitive technologies. Computer vision systems to help the visually impaired explore the world

• Sports
  • Cognitive Highlights