1.1

Human Information, Interpretation, and Influence

The US Department of Defense (DOD) Communities of Interest (COI) were established to encourage multi-agency coordination and collaboration in cross-cutting technology focus areas [5]. Among the 17 technical areas represented in the COI structure is the Human Systems (HS) COI. One Thrust area is the Human Information, Interpretation, and Influence (HI3) activity, which brings together scientists from Air Force, Army, and Navy Research Labs. The vision of the group is to use effective engagement with the dynamic human terrain to make better courses of action and predict human responses to military actions. Three objectives define the approach to that vision: 1) Process and exploit social data to understand adversary intent, 2) Inform Tactics, Techniques, and Procedures (TTPs) to assess information maneuvers and forecast adversarial social response, and 3) Rapidly deploy counter-measures against information threats. These capabilities will be delivered through the deployment of technologies that utilize Natural Language Processing (NLP) tools for extracting meaning in multimedia, and that encompass models and Machine Learning (ML) algorithms to forecast conflict/human intent/crises, that can be integrated with system-level tools and visualizations to allow understanding of computational outputs.

The HI3 Joint researchers are engaged in a number of shared activities exploring new technologies to deliver capabilities to tactical Warfighters in the near future. Research-level tools in development include software applications that rapidly and accurately extract socially-based patterns that signal intentional behaviors that threaten military courses of action. These tools perform a range of capabilities in the spectrum of understanding an unfamiliar society and accurately anticipating future actions, both from adversary groups and from the general populace. Information extraction tools provide features that identify influential actors, relationships in social networks, trending topics of discussion, sentiment expressed toward major entities, and discourse analysis in printed text. Additionally, cultural identification and representation tools provide distinct views of major cultural dispersions in an area of operations. Such tools not only allow rapid information sharing about cultural norms in an unfamiliar area, but provide a geographic indication of how cultural beliefs are arrayed in the population. Further, uncertainty assessments can be provided in two ways by this family of tools. One aspect is to identify deception in social media information flows (e.g., detecting bots and false claims). Another is to test the potential network effects that can be achieved by removing one influential node from a known network. Finally, conflict predictions based on discourse analysis and quantitative social modeling can be used to track the actual and forecasted spread of violence, or threats thereof, in an area.

The evolution of the HI3 Joint Thrust activity builds from the complex nature of OIE that are based on the foundations of Humans sharing Information that is subject to Interpretation by others and is designed to spread Influence to achieve desired outcomes. Within the Human element, we are concerned with individuals, social groups, organizations, analysts, and decision makers. Our foci there is to characterize the persons operating in a particular manner within the IE and any groups/organizations with which they are affiliated and to provide defense actors with the tools to accurately identify threatening messages and campaigns. Within the Information element, we concentrate on computer-mediated messaging that include an array of text, images, videos, geo-locations, and network associations. This is designed to help characterize the potential importance of the human engagement and to ascertain links to known threat networks. The Interpretation element contains NLP and ML approaches to include topic modeling, sentiment/social network/discourse analysis, narrative formulation, pattern of life estimation, relationship linking, and the detection of several messaging strategies to include deception detection, fake news, disinformation, misinformation, individual and coordinated bot campaigns, and distortion. The Influence element is focused on the development of ethical, truthful messaging with relevant platforms and emic perspectives (e.g., views obtained from within the perspective of the subject) and informing TTPs for countering adversary messaging. Three focus areas have been defined within the HI3 activity that are designed to address the above challenges. These are Deep Understanding of Adversarial Information Campaigns, Common Social Analytics Workflow, and Maturing Capabilities through Exercise & Experimentation. We turn next to the research priorities and future needs of these HI3 research thrusts. For each priority area we will describe the relevance of the challenge to multi-domain operations and identify technical issues associated with transitioning research results to system-level applications.

2.1

Deep Understanding of Adversarial Information Campaigns

The ability to extract, characterize, and visualize elements of an adversary’s information campaign is central to the US and Coalition forces’ situational understanding. This includes the ability to accurately identify individuals and groups associated with a specific threat, recognition of strategies employed to reach and influence others, and estimation of strategies that might be employed to counter or disrupt adversary activities within applicable rules of engagement. Within the emerging threat environments that Western democracies can be expected to face in the future this can be an extremely complex undertaking, given the complex nature of the social terrain.

The socio-cultural landscape of noncombatants is increasingly complex, as hybrid conflicts are occurring in urban environments. Indeed, it is anticipated that much of the world’s population will reside in large urban areas, or megacities, in the near future [6]. These areas are studied by defense planners and researchers because of they may serve as fertile ground for the spread of radicalism. A term used to address this type of conflict is the ‘contested urban environment’ (CUE) and is the subject of international study among defense science groups [7]. Wargaming and subsequent analysis from the North Atlantic Treaty Organization (NATO) study group concerned with exploring this emerging threat domain concluded that the essential military capabilities include “collect, communicate, process, fuse, assimilate, and distribute information from many different sources, especially HUMINT, in a responsive manner” (italics added) [7]. A description of the CUE is summarized below:

In these densely populated areas conflict can arise from many different environmental factors and often are enacted within the IE. Recent world events highlight the range of violent conflicts and the interactive/provocative role that mobile communication devices and networked social systems play in those events. The Arab Spring demonstrations that toppled leaders throughout the Arab world [8] were the initial signal that widespread use of social media could be targeted toward serious social purposes [9]. These events were followed by serious study of how the wide spectrum of social media information sources might be used for military Command and Control (C2) and decision making [10]. This awareness has led to military command and control (C2) efforts to take an integrative approach where the strengths of social media content can be combined with other, more traditional, sources [11].

2.2

Common Social Analytics Workflow

Once a set of multiple information exploitation tools is available, assembling them into a cohesive, coordinated, and modular workflow is necessary. This is aimed at reducing analyst cognitive workload with AI/ML tools to support sensemaking. These tools should help to avoid group-think and perception bias, possibly with the use of computational course of action recommenders. The goal here is to reduce the manual search and memory requirements for human analysts. One promising approach is the integration of a specific knowledge base embedded with the relevant threat model in a network graph to aid discovery and reduce the complexity of human queries. Developing workflows will require leveraging tools and algorithms described in the previous section into an ensemble and modular software architecture with common data elements, ontological identifications, and data access/storage capabilities.

A high-level workflow envisioned in the Social Terrain Modeling component of Social Analytics is the interaction between social sensing, social dynamics, and data analytics. The goal is to provide algorithms and methodologies for tactical decision makers to navigate, exploit, and anticipate conflict in a social terrain. As depicted in Figure 1, an Army challenge is the dynamic social sensing of adversary and neutral groups’ opinions and behavioral intent. The desired capabilities that support that challenge involve a range of AI/ML approaches involving social media platforms, context detection algorithms, and techniques for tailoring information and meaning to tactical Warfighters in real-time.

Figure 1.

A depiction of a future Army challenge in a contested urban environment requiring dynamic sensing of adversary and neutral groups’ opinions and behavioral intent.

A notional workflow is represented in Figure 2 for Social Terrain Modeling. This framework takes into account existing technical challenges with social exploitation and representation capabilities. One is the emphasis on analytics that are geared toward narrow aspects of social activity typically exploited through a single information channel (social media platform du jour or open sources, etc.). These pinpoint views give an incomplete picture of the wider group or social terrain and don’t combine to equal the larger societal understanding of a region. Also, exploiting only ‘available’ signals can contribute to biased interpretations of ground truth. To bring defense capabilities to bear on existing information threats, we need to leverage multiple platforms and perspectives and consider operational effects in many situations (e.g., in competition and conflict), which allow a focus on multiple influence effects such as deterrence.

Figure 2.

The Social Terrain Modeling Workflow

2.3

Maturing Capabilities through Exercise & Experimentation

The nascent OIE field brings new challenges and capabilities to traditional Warfighting exercises and experimentation. Demonstrating automated sociocultural information exploitation capabilities at Joint and International exercises is a promising and essential element to hasten doctrinal understanding of and approval for these concepts and technologies. A secondary benefit of exercising these technologies at large wargames is the enhanced interactions between operators and developers to avoid usability and military applicability errors during the design/build/test process. In this way, Best-of-Breed real-time guides to understanding adversary tactics in the sociocultural information domain can be developed and evaluated in larger multi-domain operations simulations and analyses.

The specific capabilities HI3 tools bring to the field of military experimentation and exercise include information assessment toolkits to detect patterns/trends that signal threats and to identify relevant actors, sentiment, and trends impacting military operations. Further, TTPs can be developed, tested, and refined for rapid exploration of adversary signals in open source and social media platforms. Due to the constant change in the IE, TTPs will need to be flexible and adaptive to changing signals in the data. A Digital & Social Media Playbook would serve as an interactive training guide to information assessment and adversary tactics that would inform human analysts on the underlying human motivations and machine strategies (e.g., bots) used by adversaries in the IE.