What is Open Source Intelligence?
Open Source Intelligence, often abbreviated as OSINT, is data and information that is collected legally from open and publicly available resources. Obtaining the information doesn’t require any type of clandestine effort and it is retrieved in a manner that is legal and meets copyright requirements. The raw data is collected and then analyzed to help understand something more clearly, support decision making, assess public perceptions, forecast changes and much more.
Who Engages in Open Source Intelligence gathering and analysis?
Technically, anyone who knows how to use the tools and techniques to access the information is said to have used the process. However, the process is used formally by the United States intelligence community, the military, law enforcement, IT security professionals, private businesses and private investigators.
The Central Intelligence Agency (CIA) created the Open Source Center, which builds on the established expertise of the CIA’s Foreign Broadcast Information Service (FBIS). The Open Source Center is responsible for collecting, producing and promoting OSINT.
What types of data and information are considered to be Open Source?
Open source data and information is available in a variety of place, most of which are accessible via the internet. Examples include:
- Public records databases
- Government reports, documents and websites
- The internet
- Mass media (e.g. newspapers, TV, radio, magazines and websites)
- Social networks and social media sites
- Maps and commercial imagery
- Photos and videos
- The dark web
List of Open Source Intelligence Tools
The tools and techniques used in Open Source Intelligence searching go much further than a simple Google search. Following is a list of helpful, time-saving tools.
Email Breach Lookup
https://haveibeenpwned.com/ – This site, called Have I Been Pawned?, allows you to find out if a particular email address was affected by one of the many data breaches that have occurred over the years. Just enter an email address and it will tell you which breaches the email was affected by. In addition, it gives a brief overview of the breach.
Fact Checking Websites
Hoaxy – Hoaxy is an open source search tool that visualizes the spread of articles online
Media Bugs – A media bug is something in the media (e.g. newspaper, magazine, radio, etc.) that is wrong and is correctable. Mediabugs.org allows anyone to report a bug and they will work with the media outlet to get it corrected. The site can be used in identifying fake or incorrect news, and to also look for corrected versions.
PolitiFact – Politifact is focused on fact-checking journalism. Politifact uses a helpful rating scale to rate statement made by journalists, political figures and others. The scale runs from True to Full Flop and is a helpful way to know who is telling the truth and who is lying. Read through some of the statements Politifact has investigated and you’ll quickly realize that a lot of what we hear from politicians just isn’t true. The company won a Pulitzer Prize for its work.
SciCheck – SciCheck is a feature of FactCheck.org that focuses on evaluating false and misleading scientific claims made to influence public policy. For example, when a political figure says that “global warming isn’t real”, they are making a false claim to sway public opinion. SciCheck evaluates such claims and explains whether or not they are true.
Snopes – Snopes.com is one of the oldest, best and most well-known fact checking sites. The site researches everything from urban legends, rumors, myths, questionable photos and videos, articles and claims made by public figures.
Verification Junkie – Verification Junkie is a collection of tools designed to help verify and fact check information and assess the validity of eyewitness reports. This site has a collection of very cool tools that are worth the time to dig into.
Hacking and Threat Assessment
Norse – Norse maintains the world’s largest dedicated threat intelligence network. With over eight million sensors that emulate over six thousand applications. Their network gathers data on who the attackers are and what they’re after. Their home page shows a live attack maps with real-time information on attacks.
The following tools go beyond the typical Image Search option that is offered by most major search engines.
Current Location – Current Location is a neat map search tool that allows you to select a position on the map any where in the world and then view photos that were taken within a certain radius of your location. It checks photos posted to Instagram, Flickr and 500px.
Image Identification Project – The Wolfman Image Identification Project is a search tool that uses advanced algorithms to identify images. Simply drag and drop, or upload, an image into the search form and it will attempt to tell you what it is.
TinEye – TinEye is a reverse image search that allows you to search by image and find out where that image is being used online. You ca upload an image, drag and drop to the search box, or search using a URL for an image. The index has over 30 billion images.
Search Engines / Search Tools
Google Correlate – Google Correlate can help you identify search patterns that correlate with real world trends. It can be used to identify search patterns that are similar to one another. For example, you could use the tool to find out if there is a link between new legislation and gun sales (at least in terms of the search volume related to the two topics).
Million Short – Million Short has an interesting approach to searching for information online. The interface is generally similar to a typical search engine. However, you can sort and filter the results in a number of ways. Examples include: by popularity, eCommerce, live chat, date, location, and others. Also, it automatically pulls out the top sites that usually occupy the top spots in any search result (e.g. Amazon.com, eBay, YouTube, etc.)
TalkWalkerAlerts – TalkWalkerAlerts is a great alternative to Google Alerts because in addition to monitoring the web for certain keywords, it monitors social media, blogs and forums as well. This can be helpful if you need to find mentions of someone’s name or business across the web.
Earth Cam – Earth Cam is the gold standard when it comes to aggregating the worlds public live streaming web cams and surveillance cameras in a a way that makes sense.
Insecam – Insecam aggregate access to hundreds of surveillance cameras and public web cameras around the world. The cams are not “hacked” and do not include cams attached to computers, laptops, tablets, etc. The site has cams organized by a variety of categories, but it is tedious to find exactly the area you’re looking for.
CheckUserNames – CheckUserNames is an handy tool that will check the availability of user names on more than 500 social networks. Their new site is at https://knowem.com/, which adds additional services to the mix, including searching the USPTO trademark database.
Namech_k – This search engine allows you to search domain names to see what is available, and then go through the process of registering it, or making an offer. In addition, it will check to see if a username is available on hundreds of social media networking sites. You can even have them signup for you.
Visual / Clustering Search Engines
These search engines take searching a step further by sorting and organizing results. This helps users narrow down particular areas to focus on. And, some allow users to use visualization tools to analyze data and information in different ways.
Carrot2.org – Carrot2 offers a unique approach to searching as it organizes search results into topics, and visually. The categorization of topics is done in three ways. One as a series of folder by topic. Second, it displays the topics in a color-coded circle diagram (very cool). The size of the individual pieces indicate the number of results within each topic. The third method is called “Foamtree”, which arranges the topics into a diagram by topic.
Yippy – Yippy searches multiple sources and provides a set of results, similar to any search engine. However, it also groups the results by topic for easier sorting and viewing. It is not as visual as Carrot2, but still a helpful method of culling down the results.