This lesson is in the early stages of development (Alpha version)

Harvesting Twitter Data with twarc: Setup

Data

The data we will be using in the workshop is to practice working with twitter data. Please download the data files hashtag_gasprices.jsonl and dehydratedCapitolRiotTweets.txt and keep it in an accessible place (such as your desktop). We used twarc to search for all mentions of the hashtag #gasprices and created this . The Capitol Riot tweets is a dataset from Kaggle.

Google Drive may change the file extensions when you download these. Make sure you that hashtag_gasprices and one_tweet are .jsonl, NOT .jsn. capitol_riots.txt really is .txt.
You can force this by right clicking on the page link, choose save as, then choose “all file types” and remove the “.txt” ending and keep the “.jsonl” choose the appropriate filename extension.

Additional Utilities

We will also be using an extended set of utilities that will help explore the data collected through twarc.
twarc utilities download link

Make sure you download and unzip this folder onto your desktop. Later in the workshop, we will upload this set of tools to our JupyterHub.

Getting the Necessary Privileges

Generally, we would call this ‘getting your API key’, but like many API’s, the process of getting access to Twitter’s has become a complicated process. These instructions should create a new developer account with Essential Access privileges.

To get started, you will need:

Getting developer access requires you to validate yourself as a human, so the form is particular about having a valid phone number and email address added to your Twitter account. This is so Twitter can authenticate a user.

Once you are logged on to Twitter, sign into
the Developer Portal

Fill out the application questions as listed here:

Essential Access

Make sure this page is titled ‘Just a few questions to get you Essential Access’.
We are not applying for elevated or academic researcher access.

Making your Project and App

Right after verifying your email, you will be taken to your Developer Portal Dashboard and prompted to create a new Project. Every instance of Twarc is a Twitter App and Twitter Apps live inside of Projects. After selecting ‘Exploring the API’ as your use case, you will be given “Essential Access”. Essential Access allows you to have one project with one App inside of it.

Fill out the four questions to create your first Twitter App. It will ask you for the project name, use case, and description. The project name is your App. App names must be unique across the entire platform, so if you get an error, choose a more unique name. These can be edited and amended later if you choose to change something in the App.

Acquiring Keys and Tokens

After you have completed setting up your project, you will see the App details below. You can edit the name and app icon if you’d like to personalize it.

Under the project name, you will see Settings and Keys and Tokens tab. You will need to navigate to Keys and Tokens to generate your 5 keys and tokens. You will need to click the generate button to get each of your keys and tokens. They are not automatically made for you so you must generate them the first time.

If you forget to save, or lose your tokens, you can click regenerate to regenerate your keys and tokens, and a new set is made for you. However, if you do regenerate any keys and tokens, you’ll need to reenter them when you try to access the API again.

Necessary Keys and Tokens

SAVE these keys in an accessible place like a password manager or in a document. You will not see these keys again, so this is essential. You should have:

  • API Key
  • API Secret
  • Authentication Bearer Token
  • Access Token
  • Access Token Secret

You will have succeeded when you have these 5 keys and tokens

If you are having problems generating the 5 tokens please email us immediately or else you will not be able to follow along with the lessons