I’m going to use this week’s blog post as an opportunity to try and stay abreast of some data analysis documentation, i.e. what we did on the data side of things. This week saw a few productive screen-share/voice-channel sessions for our group; the two I’ll focus on were 1. Saturday’s data understanding/wrangling session and 2. Sunday’s BeautifulSoup web scraping session.

One of Saturday’s important questions was: what is the scope of our data set? Which cases are in, and which are out? Some answers: we talked (and clicked) through it, and decided to include cases from both the modern and legacy databases, filtering by Issue Area 3 – “First Amendment”. This means there are some extra cases that are First Amendment but not freedom of speech related, but for now a little extra data is preferable to cutting out data that we may later want.

Another important question was: From what source will we gather the SCOTUS decision language? Answer: We chose Justia because it shows the full SCOTUS opinion, including dissent, searchable by the U.S. Citation.

I sort of expected this process + web-scraping to take just one meeting, but it took a long time (for me at least) to make myself familiar with the actual language of the dataset. I went back and forth a lot between the WashU Law code book and our .csv file as we began just by understanding what information is available in the data and how/from where we would scrape the SCOTUS decision language for each case.

Once we had decided on Justia, we spend time on Sunday examining the Justia website. In order to scrape our content, we needed to know 1. the URL for each case’s webpage and 2. where on the webpage the actual case content is (“where” means figuring out how the relevant HTML containers are labeled/classed).

Figuring out part 1 wasn’t too hard. All the Justia SCOTUS cases have the same format, as determined by the U.S. Citation– ex: 391 U.S. 367 corresponds to: https://supreme.justia.com/cases/federal/us/391/367/. A minor data detail — accessing each number independently required separating the single citation column into two distinct columns.

Part 2 was a little more challenging, but Joanne and I bascially… hacked it out until it worked! Sometimes it’s like that.

Finally, I think it’s worth noting that Martin and Kevin both joined Joanne and I for the first session, and Kevin chose not to join for the second. Martin did join, and listened/observed for over 2 hours with only a few sentences of input as Joanne and I mostly just worked through the questions we had. This reflects exactly the kind of group dynamic I want to be in– that group members feel comfortable both opting in AND opting out of  meetings as makes sense for their roles and personal goals. Fully online collaboration sometimes makes it feel like everyone is invited to every meeting and therefore obligated to attend every meeting and… I hate that. I’m all in for setting boundaries and protecting personal time as a key ingredient to successful collaborative relationships, at the same time that we create an environment where all feel welcome and invited to participate in satisfying ways.