Data Privacy Course

You can access the archive through the the Carnegie Mellon Library's set of on-line databases located at https://www.library.cmu.edu/Search/AZ.html. Select the New York Times Historical archive, which is a database of articles appearing in the New York Times from 1851 through 1999.

An extraction of citations from the database (along with abstracts) for articles related to privacy is available for your use. This is more complete and accurate than the results from Lab 1. Contact Prof. Sweeney for a copy of this database by sending a message to latanya@dataprivacylab.org. You will want to use this version of the database as the subject of your work.

• Activity A
  Plot the citations over time (number of citations per year). For each peak appearing in the graph, review the documents comprising that peak and provide a written summary of the topics that are the subject of discussion that is generating the peaks. You should examine at least the top 6 peaks. Your summary should be a short abstract of no more than a page describing the topic, main issues and final disposition.

  Your project presentation will show interesting results you found. Additionally, you should allow visitors to your project booth to browse the articles related to the peaks.

  
  

• Activity B
  Consider an old technology that may appear in the database that has an analogy to a new privacy invading technology. For example, sound recording in the past may be like video surveillance recordings today. Examine the articles over time that discuss the old technology. Plot the citations over time. Write a short abstract of no more than a page that compares the lessons learned from the old technology and in what ways they are likely to be the same or different to the discourse surrounding the new technology. You should examine at least 6 parallel technologies.

  Your project presentation will show interesting results you found. Additionally, you should allow visitors to your project booth to browse the articles related to the technologies.

  
  

• Activity C
  Examine the database of articles by reflecting on the topic of the articles. This is done using keywords appearing in the title or abstract. A list of keyword ghroups are provided below. Start by plotting all the citations over time (number of citations per year). Then, for each of the groups of keywords listed below, identify which citations appear for which topics (keyword groups). Report the number of plots over time for each topic. When you are done, review your results. Consider whether some topics recur, or alternatively, whether topics once discussed are then resolved. Consider whether there are some interesting patterns. Perhaps some keyword groups should be combined.

  Your project presentation will show interesting results you found. Additionally, you should allow visitors to your project booth to browse the articles that are clustered by keywords.

  Below are the keyword groups to use (though you may augment the list as you deem appropriate). A citation "belongs" to the group of keywords if one or more of the keywords in the group appears in the title or abstract of the citation. Matching may be based on parts of words --i.e., substring matching.

  • advertising, marketing
  • consent
  • constitution
  • eavesdropping
  • mail, electronic mail, email
  • encryption
  • free speech, media, reporter, press
  • genetics, DNA
  • health, medical, hospital, doctor, physician
  • insurance
  • internet, web
  • marriage
  • personal, records, data, information
  • political, politics
  • race, ethnicity, color, racial, black
  • photography, photo, photographer
  • social security number
  • recording, sound, video
  • surveillance
  • telephone, telephony, tele
  • book, library
  • school, education
  • finances, financial, tax, irs, bank
  • law enforcement, police, judge, crime
  • celebrity, celebrities
  • employ, employment, employer, job, work

  
  

• Activity D
  Write a system that finds similar articles using a data mining technique for clustering articles. You will show clustering results as part of your project presentation to show which kinds of articles were computed as being similar. For up to 6 of the larger clusters, identify the topic of the clustered articles --that is, identify what they have in common that relates them together. Your resulting system should allow a user to select some articles of interest, and then your system will find similar articles to those selected. This is the interactive part of your project demonstration.

  There are several techniques you may use to cluster articles. Here is one based on cosine similarity. It is augmented because of the brevity of the text appearing in the titles and abstracts of the citations.

  1. Make a master index of all words appearing in all citations in the database. These are ordered and numbered. For example, a master index might begin:

     [1] the
     [2] privacy
     [3] of
     [4] rights
     [5] court
     ...
     

  2. Remove words from the master index that should not be included such as "a" and "the." Keep these words together as a list of exceptions. Renumber the master index. For example, the sample master index above might now begin:

     [1] privacy
     [2] rights 
     [3] court
     ...
     

  3. For each article, record the number of occurrences of each word in the combined title and abstract fields, skipping any words that appear in the exceptions. (You should also treat plural terms and singular terms as the same word.) For example, given the master index above and a citation whose title begins privacy rights questioned, the vector associated with this citation would be:

     [1] 1
     [2] 1 
     [3] 0
     ...
     

  4. Compare each pair of articles by computing the cosine distance. This is classically done by considering the frequency counts of words as vectors and computing the cosine of two vectors. The resulting value provides a "distance" measure between the two citations.

  5. Using the distance measurement, identify clusters of citations appearing near each other.

  Modification! Because of the brevity of the titles and abstracts, there are not many words appearing for each citation. Therefore, you may want to repeat the steps above, but instead of having each word appear as the basis of the index. Combine words together to make groups of related words be the basis of the index. For example, instead of "police" and "crime" each having a separate index position, combine these so that any occurrence of {police, judge, crime, law enforcement} counts together. You may want to use the keyword groupings identified further above.