2.1 KiB
ANLP_WS24_CA1
Master MDS
Use NLP techniques you learned so far (N-gram models, basic machine learning, no neural nets) to analyse texts or to build an application. Document your approach.
Data Source
https://github.com/taivop/joke-dataset/tree/master
| File | Jokes | Tokens |
|---|---|---|
| reddit_jokes.json | 195K jokes | 7.40M tokens |
| stupidstuff.json | 3.77K jokes | 396K tokens |
| wocka.json | 10.0K jokes | 1.11M tokens |
| TOTAL | 208K jokes | 8.91M tokens |
Topic presentations (graded) (5 min)
Focus:
- What is your overall idea?
- What kind of data will you use and where do you get the data?
- Your approach, which techniques will you use?
- Expected results.
Open Questions:
- How to evaluate similarity?
- How to find structural patterns? (like phrases, setups, punchlines, or wordplay)
Possible Hypothesis:
- Similar jokes share more common n-grams, phrases, or structural patterns.
- Basic features like word frequency, sentiment, length, or punctuation can predict joke ratings.
other ideas:
-
The length of a joke (measured in words or characters) is inversely correlated with its average rating, as shortness may enhance comedic impact.
-
Highly rated jokes follow certain structural patterns (e.g., setups, punchlines, or wordplay).
Possible Tools / Techniques
-
Text Preprocessing: Tokenization, stopword removal, stemming/lemmatization.
-
Feature Extraction: Bag-of-Words, n-grams (bigram/trigram analysis), TF-IDF.
-
Similarity: Cosine similarity for finding similar jokes.
Research
Humor Detection
Humor Detection: A Transformer Gets the Last Laugh
Computationally recognizing wordplay in jokes (N - Grams)
Word2Vec combined with K-NN Human Centric Features