Clustering SFB Abstracts

This courselet introduces text mining and clustering techniques to uncover hidden structures in academic abstracts. Using data from the SFB 649 “Economic Risk” discussion papers, it demonstrates how to extract, preprocess, and cluster text data with methods like TF-IDF weighting, K-means, CLUTO, and Adaptive Weights Clustering (AWC). Learners will explore how machine learning identifies research topics and compares them to traditional JEL classifications, gaining practical insight into unsupervised learning, dimensionality reduction, and cluster evaluation metrics such as NMI and Adjusted Rand Index.

0 Rating
0 Reviews
2 Students Enrolled

Suggested citation

Lupu, Radu and Härdle, Wolfgang Karl (2025). Clustering SFB Abstracts. Q2 ecosystem. Available at https://quantinar.com/course/1020/clustering-sfb-abstracts or http://dx.doi.org/10.24818/IDA-CL/2025.160.

@misc{lupu2025, author = {Lupu, Radu and Härdle, Wolfgang Karl}, title = {Clustering SFB Abstracts}, year = {2025}, howpublished = {Q2 ecosystem}, doi = {http://dx.doi.org/10.24818/IDA-CL/2025.160.}, url = {https://quantinar.com/course/1020/clustering-sfb-abstracts} }
Wishlist

Free

Enroll Now

Tags:

Courselet Content

1 components

	Clustering SFB Abstracts (pdf)	2.45 M

Requirements

Basic knowledge of statistics Understanding of variables, distributions, and simple metrics. Familiarity with concepts like similarity/distance is a plus. ✅ Introductory experience with machine learning Awareness of what clustering is (unsupervised learning). No need for advanced ML theory, but basic intuition helps. ✅ Very basic programming or data-handling experience Abi
+ See More lity to work with datasets in a structured format (e.g., CSV, tables). Exposure to tools like R or Python is useful but not required if the courselet is conceptual. ✅ Fundamentals of text processing Knowing what terms like “token”, “stopword”, or “frequency” mean is helpful. General comfort with textual data (documents, keywords, etc.). ✅ General interest in economics or social science research Since the course uses SFB/CRC 649 research abstracts and JEL codes, familiarity with economic topics will improve comprehension.
- See Less

General Overview

Description

This courselet provides a hands-on and conceptual introduction to text mining, unsupervised learning, and cluster analysis, using real-world academic research abstracts from the SFB 649 “Economic Risk” project at Humboldt-Universität zu Berlin. The goal is to understand how machine learning algorithms can automatically discover meaningful topic structures in large collections of unstructured text.

Learners will walk through the full pipeline of a modern text analytics workflow. Starting with web scraping and data extraction, the courselet shows how to collect abstracts and metadata—such as project identifiers and JEL classification codes—from a large research database. In the data preparation stage, participants learn how to transform raw text into machine-readable form through tokenization, lowercasing, punctuation removal, stopword filtering, lemmatization, and stemming. These cleaned inputs are used to construct a term–document matrix (TDM) and TF-IDF representations, which capture the importance of words across documents.

The courselet then introduces learners to several important clustering methods tailored to high-dimensional text:

K-means, a classical partitioning algorithm,
CLUTO, a toolkit for high-quality hierarchical text clustering, and
Adaptive Weights Clustering (AWC), an advanced method that automatically determines cluster structures without pre-specifying the number of clusters.

Learners will see how these algorithms behave on real economic research texts and how well they replicate known topic labels such as JEL classifications, which are used as a “true” clustering structure for comparison.

To evaluate clustering quality, the course explains and applies widely used metrics such as Normalized Mutual Information (NMI), the Rand Index, the Misweighting Error, and the Adjusted Rand Index (AdR)—tools that are essential for assessing model performance in unsupervised settings.

By the end of the courselet, participants will understand how machine learning can reveal hidden thematic patterns in research output, detect topic clusters, and support large-scale text analysis in economics or any other domain. They will gain a solid foundation in text preprocessing, vectorization, clustering algorithms, and unsupervised evaluation techniques, all demonstrated through an applied case study using real academic abstracts.