Human Value Detection
| Title | Human Value Detection |
|---|---|
| Summary | Given a textual argument and a human value category, classify whether or not the argument draws on that category. |
| Keywords | NLP |
| TimeFrame | |
| References | https://aclanthology.org/2022.acl-long.306.pdf |
| Prerequisites | Python, ML |
| Author | |
| Supervisor | Pablo Picazo |
| Level | Flexible |
| Status | Draft |
All information can be seen: https://touche.webis.de/semeval23/touche23-web/index.html
Summary:
Given a textual argument and a human value category, classify whether or not the argument draws on that category. This task uses a set of 20 value categories compiled from the social science literature and described in our ACL paper. Participants can submit runs (also known as approaches or systems) that detect one, a subset, or all of these values in arguments. Arguments are given as premise text, conclusion text, and binary stance of the premise to the conclusion ("in favor of" or "against").
Data: Data is provided as tab-separated values files with one header line. We will continuously improve the data until early November 2022. Stay up-to-date, report problems, and suggest new data on the task mailing list.
The arguments-training/validation/testing.tsv files contain one argument per line: its unique argument ID, the conclusion, the premise's stance towards the conclusion, and the premise itself. Example with tab-separated columns highlighted:
Argument ID Conclusion Stance Premise A01010 We should prohibit school prayer against it should be allowed if the student wants to pray as long as it is not interfering with his classes A01011 We should abolish the three-strikes laws in favor of three strike laws can cause young people to be put away for life without a chance to straight out their life A01012 The use of public defenders should be mandatory in favor of the use of public defenders should be mandatory because some people don't have money for a lawyer and this would help those that don't The labels-training/validation/testing.tsv files also contain one argument per line: its unique argument ID and one column for each of the 20 value categories with a 1 meaning that the argument resorts to the value category and a 0 that not. Example with tab-separated columns highlighted:
Argument ID Self-direction: thought Self-direction: action Stimulation Hedonism Achievement Power: dominance Power: resources Face Security: personal Security: societal Tradition Conformity: rules Conformity: interpersonal Humility Benevolence: caring Benevolence: dependability Universalism: concern Universalism: nature Universalism: tolerance Universalism: objectivity A01010 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 A01011 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 1 A01012 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0
Evaluation:
Runs are evaluated on the basis of F1-score, Precision, and Recall, once averaged over all value categories, but also for each category individually. The runs are ranked according to the F1-score.
