Knowledge Base Construction from Pre-trained Language Models (LM-KBC)

News

• 25.08.23: Winning systems in each track announced.
• 09.08.23: Final deadline extension to August 14, 11:59 pm CEST time, for both system and paper submissions.
• 29.07.23: Deadline extended to August 10 for system and prediction submission, and 11 August for paper submission.
• 26.07.23: Test data subject entities have been released on the GitHub repo. Please do a git pull for getting the test dataset and updated evaluate.py script. Submit your predictions on CodaLab to get your scores.
• 11.07.23: Submit your validation data predictions on CodaLab to get a score now (this is optional and test data leaderboard will be separate and released later).
• 10.07.23: New baseline (GPT3 + Wikidata NED) added to repository.
• 22.05.23: v1.0 of dataset (train and validation splits) released.
• 17.05.23: Test output/system submission deadline extended to August 2, 2023. Take time to submit your strongest systems!

Task Description

Pretrained language models (LMs) like chatGPT have advanced a range of semantic tasks and have also shown promise for knowledge extraction from the models itself. Although several works have explored this ability in a setting called probing or prompting, the viability of knowledge base construction from LMs remains underexplored. In the 2^nd edition of this challenge, we invite participants to build actual disambiguated knowledge bases from LMs, for given subjects and relations. In crucial difference to existing probing benchmarks like LAMA (Petroni et al., 2019), we make no simplifying assumptions on relation cardinalities, i.e., a subject-entity can stand in relation with zero, one, or many object-entities. Furthermore, submissions need to go beyond just ranking predicted surface strings and materialize disambiguated entities in the output, which will be evaluated using established KB metrics of precision and recall.

Formally, given the input subject-entity (s) and relation (r), the task is to predict all the correct object-entities ({o1, o2, ..., ok}) using LM probing.

The challenge comes with two tracks:

• Track 1: a small-model track with low computational requirements (<1 billion parameters)
• Track 2: an open track, where participants can use any LM of their choice

Track 1: Small-model track (<1 billion parameters)

Participants are free to use any pretrained LM containing at most 1 billion parameters. This includes, for instance, BERT, BART, GPT-2, and variants of OPT. The input tuples can be paraphrased through prompt engineering techniques (e.g., AutoPrompt,LPAQA), and participants can also use prompt ensembles for better output generation. However, using context (e.g., verbalizing tuples using supporting sentences) is not allowed in this track.

Track 2: Open track

In the open track, the task is the same as in the small-model track. However,

• LMs of any size, e.g., GPT-3, can be probed.
• Use of context is allowed for LM-generation, e.g., context retrieval like in REALM and factual predictions with context.

(Q941293, N.E.R.D., BandHasMember, [Q14313, Q706641, Q2584176], [Pharrell Williams, Chad Hugo, Shay Haley])

(Q365, Cologne, CityLocatedAtRiver, [Q584], [Rhine])

(Q39898, NSU, CompanyHasParentOrganisation, [Q246], [Volkswagen])

(Q150843, Hexadecane, [Q623, Q556], [carbon, hydrogen])

(Q1020, Malawi, CountryBordersCountry, [Q924, Q953, Q1029], [Tanzania, Zambia, Mozambique])

(Q334, Singapore, CountryHasOfficialLanguage, [Q1860, Q5885, Q9237, Q727694], [English, Tamil, Malay, Standard Mandarin])

(Q702, Federated States of Micronesia, CountryHasStates, [Q221684, Q1785093, Q7771127, Q11342951], [Chuuk, Kosrae State, Pohnpei State, Yap State])

(Q455462, Antoine Griezmann, FootballerPlaysPosition, [Q280658], [forward])

(Q5238609, David Plotz, PersonCauseOfDeath, [ ], [ ])

(Q6279, Joe Biden, PersonHasAutobiography, [Q100221747], [Promise Me Dad])

(Q11476943, Yōichi Shimada, PersonHasEmployer, [Q4845464], [Fukui Prefectural University])

(Q65989, Wolfgang Pauli, PersonHasNobelPrize, [Q38104], [Nobel Prize in Physics])

(Q7599711, Stanley Johnson, PersonHasNumberOfChildren, [6], [6])

(Q4369225, Alina Pokrovskaya, PersonHasPlaceOfDeath, [ ], [ ])

(Q468043, Jon Elster, PersonHasProfession, [Q121594, Q188094, Q1238570, Q2306091, Q4964182], [professor, economist, political scientist, sociologist, philosopher])

(Q5111202, Chrissy Teigen, PersonHasSpouse, [Q44857], [John Legend])

(Q15994935, Emma Blackery, PersonPlaysInstrument, [Q6607, Q61285, Q17172850], [guitar, ukulele, voice])

(Q18958964, Witold Andrzejewski, PersonSpeaksLanguage, [Q809], [Polish])

(Q45403, Brahmaputra River, RiverBasinsCountry, [Q148, Q668], [People's Republic of China, India])

(Q12403564, Euphoria, SeriesHasNumberOfEpisodes, [10], [10])

(Q1204, Illinois, StateBordersState, [Q1166, Q1415, Q1537, Q1546, Q1581, Q1603], [Michigan, Indiana, Wisconsin, Iowa, Missouri, Kentucky])

Task Evaluation

For each test instance, predictions are evaluated by calculating precision and recall against ground-truth values. The final macro-averaged F1-score is used to rank the participating systems.

Baselines

We provide several baselines:

• Standard prompt for HuggingFace models with Wikidata default disambiguation: These baselines can be instantiated with various HuggingFace models (e.g., BERT, OPT), generate entity surface forms, and use the Wikidata entity disambiguation API to generate IDs.
• Few-shot GPT-3 directly predicting IDs: This baseline uses a few samples to instruct GPT-3 to directly predict Wikidata IDs.
• Few-shot GPT-3 w/ NED: Like above, but predicting surface forms that are disambiguated via Wikidata's default disambiguation.

Baseline performance:

Method	Avg. Precision	Avg. Recall	Avg. F1-score
GPT-3 NED (Curie model)	0.308	0.210	0.218
GPT-3 IDs directly (Curie model)	0.126	0.060	0.061
BERT	0.368	0.161	0.142

Organizers