Skip to content
Snippets Groups Projects
Commit dc1f90d7 authored by Christian Boulanger's avatar Christian Boulanger
Browse files

Updates

parent 899389c1
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
langchain-experiments/compare-models.ipynb
+ 25
− 16
View file @ dc1f90d7
%% Cell type:markdown id:396ebd275b60c720 tags:
# Comparing OpenAI and open LLMs
Using the text-only content of the website of the journal AUR - Agrar- und Umweltrecht, we compare the performance of
GPT-4, GPT-3.5-turbo and Models available on Huggingface.
Using the [text-only content of the website of the journal AUR - Agrar- und Umweltrecht](langchain-experiments/data/input/journal-website.txt),
we compare the performance of GPT-4, GPT-3.5-turbo and Models available on Huggingface.
## Prpeparation
## Preparation
Import dependencies, define shorthand functions, and prepare test data
%% Cell type:code id:e46d0648c1c6c96a tags:
``` python
import io
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
import pandas as pd
from dotenv import load_dotenv
load_dotenv()
def response_to_df(response):
data = io.StringIO(response)
try:
return pd.read_csv(data)
except:
raise RuntimeError(f"Error while parsing response:\n{response}")
def use_model(model, template, **params):
prompt = ChatPromptTemplate.from_template(template)
chain = (
prompt
| model
| StrOutputParser()
)
return response_to_df(chain.invoke(params))
with open('data/input/journal-website.txt', encoding='utf-8') as f:
website_text = f.read()
journal_name = "AUR - Agrar- und Umweltrecht"
```
%% Cell type:markdown id:eb805270480fbfb5 tags:
## Prompt
OpenAI's GPT-4 works perfectly with a German-language Prompt, and a very minimal prompt returns the data we need. In contrast, the open models performed miserably with that prompt. We therefore use English and provide very detailed instructions.
OpenAI's GPT-4 works perfectly with a minimal, German-language prompt, and infers the meaning of the columns
to returns the data we need:
```
Finde im folgenden Text die Herausgeber, Redaktion/Schriftleitung und Beirat der Zeitschrift '{journal_name}' und gebe sie im CSV-Format zurück mit den Spalten 'lastname', 'firstname', 'title', 'position', 'affiliation','role'. Die Spalte 'role' enthält entweder 'Herausgeber', 'Redaktion', 'Beirat', 'Schriftleitung' oder ist leer wenn nicht bestimmbar. Wenn keine passenden Informationen verfügbar sind, gebe nur den CSV-Header zurück. Setze alle Werte in den CSV-Spalten in Anführungszeichen."
````
In contrast, the open models performed miserably with such a prompt. We therefore use English and provide very detailed instructions.
%% Cell type:code id:23aef80911796078 tags:
``` python
template = """
In the following German text, which was scraped from a website, find the members of the editorial board or the advisory board of the journal '{journal_name}' as per the following rules:
- In German, typical labels for these roles are "Herausgeber", "Redaktion/Redakteur/Schriftleitung" and "Beirat".
- Return the data as comma-separated values, which can be saved to a `.csv` file. Put all values in the CSV rows in quotes.
- The CSV data must have the columns 'lastname', 'firstname', 'title', 'position', 'affiliation','role'.
- The column 'role' must contain either 'Herausgeber', 'Redaktion', 'Beirat' or is empty. Leave the column empty if you cannot determine the role. Use 'Redaktion' for the "Schriftleitung" role.
- The column 'title' should contain academic titles such as "Dr." or "Prof. Dr."
- The column 'position' should contain the job title
- The column 'affiliation' contains the institution or organization the person belongs to, or the city if one is mentioned
- If the journal is published ("herausgeben von") by an association, institute or other organization, but its name in the column 'lastname'.
- If you cannot find any information, simply return the CSV header.
- You must not output any introduction, commentary or explanation such as 'Here is the CSV data for the members of the editorial board or the advisory board of the journal'. Only return the data.
{website_text}
"""
```
%% Cell type:markdown id:8f994c771cc9b4ef tags:
## ChatGPT-4
GPT-4 delivers an almost perfect [result](data/output/editors-openai-gpt-4.csv). There are some problems left which could be
resolved by adding some more instructions to the prompt.
GPT-4 delivers an almost perfect [result](data/output/editors-openai-gpt-4.csv). There are some problems left which could be resolved by adding some more instructions to the prompt.
%% Cell type:code id:initial_id tags:
``` python
model = ChatOpenAI(model_name="gpt-4")
df = use_model(model, template, journal_name=journal_name, website_text=website_text)
df.to_csv('data/output/editors-openai-gpt-4.csv')
df
```
%% Output
lastname firstname title \
0 DGAR NaN NaN
1 Busse Christian Dr.
2 Endres Ewald Prof. Dr.
3 Francois Matthias Dr.
4 von Garmissen Bernd Dr.
5 Glas Ingo NaN
6 Graß Christiane NaN
7 Haarstrich Jens NaN
8 Koch Erich Dr.
9 Köpl Christian Dr.
10 Martinez Jose Prof. Dr.
11 Nies Volkmar NaN
12 Stephany Ralf NaN
13 Wedemeyer Harald NaN
14 Martinez José Prof. Dr.
15 Nies Volkmar LLD
position \
0 Deutsche Gesellschaft für Agrarrecht
1 Regierungsdirektor
2 NaN
3 Rechtsanwalt
4 Rechtsanwalt
5 Rechtsanwalt
6 Rechtsanwältin
7 Rechtsanwalt
8 Ltd. Verwaltungsdirektor
9 Ministerialrat
10 NaN
11 Ltd. Landwirtschaftsdirektor
12 Rechtsanwalt/Steuerberater
13 Rechtsanwalt
14 Erster Schriftleiter
15 Zweiter Schriftleiter
affiliation role
0 NaN Herausgeber
1 Bundesministerium für Ernährung und Landwirtsc... Redaktion
2 Hochschule Weihenstephan-Triesdorf, Freising Redaktion
3 Bitburg Redaktion
4 Göttingen Redaktion
5 Rostock Redaktion
6 Bonn Redaktion
7 Peine Redaktion
8 Sozialversicherung für Landwirtschaft, Forsten... Redaktion
9 Bayerisches Staatsministerium für Ernährung, L... Redaktion
10 Institut für Landwirtschaftsrecht, Georg-Augus... Redaktion
11 Landwirtschaftskammer NRW, Bonn Redaktion
12 Bonn Redaktion
13 Landvolk Niedersachsen, Hannover Redaktion
14 Institut für Landwirtschaftsrecht, Göttingen Redaktion
15 50170 Kerpen Redaktion
%% Cell type:markdown id:9cef77bb17d57b53 tags:
## ChatGPT 3.5-turbo
GPT-3.5 [performs less well](data/output/editors-openai-gpt-3.5-turbo.csv), but still ok. It gets some of the 'title' amd 'position'
column data confused, and does not recognize the institutional publisher (Herausgeber) of the journal.
%% Cell type:code id:e1aedc5ef3cab564 tags:
``` python
model = ChatOpenAI(model_name="gpt-3.5-turbo")
df = use_model(model, template, journal_name=journal_name, website_text=website_text)
df.to_csv('data/output/editors-openai-gpt-3.5-turbo.csv')
df
```
%% Output
lastname firstname title \
0 Busse Christian Dr.
1 Endres Ewald Prof. Dr.
2 Francois Matthias Dr.
3 von Garmissen Bernd Dr.
4 Glas Ingo NaN
5 Graß Christiane Rechtsanwältin
6 Haarstrich Jens Rechtsanwalt
7 Koch Erich Dr.
8 Köpl Christian Dr.
9 Martinez Jose Prof. Dr.
10 Nies Volkmar Ltd. Landwirtschaftsdirektor
11 Stephany Ralf Rechtsanwalt/Steuerberater
12 Wedemeyer Harald Rechtsanwalt
position \
0 Regierungsdirektor
1 NaN
2 Rechtsanwalt
3 Rechtsanwalt
4 Rechtsanwalt
5 NaN
6 NaN
7 Ltd. Verwaltungsdirektor
8 Ministerialrat
9 NaN
10 NaN
11 NaN
12 NaN
affiliation role
0 Bundesministerium für Ernährung und Landwirtsc... Redaktion
1 Hochschule Weihenstephan-Triesdorf, Freising Redaktion
2 Bitburg Redaktion
3 Göttingen Redaktion
4 Rostock Redaktion
5 Bonn Redaktion
6 Peine Redaktion
7 Sozialversicherung für Landwirtschaft, Forsten... Redaktion
8 Bayerisches Staatsministerium für Ernährung, L... Redaktion
9 Institut für Landwirtschaftsrecht, Georg-Augus... Redaktion
10 Landwirtschaftskammer NRW, Bonn Redaktion
11 Bonn Redaktion
12 Landvolk Niedersachsen, Hannover Redaktion
%% Cell type:markdown id:31084716e138fe06 tags:
Now, let's try the open models via the Huggingface Inference Endpoint
Now, let's try the open models via the Huggingface Inference Endpoint. For this to work, you need to deploy
endpoints via https://ui.endpoints.huggingface.co/ and update the value of `enpoint_url` below.
%% Cell type:markdown id:90ce37abf037e19f tags:
## TheBloke/Llama-2-13B-chat-GPTQ
The [LLama2 13 billion parameter model](https://huggingface.co/TheBloke/Llama-2-13B-chat-GPTQ) produces [unusuable output](data/output/editors-llama-2-13b-chat-gptq.txt).
%% Cell type:code id:f05098a4cf2aa3dc tags:
``` python
from lib.hf_llama2_13b_chat_gptq import query
from lib.hf_llama2_chat_gptq import query
llama2_template = f"<s>[INST] <<SYS>>You are a helpful assistant. No comments or explanation, just answer the question.<</SYS>>{template}[/INST]"
lines = query(template, journal_name=journal_name, website_text=website_text).split("\n")[4:]
lines
endpoint_url = "https://z8afrqamxvaaitmf.us-east-1.aws.endpoints.huggingface.cloud"
query(endpoint_url, template, journal_name=journal_name, website_text=website_text).split("\n")
```
%% Output
['Martinez, Dr. Christian Busse, Bundesministerium für Ernährung und Landwirtschaft, Bonn Agrarprodukt Recht',
'Prof. Dr. Ewald Endres, Hochschule Weihenstephan-Triesdorf Freising Forsting Forsting, Jagd, Fischerei, Fischerei',
'Lawyeranwalt Ingo Glas, Bitburg Boden Recht',
'Christiane Grass, Bonn Agrarzivil Recht',
'Jens Haarstrich, Peine Redaktionär, Rostock',
'Prof. Dr. Bernd von Garmissen, Göttingen Erb, Redaktion, Umwelt',
'Ltdr. Jose Martinez, Georg-August-Universität Göttingen, Göttingen',
'',
'',
"Note: The column 'Role' contains the following values: 'Herausgeber', 'Redaktion', 'Beirat'"]
%% Cell type:markdown id:ca33fb28f6772cbc tags:
## TheBloke/Llama-2-70B-chat-GPTQ via Huggingface Inference Endpoint
The 70 billion parameter variant [does a bit better](data/output/editors-llama-2-70b-chat-gptq.csv) but, among other things, doesn't the academic titles right. It also cannot be persuaded to [not comment on the CSV output].(data/output/editors-llama-2-70b-chat-gptq.txt)
The 70 billion parameter variant [does a bit better](data/output/editors-llama-2-70b-chat-gptq.csv) but, among other things, doesn't the academic titles right. It also cannot be persuaded to [not comment on the CSV output].(data/output/editors-llama-2-70b-chat-gptq.txt). Given that the model costs $13/h to run, that's not really that impressive.
%% Cell type:code id:b94cf62b996bf3a2 tags:
``` python
from lib.hf_llama2_70b_chat_gptq import query
query(template, journal_name=journal_name, website_text=website_text)
endpoint_url = "https://gp8iviqlqee101a0.us-east-1.aws.endpoints.huggingface.cloud"
query(endpoint_url, template, journal_name=journal_name, website_text=website_text).split("\n")
```
%% Output
' Here is the CSV data for the members of the editorial board or the advisory board of the journal \'AUR - Agrar- und Umweltrecht\':\n\n"lastname","firstname","title","affiliation","role"\n"Busse","Christian", "Regierungsdirektor", "Bundesministerium für Ernährung und Landwirtschaft, Bonn", "Herausgeber"\n"Endres","Ewald", "Prof. Dr.", "Hochschule Weihenstephan-Triesdorf, Freising", "Redaktion"\n"Francois","Matthias", "Rechtsanwalt", "Bitburg", "Redaktion"\n"Garmissen","Bernd", "Rechtsanwalt", "Göttingen", "Redaktion"\n"Graß","Christiane", "Rechtsanwältin", "Bonn", "Redaktion"\n"Haarstrich","Jens", "Rechtsanwalt", "Peine", "Redaktion"\n"Köpl","Christian", "Ministerialrat", "Bayerisches Staatsministerium für Ernährung, Landwirtschaft und Forsten, München", ""\n"Martinez","Jose", "Prof. Dr.", "Institut für Landwirtschaftsrecht, Georg-August-Universität Göttingen, Göttingen", "Herausgeber"\n"Nies","Volkmar", "Ltd. Landwirtschaftsdirektor", "Landwirtschaftskammer NRW, Bonn", "Redaktion"\n"Stephany","Ralf", "Rechtsanwalt/Steuerberater", "Bonn", "Redaktion"\n"Wedemeyer","Harald", "Rechtsanwalt", "Landvolk Niedersachsen, Hannover", "Redaktion"\n"Schell","Irina Valeska", "", "Georg-August-Universität Göttingen, Göttingen", ""\n\nNote: The column \'role\' is empty for some members, as their role could not be determined.'
......
langchain-experiments/lib/hf_llama2_70b_chat_gptq.py deleted 100644 → 0
+ 0
− 35
View file @ 899389c1
import requests
import os
from dotenv import load_dotenv
from json import JSONDecodeError
import os
load_dotenv()
API_KEY = os.getenv("HUGGINGFACEHUB_API_TOKEN")
API_URL = "https://gp8iviqlqee101a0.us-east-1.aws.endpoints.huggingface.cloud"
headers = {
"Accept" : "application/json",
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
def query(template, model_params = None, **params):
if model_params is None:
model_params = {
"temperature": 0.1,
"max_new_tokens": 1000
}
prompt = template.format_map(params)
payload = {
"inputs": f"<s>[INST] <<SYS>>You are a helpful assistant. No comments or explanation, just answer the question.<</SYS>>{prompt}[/INST]",
"parameters": model_params
}
response = requests.post(API_URL, headers=headers, json=payload)
response.raise_for_status()
try:
return response.json()[0].get("generated_text")
except JSONDecodeError:
with open('tmp/response.txt', "w", encoding='utf-8') as f:
f.write(response.text)
raise RuntimeError(f'Cannot parse response from {response.url}. See tmp/response.txt')
langchain-experiments/lib/hf_llama2_13b_chat_gptq.py → langchain-experiments/lib/hf_llama2_chat_gptq.py
+ 4
− 6
View file @ dc1f90d7
import requests
import os
from dotenv import load_dotenv
from json import JSONDecodeError
import os
......@@ -7,25 +6,24 @@ import os
load_dotenv()
API_KEY = os.getenv("HUGGINGFACEHUB_API_TOKEN")
API_URL = "https://z8afrqamxvaaitmf.us-east-1.aws.endpoints.huggingface.cloud"
headers = {
"Accept" : "application/json",
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
def query(template, model_params = None, **params):
def query(url, template, model_params = None, **params):
if model_params is None:
model_params = {
"temperature": 0.1,
"max_new_tokens": 2000
}
prompt = template.format_map(params)
inputs = template.format_map(**params)
payload = {
"inputs": f"<s>[INST] <<SYS>>You are a helpful assistant. No comments or explanation, just answer the question.<</SYS>>{prompt}[/INST]",
"inputs": inputs,
"parameters": model_params
}
response = requests.post(API_URL, headers=headers, json=payload)
response = requests.post(url, headers=headers, json=payload)
response.raise_for_status()
try:
return response.json()[0].get("generated_text")
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment