Skip to content
Snippets Groups Projects
Commit adb2dd48 authored by Christian Boulanger's avatar Christian Boulanger
Browse files

Updates

parent dc1f90d7
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
langchain-experiments/compare-models.ipynb
+ 1
− 1
View file @ adb2dd48
%% Cell type:markdown id:396ebd275b60c720 tags: %% Cell type:markdown id:396ebd275b60c720 tags:
# Comparing OpenAI and open LLMs # Comparing OpenAI and open LLMs
Using the [text-only content of the website of the journal AUR - Agrar- und Umweltrecht](langchain-experiments/data/input/journal-website.txt), Using the [text-only content of the website of the journal AUR - Agrar- und Umweltrecht](data/input/journal-website.txt),
we compare the performance of GPT-4, GPT-3.5-turbo and Models available on Huggingface. we compare the performance of GPT-4, GPT-3.5-turbo and Models available on Huggingface.
## Preparation ## Preparation
Import dependencies, define shorthand functions, and prepare test data Import dependencies, define shorthand functions, and prepare test data
%% Cell type:code id:e46d0648c1c6c96a tags: %% Cell type:code id:e46d0648c1c6c96a tags:
``` python ``` python
import io import io
from langchain_core.prompts import ChatPromptTemplate from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser from langchain_core.output_parsers import StrOutputParser
import pandas as pd import pandas as pd
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv() load_dotenv()
def response_to_df(response): def response_to_df(response):
data = io.StringIO(response) data = io.StringIO(response)
try: try:
return pd.read_csv(data) return pd.read_csv(data)
except: except:
raise RuntimeError(f"Error while parsing response:\n{response}") raise RuntimeError(f"Error while parsing response:\n{response}")
def use_model(model, template, **params): def use_model(model, template, **params):
prompt = ChatPromptTemplate.from_template(template) prompt = ChatPromptTemplate.from_template(template)
chain = ( chain = (
prompt prompt
| model | model
| StrOutputParser() | StrOutputParser()
) )
return response_to_df(chain.invoke(params)) return response_to_df(chain.invoke(params))
with open('data/input/journal-website.txt', encoding='utf-8') as f: with open('data/input/journal-website.txt', encoding='utf-8') as f:
website_text = f.read() website_text = f.read()
journal_name = "AUR - Agrar- und Umweltrecht" journal_name = "AUR - Agrar- und Umweltrecht"
``` ```
%% Cell type:markdown id:eb805270480fbfb5 tags: %% Cell type:markdown id:eb805270480fbfb5 tags:
## Prompt ## Prompt
OpenAI's GPT-4 works perfectly with a minimal, German-language prompt, and infers the meaning of the columns OpenAI's GPT-4 works perfectly with a minimal, German-language prompt, and infers the meaning of the columns
to returns the data we need: to returns the data we need:
``` ```
Finde im folgenden Text die Herausgeber, Redaktion/Schriftleitung und Beirat der Zeitschrift '{journal_name}' und gebe sie im CSV-Format zurück mit den Spalten 'lastname', 'firstname', 'title', 'position', 'affiliation','role'. Die Spalte 'role' enthält entweder 'Herausgeber', 'Redaktion', 'Beirat', 'Schriftleitung' oder ist leer wenn nicht bestimmbar. Wenn keine passenden Informationen verfügbar sind, gebe nur den CSV-Header zurück. Setze alle Werte in den CSV-Spalten in Anführungszeichen." Finde im folgenden Text die Herausgeber, Redaktion/Schriftleitung und Beirat der Zeitschrift '{journal_name}' und gebe sie im CSV-Format zurück mit den Spalten 'lastname', 'firstname', 'title', 'position', 'affiliation','role'. Die Spalte 'role' enthält entweder 'Herausgeber', 'Redaktion', 'Beirat', 'Schriftleitung' oder ist leer wenn nicht bestimmbar. Wenn keine passenden Informationen verfügbar sind, gebe nur den CSV-Header zurück. Setze alle Werte in den CSV-Spalten in Anführungszeichen."
```` ````
In contrast, the open models performed miserably with such a prompt. We therefore use English and provide very detailed instructions. In contrast, the open models performed miserably with such a prompt. We therefore use English and provide very detailed instructions.
%% Cell type:code id:23aef80911796078 tags: %% Cell type:code id:23aef80911796078 tags:
``` python ``` python
template = """ template = """
In the following German text, which was scraped from a website, find the members of the editorial board or the advisory board of the journal '{journal_name}' as per the following rules: In the following German text, which was scraped from a website, find the members of the editorial board or the advisory board of the journal '{journal_name}' as per the following rules:
- In German, typical labels for these roles are "Herausgeber", "Redaktion/Redakteur/Schriftleitung" and "Beirat". - In German, typical labels for these roles are "Herausgeber", "Redaktion/Redakteur/Schriftleitung" and "Beirat".
- Return the data as comma-separated values, which can be saved to a `.csv` file. Put all values in the CSV rows in quotes. - Return the data as comma-separated values, which can be saved to a `.csv` file. Put all values in the CSV rows in quotes.
- The CSV data must have the columns 'lastname', 'firstname', 'title', 'position', 'affiliation','role'. - The CSV data must have the columns 'lastname', 'firstname', 'title', 'position', 'affiliation','role'.
- The column 'role' must contain either 'Herausgeber', 'Redaktion', 'Beirat' or is empty. Leave the column empty if you cannot determine the role. Use 'Redaktion' for the "Schriftleitung" role. - The column 'role' must contain either 'Herausgeber', 'Redaktion', 'Beirat' or is empty. Leave the column empty if you cannot determine the role. Use 'Redaktion' for the "Schriftleitung" role.
- The column 'title' should contain academic titles such as "Dr." or "Prof. Dr." - The column 'title' should contain academic titles such as "Dr." or "Prof. Dr."
- The column 'position' should contain the job title - The column 'position' should contain the job title
- The column 'affiliation' contains the institution or organization the person belongs to, or the city if one is mentioned - The column 'affiliation' contains the institution or organization the person belongs to, or the city if one is mentioned
- If the journal is published ("herausgeben von") by an association, institute or other organization, but its name in the column 'lastname'. - If the journal is published ("herausgeben von") by an association, institute or other organization, but its name in the column 'lastname'.
- If you cannot find any information, simply return the CSV header. - If you cannot find any information, simply return the CSV header.
- You must not output any introduction, commentary or explanation such as 'Here is the CSV data for the members of the editorial board or the advisory board of the journal'. Only return the data. - You must not output any introduction, commentary or explanation such as 'Here is the CSV data for the members of the editorial board or the advisory board of the journal'. Only return the data.
{website_text} {website_text}
""" """
``` ```
%% Cell type:markdown id:8f994c771cc9b4ef tags: %% Cell type:markdown id:8f994c771cc9b4ef tags:
## ChatGPT-4 ## ChatGPT-4
GPT-4 delivers an almost perfect [result](data/output/editors-openai-gpt-4.csv). There are some problems left which could be resolved by adding some more instructions to the prompt. GPT-4 delivers an almost perfect [result](data/output/editors-openai-gpt-4.csv). There are some problems left which could be resolved by adding some more instructions to the prompt.
%% Cell type:code id:initial_id tags: %% Cell type:code id:initial_id tags:
``` python ``` python
model = ChatOpenAI(model_name="gpt-4") model = ChatOpenAI(model_name="gpt-4")
df = use_model(model, template, journal_name=journal_name, website_text=website_text) df = use_model(model, template, journal_name=journal_name, website_text=website_text)
df.to_csv('data/output/editors-openai-gpt-4.csv') df.to_csv('data/output/editors-openai-gpt-4.csv')
df df
``` ```
%% Output %% Output
lastname firstname title \ lastname firstname title \
0 DGAR NaN NaN 0 DGAR NaN NaN
1 Busse Christian Dr. 1 Busse Christian Dr.
2 Endres Ewald Prof. Dr. 2 Endres Ewald Prof. Dr.
3 Francois Matthias Dr. 3 Francois Matthias Dr.
4 von Garmissen Bernd Dr. 4 von Garmissen Bernd Dr.
5 Glas Ingo NaN 5 Glas Ingo NaN
6 Graß Christiane NaN 6 Graß Christiane NaN
7 Haarstrich Jens NaN 7 Haarstrich Jens NaN
8 Koch Erich Dr. 8 Koch Erich Dr.
9 Köpl Christian Dr. 9 Köpl Christian Dr.
10 Martinez Jose Prof. Dr. 10 Martinez Jose Prof. Dr.
11 Nies Volkmar NaN 11 Nies Volkmar NaN
12 Stephany Ralf NaN 12 Stephany Ralf NaN
13 Wedemeyer Harald NaN 13 Wedemeyer Harald NaN
14 Martinez José Prof. Dr. 14 Martinez José Prof. Dr.
15 Nies Volkmar LLD 15 Nies Volkmar LLD
position \ position \
0 Deutsche Gesellschaft für Agrarrecht 0 Deutsche Gesellschaft für Agrarrecht
1 Regierungsdirektor 1 Regierungsdirektor
2 NaN 2 NaN
3 Rechtsanwalt 3 Rechtsanwalt
4 Rechtsanwalt 4 Rechtsanwalt
5 Rechtsanwalt 5 Rechtsanwalt
6 Rechtsanwältin 6 Rechtsanwältin
7 Rechtsanwalt 7 Rechtsanwalt
8 Ltd. Verwaltungsdirektor 8 Ltd. Verwaltungsdirektor
9 Ministerialrat 9 Ministerialrat
10 NaN 10 NaN
11 Ltd. Landwirtschaftsdirektor 11 Ltd. Landwirtschaftsdirektor
12 Rechtsanwalt/Steuerberater 12 Rechtsanwalt/Steuerberater
13 Rechtsanwalt 13 Rechtsanwalt
14 Erster Schriftleiter 14 Erster Schriftleiter
15 Zweiter Schriftleiter 15 Zweiter Schriftleiter
affiliation role affiliation role
0 NaN Herausgeber 0 NaN Herausgeber
1 Bundesministerium für Ernährung und Landwirtsc... Redaktion 1 Bundesministerium für Ernährung und Landwirtsc... Redaktion
2 Hochschule Weihenstephan-Triesdorf, Freising Redaktion 2 Hochschule Weihenstephan-Triesdorf, Freising Redaktion
3 Bitburg Redaktion 3 Bitburg Redaktion
4 Göttingen Redaktion 4 Göttingen Redaktion
5 Rostock Redaktion 5 Rostock Redaktion
6 Bonn Redaktion 6 Bonn Redaktion
7 Peine Redaktion 7 Peine Redaktion
8 Sozialversicherung für Landwirtschaft, Forsten... Redaktion 8 Sozialversicherung für Landwirtschaft, Forsten... Redaktion
9 Bayerisches Staatsministerium für Ernährung, L... Redaktion 9 Bayerisches Staatsministerium für Ernährung, L... Redaktion
10 Institut für Landwirtschaftsrecht, Georg-Augus... Redaktion 10 Institut für Landwirtschaftsrecht, Georg-Augus... Redaktion
11 Landwirtschaftskammer NRW, Bonn Redaktion 11 Landwirtschaftskammer NRW, Bonn Redaktion
12 Bonn Redaktion 12 Bonn Redaktion
13 Landvolk Niedersachsen, Hannover Redaktion 13 Landvolk Niedersachsen, Hannover Redaktion
14 Institut für Landwirtschaftsrecht, Göttingen Redaktion 14 Institut für Landwirtschaftsrecht, Göttingen Redaktion
15 50170 Kerpen Redaktion 15 50170 Kerpen Redaktion
%% Cell type:markdown id:9cef77bb17d57b53 tags: %% Cell type:markdown id:9cef77bb17d57b53 tags:
## ChatGPT 3.5-turbo ## ChatGPT 3.5-turbo
GPT-3.5 [performs less well](data/output/editors-openai-gpt-3.5-turbo.csv), but still ok. It gets some of the 'title' amd 'position' GPT-3.5 [performs less well](data/output/editors-openai-gpt-3.5-turbo.csv), but still ok. It gets some of the 'title' amd 'position'
column data confused, and does not recognize the institutional publisher (Herausgeber) of the journal. column data confused, and does not recognize the institutional publisher (Herausgeber) of the journal.
%% Cell type:code id:e1aedc5ef3cab564 tags: %% Cell type:code id:e1aedc5ef3cab564 tags:
``` python ``` python
model = ChatOpenAI(model_name="gpt-3.5-turbo") model = ChatOpenAI(model_name="gpt-3.5-turbo")
df = use_model(model, template, journal_name=journal_name, website_text=website_text) df = use_model(model, template, journal_name=journal_name, website_text=website_text)
df.to_csv('data/output/editors-openai-gpt-3.5-turbo.csv') df.to_csv('data/output/editors-openai-gpt-3.5-turbo.csv')
df df
``` ```
%% Output %% Output
lastname firstname title \ lastname firstname title \
0 Busse Christian Dr. 0 Busse Christian Dr.
1 Endres Ewald Prof. Dr. 1 Endres Ewald Prof. Dr.
2 Francois Matthias Dr. 2 Francois Matthias Dr.
3 von Garmissen Bernd Dr. 3 von Garmissen Bernd Dr.
4 Glas Ingo NaN 4 Glas Ingo NaN
5 Graß Christiane Rechtsanwältin 5 Graß Christiane Rechtsanwältin
6 Haarstrich Jens Rechtsanwalt 6 Haarstrich Jens Rechtsanwalt
7 Koch Erich Dr. 7 Koch Erich Dr.
8 Köpl Christian Dr. 8 Köpl Christian Dr.
9 Martinez Jose Prof. Dr. 9 Martinez Jose Prof. Dr.
10 Nies Volkmar Ltd. Landwirtschaftsdirektor 10 Nies Volkmar Ltd. Landwirtschaftsdirektor
11 Stephany Ralf Rechtsanwalt/Steuerberater 11 Stephany Ralf Rechtsanwalt/Steuerberater
12 Wedemeyer Harald Rechtsanwalt 12 Wedemeyer Harald Rechtsanwalt
position \ position \
0 Regierungsdirektor 0 Regierungsdirektor
1 NaN 1 NaN
2 Rechtsanwalt 2 Rechtsanwalt
3 Rechtsanwalt 3 Rechtsanwalt
4 Rechtsanwalt 4 Rechtsanwalt
5 NaN 5 NaN
6 NaN 6 NaN
7 Ltd. Verwaltungsdirektor 7 Ltd. Verwaltungsdirektor
8 Ministerialrat 8 Ministerialrat
9 NaN 9 NaN
10 NaN 10 NaN
11 NaN 11 NaN
12 NaN 12 NaN
affiliation role affiliation role
0 Bundesministerium für Ernährung und Landwirtsc... Redaktion 0 Bundesministerium für Ernährung und Landwirtsc... Redaktion
1 Hochschule Weihenstephan-Triesdorf, Freising Redaktion 1 Hochschule Weihenstephan-Triesdorf, Freising Redaktion
2 Bitburg Redaktion 2 Bitburg Redaktion
3 Göttingen Redaktion 3 Göttingen Redaktion
4 Rostock Redaktion 4 Rostock Redaktion
5 Bonn Redaktion 5 Bonn Redaktion
6 Peine Redaktion 6 Peine Redaktion
7 Sozialversicherung für Landwirtschaft, Forsten... Redaktion 7 Sozialversicherung für Landwirtschaft, Forsten... Redaktion
8 Bayerisches Staatsministerium für Ernährung, L... Redaktion 8 Bayerisches Staatsministerium für Ernährung, L... Redaktion
9 Institut für Landwirtschaftsrecht, Georg-Augus... Redaktion 9 Institut für Landwirtschaftsrecht, Georg-Augus... Redaktion
10 Landwirtschaftskammer NRW, Bonn Redaktion 10 Landwirtschaftskammer NRW, Bonn Redaktion
11 Bonn Redaktion 11 Bonn Redaktion
12 Landvolk Niedersachsen, Hannover Redaktion 12 Landvolk Niedersachsen, Hannover Redaktion
%% Cell type:markdown id:31084716e138fe06 tags: %% Cell type:markdown id:31084716e138fe06 tags:
Now, let's try the open models via the Huggingface Inference Endpoint. For this to work, you need to deploy Now, let's try the open models via the Huggingface Inference Endpoint. For this to work, you need to deploy
endpoints via https://ui.endpoints.huggingface.co/ and update the value of `enpoint_url` below. endpoints via https://ui.endpoints.huggingface.co/ and update the value of `enpoint_url` below.
%% Cell type:markdown id:90ce37abf037e19f tags: %% Cell type:markdown id:90ce37abf037e19f tags:
## TheBloke/Llama-2-13B-chat-GPTQ ## TheBloke/Llama-2-13B-chat-GPTQ
The [LLama2 13 billion parameter model](https://huggingface.co/TheBloke/Llama-2-13B-chat-GPTQ) produces [unusuable output](data/output/editors-llama-2-13b-chat-gptq.txt). The [LLama2 13 billion parameter model](https://huggingface.co/TheBloke/Llama-2-13B-chat-GPTQ) produces [unusuable output](data/output/editors-llama-2-13b-chat-gptq.txt).
%% Cell type:code id:f05098a4cf2aa3dc tags: %% Cell type:code id:f05098a4cf2aa3dc tags:
``` python ``` python
from lib.hf_llama2_chat_gptq import query from lib.hf_llama2_chat_gptq import query
llama2_template = f"<s>[INST] <<SYS>>You are a helpful assistant. No comments or explanation, just answer the question.<</SYS>>{template}[/INST]" llama2_template = f"<s>[INST] <<SYS>>You are a helpful assistant. No comments or explanation, just answer the question.<</SYS>>{template}[/INST]"
endpoint_url = "https://z8afrqamxvaaitmf.us-east-1.aws.endpoints.huggingface.cloud" endpoint_url = "https://z8afrqamxvaaitmf.us-east-1.aws.endpoints.huggingface.cloud"
query(endpoint_url, template, journal_name=journal_name, website_text=website_text).split("\n") query(endpoint_url, template, journal_name=journal_name, website_text=website_text).split("\n")
``` ```
%% Output %% Output
['Martinez, Dr. Christian Busse, Bundesministerium für Ernährung und Landwirtschaft, Bonn Agrarprodukt Recht', ['Martinez, Dr. Christian Busse, Bundesministerium für Ernährung und Landwirtschaft, Bonn Agrarprodukt Recht',
'Prof. Dr. Ewald Endres, Hochschule Weihenstephan-Triesdorf Freising Forsting Forsting, Jagd, Fischerei, Fischerei', 'Prof. Dr. Ewald Endres, Hochschule Weihenstephan-Triesdorf Freising Forsting Forsting, Jagd, Fischerei, Fischerei',
'Lawyeranwalt Ingo Glas, Bitburg Boden Recht', 'Lawyeranwalt Ingo Glas, Bitburg Boden Recht',
'Christiane Grass, Bonn Agrarzivil Recht', 'Christiane Grass, Bonn Agrarzivil Recht',
'Jens Haarstrich, Peine Redaktionär, Rostock', 'Jens Haarstrich, Peine Redaktionär, Rostock',
'Prof. Dr. Bernd von Garmissen, Göttingen Erb, Redaktion, Umwelt', 'Prof. Dr. Bernd von Garmissen, Göttingen Erb, Redaktion, Umwelt',
'Ltdr. Jose Martinez, Georg-August-Universität Göttingen, Göttingen', 'Ltdr. Jose Martinez, Georg-August-Universität Göttingen, Göttingen',
'', '',
'', '',
"Note: The column 'Role' contains the following values: 'Herausgeber', 'Redaktion', 'Beirat'"] "Note: The column 'Role' contains the following values: 'Herausgeber', 'Redaktion', 'Beirat'"]
%% Cell type:markdown id:ca33fb28f6772cbc tags: %% Cell type:markdown id:ca33fb28f6772cbc tags:
## TheBloke/Llama-2-70B-chat-GPTQ via Huggingface Inference Endpoint ## TheBloke/Llama-2-70B-chat-GPTQ via Huggingface Inference Endpoint
The 70 billion parameter variant [does a bit better](data/output/editors-llama-2-70b-chat-gptq.csv) but, among other things, doesn't the academic titles right. It also cannot be persuaded to [not comment on the CSV output].(data/output/editors-llama-2-70b-chat-gptq.txt). Given that the model costs $13/h to run, that's not really that impressive. The 70 billion parameter variant [does a bit better](data/output/editors-llama-2-70b-chat-gptq.csv) but, among other things, doesn't the academic titles right. It also cannot be persuaded to [not comment on the CSV output].(data/output/editors-llama-2-70b-chat-gptq.txt). Given that the model costs $13/h to run, that's not really that impressive.
%% Cell type:code id:b94cf62b996bf3a2 tags: %% Cell type:code id:b94cf62b996bf3a2 tags:
``` python ``` python
endpoint_url = "https://gp8iviqlqee101a0.us-east-1.aws.endpoints.huggingface.cloud" endpoint_url = "https://gp8iviqlqee101a0.us-east-1.aws.endpoints.huggingface.cloud"
query(endpoint_url, template, journal_name=journal_name, website_text=website_text).split("\n") query(endpoint_url, template, journal_name=journal_name, website_text=website_text).split("\n")
``` ```
%% Output %% Output
' Here is the CSV data for the members of the editorial board or the advisory board of the journal \'AUR - Agrar- und Umweltrecht\':\n\n"lastname","firstname","title","affiliation","role"\n"Busse","Christian", "Regierungsdirektor", "Bundesministerium für Ernährung und Landwirtschaft, Bonn", "Herausgeber"\n"Endres","Ewald", "Prof. Dr.", "Hochschule Weihenstephan-Triesdorf, Freising", "Redaktion"\n"Francois","Matthias", "Rechtsanwalt", "Bitburg", "Redaktion"\n"Garmissen","Bernd", "Rechtsanwalt", "Göttingen", "Redaktion"\n"Graß","Christiane", "Rechtsanwältin", "Bonn", "Redaktion"\n"Haarstrich","Jens", "Rechtsanwalt", "Peine", "Redaktion"\n"Köpl","Christian", "Ministerialrat", "Bayerisches Staatsministerium für Ernährung, Landwirtschaft und Forsten, München", ""\n"Martinez","Jose", "Prof. Dr.", "Institut für Landwirtschaftsrecht, Georg-August-Universität Göttingen, Göttingen", "Herausgeber"\n"Nies","Volkmar", "Ltd. Landwirtschaftsdirektor", "Landwirtschaftskammer NRW, Bonn", "Redaktion"\n"Stephany","Ralf", "Rechtsanwalt/Steuerberater", "Bonn", "Redaktion"\n"Wedemeyer","Harald", "Rechtsanwalt", "Landvolk Niedersachsen, Hannover", "Redaktion"\n"Schell","Irina Valeska", "", "Georg-August-Universität Göttingen, Göttingen", ""\n\nNote: The column \'role\' is empty for some members, as their role could not be determined.' ' Here is the CSV data for the members of the editorial board or the advisory board of the journal \'AUR - Agrar- und Umweltrecht\':\n\n"lastname","firstname","title","affiliation","role"\n"Busse","Christian", "Regierungsdirektor", "Bundesministerium für Ernährung und Landwirtschaft, Bonn", "Herausgeber"\n"Endres","Ewald", "Prof. Dr.", "Hochschule Weihenstephan-Triesdorf, Freising", "Redaktion"\n"Francois","Matthias", "Rechtsanwalt", "Bitburg", "Redaktion"\n"Garmissen","Bernd", "Rechtsanwalt", "Göttingen", "Redaktion"\n"Graß","Christiane", "Rechtsanwältin", "Bonn", "Redaktion"\n"Haarstrich","Jens", "Rechtsanwalt", "Peine", "Redaktion"\n"Köpl","Christian", "Ministerialrat", "Bayerisches Staatsministerium für Ernährung, Landwirtschaft und Forsten, München", ""\n"Martinez","Jose", "Prof. Dr.", "Institut für Landwirtschaftsrecht, Georg-August-Universität Göttingen, Göttingen", "Herausgeber"\n"Nies","Volkmar", "Ltd. Landwirtschaftsdirektor", "Landwirtschaftskammer NRW, Bonn", "Redaktion"\n"Stephany","Ralf", "Rechtsanwalt/Steuerberater", "Bonn", "Redaktion"\n"Wedemeyer","Harald", "Rechtsanwalt", "Landvolk Niedersachsen, Hannover", "Redaktion"\n"Schell","Irina Valeska", "", "Georg-August-Universität Göttingen, Göttingen", ""\n\nNote: The column \'role\' is empty for some members, as their role could not be determined.'
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment