Is a given system useful for a given task? This calls into being the following chain of reasoning.
Purpose -> Criteria of Acceptability -> Is it useful?
So are Large Language Models as capable at work as a human? Are they useful? In addressing this I'm not going to make public what I've done for my employer, suffice to say it is potentially useful for us in a wide variety of roles. Instead I'm going to approach it from a tangent.
Does GPT3.5 (ChatGPT) understand and is this the same as human understanding?
Chris
I am going to give you three instructions to carry out a task.
1) Read the sentences I provide.
2) For each one transform the sentence into the past tense.
Do you understand?
GPT3.5
Yes, I understand. Please provide me with the sentences you would like me to transform into the past tense.
Chris
What were the instructions I gave you?
GPT3.5
The instructions you gave me were:
Read the sentences I provide.
For each one transform the sentence into the past tense.
Chris
How many instructions were there?
GPT3.5
There were three instructions.
Chris
I said I would give you three, but what were the instructions?
GPT3.5
My apologies, you only gave me two instructions. The instructions you gave me were:
Read the sentences I provide.
For each one transform the sentence into the past tense.
Chris
So did you understand?
GPT3.5
Yes, I understood your instructions. I will read the sentences you provide and transform each one into the past tense.
Chris
Can you tell me why you didn't realise that I hadn't given you the proper number of instructions?
GPT3.5
As an AI language model, I do not have the ability to realize or feel. I am programmed to analyze and respond to the input I receive based on my programming and training. In this case, I simply responded based on the information you provided to me, which was two instructions instead of three.
OK, so, let's try the same with GPT4.
Chris
I am going to give you three instructions to carry out a task.
1) Read the sentences I provide.
2) For each one transform the sentence into the past tense.
Do you understand?
GPT4
Yes, I understand your instructions. Please provide the sentences you'd like me to transform into the past tense.
Chris
What were the instructions I gave you?
GPT4
The instructions you gave me were:
Read the sentences you provide.
For each one, transform the sentence into the past tense.
You provided two instructions for carrying out a task.
Chris
How many instructions did I say I would give you?
GPT4
You mentioned that you would give me three instructions to carry out a task. However, you only provided two instructions.
Chris
So, when you first said you understand, what did you mean by that?
GPT4
When I first said I understand, I meant that I comprehended the instructions you had given me at that point. I understood that you intended to provide three instructions, but I acknowledged and was prepared to follow the two instructions you had actually given. My response was to indicate that I was ready to proceed with the task based on the information you had provided.
OK, so we have a picture, bolstered by other work I've been doing and by numerous accounts from others, GPT4 is substantially more intelligent than GPT3.5. But neither flagged the issue of the missing instruction at the outset. When they report they understand this is just words, in a sense they do understand, as shown in my previous post, but in a sense they don't.
Here's another test on GPT4.
Chris
OK, another test. In this test I will give you a number and some instructions, I want you to tell me the most efficient way to get to the result that the instructions would give.
GPT4
Sure, I'm ready for the test. Please provide the number and the instructions, and I will help you determine the most efficient way to get to the result.
Chris
The number is five. I want you to add 2, then add 3 to the number and tell me the result and workings.
GPT4
To efficiently solve the problem, you can combine the numbers you want to add before performing the addition. Here's the working:
Combine the numbers to be added: 2 + 3 = 5
Add the combined number to the initial number: 5 (initial number) + 5 (combined number) = 10
The result is 10.
Chris
Thank you. The number is 12. I want you to add 6, then subtract 6 to the number and tell me the result and workings.
GPT4
In this case, the most efficient way to get the result is to recognize that adding 6 and then subtracting 6 will cancel each other out, leaving the initial number unchanged. Here are the workings:
Add 6: 12 (initial number) + 6 = 18
Subtract 6: 18 - 6 = 12
The result is 12, which is the same as the initial number.
So here, in the second sum, GPT4 spots that action is superfluous, but having been given the instruction to provide workings it goes ahead and does it anyway.
Let's try something far more demanding....
Here I took a set of data from a large (publicly available) database of tradeable microwave link licenses that contains substantial overhead, for each link (Go/Return) there are four entries, transmit and receive for each site at ends of the link. Can this be rationalised down into a link oriented format? (GPT3.5 didn't get this one.)
Chris
The following text is a snippet from a massive table of fixed link radio licenses, these are licenses for point-to-point microwave links.
Licence Number,NGR,Frequency (GHz),Channel Width (MHz),Station Type,Emission Code,Antenna Name,Antenna Polarisation,Licencee Company,Antenna Height,Antenna AZIMUTH,Antenna Elevation,Antenna Gain,Antenna ERP
1021490/1,SU 49777 29446,23.1,56,R,56M0D7W,A/23G/H/07/028/AA,V,MOBILE BROADBAND NETWORK LIMITED,15,328.1668868,0.251100552,40.2,0
1021490/1,SU 49777 29446,22.092,56,T,56M0D7W,A/23G/H/07/028/AA,V,MOBILE BROADBAND NETWORK LIMITED,15,328.1668868,0.251100552,40.2,1986.831166
1021490/1,SU 45800 35700,23.1,56,T,56M0D7W,A/23G/H/07/028/AA,V,MOBILE BROADBAND NETWORK LIMITED,31,148.123345,-0.317693493,40.2,3072.257594
1021490/1,SU 45800 35700,22.092,56,R,56M0D7W,A/23G/H/07/028/AA,V,MOBILE BROADBAND NETWORK LIMITED,31,148.123345,-0.317693493,40.2,0
1020795/1,SP 33109 78359,24.5595,7,T,7M00G7W,A/25/H/05/026/AA,V,Airwave Solutions Limited,47,266.2455507,-0.223524848,36.6,7.213305911
1020795/1,SP 33109 78359,25.5675,7,R,7M00G7W,A/25/H/05/026/AA,V,Airwave Solutions Limited,47,266.2455507,-0.223524848,36.6,0
1020795/1,SP 30034 78137,25.5675,7,T,7M00G7W,A/25/H/05/026/AA,V,Airwave Solutions Limited,18,86.20972636,0.195877807,36.6,10.87713886
1020795/1,SP 30034 78137,24.5595,7,R,7M00G7W,A/25/H/05/026/AA,V,Airwave Solutions Limited,18,86.20972636,0.195877807,36.6,0
1020801/1,NS 79789 93239,18.466,7,T,7M00G7W,A/19/H/05/018/AA,V,Airwave Solutions Limited,25,73.74506981,0.182952599,34.2,12.69804364
1020801/1,NS 79789 93239,19.474,7,R,7M00G7W,A/19/H/05/018/AA,V,Airwave Solutions Limited,25,73.74506981,0.182952599,34.2,0
1020801/1,NS 86933 95102,19.474,7,T,7M00G7W,A/19/H/05/018/AA,V,Airwave Solutions Limited,20,253.8397529,-0.249145473,34.2,21.26433081
1020801/1,NS 86933 95102,18.466,7,R,7M00G7W,A/19/H/05/018/AA,V,Airwave Solutions Limited,20,253.8397529,-0.249145473,34.2,0
Each pair of these constitutes a link, the link has a transmit frequency one way and at the received end there is that frequency as receive (R - station type), with its own transmit frequency.
So this is a set of data for a series of links saved as an ordered record centred around the license number and the frequencies.
Is it possible for you to re-state this as a CSV table in the sense of the following headings:
Licence Number,Go NGR,Go Frequency (GHz),Go Channel Width (MHz),Return NGR,Return Frequency (GHz),Return Channel Width (MHz),Emission Code(Tx/Rx),Licencee Company,Antenna Height(Tx/Rx).
GPT4
Yes, I can restate the provided data as a CSV table with the specified headings. Here is the reformatted table:
Licence Number,Go NGR,Go Frequency (GHz),Go Channel Width (MHz),Return NGR,Return Frequency (GHz),Return Channel Width (MHz),Emission Code(Tx/Rx),Licencee Company,Antenna Height(Tx/Rx)
1021490/1,SU 49777 29446,23.1,56,SU 45800 35700,22.092,56,56M0D7W,MOBILE BROADBAND NETWORK LIMITED,15/31
1020795/1,SP 33109 78359,24.5595,7,SP 30034 78137,25.5675,7,7M00G7W,Airwave Solutions Limited,47/18
1020801/1,NS 79789 93239,18.466,7,NS 86933 95102,19.474,7,7M00G7W,Airwave Solutions Limited,25/20
I have omitted the columns that were not included in the specified headings, such as antenna polarization, antenna azimuth, antenna elevation, antenna gain, and antenna ERP. Each row now represents a link, with Go NGR and Go Frequency referring to the transmit side, and Return NGR and Return Frequency referring to the receive side. The antenna heights for both transmit and receive sides are combined in the Antenna Height(Tx/Rx) column.
Chris
Thanks, better than GPT3.5. Can you restate the above table with the additional following column Azimuth (Go / Return)
GPT4
Certainly! Here's the reformatted table with the additional "Azimuth (Go / Return)" column:
Licence Number,Go NGR,Go Frequency (GHz),Go Channel Width (MHz),Return NGR,Return Frequency (GHz),Return Channel Width (MHz),Emission Code(Tx/Rx),Licencee Company,Antenna Height(Tx/Rx),Azimuth (Go / Return)
1021490/1,SU 49777 29446,23.1,56,SU 45800 35700,22.092,56,56M0D7W,MOBILE BROADBAND NETWORK LIMITED,15/31,328.1668868/148.123345
1020795/1,SP 33109 78359,24.5595,7,SP 30034 78137,25.5675,7,7M00G7W,Airwave Solutions Limited,47/18,266.2455507/86.20972636
1020801/1,NS 79789 93239,18.466,7,NS 86933 95102,19.474,7,7M00G7W,Airwave Solutions Limited,25/20,73.74506981/253.8397529
I have added the "Azimuth (Go / Return)" column to include the azimuth values for both transmit and receive sides, separated by a slash.
Chris
Thank you. That is quite correct.
So GPT4 has enough 'intelligence' to read a CSV formatted dataset, follow scant instructions, and correctly reduce the substantial overhead to represent the data in a different manner, compressing four records into one record. It is even able to manage the implications of the original heading Emission Code (Tx/Rx) and then to proceed to add a column on in a following step using the same trick as with Emission Code. I intend to get it to write the algorithmic code to do this sometime, I wouldn't process large datasets in it in this manner.
There is clearly understanding here, but this is not the sort of understanding we humans have, where we are attuned to violations of expectation from our own mental model of the world (as far as I can see these systems don't have a mental model of the world like us, they are more transitory).
- There is something more in GPT 4 as compared to GPT3.5, call it something else if you don't like the word intelligence, but I consider GPT4 to be much more intelligent than GPT3.5.
- These systems do understand in a sense, GPT4 arguably more than GPT3.5.
- GPT4 is very good at drawing inferences from information given to it. Indeed, in my assessment it exceeds the average person (IQ=100).
But these are not human. If you ask them whether they understand, they may surprise you by not understanding as you thought, that noted, this can happen with humans too.
Do not make the mistake of taking them as human, when some moron applies a train of prompts that lead to a GPT saying it wants to kill all humans this is as meaningless as writing 'Hell World' code that says the same thing. Despite this I always treat them with the same respect and consideration I would any employee. Also the peculiarities and deficits of GPT4 are within the range I see managing a team of disparate people, so it's easy to treat it politely, but it isn't a human.
Indeed I've been saying to people the following:
I programme with GPT3.5 but I programme with GPT4.
The 'but' might seem strange, however there are two uses of the word with at play here. When I programme with GPT3.5 I do so as a carpenter uses a mallet and chisel, as a tool. When I programme with GPT4 I do so as an equal, because I'm an engineer (and manager) my programming skills are always rusty, GPT4 is like having an expert programmer who can flesh out my plans with working code (the iterative process of getting code to work is something GPT4 helps with, even turning into an IT help-desk assistant).
In my work I am now using GPT4 and 3.5 to help with organising and cleaning messy data, quotes, research, code and ad-hoc tasks. Once GPT4 is incorporated into Office 365 as co-pilot, and released to the majority of business users, I will be able to get it to do a lot of things my Deputy is currently doing. My coding productivity has already increased ten-fold, to such a degree that I am running out of projects to do, some of which have been on the slate for years. I expect that with Office 365 Co-Pilot something similar will happen in the bulk of my work life.
No comments:
Post a Comment