Is Github Co-pilot worth the money? IMO spend elswhere.

TLDR: you might as well pay for a GPT-4 subscription instead and copy and paste stuff to your browser, you’ll spend time doing that in VS code anyway which negates the added convenience and you’ll probably get better results.

Why look at this?

After a couple of week of using copilot (and paying my first subscription fee, “because it makes me code faster”) I started to wonder, is this actually worth paying for? And are there other ways I can push my use of this tool beyond fixing single line syntax errors. Could it take the headache out of SQL to supabase-js query creation and save time?

What is github copilot?

Copilot is effectively ChatGPT by OpenAI resold by Github, possibly with some additional conditioning and nice formatting if you use it in VSCode with the provided extensions.

Use Case

To save time, I want to try and convert an SQL query I’ve already generated in DBVisualiser using Query Builder, on a simple test Postgres DB schema with an M-2-M relation in into a format described in the Supabase-js V2 docs.

sharpi_sh:

convert this SQL query to supabase-js format

SELECT schools.school_name, schools.school_postcode, students.student_name, students.student_title, students.student_id, schools.school_id FROM schools JOIN school_student_pairing ON schools.school_id = school_student_pairing.schoool_fid JOIN students ON school_student_pairing.student_fid = students.student_id WHERE LOWER(students.student_name) = 'sally gunnel';

What i got back…

Problem

What I get back from Copilot is useless. Utter garbage.

I suspect it was trained before the release of V2. (circa 16th Aug 2022)

Training Data, its out of date

Ok first step, I want to know when it was last trained, does it even know the latest SDK docs? So I asked ChatGPT first..

sharpi_sh:

When was your training cut-off?

ChatGPT-3.5:

The specific date of the training cut-off for my model wasn’t publicly disclosed by OpenAI. However, I can inform you that the training data includes information up to January 2022. Beyond that point, I do not have access to events or information that occurred.

So ChatGPT 3.5 is out of date. Note that GPT-4 has been trained up to December 2023 which could potentially include V2 updates to Supabase.

Github Hidden Truths

It seems that Github have deliberately hidden the training dates for their Copilot, with it flatly refusing to answer the same question.

As of my last update in January 2022, the last known version of the Supabase JavaScript SDK was 1.23.0. However, there may have been newer versions released since then.

As an AI developed by OpenAI, I don’t have the ability to access real-time data or updates. My responses are based on the data I was trained on, which includes a diverse range of sources, including public code repositories and documentation. However, I don’t have the ability to access or retrieve information about specific versions of libraries like the Supabase JS SDK from my training data. For the most accurate and up-to-date information, I recommend checking the official Supabase documentation or their GitHub repository.

Going to lengths to hide this suggest wreaks of filthy skulduggery. Now I feel like investigating….

Copilot:

As an AI developed by OpenAI, I don’t have real-time access to the internet or databases, so I can’t provide the specific differences between Supabase v1 and v2. My training data includes a diverse range of sources, including public code repositories… bla bla bla

Ok so what about jQuery versions that I know were out before GPT-3.5 was trained….

sharpi_sh:

Name some differences between jquery version 2 and version 3.0

Copilot:

Here are some of the major differences between jQuery 2.x and jQuery 3.0:

1. Deprecated Event Aliases: .load, .unload, and .error, deprecated since jQuery 1.8, are no more. Use .on() to register listeners.

A clear and simple answer with examples. It seems to be hiding the fact that it doesn’t know about the (upcoming) V2 release, replacing the date of training cut off with waffle about data “including a diverse range of sources”. Anyway this is tangential nonsense, the bottom line is it doesn’t have up to date docs and can’t reliably help.

Inline Edit – Fairly Useless

I will dig out examples of this next time I’m working on VScode and post here. While it can fix some simple anchor tags syntax error when I’m feeling lazy, I find the change acceptance UI really confusing and it will often reformat larger portions of code creating more problems. Solid examples to come.

Project Folder Access – Nope.

I followed the hype and tried Cpoliot expecting it to have access to all of my project files and be slightly better than using the ChatGPT web interface.

I was disappointed to find the chat window only takes the current file as context, so forget working on harder topics that cover multiple files.

I find that I often have to copy the relevant block of code and paste it into the chat window because describing the function name in the request and relying on the file to be passed as context won’t get a desired result. Eg.

sharpi_sh:

Change the only <Link> tag in this .tsx file to use anchor instead

Would get a generic response about how to reformat a generic link, rather than the specific code I need to change my <Link> and format all of its attributes correctly.

Where it is useful, the reign of StackOverflow is coming to an end.

I’m in the process of picking up TypeScript after a lifetime hiding out on the server side as a PHP developer who went all in with jQuery.

Trivial problems. Eg. syntax errors that are highlighted by VS code have a handy “explain this” which turns generic errors into a nice human readable explanation.  Where this has helped is saving minutes otherwise spent trawling through irrelevant details and the toxic atmosphere of people getting burned and downvoted on Stackoverflow. This is good if I can’t be bothered to google or read the docs. How much I learn from this easy solution is in question because TBH I don’t even read whats it’s done, just test and move on.

Simple reformatting that aids laziness like

sharpi_sh:

add

className="example"

to all cells (copy and paste table code into chat) in table

This feels good at times because it’s possible to write in broken English with incomplete sentences and it still understands, enabling me to fully bathe in the sloppiness.

Other uses Reformatting CSV to JSON

I’ll write more on this on a separate post but basic data reformatting tasks can be executed much faster with ChatGPT itself. Where it falls down is it won’t follow direct instructions consistently across every row, with a large dataset if a few rows need manual checking, the whole dataset cant’ be trusted.