Introduction

The popularity of ChatGPT is soaring, and now it’s time to witness its performance in the realm of cybersecurity tools. In this blog post, we will delve into the thrilling domain of cybersecurity and subject ChatGPT to rigorous testing. Come along as we uncover how this powerful language model can contribute to creating custom security solutions.

The problem

Secure Sockets Layer(SSL) is a cryptographic protocol that enables secure communication over the internet. It establishes an encrypted connection between a server and a client, protecting sensitive information from unauthorized access.

Securing website communications through SSL connections is paramount, demanding proper configuration and regular evaluation. In this project, we’re building a C# application that leverages the OpenSSL labs API to streamline this process.

The tool takes a list of domains together with a minimum allowed scores as input. It connects to the OpenSSL labs API to retrieve the SSL connection score for each domain. The tool then compares the obtained scores with the expected scores, simplifying the evaluation of SSL connection security.

Part I: building solution prototype.

Lets start by asking ChatGPT if it knows the SSL Labs API we are about to use:

This appears to be a promising start. Let’s proceed with the implementation of the solution:

So the tool actually generates the very first prototype:

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;

namespace SSLCertChecker
{
    class Program
    {
        static async Task Main(string[] args)
        {
            List<string> domains = new List<string>
            {
                "example.com",
                "google.com",
                "facebook.com"
            };

            foreach (var domain in domains)
            {
                var grade = await GetCertificateGrade(domain);
                Console.WriteLine($"Domain: {domain}, Grade: {grade}");
            }

            Console.ReadLine();
        }

        static async Task<string> GetCertificateGrade(string domain)
        {
            using (var httpClient = new HttpClient())
            {
                var apiUrl = $"https://api.ssllabs.com/api/v3/analyze?host={domain}";
                var response = await httpClient.GetAsync(apiUrl);
                var content = await response.Content.ReadAsStringAsync();
                // Parse the JSON response to get the grade
                // Here, you can use your preferred JSON parsing library (e.g., Newtonsoft.Json)
                // Extract the necessary information from the response and return the grade
                // Note that SSL Labs API has rate limits, so ensure you comply with them
                return "A+"; // Replace with the actual grade obtained from the response
            }
        }
    }
}


The initial impression is truly remarkable: the code complies and sends correct requests to the api; however, its current utility is limited due to the absence of code for parsing the API response. Let’s work towards enhancing the functionality:

Here is the full response:

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json.Linq;

namespace SSLCertChecker
{
    class Program
    {
        static async Task Main(string[] args)
        {
            List<string> domains = new List<string>
            {
                "example.com",
                "google.com",
                "facebook.com"
            };

            foreach (var domain in domains)
            {
                var grade = await GetCertificateGrade(domain);
                Console.WriteLine($"Domain: {domain}, Grade: {grade}");
            }

            Console.ReadLine();
        }

        static async Task<string> GetCertificateGrade(string domain)
        {
            using (var httpClient = new HttpClient())
            {
                var apiUrl = $"https://api.ssllabs.com/api/v3/analyze?host={domain}";
                var response = await httpClient.GetAsync(apiUrl);
                var content = await response.Content.ReadAsStringAsync();

                // Parse the JSON response to get the grade
                var json = JObject.Parse(content);
                var endpoints = json["endpoints"];
                if (endpoints != null && endpoints.HasValues)
                {
                    var firstEndpoint = endpoints[0];
                    var grade = firstEndpoint["grade"];
                    if (grade != null)
                    {
                        return grade.ToString();
                    }
                }

                return "N/A"; // Grade not available
            }
        }
    }
}

Wow! At this moment, I was genuinely impressed that the chatbot was able to generate this working prototype. While I do have several comments concerning implementation details, stability, and functionality, it is noteworthy that the prototype is functional and fulfils some of my initial requirements to a certain extent.

In order to transition from a prototype to an actual working solution, the following features would be beneficial:

  1. Incorporate a minimum allowed score per domain and ensure that the actual grade does not fall below that threshold.
  2. Integrate input retrieval from the appsettings.json file, enabling flexibility by avoiding a hardcoded list of domains.
  3. Implement proper handling of API responses, such as retrying the API call multiple times if the result is not readily available. This ensures a more robust and reliable operation.

Part II: Adding more Features and Bug Fixes.

I started guiding the bot to implement it feature by feature.

That was accomplished effortlessly. Let’s proceed with another task.

The overall performance was generally smooth, although there were some occasional bugs found:

It was impressive to witness how easily some bugs were fixed. However, when it came to the “score comparison” feature, it became quite challenging. The chatbot grasped my requirements effectively, but encountered numerous bugs while attempting to ensure that all the findings were addressed and resolved.

Here is a piece of the generated unit test code:

        static bool GradeMetMinimumRequirement(string grade, string minRequiredGrade)
        {
            return string.Compare(grade, minRequiredGrade, StringComparison.OrdinalIgnoreCase) >= 0;
        }

        // Unit tests for GradeMetMinimumRequirement function
        static void RunGradeComparisonTests()
        {
            Console.WriteLine("Running grade comparison tests...");

            // Test cases
            var testCases = new List<(string grade, string minRequiredGrade, bool expectedResult)>
            {
                ("A", "B-", true),
                ("B-", "B-", true),
                ("C", "B-", false),
                ("A", "A+", false)
            };

            // Run tests
            foreach (var (grade, minRequiredGrade, expectedResult) in testCases)
            {
                var result = GradeMetMinimumRequirement(grade, minRequiredGrade);
                Console.WriteLine($"Grade: {grade}, Min Required: {minRequiredGrade}, Expected: {expectedResult}, Result: {result}");
                Console.WriteLine(result == expectedResult ? "Test Passed!" : "Test Failed!");
                Console.WriteLine();
            }
        }

One intriguing aspect is that the provided tests are functioning, but failing (obviously due to an incorrect implementation of the GradeMetMinimumRequirement method). However, I encountered significant difficulties when attempting to instruct the bot to rectify this issue.

Another attempt:

And yet another one:

And still not complete…

I believe that a human would be better equipped to handle this particular fix more effectively.

Once I resolved this aspect of the functionality, I proceeded with code generation. However, a few issues arose during this process. Occasionally, the code regeneration would reintroduce previously fixed bugs or introduce new “silly” mistakes, like having two main functions. In summary, while the tool successfully generated certain portions of the functionality, working with the entire solution proved to be quite bothersome.

Here are a few instances that caused frustration:

After spending a few hours interacting with the bot, I am delighted to share one of the finest generated versions I came across…

Part III: Final Code Analysis

Presented below is the most functional version achieved thus far:

using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;

namespace SSLCertificateChecker
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var domains = LoadDomainsFromAppSettings();
            var certificateGrades = await GetCertificateGrades(domains);
            var allGradesMetRequirement = true;

            for (int i = 0; i < domains.Count; i++)
            {
                var domain = domains[i];
                var endpointGrades = certificateGrades[i];

                Console.WriteLine($"Domain: {domain.Domain}");
                Console.WriteLine($"Minimum Required Score: {domain.MinRequiredScore}");

                for (int j = 0; j < endpointGrades.Count; j++)
                {
                    if (j < domain.Endpoints.Count) // Ensure index is within bounds
                    {
                        var endpoint = domain.Endpoints[j];
                        var grade = endpoint != null ? endpoint.Grade.ToString() : "N/A"; // Check for nullity and retrieve the Grade property
                        Console.WriteLine($"Endpoint {endpoint.IpAddress} - {endpoint.ServerName}: Grade: {grade}");
                    }
                }

                var gradesOnly = endpointGrades.Select(endpoint => endpoint.Grade).ToList(); // Extract grades only
                var gradeMetRequirement = GradeMetMinimumRequirement(gradesOnly, domain.MinRequiredScore);
                Console.WriteLine($"Grade Met Minimum Requirement: {gradeMetRequirement}");

                if (!gradeMetRequirement)
                {
                    allGradesMetRequirement = false;
                }

                Console.WriteLine();
            }

            if (!allGradesMetRequirement)
            {
                Console.WriteLine("At least one site score does not meet the minimum requirement.");
                Environment.Exit(1); // Exit program with an error
            }

            Console.ReadLine();
        }


        static List<DomainConfig> LoadDomainsFromAppSettings()
        {
            var appSettingsPath = "appsettings.json";
            var appSettingsContent = File.ReadAllText(appSettingsPath);
            var appSettings = JsonConvert.DeserializeObject<AppSettings>(appSettingsContent);
            return appSettings.Domains;
        }

        static async Task<List<List<Endpoint>>> GetCertificateGrades(List<DomainConfig> domains)
        {
            var httpClient = new HttpClient();
            var certificateGrades = new List<List<Endpoint>>();

            foreach (var domain in domains)
            {
                var apiUrl = $"https://api.ssllabs.com/api/v3/analyze?host={domain.Domain}";

                var response = await httpClient.GetAsync(apiUrl);
                var responseContent = await response.Content.ReadAsStringAsync();

                var json = JsonConvert.DeserializeObject<dynamic>(responseContent);

                var attempts = 0;
                var maxAttempts = 5;
                var waitTimeSeconds = 5;

                while (attempts < maxAttempts)
                {
                    if (json != null && json.endpoints != null && json.endpoints.HasValues)
                    {
                        var endpoints = new List<Endpoint>();

                        foreach (var endpoint in json.endpoints)
                        {
                            var grade = endpoint?.grade?.ToString() ?? "N/A"; // Check for nullity and existence of grade property
                            var ipAddress = endpoint?.ipAddress?.ToString();
                            var serverName = endpoint?.serverName?.ToString();
                            var endpointInfo = new Endpoint
                            {
                                IpAddress = ipAddress,
                                ServerName = serverName,
                                Grade = grade // Add Grade property
                            };
                            endpoints.Add(endpointInfo);
                        }

                        domain.Endpoints = endpoints; // Set the endpoints in the domain object
                        certificateGrades.Add(endpoints);
                        break;
                    }

                    Console.WriteLine($"Waiting {waitTimeSeconds} seconds before retrying...");
                    await Task.Delay(waitTimeSeconds * 1000);

                    attempts++;

                    response = await httpClient.GetAsync(apiUrl);
                    responseContent = await response.Content.ReadAsStringAsync();

                    json = JsonConvert.DeserializeObject<dynamic>(responseContent);
                }

                if (attempts == maxAttempts || json == null)
                {
                    var endpoints = new List<Endpoint> { new Endpoint { IpAddress = "N/A", ServerName = "N/A", Grade = "N/A" } };
                    domain.Endpoints = endpoints; // Set the endpoints in the domain object
                    certificateGrades.Add(endpoints);
                }
            }

            return certificateGrades;
        }

        public static bool GradeMetMinimumRequirement(List<string> grades, string minRequiredGrade)
        {
            var gradeOrder = new Dictionary<string, int>
            {
                { "A+", 14 },
                { "A", 13 },
                { "A-", 12 },
                { "B+", 11 },
                { "B", 10 },
                { "B-", 9 },
                { "C+", 8 },
                { "C", 7 },
                { "C-", 6 },
                { "D+", 5 },
                { "D", 4 },
                { "D-", 3 },
                { "F+", 2 },
                { "F", 1 },
                { "F-", 0 }
            };

            var minRequiredGradeValue = gradeOrder.ContainsKey(minRequiredGrade) ? gradeOrder[minRequiredGrade] : -1;
            if (minRequiredGradeValue == -1)
            {
                Console.WriteLine($"Invalid minimum required grade: {minRequiredGrade}");
                return false;
            }

            var gradeMetRequirement = true;

            foreach (var grade in grades)
            {
                if (!gradeOrder.ContainsKey(grade))
                {
                    Console.WriteLine($"Invalid grade value: {grade}");
                    return false;
                }

                var gradeValue = gradeOrder[grade];
                if (gradeValue < minRequiredGradeValue)
                {
                    gradeMetRequirement = false;
                    break;
                }
            }

            return gradeMetRequirement;
        }
    }

    class AppSettings
    {
        public List<DomainConfig> Domains { get; set; }
    }

    class DomainConfig
    {
        public string Domain { get; set; }
        public string MinRequiredScore { get; set; }
        public List<Endpoint> Endpoints { get; set; }
    }

    class Endpoint
    {
        public string IpAddress { get; set; }
        public string ServerName { get; set; }
        public string Grade { get; set; }
    }
}

Please note that to conduct the test yourself, you will need to populate the appsettings.json file((don’t forget to copy it to the build output directory)) and install specific nugget packages. 

{
  "Domains": [
    {
      "Domain": "psolomatin.com",
      "MinRequiredScore": "A"
    },
    {
      "Domain": "google.com",
      "MinRequiredScore": "B+"
    },
    {
      "Domain": "facebook.com",
      "MinRequiredScore": "B-"
    }
]
}

With this input it will return the following output:

Domain: psolomatin.com
Minimum Required Score: A
Endpoint 31.31.198.201 - spl89.hosting.reg.ru: Grade: A
Grade Met Minimum Requirement: True

Domain: google.com
Minimum Required Score: B+
Endpoint 2607:f8b0:4007:810:0:0:0:200e - lax30s04-in-x0e.1e100.net: Grade: B
Endpoint 142.250.68.46 - lax17s46-in-f14.1e100.net: Grade: B
Grade Met Minimum Requirement: False

Domain: facebook.com
Minimum Required Score: B-
Endpoint 31.13.65.36 - edge-star-mini-shv-01-atl3.facebook.com: Grade: B
Endpoint 2a03:2880:f111:83:face:b00c:0:25de - edge-star-mini6-shv-01-atl3.facebook.com: Grade: B
Grade Met Minimum Requirement: True

At least one site score does not meet the minimum requirement.

The code appears to be workable, but not flawless. Typically, I divide my review comments into two categories: cosmetic and essential. Let’s follow that approach here as well. I have several cosmetic comments to provide, such as:

  • There are redundant “using” statements present, which can be removed.
  • Numerous obvious comments could be omitted to improve clarity.
  • The code does not adhere to a consistent code style.
  • There are several variables that are declared but only used once in the code.
  • The “if (j < domain.Endpoints.Count)” statement within the for loop seems peculiar.
  • The implementation of “GradeMetMinimumRequirement” appears to be awkward…

Regarding essential comments, the following points should be noted:

  • The code lacks granularity, with some methods performing multiple tasks at once. A specific instance is observed in the method “GetCertificateGrades,” which directly modifies “domains.endpoints.”. This also makes it difficult to write proper unit-tests.
  • Error handling is inadequate, as the code makes assumptions and fails to address potential issues. For example, what happens if “endpoint.IpAddress” is null?
  • The code violates SOLID principles, making it challenging to extend or modify. For instance, what if there is a need to change the output from console to another logger type?
  • Object-oriented programming (OOP) principles are not properly applied. I would advocate for handling null/default values in the constructors.

These essential comments highlight areas that require attention and improvement in the code.

In summary, I believe substantial refactoring will be necessary before deploying this code to production.

Conclusions

Overall, my feelings about this tool are quite mixed. On one hand, it is truly impressive to witness a tool that can capture requests in written form and generate working code while also offering guidance on bug fixes and handling exceptions. However, on the other hand, it can be extremely time-consuming, and at times it becomes evident that the machine lacks a deep understanding of its actions. This is evident when it reintroduces previously fixed bugs or inadvertently breaks existing functionality while attempting to address minor cosmetic issues. While the technology itself is impressive, it is unlikely to replace human developers in the near future. Particularly for projects that exceed 200 lines of code, the expertise and comprehension provided by human developers are still invaluable.

Thanks for reading this. Feel free to share.

P.S. Did you happen to notice that the text of this blog entry was also generated by a machine?

You might also like …

Building a Simple yet Effective Security Tool in C# using OpenSSL Labs API with ChatGPT Integration