Retrieving Open Graph data from a URL in C# and .NET Core

OpenGraph metadata is data that allows (among other things) creating a preview for a link when it's shared on the Internet.

More info in our article dedicated to OpenGraph: https://code-garage.fr/blog/comment-personnaliser-le-partage-de-votre-site-sur-les-reseaux-sociaux-avec-opengraph

If you handle links in your software, you probably want to be able to create a preview like this one:

Step 1: Prepare the project

To carry out the project successfully, you'll need two modules:

using System.Web; //Used to load the page
using HtmlAgilityPack; //Used to parse the page

For HtmlAgilityPack you'll need to install it via your NuGet package manager

Next, we'll prepare a simple data model to hold our Open Graph metadata, like this:

public record LinkPreview
{
    public int? Id { get; set; }
    public string Url { get; init; } = String.Empty;
    public string Title { get; init; } = String.Empty;
    public string Description { get; init; } = String.Empty;
    public string ImageUrl { get; init; } = String.Empty;
    public string SiteName { get; init; } = String.Empty;
    public DateTime LastUpdate { get; init; }
}

Then we'll declare our OpenGraphLoader class and its main function, which will take a url as input and return (if it exists) an instance of LinkPreview:

public class OpenGraphLoader 
{
	readonly HttpClient _client;

	public OpenGraphLoader ()
    {
		// Initialize our HTTP client
		_client = new HttpClient();
    }

	public async Task<LinkPreview?> GetLinkPreview(string url)
	{
		return null;
	}
}

Our class will contain an instance of an HTTP client so we can make the request to fetch the web page.

For now, the main function is empty, and we're now going to write its logic!

Step 2: Load the web page

To load the HTML content of our web page, we'll need to use the HTTP client initialized earlier, and pass it our URL.

The response content is simply a sequence of bytes, so we'll need to convert these bytes back into a String to retrieve our HTML code:

public async Task<LinkPreview?> GetLinkPreview(string url)
{
	// Execute the request to the web page's url
	var response = await _client.GetAsync(url);
	// Parse the response body (the page) as a string
	var html = await response.Content.ReadAsStringAsync();

	/* 
		Here we will parse 
		our metadata 
	*/

	return null;
}

All that's left now is to dig through this HTML code in search of our metadata and their values

Step 3: Parse the metadata

For this step we'll use the HtmlAgilityPack package, which will allow us to load the HTML code and turn it into a real HTML document that we can browse through:

// Load the HTML content into a HtmlDocument
var doc = new HtmlDocument();
doc.LoadHtml(html);

Then we'll be able to select the HTML elements we're interested in (the famous OpenGraph tags), and retrieve their values one by one:

// Sélectionne tous les éléments <meta> avec l'attribut property="og:*"
var metaTags = doc.DocumentNode.SelectNodes("//meta[starts-with(@property, 'og:')]");

// Créer un dictionnaire pour stocker tous les noms des éléments et leur valeur
var openGraphData = new Dictionary<string, string>();

// Extrait les données et les ajoute au dictionnaire une par une
if (metaTags != null)
{
    foreach (var tag in metaTags)
    {
        var property = tag.GetAttributeValue("property", "");
        var content = tag.GetAttributeValue("content", "");
        if (!string.IsNullOrEmpty(property) && !string.IsNullOrEmpty(content))
        {
            // Supprime le prfixe 'og:' du nom de la propriété
            property = property.Replace("og:", "");
            openGraphData[property] = content;
        }
    }
} else {
	return null;
}

If no og:* tag is found, the function will return null, but you may choose to return an empty LinkPreview instead!

And finally, all that's left is to pass all this data to our LinkPreview object (if metadata was found) and return it at the end of our function:

return new LinkPreview()
{
    Url = url,
    Title = openGraphData["title"],
    Description = openGraphData["description"],
    ImageUrl = openGraphData["image"],
    SiteName = openGraphData["site_name"],
    LastUpdate = DateTime.Now
};

And there you go! You can now retrieve the Open Graph tags from any web page (provided they exist)!

And of course, you can adapt this code to retrieve other information present on the page.

The complete code

Here's the entire code for this tutorial, with all the comments:

using System.Web; //Used to load the page
using HtmlAgilityPack; //Used to parse the page

namespace App;

public record LinkPreview
{
    public int? Id { get; set; }
    public string Url { get; init; } = String.Empty;
    public string Title { get; init; } = String.Empty;
    public string Description { get; init; } = String.Empty;
    public string ImageUrl { get; init; } = String.Empty;
    public string SiteName { get; init; } = String.Empty;
    public DateTime LastUpdate { get; init; }
}

public class OpenGraphLoader 
{
	readonly HttpClient _client;

	public OpenGraphLoader ()
  {
		// Initialize our HTTP client
		_client = new HttpClient();
  }

	public async Task<LinkPreview?> GetLinkPreview(string url)
	{
		// Execute the request to the web page's url
		var response = await _client.GetAsync(url);
		// Parse the response body (the page) as a string
		var html = await response.Content.ReadAsStringAsync();

		// Load the HTML content into a HtmlDocument
		var doc = new HtmlDocument();
		doc.LoadHtml(html);

		// Select all meta tags with property attribute starting with 'og:'
		var metaTags = doc.DocumentNode.SelectNodes("//meta[starts-with(@property, 'og:')]");
		
		// Create a dictionary to store OpenGraph metadata
		var openGraphData = new Dictionary<string, string>();
		
		// Extract content from meta tags and add it to the dictionary
		if (metaTags != null)
		{
		    foreach (var tag in metaTags)
		    {
		        var property = tag.GetAttributeValue("property", "");
		        var content = tag.GetAttributeValue("content", "");
		        if (!string.IsNullOrEmpty(property) && !string.IsNullOrEmpty(content))
		        {
		            // Remove 'og:' prefix from property
		            property = property.Replace("og:", "");
		            openGraphData[property] = content;
		        }
		    }
		} else {
			return null;
		}
	
		return new LinkPreview()
		{
		    Url = url,
		    Title = openGraphData["title"],
		    Description = openGraphData["description"],
		    ImageUrl = openGraphData["image"],
		    SiteName = openGraphData["site_name"],
		    LastUpdate = DateTime.Now
		};
	}
}

You can directly copy-paste this code, but if you don't understand certain parts, remember to re-read the whole tutorial!

Before finishing up

Each url you preview may take more or less time to load, which can slow down your software.

You could of course make this process non-blocking, but that's not always possible depending on your use case!

The ideal approach is to store the result of these calls in a database (using the url as identifier), to create a temporary cache, and minimize the number of requests (and thus the execution time) for calls to this function!