Signal Sciences Web Application Firewall /
basic terms and 101

In this article Curious about Signal Sciences Web Application Firewall? Let's break it down! This article will guide you through the essentials of Signal Sciences Web Application Firewall.

Signal Science Signal :

A Signal is a descriptive tag about a request, a request gets one or multiple tags based on rules a request triggers.

Defined Systems signals

Attack Signal

Triggered by malicious request attack payloads, to hack, destroy, disable, steal…

Here’s an example of such request with a URL with an attempt of SQL Injection :

https://www.example.com/search?q=SELECT+*+FROM+users+WHERE+username='$username'

Anomaly Signal

These signals indicate unusual or potentially malicious activities that don’t fit into specific attack categories, but are still suspicious.

Examples include:

like https://example.com/wp-admin

UTF-8: A common exploit vector involves sending invalid byte sequences that break UTF-8 encoding rules. For example, a lone byte %C3 which is not followed by a valid continuation byte.

Request example :

POST /login HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 40

username=%27%20OR%201=1;%20--%20&password=test

Here, %27 represents an apostrophe, which could be part of a SQL injection payload.

Exploit: The application might concatenate the injected payload directly into a SQL query, leading to SQL injection.

Like incorrect JSON data send to an API in order to cause the server to throw an error, potentially leading to a denial of service if the server does not handle such errors gracefully. In some cases, improperly handled parsing errors could be exploited to reveal internal server information or cause unexpected behavior.

Both Attack Signals and Anomaly Signals are referred to as System Signals, as they are built into the system.

SigSci Custom Signals

Defined by users, are tagged to requests that match user-defined conditions.

Custom signals are created at either the corporation level (Corp Signals) or the site level (Site Signals). They can be used to tag specific requests based on custom rules. This tagging helps in identifying and managing unusual or malicious traffic more effectively.

Examples :

Blocking Web Crawlers:

Tagging Anonymous Email Domains :

Thresholds

Thresholds determine when to trigger an action. For example, a threshold could be « 10 or more requests within 60 seconds. » Default attack thresholds are provided but can be customized. The default thresholds are:

Signal Science Rules :

Rules are settings that define conditions to block, allow, or tag requests either permanently or for a certain period.

System Rules

System rules are predefined in the tool, when applied they can generate System Signals for an Attac or Anomaly.

Request Rules

These rules let you set specific conditions to block, allow, or tag requests either permanently or for a certain time. In the UI, these rules are grouped together.

Type :

Conditions :

Actions :

Select the Action(s) that will happen when the conditions are met

Exclusion Rules

Signal Exclusion Rules let users specify which requests should have System Signals tags removed from them. This helps reduce false positives. For instance, you can remove the XSS Attack Signal from endpoints where users are expected to submit valid scripts.

Rate Limit Rules

Rate Limit Rules allow users to set conditions to either block requests or log them with a rate-limit signal if they exceed a specified threshold within a certain time window. For instance, a threshold could be « 10 or more requests within 60 seconds. » If any IP sends 10 or more requests within the last 60 seconds, it triggers related rules, such as Custom Signals or blocking requests from that IP. Many of our rules rely on thresholds like this for their logic.

Rate Limit applies at the Site Level

Templated Rules

Partially pre-made rules. When you fill them out, they let you block requests, flag IPs for blocking or logging, or tag specific types of requests. These rules are useful for managing registrations, logins, and virtual patches in customer applications by setting up simple rules. There are three types of templates: API, ATO (Account Take Over), & CVE (Common Vulnerabilities and Exposures).

Systems Alters and Site Alerts

System Alerts are preset thresholds for the number of requests with attack signals over time. You can adjust these thresholds in your Site Settings under Attack Thresholds. This rule lets users pick any attack, anomaly, or custom signal and set a threshold and interval for when to flag and alert, and optionally block subsequent attacks from an IP.

SigSci’s architecture

SigSci is built to be flexible. This means you can choose where to put the software that deals with incoming requests. This software comes in two parts: Modules and Agents. They work together to sort through requests and block bad ones.

Step 1

A request arrives, then the barrel man (module) detects the request and informs the captain (agent) about the request.

The module (barrel man) : Receives the Request, Sends Req to Agent and Enforces Agent Decision

Step 2

The captain (agent) asks the Admiralty (sites) what should be done with that request, intercept it, let it though.

The agent (Captain) : Applies Rules, Generates Signals, Decide Allow/Block

The sites (Admiralty) : Those Signals and a host of other data will migrate up to the Site level and the Corp level.

Corp level

Starting with the Corp, this is the customer account. Collected here are settings for common account elements like Access Tokens and User Management, along with Signal Sciences specifics like Corp Monitoring, Corp Rules, and Sites. Corp Monitoring allows users to see metrics at an account level.

The Corp Rules are all the user-defined rules, as discussed earlier, that will be applied across the Corp via each Site

Within each Corp, users can set up multiple Sites. A Site, aka workspace, can be for a single web application, a bundle of web applications, API, or microservice that Signal Sciences can protect from attacks.

Corp – Sites

Like with Corps, Sites can have their own level of Rules that can be configured. The Site’s Rules are bundled with the Corp’s Rules and passed to the linked Agents. Agents send back data to pass through the Site for Site level monitoring, which could then be rolled up into Corp level Monitoring too.

Agent Mode

Blocking Mode : Applying rules to : Block request yes – Log request yes

Logging Mode : Applying rules to : Block request no – Log request yes

Off Mode : Applying rules to : Block request no – Log request no

All agents under a site can be set to one of these modes. When an Agent is set to Off Mode, the linked Module will fail-open ensuring traffic is not blocked.

Monitor attacks

Flagged IPs will stay flagged for 24 hours, unless configured otherwise. Even if the number of requests with attack signals falls below the threshold, the IP will stay Flagged until the configure duration finishe

We’re exploring the Error, Restart, and Log subroutines, but a heads-up: Restart isn’t technically a subroutine in the usual sense.

For our learning objectives, let’s focus on understanding these elements:

Where they fit into the overall process
What they’re supposed to do
How they interact with each other and other parts of the system.

VCL_error

How requests come in VCL_error ?

As you can see from the diagram, we can call vcl_error from almost everywhere, except for the deliver and log processes.You can spot this on the diagram by looking for the red arrows.

How requests come out of VCL_error ?

The default exit for vcl_erorr is return(deliver), or it can also be return(deliver_stale) if there’s stale content. And restart to return back to recv

VCL_error use cases

Here’s some uses cases for VCL_error :

Synthetic responses

A synthetic response is one that Varnish can generate without needing to connect to your origin server. For example, it can create error responses like 404, 500, or redirects like 301. Typically, with synthetic responses, we create a custom HTML page that gets stored in our VCL file.

Here’s an example, in vcl_recv I trigger an error 600 if req.url.path == "/anything/404"

sub vcl_recv {
  #FASTLY recv

  if(req.url.path == "/anything/404"){
    error 600;
  }


  return(lookup);
}

🔍 Maybe you are wondering where this 600 code comes from…

The status codes used for Error handling in Fastly typically fall into specific ranges. Fastly reserves the 800 range for internal codes, which is why the pre-built « Force TLS Error » uses code 801. Developers should use codes in the 600 to 700 range when triggering Error from other subroutines.

Then we end up in the error subroutine, where I can build my synthetic response :

sub vcl_error {
  #FASTLY error

  if(obj.status == 600){
    set obj.status = 404; 
    set obj.http.Content-Type = "text/html";
    synthetic {"
    <html>
        <head>
            <meta charset="UTF-8">
        </head>

        <body>
                <h1>Not Found 🤷‍♂️ </h1>
        </body>
    </html>
    "};
  }

  return(deliver);
}

VCL_error recap :

Restart

Firstly Restart is not a subroutine, it’s not on list. Restart it’s its own transition call

How restart works :

• Restart goes back to the start of vcl_recv.
• Restart goes back to the Delivery Node.
• Limited to THREE restarts per POP.
• Restart disables : Clustering and Shielding
• Benefits of Disabling Clustering and Shielding.

Restart goes back to the Delivery Node

As you can see here, when a restart occurs, we go back to the delivery node :

Limited to THREE restarts per POP

With shielding ON you can restart up to 6 time, 3 time on one edge pop and 3 time on the shielding one.

Clustering and shielding with restart

We get a request, it hits one ofour Edge POPs, it gets assigned to a Delivery Node, Delivery Node goes through a
whole process of finding the Fetch Node on the Edge POP, Edge POP doesn’thave it, goes to the Shield POP, rinse and repeat for Delivery and Fetch.

Step 1

So our Fetch Node on the Shield POP makes the request, the origin gets back a 5xx error,either a 500 or a 503. (We are then a step one (1) in the diagram)

Step 2

You can see that logic there in vcl_fetch. We see the 5xx, and the Fetch Nodes says, « Ah, we got to restart. » So now it’s headed back to theDelivery Node on the Shield POP.

The Delivery Node then runs through VCL and makes its own attempt to reach the origin so it’s a fallback. We get the same 5xx error, and the Delivery Node goes, « All right. Well, I can’t restart anymore. »

Step 3

It passes it back to the Edge POP, the Fetch Node runs vcl_fetch on the response, sees again the 503 that was handed back, restarts, Delivery Node now attempts to make its own connection and gets back the last 503 or 500 error. You then, if there’s any logic to rewrite it or anything else, clean it up with a synthetic response, you hand off the eventual response to the end user

NB : in this step (3) the delivery nod eon the edge pop goes directly to the origin because we now already that the path though the shield node already failed.

Use cases for Restart

VCL_log

« `vcl_log«  is a function used to log messages for debugging, monitoring, or auditing purposes. It allows Varnish administrators and developers to track the behavior of the Cache by recording specific events, actions, or errors.

vcl_log is a function used in Varnish Cache’s configuration language (VCL) to log messages for debugging, monitoring, or auditing purposes. It allows Varnish administrators and developers to track the behavior of Varnish Cache by recording specific events, actions, or errors.

Here are some situations where vcl_log can be useful:

  1. Debugging: When troubleshooting issues with Varnish Cache, vcl_log can be used to output relevant information to the log files. This can help in diagnosing problems such as cache misses, backend connection issues, or incorrect VCL configurations.

  2. Monitoring: By strategically placing vcl_log statements in the VCL code, administrators can monitor the flow of requests through Varnish Cache. This can provide insights into traffic patterns, request handling times, and cache hit rates, aiding in performance optimization and capacity planning.

  3. Auditing: vcl_log can also be used to log specific events or actions for auditing purposes. For example, logging requests that match certain criteria, recording cache invalidations, or tracking user sessions can help maintain accountability and ensure compliance with security or regulatory requirements.

  4. Custom Logging: VCL allows for flexible customization of logging based on specific requirements. With vcl_log, administrators can define their own log formats and selectively log relevant information, tailoring the logging behavior to suit the needs of their infrastructure and applications.

How requests come in VCL_log ?


The transition call from deliver into log is return(deliver)

How requests come out of VCL_log ?

The transition call to end vcl_log is also return(deliver)

Conclusion

Okay, let’s go over what we’ve learned. First, Error lets us use fake responses on the Edge POPs. When we Restart, it sends the request back to Recv on the Delivery Node and turns off Clustering and Shielding. Each POP can only Restart three times (three on the Edge, three on the Shield). Finally, Log runs at the end, so you can set up and gather logging information.

How to install Terraform

On a Mac :

brew install hashicorp/tap/terraform

Verify Terraform is set up :

terraform -help

Create a the main terraform file

Let’s create our terraform project in his own folder :

mkdir terraform-tutorial && cd terraform-tutorial

Then let’s create our main.tf

touch main.tf

Add Fastly as a provider :

terraform {
  required_providers {
    fastly = {
      source  = "fastly/fastly"
      version = ">= 5.7.3"
    }
  }
}

Add a resource :

In the following code you can personalize the demo_tutorialpart the name and the domain name.

domain

For the domain name you can pick what ever you want if you want Fastly to create a temp domain name for you, but you have to follow the format : <name>.global.ssl.fastly.net

force_destroy = true

In Terraform, the force_destroy argument is used in conjunction with certain resources, typically those that manage persistent storage or long-lived infrastructure components. When force_destroy is set to true, it instructs Terraform to destroy the resource even if it has non-Terraform-managed changes or it’s not empty.

For example, if you have an AWS S3 bucket managed by Terraform and you want to delete it, but it still contains objects, Terraform will refuse to delete it by default because it wants to prevent accidental data loss. However, if you set force_destroy to true, Terraform will proceed with the deletion, permanently removing all data from the bucket and then destroying the bucket itself.

The backend block is for the address of your origin :

resource "fastly_service_vcl" "demo_tutorial" {
  name = "Super tutorial"

  domain {
    name    = "antoine-super-tutorial.global.ssl.fastly.net"
  }

  backend {
    address = "httpbin.org"
    name    = "Httbin dummy server"
  }


  force_destroy = true


}

Add an output

In Terraform, the output block is used to define values that should be exposed to the user after Terraform has executed its operations. These values can be useful for displaying important information or passing data between Terraform configurations.

output "active" {
  value = fastly_service_vcl.demo_tutorial.active_version
}

Add the Fastly API Key

provider "fastly" {
  api_key = "NyVYPuAb2Jb3nu_tsQblrMtmk-gw-oBd"
}

The complete main.tf

terraform {
  required_providers {
    fastly = {
      source  = "fastly/fastly"
      version = ">= 5.7.3"
    }
  }
}


# Configure the Fastly Provider
provider "fastly" {
  api_key = "NyVYPuAb2Jb3nu_tsQblrMtmk-gw-oBd"
}



resource "fastly_service_vcl" "demo_tutorial" {
  name = "Super tutorial"

  domain {
    name    = "antoine-super-tutorial.global.ssl.fastly.net"
  }

  backend {
    address = "httpbin.org"
    name    = "Httbin dummy server"
  }



  force_destroy = true


}

output "active" {
  value = fastly_service_vcl.demo_tutorial.active_version
}

Deploy the project on Fastly

terraform init

Terraform relies on plugins called providers to interact with various cloud platforms, APIs, and services. When you run terraform init, it downloads the necessary provider plugins specified in your configuration files and installs them locally. This ensures that Terraform has the required tools to manage your infrastructure.

Run :

terraform init 

tarraform plan

The terraform plan command is used in Terraform to create an execution plan. It essentially simulates the execution of the configuration and displays what actions Terraform will take to achieve the desired state defined in the configuration files

terraform plan

tarraform apply

terraform apply is a command used in Terraform to apply the changes specified in your Terraform configuration to your infrastructure.Terraform executes the actions outlined in the execution plan. This may involve creating new resources, updating existing ones, or destroying resources that are no longer needed.

terraform apply

Add a custom VCL

In the resource fastly_service_vcl we can specify a vcl block to add a custom vcl as specified in the doc

    vcl {
        name    = "my_custom_main_vcl"
        content = file("${path.module}/vcl/main.vcl")
        main    = true
    }

name : the name of your VCL
content : The path to your vcl code. In my case my vcl is stored in a vcl folder and in a main.vcl file.
main : is it the main vcl ?

The complete resource block :

resource "fastly_service_vcl" "test_service" {
    name = "An Example Service"

    domain {
    name = "hello-antoine.global.ssl.fastly.net"
    }

    backend {
    address = "httpbin.org"
    name= "My Test Backend"
    }

    vcl {
        name    = "my_custom_main_vcl"
        content = file("${path.module}/vcl/main.vcl")
        main    = true
    }

    force_destroy = true

}

The custom vcl

the vcl_recv subroutine. This subroutine is executed whenever Varnish receives an incoming HTTP request.

  if (req.url.path == "/anything/here") {
    set req.http.X-CustomHeader = "example";
  }

This if block checks if the request URL path is exactly /anything/here. If it is, then it sets a custom HTTP header named « X-CustomHeader » with the value « example » in the request. This header modification allows for customization of the request handling based on the requested URL.

if (req.url.path == "/anything/not/found") {
    error 600;
}

This if block checks if the request URL path is exactly « /anything/not/found ». If it is, then it generates a synthetic error with status code 600

sub vcl_error { }

This VCL (Varnish Configuration Language) code snippet defines a subroutine vcl_error, which is executed when an error occurs during the processing of a request


if (obj.status == 600) { }

This block of code generates a synthetic response body using the synthetic keyword. It returns a simple HTML document containing an UTF-8 charset meta tag and a « Not Found » message in an <h1> element. This synthetic response is sent to the client when the error condition specified in the if block is met.

Our complete VCL

// vcl/main.vcl

   sub vcl_recv {
    #FASTLY recv

        if (req.url.path == "/anything/here") {
            set req.http.X-CustomHeader = "example";
        }

        if (req.url.path == "/anything/not/found") {
            error 600;
        }

        return(lookup);
    }

    sub vcl_error {
        #FASTLY error
        if (obj.status == 600) {

            set obj.status = 404;
            set obj.http.Content-Type = "text/html";
            synthetic {"
                <html>
                    <head>
                        <meta charset="UTF-8">
                    </head>

                    <body>
                            <h1>Not Found 🤷‍♂️ </h1>
                    </body>
                </html>
            "};

            return(deliver);
        }
}

To deploy as said before :

terraform plan

And then

terraform apply

Diagram and project goal

In this article, my goal is to use compute at edge from Faslty to protect my origin and cache my content. Instead of using Varnish Configuration Language (VCL) to set my caching rules I will use JavaScript with compute.

Start a new compute / project

To start a new compute project, you will need to have the Faslty CLI installed

fastly compute init

After hitting the above command, you will get prompted few questions

Author (email): [antoine@gmail.com]

Language:
(Find out more about language support at https://www.fastly.com/documentation/guides/compute)
[1] Rust
[2] JavaScript
[3] Go
[4] Other ('bring your own' Wasm binary)

👉 option: 2

The started kit :

Starter kit:
[1] Default starter for JavaScript
    A basic starter kit that demonstrates routing, simple synthetic responses and overriding caching rules.
    https://github.com/fastly/compute-starter-kit-javascript-default

👉 Default starter for JavaScript

Post init :


INFO: This project has a custom post_init script defined in the fastly.toml manifest: npm install Do you want to run this now? [y/N]

👉 yes

JavaScript project / setup

npm install @fastly/expressly

Add a dev command in the npm scripts :

To add the « dev » script to your package.json file, simply insert the following line within the « scripts » object:

  "scripts": {
    "dev": "fastly compute serve --watch"
  }

💡 By default the port used is 7676 if you want to use an other one use the addr flag like so :


fastly compute serve --watch --addr=127.0.0.1:7777

This will allow you to run the command npm run dev to start your JavaScript projector using Fastly Compute.

Import Expressly / to manage the routes

Expressly is a router similar to Express in Node.js, If you ever used express Expressly will feel very confortable to use. You can check the documentation here

In /src/index.js add :

import { Router } from "@fastly/expressly";

const router = new Router();


router.get("/", async (req, res) => {

  res.send("hello");

});


router.listen();

This should return hello when you visit : http://localhost:7676

Publish on Faslty / and connect your backend

npm run deploy

Or

fastly compute publish

Accept and add a backend in my case I added :

fast.antoinebrossault.com

With the name :

fast_antoinebrossault_com

Add the backends in the .toml file

At the root of your project, you should have a fastly.tom file. In this file, add your backends if it’s not already the case.

Here I have my local backend and my production one

[local_server]
  [local_server.backends]
    [local_server.backends.fast_antoinebrossault_com]
      override_host = "fast.antoinebrossault.com"
      url = "https://fast.antoinebrossault.com"

[setup]
  [setup.backends]
    [setup.backends.fast_antoinebrossault_com]
      address = "fast.antoinebrossault.com"
      port = 443

Check your host config in the UI

Mine looks like this, If you get 421 Misdirected Request Error with Fastly compute, double-check the SNI hostname and certificate hostname configuration part.

Visit the url to check your deployment

If everything worked, you can visit the URL you got in the CLI to see the same result we had locally but this time on Fastly.

https://likely-destined-satyr.edgecompute.app/

Query / your backend

Modify the route we created earlier with :

router.get("/", async (req, res) => {

    const beResp = await fetch(req, {
        backend: "fast_antoinebrossault_com"
    });

    res.send(beResp);

});

Here this code query our backend and send back the result.

I your backend sends back an HTML page, the page will be broken as all the CSS and JavaScript files are not intercepted by our routes.

Let’s add routes to handle CSS and JavaScript files

This code defines a route in a router that handles requests for specific file types (JPEG, PNG, GIF, JPG, CSS, JS) and responds with the fetched content from a specified backend server (« fast_antoinebrossault_com »).

Let’s add this to our code :

router.get(/\.(jpe?g|png|gif|jpg|css|js)$/, async (req, res) => {
  res.send(await fetch(req, {
      backend: "fast_antoinebrossault_com"
  }));
});

Now our images, CSS and JavaScript files work, and all others files matching our regex will work too :

Another feature is broken on our site, this page fetches a Joke with an AJAX request to the API and insert it into the page.

Let’s handle front-end API calls

Let’s add another route to manage our API calls :

Let’s use this new route to handle the requests to the API, I created a tour to manage all the request which go to the API .

This regular expression (/^.*\/api\/.*$/ matches any string containing « /api/ » within it, regardless of what comes before or after « /api/ ».

router.get(/^.*\/api\/.*$/, async (req, res) => {
    res.send(await fetch(req, {
        backend: "fast_antoinebrossault_com"
    }));
});

Our API call now works :

We can now publish our changes with :

npm run deploy

As we can see, everything now works like a charm :

Add caching

We want Fastly to cache our content for a specific amount of time, let’s do this in our compute project.

To cache our URLs we can use CacheOverride

First import it :

import { CacheOverride } from "fastly:cache-override";

No need to install anything with npm install as cache-override exists in the fastly library we already have in our project.

We want to cache our home page for 50sec to do so we adapt the code inside our route :


router.get("/", async (req, res) => { const beResp = await fetch(req, { backend: "fast_antoinebrossault_com", // cache this request for 50 sec cacheOverride: new CacheOverride("override", { ttl: 50 }) }); res.send(beResp); });

Now we re-deploy with npm deploy to see if it worked

It worked 🎉 ! As you can see, this request is now cached :

antoine@macbook / % curlHeaders https://likely-destined-satyr.edgecompute.app/
HTTP/2 200
date: Mon, 08 Apr 2024 18:16:41 GMT
server: Apache/2.4.57 (Debian)
x-powered-by: Express
content-type: text/html; charset=utf-8
x-served-by: cache-mrs10549-MRS
etag: W/"b1e-A4dLyj+Lkq4bnJSZB7a7fCcwunw"
vary: Accept-Encoding
age: 1 👈👈
accept-ranges: bytes
x-cache: MISS 👈👈
x-cache-hits: 0 👈👈
content-length: 2846

antoine@macbook / % curlHeaders https://likely-destined-satyr.edgecompute.app/
HTTP/2 200
date: Mon, 08 Apr 2024 18:16:41 GMT
server: Apache/2.4.57 (Debian)
x-powered-by: Express
content-type: text/html; charset=utf-8
x-served-by: cache-mrs1050108-MRS
etag: W/"b1e-A4dLyj+Lkq4bnJSZB7a7fCcwunw"
vary: Accept-Encoding
age: 9 👈👈
accept-ranges: bytes
x-cache: HIT 👈👈
x-cache-hits: 1 👈👈
content-length: 2846

We can see the first request is a x-cache: MISS and the second is a x-cache: HIT. It will remain in the cache until the age reaches 50, which is the TTL we put in the code.

If you are wondering what command I use to only get the headers with curl, I use this in my bash profile :

# curl-headers
#
# will return the headers only 
# @example : curl-headers <url> 
curl-headers() {
    curl -sSL -D - $1 -o /dev/null
}

alias curlHeaders="curl-headers";

Using this method to put something in cache is no different from using VCL, just nicer and more elegant. To prove it, you can go to the Faslty UI and check if the URL is in the cache :

How to add custom headers ?

To add custom headers, it’s dead simple, let’s add headers to the response we get from the home page.

To do so simply use the headers.set() method on a backend response like so :


router.get("/", async (req, res) => { // Forward the request to a backend. const beResp = await fetch(req, { backend: "fast_antoinebrossault_com", cacheOverride: new CacheOverride("override", { ttl: 50 }) }); beResp.headers.set('Hello', "Is it me you are loooking for ?"); res.send(beResp); });

And it works ! 🎉

antoine@macbook / % curlHeaders curlHeaders https://likely-destined-satyr.edgecompute.app/
HTTP/2 200
date: Mon, 08 Apr 2024 18:36:04 GMT
server: Apache/2.4.57 (Debian)
x-powered-by: Express
content-type: text/html; charset=utf-8
hello: Is it me you are loooking for ? 👈👈👈
etag: W/"b34-1zh57py/zCElqztfbzqM3oXO/A4"
vary: Accept-Encoding
age: 6
accept-ranges: bytes
x-cache: HIT
x-cache-hits: 1
x-served-by: cache-mrs1050097-MRS
content-length: 2868

As you can see we have a new header in the response : hello: Is it me you are loooking for ?

Use Config stores / a key value pair storage

Fastly Config stores act like mini-databases at the edge of their network. They store frequently used configuration settings (like feature flags or A/B testing values) that your edge applications can access quickly and easily. This allows you to manage configurations centrally and deliver them to your services with minimal delay.

Create a config store

Here I created a config store named Features

Here you can add your key value data.

I createad a key html_404 with as a value a bit of HTML representig a 404 page. My goal is to use this HTML code asa synthetic response.

Use the Config Store

It’s important to kepp in mind the local code we run with npm run dev doesn’t have access to our backend in the same way our deployed code does. That’s why for the backend end we have a local one specified

Local

[local_server]
  [local_server.backends]
    [local_server.backends.fast_antoinebrossault_com]
      override_host = "fast.antoinebrossault.com"
      url = "https://fast.antoinebrossault.com"

[setup]
  [setup.backends]
    [setup.backends.fast_antoinebrossault_com]
      address = "fast.antoinebrossault.com"
      port = 443

It’s the same idea for config store, here’s the same data used in my config store but stored locally in my fastly.toml file :


[local_server.config_stores.Features] format = "inline-toml" [local_server.config_stores.Features.contents] "html_404" = '<!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><title>404 Not Found</title><link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"></head><body class="bg-gray-100"><div class="flex items-center justify-center h-screen"><div class="text-center"><h1 class="text-4xl font-bold text-gray-800">404</h1><p class="text-lg text-gray-600">Page Not Found</p></div></div></body></html>'

Query the config store from our code :

First thing first import the module for Config Store :

import { ConfigStore } from 'fastly:config-store';

Then Initialize the config store and get the value :

  const config = new ConfigStore('Features');
  const value = config.get('html_404')

I created a /404route to demonstrate this :

router.get('/404', async (req,res) => {

  const config = new ConfigStore('Features');

  res.withStatus(404).html(config.get('html_404'));

});

It worth noting I send a 404 status code with https://expressly.edgecompute.app/docs/handling-data/response#reswithstatus And instead of setting the headers for HTML myself I used .html()

You can now deploy it to see the result live.

Add geolocation / personalize the content based on location

Identifying user location through IP can enhance website experience by offering localized content, relevant services, and targeted ads. It enables personalized recommendations based on regional preferences, language, and cultural nuances, fostering user engagement and satisfaction. Additionally, it aids in fraud prevention and compliance with geo-specific regulations.

Import our getGeolocationForIpAddress function from the package :

import { getGeolocationForIpAddress } from "fastly:geolocation"

Add a new route :

router.get("/geo", async (req, res) => {
  if(!req.ip) res.withStatus(403).json({error: "No ip set"});
  res.json(getGeolocationForIpAddress(req.ip));
});

Here’s the available data returned by getGeolocationForIpAddress()

{
"as_name": "netskope inc",
"as_number": 55256,
"area_code": 0,
"city": "paris",
"conn_speed": "broadband",
"conn_type": "wired",
"continent": "EU",
"country_code": "FR",
"country_code3": "FRA",
"country_name": "france",
"gmt_offset": 200,
"latitude": 48.86,
"longitude": 2.34,
"metro_code": 250075,
"postal_code": "75001",
"proxy_description": "cloud-security",
"proxy_type": "hosting",
"region": "IDF",
"utc_offset": 200
}

Detect the device / personalize the content based device type

Dynamic serving or content negotiation is crucial for delivering a tailored user experience across various devices and platforms. By detecting characteristics like user agent or device type, websites can serve different content optimized for each, enhancing usability and performance. For instance, a website might offer a mobile-friendly layout for smartphones, a desktop version for computers, and even an alternative format for accessibility purposes

Import the device lib

import {Device} from "fastly:device"

Use the Device class :

router.get("/device", async (req, res) => {

  const ua = req.headers.get('user-agent')

  const machine = Device.lookup(ua)

  res.json({
    ua,
    machine
  });

});

The route will send back those data when connecting with a Google Pixel 7 user-agent :

{
"ua": "Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36",
"machine": {
"name": "Google Pixel 7",
"brand": "Google",
"model": "Pixel 7",
"hardwareType": null,
"isDesktop": null,
"isGameConsole": null,
"isMediaPlayer": null,
"isMobile": null,
"isSmartTV": null,
"isTablet": null,
"isTouchscreen": null
}

Read environement variables / set by Fastly

You can access those env variables within your compute code :

FASTLY_CACHE_GENERATION
FASTLY_CUSTOMER_ID
FASTLY_HOSTNAME
FASTLY_POP
FASTLY_REGION
FASTLY_SERVICE_ID
FASTLY_SERVICE_VERSION
FASTLY_TRACE_ID

To so so import :

import { env } from "fastly:env";

Then use is like so :


router.get("/env", async (req, res) => { const hostname = env("FASTLY_HOSTNAME"); const serviceId = env('FASTLY_SERVICE_ID'); const cacheGeneration = env('FASTLY_CACHE_GENERATION'); const pop = env('FASTLY_POP'); const region = env('FASTLY_REGION'); const serviceVersion = env('FASTLY_SERVICE_VERSION'); res.json({ hostname, serviceId, cacheGeneration, region, pop, serviceVersion }); });

This code returns :

{
"hostname": "cache-par-lfpg1960091",
"serviceId": "abqGMGiPUkqTNvD7cgRiz4",
"cacheGeneration": "0",
"region": "EU-West",
"pop": "PAR",
"serviceVersion": "37"
}

Edge Rate limiter / set rate limit at the edge

A rate limiter is like a gatekeeper that controls how often people or programs can access your app or website. It helps keep out bad behavior like too many requests from hackers or automated tools trying to grab all your data or slow down your site. By limiting requests, you protect your server, ensure everyone gets a fair chance to use your service, and keep your app or website running smoothly without overloads.

Import the device lib :

import { RateCounter, PenaltyBox, EdgeRateLimiter } from 'fastly:edge-rate-limiter';

Then use is like so :

router.get('/rateLimit', async (req, res) => {

    const rc = new RateCounter('rc')
    const pb = new PenaltyBox('pb')
    const limiter = new EdgeRateLimiter(rc, pb);

    if (!req.ip) res.withStatus(403).json({ error: "No ip set" });

    const shouldBeBlocked = limiter.checkRate(
        req.ip, // The client to rate limit.
        1, // The number of requests this execution counts as.
        10, // The time window to count requests within.
        100, // The maximum average number of requests per second calculated over the rate window.
        1 // The duration in minutes to block the client if the rate limit is exceeded.
    );

    if(shouldBeBlocked) res.withStatus(403).json({ shouldBeBlocked, error: "Number of requests exceeded" });

    res.json({
        shouldBeBlocked
    })

});

See it in action

To test the rate limiter I used Siege to make hundreds of HTTP calls with the following command :

siege "https://likely-destined-satyr.edgecompute.app/rateLimit"

As you can see in the video underneath we get some 200 and then quicly we get only 403 :

Conclusion

With Fastly Compute, JavaScript, and Expressly, we can handle our requests and rules at the edge. I find this method efficient and sleek compared to using VCL. As a JavaScript developer, it feels natural and easy to use. If you’re familiar with PWAs and Service Workers, you’ll notice that Fastly’s approach to handling requests is similar to the Service Worker API they aim to implement.

The idea behind autocomplete

The main goal of this application was to help HubSpot users easily create companies in their CRM with auto-completion features. Currently, when you create a company in HubSpot or any other object like a contact, there’s no auto-completion available. For instance, you can’t use the Google Maps API to auto-complete an address. Moreover, there’s no direct connection to the official company registry of your country to retrieve the correct tax number, etc. So, my idea was straightforward: create a website, app, or page with a form that incorporates auto-completion.

I approached this in a more sophisticated manner. To link the form to the HubSpot portal, you typically need to send the data through an API call. However, to make this app accessible to the general public, including those who may not be comfortable with coding or manipulating API keys, integrating with the HubSpot OAuth flow was the solution

Demo

Diagram

The stack and strategy

The most challenging aspect of this project was creating the public app with the OAuth flow. Managing the various tokens and refreshing them required a bit of effort.

The oAuth flow

To create a public app in HubSpot using OAuth, first register your app in the HubSpot Developer Portal. Then, implement the OAuth 2.0 authorization code grant flow. This involves redirecting users to the HubSpot authorization URL, where they’ll authenticate and authorize your app’s access to their HubSpot data. Upon authorization, HubSpot will redirect users back to your app with an authorization code that you’ll exchange for an access token, enabling your app to make authenticated requests to the HubSpot API on behalf of the user.

Firebase

I started by using the demo OAuth flow in Node.js provided by HubSpot as a foundation. Then, I re-implemented the functions in my own coding style, making the code clearer and cleaner.

For storing the tokens, I chose to use Firebase Firestore, which is a NoSQL database provided by Google.

To handle account creation and sign-in with Google, I chose to rely on Firebase’s functions to save time.

Frontend

For the front end, instead of hosting it on Firebase, I decided to host it on my own server.

The front-end is a Single Page Application (SPA) made with Nuxt. I used Vuex to manage my states, and for styling, I relied on Tailwind CSS.

Backends

I created a Node application using Express to manage the OAuth flow and my backend API. The REST API I built is consumed by a Nuxt.js frontend.

For the data, I created a service with an API which consumes another API with the tax information data.

Docker and deployment

Each backend is containerized with Docker. I encountered an interesting challenge related to the OAuth flow. Since all the OAuth logic is handled by my main backend, but it’s completely detached from my front end, the callback URL required didn’t match.

To address this, I utilized a reverse proxy with Apache. All requests coming from https://autocomplete.antoinebrossault.com/app are forwarded to the main backend, while requests for https://autocomplete.antoinebrossault.com/ are handled outside of the container by the Apache Server installed on the VPS.

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerAdmin webmaster@autocomplete.antoinebrossault.com
    ServerName autocomplete.antoinebrossault.com
    DocumentRoot "/home/where/apps/autocomplete.antoinebrossault.com"
    ErrorLog ${APACHE_LOG_DIR}/.log
    CustomLog ${APACHE_LOG_DIR}/.log combined

    <Directory /home/where/apps/autocomplete.antoinebrossault.com >
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>


     ProxyPass /app http://localhost:3012
     ProxyPassReverse /app http://localhost:3012


SSLCertificateFile /etc/letsencrypt/live/autocomplete.antoinebrossault.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/autocomplete.antoinebrossault.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Create a Dockerfile

Create a Dockerfile in your project :

FROM arm64v8/node:14-alpine


WORKDIR /app
COPY . .

# If you have native dependencies, you'll need extra tools
# RUN apk add --no-cache make gcc g++ python

RUN apk update && apk upgrade

RUN apk add curl

RUN npm install

EXPOSE 8181

CMD [ "node", "/app/server.js" ]

This Dockerfile is used to create a Docker image for running a Node.js application on ARM64 architecture using Alpine Linux as the base image.

FROM arm64v8/node:14-alpine This line specifies the base image for the Dockerfile. It pulls the image tagged 14-alpine from the Docker Hub repository arm64v8/node, which is a Node.js image specifically built for ARM64 architecture

This Dockerfile is used to create a Docker image for running a Node.js application on ARM64 architecture using Alpine Linux as the base image. Let’s break down each instruction:

WORKDIR /app: This sets the working directory inside the container to /app. Any subsequent commands will be executed relative to this directory.

COPY . .: This copies all the files from the host machine’s current directory (where the Dockerfile resides) to the /app directory inside the container.

RUN apk update && apk upgrade: This updates the package index and upgrades installed packages in the Alpine Linux system.

RUN apk add curl: This installs the curl command-line tool on the Alpine Linux system.

RUN npm install: This installs Node.js dependencies specified in the package.json file in the /app directory. It assumes there’s a package.json file in the root directory of the copied files.

EXPOSE 8181: This exposes port 8181 on the container. It doesn’t actually publish the port to the host machine, but it serves as documentation for users of the image to know which port to map when running a container based on this image.

CMD [ "node", "/app/server.js" ]: This specifies the command to run when a container is started from this image. It runs the Node.js application, assuming that the entry point of the application is located at /app/server.js inside the container.

Build the image

docker build -t antoine/shortgame-api . --force-rm;

In summary, the docker build command builds a Docker image based on the Dockerfile in the current directory, tags the image with antoine/shortgame-api, and removes intermediate containers after a successful build.

Run the image

Use a Makefile to save time

A Makefile is a simple text file used to define a set of tasks or instructions for building, compiling, and managing software projects. It’s commonly used in software development to automate repetitive tasks such as compiling code, running tests, and deploying applications

Here’s mine for managing my simple Docker steps :

build:
    docker build -t antoine/shortgame-api . --force-rm;

run:
    docker run -d -p 3081:8181 -it --name shortgame-api antoine/shortgame-api

Serve the App through HTTPS

To do so you need to have apache2 setup and certbot

Install Apache2

apt-get install apache2

Install the proxy mods

These Apache2 mods enable the server to act as a proxy, facilitating routing and forwarding of HTTP requests to other servers.

a2enmod proxy proxy_http

Restart Apache2

systemctl restart apache2

Install Certbot

Certbot is a free, open-source tool designed to automate the process of obtaining and renewing SSL/TLS certificates, which are essential for securing websites and enabling HTTPS encryption. It simplifies the management of SSL certificates by handling the configuration.

Install cert-bot to get let’s encrypt certificates, follow the instructions

Install snap

sudo apt update
sudo apt install snapd

Install certbot :

snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot
sudo snap install --classic certbot

Use this Bash script to automate the Apache config and certification

#!/bin/bash

# Check if the script is run as root
if [ "$EUID" -ne 0 ]; then
  echo "Please run this script as root."
  exit 1
fi

# Prompt for the domain name and directory
read -p "Enter the domain name for your website (e.g., example.com): " domain
read -p "Enter the path to the application directory (e.g., /var/www/nodeapp): " app_dir
read -p "Enter the application port (e.g., 3000): " app_port

# Create the Node.js application directory
mkdir -p $app_dir
chown -R www-data:www-data $app_dir
chmod -R 755 $app_dir

# Create a new Apache configuration file
config_file="/etc/apache2/sites-available/$domain.conf"
touch $config_file

# Define the Apache VirtualHost configuration with reverse proxy
cat > $config_file <<EOL
<VirtualHost *:80>
    ServerAdmin webmaster@$domain
    ServerName $domain

    # ProxyPass for Node.js application
    ProxyPass / http://127.0.0.1:$app_port/
    ProxyPassReverse / http://127.0.0.1:$app_port/

    DocumentRoot $app_dir
    ErrorLog \${APACHE_LOG_DIR}/$domain_error.log
    CustomLog \${APACHE_LOG_DIR}/$domain_access.log combined

    <Directory $app_dir>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>
EOL

# Enable the site configuration
a2ensite $domain

# Reload Apache to apply the changes
systemctl reload apache2

# Obtain SSL certificate and update Apache configuration with Certbot
certbot --apache --agree-tos --redirect --non-interactive --domain $domain

# Provide instructions to the user
echo "Node.js application configuration for $domain has been created and enabled."
echo "You can now start your application and configure it to listen on port $app_port."
echo "Don't forget to update your DNS records to point to this server."

Docker-compose is a powerful tool for managing multi-container Docker applications. In this article, we’ll explore how to set up a MySQL database using Docker-compose and tackle common issues that may arise during setup. We’ll cover troubleshooting steps for errors such as « not allowed to connect to this MySQL server » and dealing with authentication plugin loading problems. By following these steps, you can ensure smooth deployment and management of your MySQL containers.

Docker-compose

version: '3.8'

services:
  mysqldb:
    image: mysql:latest
    container_name: mysql-db
    restart: unless-stopped
    env_file: ./.env
    command: mysqld --default-authentication-plugin=mysql_native_password
    environment:
      - MYSQL_ROOT_PASSWORD=$MYSQLDB_ROOT_PASSWORD
      - MYSQL_DATABASE=$MYSQLDB_DATABASE
    ports:
      - $MYSQLDB_LOCAL_PORT:$MYSQLDB_DOCKER_PORT
    volumes:
      - /home/antoine/databases/mysql/data:/var/lib/mysql

Set .env

.env is at the root of my project

MYSQLDB_USER=root
MYSQLDB_ROOT_PASSWORD=qqddqsqddqqdqdsqds
MYSQLDB_LOCAL_PORT=3306
MYSQLDB_DOCKER_PORT=3306

Troubleshooting

not allowed to connect to this MySQL server

Host '172.18.0.1' is not allowed to connect to this MySQL server 

If you get this error when trying to connect to your fresh MySQL container

Enter on the container :

docker exec -it mysql-db bash

Login to mysql

mysql -u root -p

Query the myql.user database

SELECT host, user FROM mysql.user;

You will get this

+------------+------------------+
| host       | user             |
+------------+------------------+
| %          | root             |
| 127.0.0.1  | root             |
| ::1        | root             |
| localhost  | mysql.sys        |
| localhost  | root             |
| localhost  | sonar            |
+------------+------------------+

Create a new user

CREATE USER 'antoine'@'%' IDENTIFIED WITH mysql_native_password BY '<thePassword>';
grant all on *.* to 'antoine'@'%';

Alternatively :

ALTER USER 'antoine'@'%' IDENTIFIED WITH mysql_native_password BY '<thePassword>';

mysql docker uthentication plugin ‘caching_sha2_password’ cannot be loaded:

If you get an erorr which says :

mysql docker uthentication plugin 'caching_sha2_password' cannot be loaded: dlopen(/usr/local/lib/plugin/caching_sha2_password.s

Add this command to the docker-compose

    command: mysqld --default-authentication-plugin=mysql_native_password

Issue with Sequel Pro

If you encounter issues when trying to connect to your database with Sequel Pro, it’s likely a bug from sequel pro with the latest MySQL version.

Instead uses Sequel Ace

What’s docker-compose ?

Docker Compose is a tool that simplifies the management of multi-container Docker applications. It allows users to define and run multiple Docker containers as a single application, making it easier to orchestrate and manage complex deployments. With Docker Compose, users can describe their application’s services, networks, and volumes in a single file, making it straightforward to replicate environments across different systems.

Find the right binary

Identify your distribution and architecture :

uname -s

Returns :

Linux

Identify architecture :

uname -m

Returns :

aarch64

Now we know our systems is Linux aarch64

Go on the docker-compose release page on github :

https://github.com/docker/compose/releases

Identify the right release in my case :

docker-compose-linux-aarch64

Copy the download link :

https://github.com/docker/compose/releases/download/v2.26.1/docker-compose-linux-aarch64

Download it and place it to the right place :

curl -L "https://github.com/docker/compose/releases/download/v2.26.1/docker-compose-linux-aarch64" -o  /usr/local/bin/docker-compose

Make it executable :

chmod +x /usr/local/bin/docker-compose

Test the installation :

docker-compose -v 

Default DNS Management Simplified

When you buy a domain from places like Google Domains, handling your DNS records is done right there. It’s easy for many folks, but it has its limits. For example, there’s no fancy tool like a REST API for managing your domains. This means you can’t automate tasks such as creating subdomains using scripts or smoothly deploying projects from your command line.

Imagine a scenario where you want a new subdomain every time you start a new project. Sadly, with the basic DNS management provided by default, that’s not possible.

Custom DNS Management: A Better Option

The world of DNS management isn’t just limited to what domain registrars offer. There are third-party services like Cloudflare DNS, Amazon Route 53, or DNSimple. These services bring in a lot more flexibility and tools for managing your DNS records.

One great thing about these custom DNS services is that they come with REST APIs. This means developers can talk to their DNS settings using code, opening up a whole new world of automation. Imagine making a simple request to create a new subdomain or updating your DNS records automatically when your infrastructure changes. That’s the power custom DNS management offers.

Moving to Custom DNS Servers

If you want to switch to a custom DNS service, you’ll need to redirect your domain’s DNS queries from your registrar’s default servers to the servers of your chosen DNS provider.

In my situation, I decided to have Hetzner handle my DNS, but my domain is currently looked after by Google.

To make this change, I had to update the name servers, following the instructions provided in the screenshots.

Hetzner new server name :

Google domain registrar :

How to create a record with the Hetzner API

Here’s a small script to get the zones and to create a new record.

The api key is store in a .env. file at the root of my project.

You need to install dotenv and axios

npm install dotenv 
npm install axios 
touch .env

The .env file looks like this :

hetznerDNSToken = sbfwolkdnazhdhl9833

The script :

const dotenv = require('dotenv');
dotenv.config();

const axios = require('axios');



const axiosConfig = {
    headers: {
        'Auth-API-Token': process.env.hetznerDNSToken
      }
}

/**
 * Retrieves all DNS zones from Hetzner DNS API.
 * @async
 * @returns {Promise<Array>} An array containing information about all DNS zones.
 * @throws {Error} If an error occurs during the fetch process or if no zones are found.
 */
const getZones = async () => {

    const response = await axios.get('https://dns.hetzner.com/api/v1/zones', axiosConfig);

    if(!response.data) throw new Error('Error fetching zones:', response.data);

    const {zones} = response.data;

    if(!zones) throw new Error('No zones found');

    return zones;
};

/**
 * Retrieves all records associated with a specific DNS zone from Hetzner DNS API.
 * @async
 * @param {string} zoneId - The ID of the DNS zone to retrieve records from.
 * @returns {Promise<Object>} An object containing information about DNS records.
 * @throws {Error} If an error occurs during the fetch process.
 */
const getRecords = async (zoneId) => {

    const response = await axios.get(`https://dns.hetzner.com/api/v1/records?zone_id=${zoneId}`, axiosConfig);

    if(!response.data) throw new Error('Error fetching records:', response.data);

    return response.data;
};

/**
 * Creates a DNS record within a specific DNS zone using the provided data.
 * @async
 * @param {object} requestData - An object containing data required to create a DNS record.
 * @param {string} requestData.value - The value of the DNS record.
 * @param {number} requestData.ttl - The Time-To-Live (TTL) of the DNS record.
 * @param {string} requestData.type - The type of the DNS record (e.g., 'A', 'CNAME', 'MX').
 * @param {string} requestData.name - The name of the DNS record.
 * @param {string} requestData.zone_id - The ID of the DNS zone where the record will be created.
 * @returns {Promise<Object>} An object containing information about the created DNS record.
 * @throws {Error} If any required property is missing or if an error occurs during the creation process.
 */
const createRecord = async (requestData) => {

    const requiredProperties = ['value', 'ttl', 'type', 'name', 'zone_id'];

    for (const prop of requiredProperties) {
        if (!(prop in requestData)) {
          throw new Error(`Missing required property: ${prop}`);
        }
      }

    const response = await axios.post('https://dns.hetzner.com/api/v1/records', requestData, axiosConfig);

    return response.data;
};




  (async () => {


    const zones = await getZones();

    console.log(zones);


    const records = await getRecords(zones[0].id);

    console.log(records);


    const response = await createRecord({
        zone_id: zones[0].id,
        ttl: 86400,
        name: 'testa',
        type: 'A',
        value: '51.68.227.24'
    });

    console.log(response);

  })().catch(console.log);



Test your custom DNS server with dig

dig @1.0.0.1 lancieux.antoinebrossault.com. AAAA

This command is like asking a specific phone book (the DNS server at 1.0.0.1) for the phone number (IPv6 address) associated with a particular name (lancieux.antoinebrossault.com).

The part AAAA specifies that we’re specifically looking for an IPv6 address.

So, the command is basically saying: « Hey, DNS server at 1.0.0.1, tell me the IPv6 address for the name lancieux.antoinebrossault.com. »

Result


; <<>> DiG 9.10.6 <<>> @1.0.0.1 lancieux.antoinebrossault.com. AAAA ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54149 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;lancieux.antoinebrossault.com. IN AAAA ;; AUTHORITY SECTION: antoinebrossault.com. 3600 IN SOA hydrogen.ns.hetzner.com. dns.hetzner.com. 2024033004 86400 10800 3600000 3600 ;; Query time: 113 msec ;; SERVER: 1.0.0.1#53(1.0.0.1) ;; WHEN: Sat Mar 30 22:28:50 CET 2024 ;; MSG SIZE rcvd: 118

Under the Authority section we can see our new servers


;; AUTHORITY SECTION: antoinebrossault.com. 3600 IN SOA hydrogen.ns.hetzner.com. dns.hetzner.com. 2024033004 86400 10800 3600000 3600

Create the docker container


docker run --name varnish -p 3456:80 varnish

This command runs a Docker container named « varnish » using the « varnish » image. It maps port 3456 on the host to port 80 in the container, allowing access to the containerized varnish service.

Start the container


docker start varnish

Find your local ip address


ifconfig | grep 192

Returns :

inet 192.168.1.105 netmask 0xffffff00 broadcast 192.168.1.255
inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255

Here is the ip I will use as hostname : 192.168.1.105

Create a default VCL

On my local machine I created : default.vcl


vcl 4.0; backend default { .host = "192.168.64.1"; .port = "3455"; }

The host and the port are the one from my application behind Varnish

Copy the VCL in the container

docker cp default.vcl varnish:/etc/varnish

Restart the container

docker restart varnish

Start to play with the VCL

vcl 4.0;

backend default {
  .host = "192.168.64.1";
  .port = "3455";
}




sub vcl_recv {

    if (req.url ~ "keep-fresh") {
        return (pass);
    }

}



sub vcl_backend_response {

    # Set a default
    set beresp.ttl = 30s;

    if(bereq.url ~ "data"){
        set beresp.ttl = 8s;
    }

}
sub vcl_deliver {

    set resp.http.Antoine = "Hello";
}

Copy the custom VCL and restart in one command

docker cp default.vcl varnish:/etc/varnish && docker restart varnish

Do not cache a specific url :

sub vcl_recv {

    if (req.url ~ "keep-fresh") {
        return (pass);
    }

}

When I call http://localhost:3456/keep-fresh it’s never cached :

abrossault@macbook ~ % curlHeaders http://localhost:3456/keep-fresh
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 35
ETag: W/"23-gZVVGSUOoG7U6R/CPEAL+l/wRng"
Date: Tue, 26 Mar 2024 13:16:17 GMT
X-Varnish: 492
Age: 0
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Antoine: Hello
Connection: keep-alive

Only cache an url for 8sec

sub vcl_backend_response {

    # Set a default
    set beresp.ttl = 30s;

    if(bereq.url ~ "data"){
        set beresp.ttl = 8s;
    }

}

When I call http://localhost:3456/data it will remain in cache for only 8sec

abrossault@macbook ~ % curlHeaders http://localhost:3456/data
HTTP/1.1 200 OK
X-Powered-By: Express
ETag: W/"5fc7-3rbdk4/NvVpLo6VEDXT1gH26XH8"
Date: Tue, 26 Mar 2024 13:18:44 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 24519
X-Varnish: 98308 32836
Age: 8
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Accept-Ranges: bytes
Antoine: Hello
Connection: keep-alive

Here the Age is 8 Age: 8

Then when I call this URL again the Age header is reset :

abrossault@macbook ~ % curlHeaders http://localhost:3456/data
HTTP/1.1 200 OK
X-Powered-By: Express
ETag: W/"5fc7-3rbdk4/NvVpLo6VEDXT1gH26XH8"
Date: Tue, 26 Mar 2024 13:18:53 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 24519
X-Varnish: 505 98309
Age: 2
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Accept-Ranges: bytes
Antoine: Hello
Connection: keep-alive

Cache static assets for 7 days

All the assets finish with (css|js|png|jpg|jpeg|gif|ico|woff|woff2|svg|otf|ttf|eot) will be cached for 7 days, and we put a cache-control header

sub vcl_backend_response {

    # Set a default
    set beresp.ttl = 30s;

    if (bereq.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|woff|woff2|svg|otf|ttf|eot)$") {
        set beresp.ttl = 7d; // Cache static assets for 7 days
        set beresp.http.Cache-Control = "public, max-age=604800"; // Set cache-control header
    } 

}

As you can see after few minutes my Age header went up by more than the default ttl

abrossault@macbook ~ % curlHeaders http://localhost:3456/js/app.js
HTTP/1.1 200 OK
X-Powered-By: Express
Last-Modified: Tue, 26 Mar 2024 13:39:12 GMT
ETag: W/"15-18e7afc80c4"
Content-Type: application/javascript; charset=UTF-8
Content-Length: 21
Date: Tue, 26 Mar 2024 13:55:58 GMT
Cache-Control: public, max-age=604800
X-Varnish: 77 32771
Age: 669
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Accept-Ranges: bytes
Antoine: Hello
Connection: keep-alive

Redirect a URL

sub vcl_recv {

    if (req.url == "/old-url") {
        return (synth(301, "/new-url")); // Redirect to new URL
    }

}

And

sub vcl_synth {
    if (resp.status == 301) {
        set resp.http.location = resp.reason;
        set resp.reason = "Moved";
        return (deliver);
    }
}

Result :

abrossault@macbook ~ % curlHeaders http://localhost:3456/old-url
HTTP/1.1 301 Moved
Date: Tue, 26 Mar 2024 14:42:28 GMT
Server: Varnish
X-Varnish: 32773
location: /new-url
Content-Length: 0
Connection: keep-alive

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 24
ETag: W/"18-/cE9SPz52ciEALTCHs8hXDUSICk"
Date: Tue, 26 Mar 2024 14:42:28 GMT
X-Varnish: 32774
Age: 0
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Accept-Ranges: bytes
Antoine: Hello
Connection: keep-alive

Purge

If I want to purge a specific URL :

First set an acl with the authorized Ips


acl purge { "localhost"; "192.168.1.105"; "192.168.65.1"; } sub vcl_recv { # Purge if (req.method == "PURGE") { if (client.ip !~ purge) { return (synth(405, "Method Not Allowed for : "+client.ip)); } return (purge); } }

Then use this curl command to purge

curl  -X PURGE http://192.168.64.1:3456/data -i

Which will return :

HTTP/1.1 200 Purged
Date: Tue, 26 Mar 2024 16:03:53 GMT
Server: Varnish
X-Varnish: 32773
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 240
Connection: keep-alive

<!DOCTYPE html>
<html>
  <head>
    <title>200 Purged</title>
  </head>
  <body>
    <h1>Error 200 Purged</h1>
    <p>Purged</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 32773</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

Dynamic backend

If you want to select a backend based on an header, here’s one solution.

Here I have my two apps running on different ports : 192.168.64.1:3454 and 192.168.64.1:3455

First I define my backends :

backend backend_one {
    .host = "192.168.64.1";
    .port = "3454";
}

backend backend_two {
    .host = "192.168.64.1";
    .port = "3455";
}



backend default {
  .host = "192.168.64.1";
  .port = "3454";
}

Then I select the backend I want based on a header in vcl_recv, with varnish you can set the backend by using req.backend_hint as described in the documentation. Note that if you use Fastly flavored VCL you have to use set req.backend = <backendName> as described in the doc

sub vcl_recv {

    if (req.http.X-Custom-Backend) {
        if (req.http.X-Custom-Backend == "one") {
            set req.backend_hint = backend_one;
        } else if (req.http.X-Custom-Backend == "two") {
            set req.backend_hint = backend_two;
        } else {
            # If the header is set but unrecognized, use the default
            set req.backend_hint = default;
        }
    } else {
        # If no header is present, use the default
        set req.backend_hint = default;
    }
}

To test it :

curl -H "X-Custom-Backend: two" http://192.168.64.1:3456/

or

curl -H "X-Custom-Backend: one" http://192.168.64.1:3456/