Simulate attacks on your server with Nikto /
quickly test your WAF setup

Nikto is a powerful, open-source web server scanner designed to identify potential security issues and vulnerabilities in web servers. It plays a crucial role in assessing the security posture of web applications by detecting outdated software versions, misconfigurations, and dangerous files. One of its valuable applications is in testing the implementation and effectiveness of Web Application Firewalls (WAFs), ensuring they provide the intended security protections.

Introduction

Nikto is a powerful, open-source web server scanner designed to identify potential security issues and vulnerabilities in web servers. It plays a crucial role in assessing the security posture of web applications by detecting outdated software versions, misconfigurations, and dangerous files. One of its valuable applications is in testing the implementation and effectiveness of Web Application Firewalls (WAFs), ensuring they provide the intended security protections.

Use Cases

Identifying Vulnerable Software Versions: Nikto scans for outdated versions of web servers and software that might be susceptible to known vulnerabilities.

Detecting Insecure Files and Scripts: It identifies default and potentially dangerous files/scripts that might be inadvertently left on the server.

Server Configuration Analysis: The tool checks for common misconfigurations that could lead to security issues.

Testing Web Application Firewalls: By simulating various attack patterns, Nikto can help assess the effectiveness of a WAF in blocking malicious traffic.

Security Audits and Compliance: Useful for conducting regular security audits and ensuring compliance with security standards.

Setting Up Nikto with Docker

Using Docker simplifies the setup and ensures a consistent environment for running Nikto. Here’s how to set it up:

Install Docker: Make sure Docker is installed on your system. You can download and install it from Docker’s official website.

Pull the Nikto Project : Open a terminal and pull the Nikto repo from GitHub with the following command:

git clone https://github.com/sullo/nikto.git

Go to the folder :

cd nikto

Build the image :

docker build -t sullo/nikto .

Run Nikto: You can run Nikto against a target web server using the pulled Docker image:

docker run --rm sullo/nikto -Display V -h [target_ip_or_domain]

Useful Options

Target Host (-h): Specify the target host to scan.

  docker run --rm sullo/nikto -h example.com

Verbose (-Display V): Print each request on the screen.

 docker run --rm sullo/nikto -Display V -h example.com

Port (-p): Define the port to scan (default is 80).

  docker run --rm sullo/nikto -h example.com -p 8080

Output Format (-o and -Format): Save the scan results in various formats such as HTML, CSV, or XML.

docker run --rm sullo/nikto -h example.com -o results.html -Format html

Plugins (-Plugins): Run specific plugins for more targeted testing.

  docker run --rm sullo/nikto -h example.com -Plugins outdated

Conclusion

Nikto is a versatile and essential tool for web server security assessments, particularly useful for verifying the implementation and effectiveness of Web Application Firewalls. Its comprehensive scanning capabilities and ease of use, especially when set up with Docker, make it a valuable asset for security professionals aiming to safeguard web applications. Whether for routine security audits or compliance checks, Nikto helps in identifying and mitigating potential vulnerabilities effectively.

Understanding Docker Entrypoints, Dockerfiles, and Environment Variables with Node.js

Docker has become an essential tool for developers looking to build, ship, and run applications consistently across different environments. In this article, we will explore how to use Dockerfiles, entrypoints, and environment variables in the context of a Node.js application. We’ll use a specific Dockerfile and entrypoint script as an example to illustrate these concepts.

1. What is a Dockerfile?

A Dockerfile is a script containing a series of instructions on how to build a Docker image. It specifies the base image, software dependencies, configuration settings, and commands to be run in the container.

Example Dockerfile

Here is an example Dockerfile for a Node.js application:

# Use an official Node runtime as a parent image
FROM arm64v8/node:14-alpine

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container
COPY . .

# Update and upgrade apk package manager
RUN apk update && apk upgrade

# Install curl
RUN apk add curl

# Install ffmpeg
RUN apk add ffmpeg

# Install necessary npm packages
RUN npm install

# Install nodemon globally
RUN npm install -g nodemon

# Copy entrypoint script into the container
COPY entrypoint.sh /usr/local/bin/entrypoint.sh

# Make the entrypoint script executable
RUN chmod +x /usr/local/bin/entrypoint.sh

# Make port 3008 available to the world outside this container
EXPOSE 3008

# Define the entrypoint script to be executed
ENTRYPOINT ["entrypoint.sh"]

2. Entrypoints in Docker

The ENTRYPOINT instruction in a Dockerfile allows you to configure a container to run as an executable. Unlike the CMD instruction, which provides defaults that can be overridden, ENTRYPOINT instructions are always executed when the container starts.

In this example, we have defined an entrypoint script entrypoint.sh which will be executed every time the container starts.

Example Entrypoint Script

Here is an example of what the entrypoint.sh script might look like:

#!/bin/sh

# Check for environment variable and set command accordingly
if [ "$ENV" = "development" ]; then
  COMMAND="nodemon app.js"
elif [ "$ENV" = "production" ]; then
  COMMAND="npm run start"
fi

# Execute the command
exec $COMMAND

This script starts the Node.js application using nodemon, which is a tool that helps develop node.js based applications by automatically restarting the node application when file changes in the directory are detected if $ENV is equal to « development ».

3. Using Environment Variables

Environment variables are a way to pass configuration into your Docker containers. They can be defined in the Dockerfile using the ENV instruction or passed at runtime using the -e flag.

Defining Environment Variables in Dockerfile

In this example, environment variables are not defined in the Dockerfile, but you can easily add them using the ENV instruction, But the whole idea of the entry point is to do this dynamically based on the environment.

Passing Environment Variables at Runtime

You can also pass environment variables to your container at runtime:

docker run -e NODE_ENV=development -e PORT=3008 my-node-app

4. Practical Example: Node.js Application

Let’s put everything together in a more concrete example. We’ll create a simple Node.js application that uses environment variables and demonstrates the Dockerfile and entrypoint script provided.

Directory Structure

my-node-app/
│
├── Dockerfile
├── entrypoint.sh
├── app.js
├── package.json
└── package-lock.json

Dockerfile

# Use an official Node runtime as a parent image
FROM arm64v8/node:14-alpine

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container
COPY . .

# Update and upgrade apk package manager
RUN apk update && apk upgrade

# Install curl
RUN apk add curl

# Install ffmpeg
RUN apk add ffmpeg

# Install necessary npm packages
RUN npm install

# Install nodemon globally
RUN npm install -g nodemon

# Copy entrypoint script into the container
COPY entrypoint.sh /usr/local/bin/entrypoint.sh

# Make the entrypoint script executable
RUN chmod +x /usr/local/bin/entrypoint.sh

# Make port 3008 available to the world outside this container
EXPOSE 3008

# Define the entrypoint script to be executed
ENTRYPOINT ["entrypoint.sh"]

entrypoint.sh

#!/bin/sh

# Check for environment variable and set command accordingly
if [ "$ENV" = "development" ]; then
  COMMAND="nodemon app.js"
elif [ "$ENV" = "production" ]; then
  COMMAND="npm run start"
fi

# Execute the command
exec $COMMAND

app.js

const http = require('http');
const port = process.env.PORT || 3000;
const environment = process.env.ENV || 'development';

const requestHandler = (request, response) => {
    response.end(`Hello, World! Running in ${environment} mode on port ${port}.`);
};

const server = http.createServer(requestHandler);

server.listen(port, (err) => {
    if (err) {
        return console.log('something bad happened', err);
    }
    console.log(`Server is listening on ${port}`);
});

package.json

{
  "name": "my-node-app",
  "version": "1.0.0",
  "description": "A simple Node.js application",
  "main": "app.js",
  "scripts": {
    "start": "node app.js"
  },
  "dependencies": {}
}

Building and Running the Docker Container

Build the Docker Image :

docker build -t my-node-app .

Run the Docker Container :

docker run -e ENV=development -e PORT=3008 -v $(shell pwd)/app.js:/app/app.js -d -p 3090:3008 my-node-app

Makefile :

build:
    docker build -t my-node-app . --force-rm

runprod:
    docker run -e ENV=production -e PORT=3008 -d -p 3090:3008 my-node-app

rundev:
    docker run -e ENV=development -e PORT=3008 -v $(shell pwd)/app.js:/app/app.js -d -p 3090:3008 my-node-app

Now, if you navigate to http://localhost:3008, you should see Hello, World! Running in development mode on port 3008.

Conclusion

Understanding Dockerfiles, entrypoints, and environment variables is crucial for building robust and flexible Docker containers. By leveraging these tools, you can create containers that are not only easy to configure and deploy but also capable of handling a wide variety of use cases.

With Docker, you can ensure your Node.js applications run consistently across different environments, making your development and deployment processes more efficient and reliable.

In this example, we used a Dockerfile to set up a Node.js environment, installed necessary dependencies, and defined an entrypoint script to manage the application startup process. By utilizing environment variables, we made the application configuration flexible and easy to manage.

A deployment with CLI

To deploy the WAF on the edge, you need to call the WAF API. The official Fastly documentation provides CURL commands that you can execute in your terminal. Instead of using CURL commands, I decided to create a small CLI program in Node.js.
By the way it’s worth noting you can also deploy the WAF with Terraform.

The CLI

The code for the CLI is available in this GitHub repository. If you look at the code, you’ll see it is straightforward and simply converts the CURL commands into JavaScript HTTP calls.

Prerequisites for the CLI:

• You need to have Node.js installed with npm.
• You need access to your Fastly API key and your Signal Science (WAF) API key.
• You should have a site created on the Signal Science dashboard.
• You need an existing Fastly service for delivery.

Install the CLI

git clone https://github.com/Antoinebr/Fastly-WAF-Edge-deployement.git 

Go in the folder :

cd Fastly-WAF-Edge-deployement

Install the dependecies :

npm install 

Fill the .env

Copy the .env.sample and rename it .env then fill the informations with your own credentials.

cp .env.sample  .env

Replace those (dummy data) with your credentials :

SIGSCI_EMAIL="yourEmail@provider.com"
SIGSCI_TOKEN="3dd2-b927-3fde-349dq-dss922d"
FASTLY_KEY="dsddIIOLddsdbndfqlqs-G92_221_K-o"



corpName = "antoine_lab"
siteName = "faslty.antoinebrossault.com"
fastlySID = "eGdsdddd20002FwuTfjn66"

Run the CLI

npm run cli

Create the security service

Set up a new edge security service by using the edge deployment API. This API call will create a new edge security service linked to your corp and site.

In the CLI choose option 1 :

    -----------------------------------------------------
    Menu
    -----------------------------------------------------

    🌎 : edgeSecurityServiceCreation - [1]

    🔒 : getGetSecurityService - [2]

    🔗 : mapEdgeSecurityServiceToFastly - [3]

    -----------------------------------------------------

Choose an option by inputing the number, then hit enter : 1

If everything went OK it should return the following message :

✅ edgeSecurityServiceCreation : Service created 🌎

Check the security service

To check if the creation worked, you can select the getGetSecurityService - [2] option.

    -----------------------------------------------------
    Menu
    -----------------------------------------------------

    🌎 : edgeSecurityServiceCreation - [1]

    🔒 : getGetSecurityService - [2]

    🔗 : mapEdgeSecurityServiceToFastly - [3]

    -----------------------------------------------------

Choose an option by inputing the number, then hit enter :2
Getting security service for antoine_lab and siteName faslty.antoinebrossault.com

If everything went OK it should return the following message :

{
  AgentHostName: 'se--antoine-lab--65df71.edgecompute.app',
  ServicesAttached: [
    {
      id: 'eGI13sdd922Tfjn66',
      accountID: '5FCbddssSuUxxSa4faLnP',
      created: '2024-05-27T05:22:20Z',
      createdBy: 'antoinebrossault@gmail.com'
    }
  ]
}

Map the Security Service to your Fastly delivery service

    -----------------------------------------------------
    Menu
    -----------------------------------------------------

    🌎 : edgeSecurityServiceCreation - [1]

    🔒 : getGetSecurityService - [2]

    🔗 : mapEdgeSecurityServiceToFastly - [3]

    -----------------------------------------------------

Choose an option by inputing the number, then hit enter :3

You are about to mapEdgeSecurityServiceToFastly, for corpName : antoine_lab, siteName faslty.antoinebrossault.com and fastlySID eGI13FcVmYzg3FwuTfjn66 continue ? [Y/N]y

If everything went OK it should return the following message :

{
  fastlyServices: [
    {
      id: 'eGI13sdd922Tfjn66',
      accountID: '5FCbddssSuUxxSa4faLnP',
      created: '2024-05-27T05:22:20Z',
      createdBy: 'antoinebrossault@gmail.com'
    }
  ]
}

Send the traffic to the WAF

By default, the service will be activated and set to 0% traffic ramping, you can add traffic by updating the Enabled value in a newly created dictionary called Edge_Security

Here the 100% value means I send 100% of the traffic to the WAF, worth noting this modification to the dictionary doesn’t require an activation.

Test the deployment

An easy way to test the deployment is to send malicious requests to your domain to see if the WAF is able to identify them.

An example of such request :

curl https://yourdomain.com/.env 

If you run this request multiple time you should get something like this :

Signal Science Signal :

A Signal is a descriptive tag about a request, a request gets one or multiple tags based on rules a request triggers.

Defined Systems signals

Attack Signal

Triggered by malicious request attack payloads, to hack, destroy, disable, steal…

Here’s an example of such request with a URL with an attempt of SQL Injection :

https://www.example.com/search?q=SELECT+*+FROM+users+WHERE+username='$username'

Anomaly Signal

These signals indicate unusual or potentially malicious activities that don’t fit into specific attack categories, but are still suspicious.

Examples include:

• Failed attempts to access admin page :

like https://example.com/wp-admin

• Requests with Invalid encoding :

UTF-8: A common exploit vector involves sending invalid byte sequences that break UTF-8 encoding rules. For example, a lone byte %C3 which is not followed by a valid continuation byte.

Request example :

POST /login HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 40

username=%27%20OR%201=1;%20--%20&password=test

Here, %27 represents an apostrophe, which could be part of a SQL injection payload.

Exploit: The application might concatenate the injected payload directly into a SQL query, leading to SQL injection.

• Malformed data in the request body, which might indicate a malicious request

Like incorrect JSON data send to an API in order to cause the server to throw an error, potentially leading to a denial of service if the server does not handle such errors gracefully. In some cases, improperly handled parsing errors could be exploited to reveal internal server information or cause unexpected behavior.

Both Attack Signals and Anomaly Signals are referred to as System Signals, as they are built into the system.

SigSci Custom Signals

Defined by users, are tagged to requests that match user-defined conditions.

Custom signals are created at either the corporation level (Corp Signals) or the site level (Site Signals). They can be used to tag specific requests based on custom rules. This tagging helps in identifying and managing unusual or malicious traffic more effectively.

Examples :

Blocking Web Crawlers:

• Scenario : If a site owner notices vulnerability scans from requests that do not use the proper hostname, they can create a custom signal to tag valid hostname requests.

• Rule : Tag traffic that includes the domain acmecorp.com with a signal like Organization Domains.

• Blocking Rule : Any request without the Organization Domains signal is blocked, effectively filtering out suspicious traffic that targets the IP directly.

Tagging Anonymous Email Domains :

• Scenario : To reduce fraudulent account registrations, a rule can be created to tag sign-ups using disposable email domains.

• Rule : Tag requests with emails from known disposable domains with a signal like Disposable Email.

• Action : Block or further inspect these requests, preventing the creation of fake accounts.

Thresholds

Thresholds determine when to trigger an action. For example, a threshold could be « 10 or more requests within 60 seconds. » Default attack thresholds are provided but can be customized. The default thresholds are:

Signal Science Rules :

Rules are settings that define conditions to block, allow, or tag requests either permanently or for a certain period.

System Rules

System rules are predefined in the tool, when applied they can generate System Signals for an Attac or Anomaly.

Request Rules

These rules let you set specific conditions to block, allow, or tag requests either permanently or for a certain time. In the UI, these rules are grouped together.

Type :

Conditions :

Actions :

Select the Action(s) that will happen when the conditions are met

Exclusion Rules

Signal Exclusion Rules let users specify which requests should have System Signals tags removed from them. This helps reduce false positives. For instance, you can remove the XSS Attack Signal from endpoints where users are expected to submit valid scripts.

Rate Limit Rules

Rate Limit Rules allow users to set conditions to either block requests or log them with a rate-limit signal if they exceed a specified threshold within a certain time window. For instance, a threshold could be « 10 or more requests within 60 seconds. » If any IP sends 10 or more requests within the last 60 seconds, it triggers related rules, such as Custom Signals or blocking requests from that IP. Many of our rules rely on thresholds like this for their logic.

Rate Limit applies at the Site Level

Templated Rules

Partially pre-made rules. When you fill them out, they let you block requests, flag IPs for blocking or logging, or tag specific types of requests. These rules are useful for managing registrations, logins, and virtual patches in customer applications by setting up simple rules. There are three types of templates: API, ATO (Account Take Over), & CVE (Common Vulnerabilities and Exposures).

Systems Alters and Site Alerts

System Alerts are preset thresholds for the number of requests with attack signals over time. You can adjust these thresholds in your Site Settings under Attack Thresholds. This rule lets users pick any attack, anomaly, or custom signal and set a threshold and interval for when to flag and alert, and optionally block subsequent attacks from an IP.

SigSci’s architecture

SigSci is built to be flexible. This means you can choose where to put the software that deals with incoming requests. This software comes in two parts: Modules and Agents. They work together to sort through requests and block bad ones.

Step 1

A request arrives, then the barrel man (module) detects the request and informs the captain (agent) about the request.

The module (barrel man) : Receives the Request, Sends Req to Agent and Enforces Agent Decision

Step 2

The captain (agent) asks the Admiralty (sites) what should be done with that request, intercept it, let it though.

The agent (Captain) : Applies Rules, Generates Signals, Decide Allow/Block

The sites (Admiralty) : Those Signals and a host of other data will migrate up to the Site level and the Corp level.

Corp level

Starting with the Corp, this is the customer account. Collected here are settings for common account elements like Access Tokens and User Management, along with Signal Sciences specifics like Corp Monitoring, Corp Rules, and Sites. Corp Monitoring allows users to see metrics at an account level.

The Corp Rules are all the user-defined rules, as discussed earlier, that will be applied across the Corp via each Site

Within each Corp, users can set up multiple Sites. A Site, aka workspace, can be for a single web application, a bundle of web applications, API, or microservice that Signal Sciences can protect from attacks.

Corp – Sites

Like with Corps, Sites can have their own level of Rules that can be configured. The Site’s Rules are bundled with the Corp’s Rules and passed to the linked Agents. Agents send back data to pass through the Site for Site level monitoring, which could then be rolled up into Corp level Monitoring too.

Agent Mode

Blocking Mode : Applying rules to : Block request yes – Log request yes

Logging Mode : Applying rules to : Block request no – Log request yes

Off Mode : Applying rules to : Block request no – Log request no

All agents under a site can be set to one of these modes. When an Agent is set to Off Mode, the linked Module will fail-open ensuring traffic is not blocked.

Monitor attacks

• Suspicious IPs are IPs with requests, containing signals, numbering below the configured Attack Thresholds.
• Flagged IPs are IPs with requests, containing signals, numbering at or above the configured Attack Thresholds.

Flagged IPs will stay flagged for 24 hours, unless configured otherwise. Even if the number of requests with attack signals falls below the threshold, the IP will stay Flagged until the configure duration finishe

We’re exploring the Error, Restart, and Log subroutines, but a heads-up: Restart isn’t technically a subroutine in the usual sense.

For our learning objectives, let’s focus on understanding these elements:

Where they fit into the overall process
What they’re supposed to do
How they interact with each other and other parts of the system.

VCL_error

How requests come in VCL_error ?

As you can see from the diagram, we can call vcl_error from almost everywhere, except for the deliver and log processes.You can spot this on the diagram by looking for the red arrows.

How requests come out of VCL_error ?

The default exit for vcl_erorr is return(deliver), or it can also be return(deliver_stale) if there’s stale content. And restart to return back to recv

VCL_error use cases

Here’s some uses cases for VCL_error :

Synthetic responses

A synthetic response is one that Varnish can generate without needing to connect to your origin server. For example, it can create error responses like 404, 500, or redirects like 301. Typically, with synthetic responses, we create a custom HTML page that gets stored in our VCL file.

Here’s an example, in vcl_recv I trigger an error 600 if req.url.path == "/anything/404"

sub vcl_recv {
  #FASTLY recv

  if(req.url.path == "/anything/404"){
    error 600;
  }


  return(lookup);
}

🔍 Maybe you are wondering where this 600 code comes from…

The status codes used for Error handling in Fastly typically fall into specific ranges. Fastly reserves the 800 range for internal codes, which is why the pre-built « Force TLS Error » uses code 801. Developers should use codes in the 600 to 700 range when triggering Error from other subroutines.

Then we end up in the error subroutine, where I can build my synthetic response :

sub vcl_error {
  #FASTLY error

  if(obj.status == 600){
    set obj.status = 404; 
    set obj.http.Content-Type = "text/html";
    synthetic {"
    <html>
        <head>
            <meta charset="UTF-8">
        </head>

        <body>
                <h1>Not Found 🤷‍♂️ </h1>
        </body>
    </html>
    "};
  }

  return(deliver);
}

VCL_error recap :

• Error responses go directly to the delivery stage, skipping any caching processes.

Restart

Firstly Restart is not a subroutine, it’s not on list. Restart it’s its own transition call

How restart works :

• Restart goes back to the start of vcl_recv.
• Restart goes back to the Delivery Node.
• Limited to THREE restarts per POP.
• Restart disables : Clustering and Shielding
• Benefits of Disabling Clustering and Shielding.

Restart goes back to the Delivery Node

As you can see here, when a restart occurs, we go back to the delivery node :

Limited to THREE restarts per POP

With shielding ON you can restart up to 6 time, 3 time on one edge pop and 3 time on the shielding one.

Clustering and shielding with restart

We get a request, it hits one ofour Edge POPs, it gets assigned to a Delivery Node, Delivery Node goes through a
whole process of finding the Fetch Node on the Edge POP, Edge POP doesn’thave it, goes to the Shield POP, rinse and repeat for Delivery and Fetch.

Step 1

So our Fetch Node on the Shield POP makes the request, the origin gets back a 5xx error,either a 500 or a 503. (We are then a step one (1) in the diagram)

Step 2

You can see that logic there in vcl_fetch. We see the 5xx, and the Fetch Nodes says, « Ah, we got to restart. » So now it’s headed back to theDelivery Node on the Shield POP.

The Delivery Node then runs through VCL and makes its own attempt to reach the origin so it’s a fallback. We get the same 5xx error, and the Delivery Node goes, « All right. Well, I can’t restart anymore. »

Step 3

It passes it back to the Edge POP, the Fetch Node runs vcl_fetch on the response, sees again the 503 that was handed back, restarts, Delivery Node now attempts to make its own connection and gets back the last 503 or 500 error. You then, if there’s any logic to rewrite it or anything else, clean it up with a synthetic response, you hand off the eventual response to the end user

NB : in this step (3) the delivery nod eon the edge pop goes directly to the origin because we now already that the path though the shield node already failed.

Use cases for Restart

• Retry an unsuccessful request to the same or an alternative
• Perform authentication at the edge. See https://www.fastly.com/blog/patterns-for-authentication-at-the-edge
• Follow redirects when the origin returns a 301/302.
• Ignore a previously cached object and go to the origin even for a hit, without purging the object.

VCL_log

« `vcl_log«  is a function used to log messages for debugging, monitoring, or auditing purposes. It allows Varnish administrators and developers to track the behavior of the Cache by recording specific events, actions, or errors.

vcl_log is a function used in Varnish Cache’s configuration language (VCL) to log messages for debugging, monitoring, or auditing purposes. It allows Varnish administrators and developers to track the behavior of Varnish Cache by recording specific events, actions, or errors.

Here are some situations where vcl_log can be useful:

1. Debugging: When troubleshooting issues with Varnish Cache, vcl_log can be used to output relevant information to the log files. This can help in diagnosing problems such as cache misses, backend connection issues, or incorrect VCL configurations.

2. Monitoring: By strategically placing vcl_log statements in the VCL code, administrators can monitor the flow of requests through Varnish Cache. This can provide insights into traffic patterns, request handling times, and cache hit rates, aiding in performance optimization and capacity planning.

3. Auditing: vcl_log can also be used to log specific events or actions for auditing purposes. For example, logging requests that match certain criteria, recording cache invalidations, or tracking user sessions can help maintain accountability and ensure compliance with security or regulatory requirements.

4. Custom Logging: VCL allows for flexible customization of logging based on specific requirements. With vcl_log, administrators can define their own log formats and selectively log relevant information, tailoring the logging behavior to suit the needs of their infrastructure and applications.

How requests come in VCL_log ?

The transition call from deliver into log is return(deliver)

How requests come out of VCL_log ?

The transition call to end vcl_log is also return(deliver)

Conclusion

Okay, let’s go over what we’ve learned. First, Error lets us use fake responses on the Edge POPs. When we Restart, it sends the request back to Recv on the Delivery Node and turns off Clustering and Shielding. Each POP can only Restart three times (three on the Edge, three on the Shield). Finally, Log runs at the end, so you can set up and gather logging information.

How to install Terraform

On a Mac :

brew install hashicorp/tap/terraform

Verify Terraform is set up :

terraform -help

Create a the main terraform file

Let’s create our terraform project in his own folder :

mkdir terraform-tutorial && cd terraform-tutorial

Then let’s create our main.tf

touch main.tf

Add Fastly as a provider :

terraform {
  required_providers {
    fastly = {
      source  = "fastly/fastly"
      version = ">= 5.7.3"
    }
  }
}

Add a resource :

In the following code you can personalize the demo_tutorialpart the name and the domain name.

domain

For the domain name you can pick what ever you want if you want Fastly to create a temp domain name for you, but you have to follow the format : <name>.global.ssl.fastly.net

force_destroy = true

In Terraform, the force_destroy argument is used in conjunction with certain resources, typically those that manage persistent storage or long-lived infrastructure components. When force_destroy is set to true, it instructs Terraform to destroy the resource even if it has non-Terraform-managed changes or it’s not empty.

For example, if you have an AWS S3 bucket managed by Terraform and you want to delete it, but it still contains objects, Terraform will refuse to delete it by default because it wants to prevent accidental data loss. However, if you set force_destroy to true, Terraform will proceed with the deletion, permanently removing all data from the bucket and then destroying the bucket itself.

The backend block is for the address of your origin :

resource "fastly_service_vcl" "demo_tutorial" {
  name = "Super tutorial"

  domain {
    name    = "antoine-super-tutorial.global.ssl.fastly.net"
  }

  backend {
    address = "httpbin.org"
    name    = "Httbin dummy server"
  }


  force_destroy = true


}

Add an output

In Terraform, the output block is used to define values that should be exposed to the user after Terraform has executed its operations. These values can be useful for displaying important information or passing data between Terraform configurations.

output "active" {
  value = fastly_service_vcl.demo_tutorial.active_version
}

Add the Fastly API Key

provider "fastly" {
  api_key = "NyVYPuAb2Jb3nu_tsQblrMtmk-gw-oBd"
}

The complete main.tf

terraform {
  required_providers {
    fastly = {
      source  = "fastly/fastly"
      version = ">= 5.7.3"
    }
  }
}


# Configure the Fastly Provider
provider "fastly" {
  api_key = "NyVYPuAb2Jb3nu_tsQblrMtmk-gw-oBd"
}



resource "fastly_service_vcl" "demo_tutorial" {
  name = "Super tutorial"

  domain {
    name    = "antoine-super-tutorial.global.ssl.fastly.net"
  }

  backend {
    address = "httpbin.org"
    name    = "Httbin dummy server"
  }



  force_destroy = true


}

output "active" {
  value = fastly_service_vcl.demo_tutorial.active_version
}

Deploy the project on Fastly

terraform init

Terraform relies on plugins called providers to interact with various cloud platforms, APIs, and services. When you run terraform init, it downloads the necessary provider plugins specified in your configuration files and installs them locally. This ensures that Terraform has the required tools to manage your infrastructure.

Run :

terraform init 

tarraform plan

The terraform plan command is used in Terraform to create an execution plan. It essentially simulates the execution of the configuration and displays what actions Terraform will take to achieve the desired state defined in the configuration files

terraform plan

tarraform apply

terraform apply is a command used in Terraform to apply the changes specified in your Terraform configuration to your infrastructure.Terraform executes the actions outlined in the execution plan. This may involve creating new resources, updating existing ones, or destroying resources that are no longer needed.

terraform apply

Add a custom VCL

In the resource fastly_service_vcl we can specify a vcl block to add a custom vcl as specified in the doc

    vcl {
        name    = "my_custom_main_vcl"
        content = file("${path.module}/vcl/main.vcl")
        main    = true
    }

name : the name of your VCL
content : The path to your vcl code. In my case my vcl is stored in a vcl folder and in a main.vcl file.
main : is it the main vcl ?

The complete resource block :

resource "fastly_service_vcl" "test_service" {
    name = "An Example Service"

    domain {
    name = "hello-antoine.global.ssl.fastly.net"
    }

    backend {
    address = "httpbin.org"
    name= "My Test Backend"
    }

    vcl {
        name    = "my_custom_main_vcl"
        content = file("${path.module}/vcl/main.vcl")
        main    = true
    }

    force_destroy = true

}

The custom vcl

the vcl_recv subroutine. This subroutine is executed whenever Varnish receives an incoming HTTP request.

  if (req.url.path == "/anything/here") {
    set req.http.X-CustomHeader = "example";
  }

This if block checks if the request URL path is exactly /anything/here. If it is, then it sets a custom HTTP header named « X-CustomHeader » with the value « example » in the request. This header modification allows for customization of the request handling based on the requested URL.

if (req.url.path == "/anything/not/found") {
    error 600;
}

This if block checks if the request URL path is exactly « /anything/not/found ». If it is, then it generates a synthetic error with status code 600

sub vcl_error { }

This VCL (Varnish Configuration Language) code snippet defines a subroutine vcl_error, which is executed when an error occurs during the processing of a request

if (obj.status == 600) {

}

This block of code generates a synthetic response body using the synthetic keyword. It returns a simple HTML document containing an UTF-8 charset meta tag and a « Not Found » message in an <h1> element. This synthetic response is sent to the client when the error condition specified in the if block is met.

Our complete VCL

// vcl/main.vcl

   sub vcl_recv {
    #FASTLY recv

        if (req.url.path == "/anything/here") {
            set req.http.X-CustomHeader = "example";
        }

        if (req.url.path == "/anything/not/found") {
            error 600;
        }

        return(lookup);
    }

    sub vcl_error {
        #FASTLY error
        if (obj.status == 600) {

            set obj.status = 404;
            set obj.http.Content-Type = "text/html";
            synthetic {"
                <html>
                    <head>
                        <meta charset="UTF-8">
                    </head>

                    <body>
                            <h1>Not Found 🤷‍♂️ </h1>
                    </body>
                </html>
            "};

            return(deliver);
        }
}

To deploy as said before :

terraform plan

And then

terraform apply

Add a VCL snippet

Instead of writing and importing a full VCL configuration in a separate file, it could make sense to split your logic, or parts of your logic, into multiple VCL snippets. Here’s how to proceed with Terraform:

  snippet {
    name     = "custom-snippet" # Name of the snippet 
    type     = "recv" # In which subroutine should the snippet reside
    content  = <<EOT
    # Custom VCL logic
    if (req.http.X-Custom-Header) {
      set req.http.X-Response-Header = "Custom response";
    }
    EOT
    priority = 10
  }

Following our previous example the full Terraform file should look like this :

resource "fastly_service_vcl" "demo_tutorial" {
  name = "Super tutorial"

  domain {
    name    = "antoine-super-tutorial.global.ssl.fastly.net"
  }

  backend {
    address = "httpbin.org"
    name    = "Httbin dummy server"
  }


  vcl {
    name    = "my_custom_main_vcl"
    content = file("${path.module}/vcl/main.vcl")
    main    = true
  }

   # snippet START
   snippet {
    name     = "custom-snippet"
    type     = "recv" # Indique où le snippet sera appliqué (recv, deliver, etc.)
    content  = <<EOT
    # Custom VCL logic
    if (req.http.X-Custom-Header) {
      set req.http.X-Response-Header = "Custom response";
    }
    EOT
    priority = 10
  }
  # snippet END 


  force_destroy = true


}

Diagram and project goal

In this article, my goal is to use compute at edge from Faslty to protect my origin and cache my content. Instead of using Varnish Configuration Language (VCL) to set my caching rules I will use JavaScript with compute.

Start a new compute / project

To start a new compute project, you have two options.

You can use :

npm init @fastly/compute

Or you can kickstart your project with the Faslty CLI, that you need to install.

fastly compute init

After hitting the above command, you will get prompted few questions

Author (email): [antoine@gmail.com]

Language:
(Find out more about language support at https://www.fastly.com/documentation/guides/compute)
[1] Rust
[2] JavaScript
[3] Go
[4] Other ('bring your own' Wasm binary)

👉 option: 2

The started kit :

Starter kit:
[1] Default starter for JavaScript
    A basic starter kit that demonstrates routing, simple synthetic responses and overriding caching rules.
    https://github.com/fastly/compute-starter-kit-javascript-default

👉 Default starter for JavaScript

Post init :

INFO: This project has a custom post_init script defined in the fastly.toml manifest:
    npm install

Do you want to run this now? [y/N]

👉 yes

JavaScript project / setup

npm install @fastly/expressly

Add a dev command in the npm scripts :

To add the « dev » script to your package.json file, simply insert the following line within the « scripts » object:

  "scripts": {
    "dev": "fastly compute serve --watch"
  }

💡 By default the port used is 7676 if you want to use an other one use the addr flag like so :

fastly compute serve --watch --addr=127.0.0.1:7777 

This will allow you to run the command npm run dev to start your JavaScript projector using Fastly Compute.

Import Expressly / to manage the routes

Expressly is a router similar to Express in Node.js, If you ever used express Expressly will feel very confortable to use. You can check the documentation here

In /src/index.js add :

import { Router } from "@fastly/expressly";

const router = new Router();


router.get("/", async (req, res) => {

  res.send("hello");

});


router.listen();

Start the project by typing :

npm run dev 

Or

fastly compute serve --watch

This should return hello when you visit : http://localhost:7676

Publish on Faslty / and connect your backend

npm run deploy

Or

fastly compute publish

Accept and add a backend in my case I added :

fast.antoinebrossault.com

With the name :

fast_antoinebrossault_com

Add the backends in the .toml file

At the root of your project, you should have a fastly.tom file. In this file, add your backends if it’s not already the case.

Here I have my local backend and my production one

[local_server]
  [local_server.backends]
    [local_server.backends.fast_antoinebrossault_com]
      override_host = "fast.antoinebrossault.com"
      url = "https://fast.antoinebrossault.com"

[setup]
  [setup.backends]
    [setup.backends.fast_antoinebrossault_com]
      address = "fast.antoinebrossault.com"
      port = 443

Check your host config in the UI

Mine looks like this, If you get 421 Misdirected Request Error with Fastly compute, double-check the SNI hostname and certificate hostname configuration part.

Visit the url to check your deployment

If everything worked, you can visit the URL you got in the CLI to see the same result we had locally but this time on Fastly.

https://likely-destined-satyr.edgecompute.app/

Query / your backend

Modify the route we created earlier with :

router.get("/", async (req, res) => {

    const beResp = await fetch(req, {
        backend: "fast_antoinebrossault_com"
    });

    res.send(beResp);

});

Here this code query our backend and send back the result.

I your backend sends back an HTML page, the page will be broken as all the CSS and JavaScript files are not intercepted by our routes.

Let’s add routes to handle CSS and JavaScript files

This code defines a route in a router that handles requests for specific file types (JPEG, PNG, GIF, JPG, CSS, JS) and responds with the fetched content from a specified backend server (« fast_antoinebrossault_com »).

Let’s add this to our code :

router.get(/\.(jpe?g|png|gif|jpg|css|js)$/, async (req, res) => {
  res.send(await fetch(req, {
      backend: "fast_antoinebrossault_com"
  }));
});

Now our images, CSS and JavaScript files work, and all others files matching our regex will work too :

Another feature is broken on our site, this page fetches a Joke with an AJAX request to the API and insert it into the page.

Let’s handle front-end API calls

Let’s add another route to manage our API calls :

Let’s use this new route to handle the requests to the API, I created a tour to manage all the request which go to the API .

This regular expression (/^.*\/api\/.*$/ matches any string containing « /api/ » within it, regardless of what comes before or after « /api/ ».

router.get(/^.*\/api\/.*$/, async (req, res) => {
    res.send(await fetch(req, {
        backend: "fast_antoinebrossault_com"
    }));
});

Our API call now works :

We can now publish our changes with :

npm run deploy

As we can see, everything now works like a charm :

Add caching

We want Fastly to cache our content for a specific amount of time, let’s do this in our compute project.

To cache our URLs we can use CacheOverride

First import it :

import { CacheOverride } from "fastly:cache-override";

No need to install anything with npm install as cache-override exists in the fastly library we already have in our project.

We want to cache our home page for 50sec to do so we adapt the code inside our route :

router.get("/", async (req, res) => {
    const beResp = await fetch(req, {
        backend: "fast_antoinebrossault_com",

        // cache this request for 50 sec 
        cacheOverride: new CacheOverride("override", {
            ttl: 50
        })
    });

    res.send(beResp);

});

Now we re-deploy with npm deploy to see if it worked

It worked 🎉 ! As you can see, this request is now cached :

antoine@macbook / % curlHeaders https://likely-destined-satyr.edgecompute.app/
HTTP/2 200
date: Mon, 08 Apr 2024 18:16:41 GMT
server: Apache/2.4.57 (Debian)
x-powered-by: Express
content-type: text/html; charset=utf-8
x-served-by: cache-mrs10549-MRS
etag: W/"b1e-A4dLyj+Lkq4bnJSZB7a7fCcwunw"
vary: Accept-Encoding
age: 1 👈👈
accept-ranges: bytes
x-cache: MISS 👈👈
x-cache-hits: 0 👈👈
content-length: 2846

antoine@macbook / % curlHeaders https://likely-destined-satyr.edgecompute.app/
HTTP/2 200
date: Mon, 08 Apr 2024 18:16:41 GMT
server: Apache/2.4.57 (Debian)
x-powered-by: Express
content-type: text/html; charset=utf-8
x-served-by: cache-mrs1050108-MRS
etag: W/"b1e-A4dLyj+Lkq4bnJSZB7a7fCcwunw"
vary: Accept-Encoding
age: 9 👈👈
accept-ranges: bytes
x-cache: HIT 👈👈
x-cache-hits: 1 👈👈
content-length: 2846

We can see the first request is a x-cache: MISS and the second is a x-cache: HIT. It will remain in the cache until the age reaches 50, which is the TTL we put in the code.

If you are wondering what command I use to only get the headers with curl, I use this in my bash profile :

# curl-headers
#
# will return the headers only 
# @example : curl-headers <url> 
curl-headers() {
    curl -sSL -D - $1 -o /dev/null
}

alias curlHeaders="curl-headers";

Using this method to put something in cache is no different from using VCL, just nicer and more elegant. To prove it, you can go to the Faslty UI and check if the URL is in the cache :

How to add custom headers ?

To add custom headers, it’s dead simple, let’s add headers to the response we get from the home page.

To do so simply use the headers.set() method on a backend response like so :

router.get("/", async (req, res) => {
    // Forward the request to a backend.
    const beResp = await fetch(req, {
        backend: "fast_antoinebrossault_com",
        cacheOverride: new CacheOverride("override", {
            ttl: 50
        })
    });

    beResp.headers.set('Hello', "Is it me you are loooking for ?");

    res.send(beResp);

});

And it works ! 🎉

antoine@macbook / % curlHeaders curlHeaders https://likely-destined-satyr.edgecompute.app/
HTTP/2 200
date: Mon, 08 Apr 2024 18:36:04 GMT
server: Apache/2.4.57 (Debian)
x-powered-by: Express
content-type: text/html; charset=utf-8
hello: Is it me you are loooking for ? 👈👈👈
etag: W/"b34-1zh57py/zCElqztfbzqM3oXO/A4"
vary: Accept-Encoding
age: 6
accept-ranges: bytes
x-cache: HIT
x-cache-hits: 1
x-served-by: cache-mrs1050097-MRS
content-length: 2868

As you can see we have a new header in the response : hello: Is it me you are loooking for ?

Rewrite the content of a backend reponse / edit the HTML or JSON...

Rewriting the response from the backend can be very useful in various scenarios. For example:

1. Injecting Scripts: If you need to add a script (like a tracking code) but don’t have direct access to your backend code, you can use an edge function to inject the script into the HTML response.

2. Modifying API Responses: If you want to change the structure of a JSON response from an API to match the format your frontend expects, you can rewrite the response.

3. Adding Custom Headers: You might want to add custom headers to the response for security or analytics purposes, such as adding a Content Security Policy (CSP) header.

4. Personalizing Content: Based on user data, you can rewrite the HTML response to deliver personalized content without changing the backend code.

5. A/B Testing: For A/B testing, you can modify the HTML response to show different versions of the content to different users.

6. Localization: If your site serves multiple regions, you can rewrite the response to include region-specific content or translations.

7. Feature Flags: You can use edge functions to enable or disable features in the response based on feature flags, allowing for easier feature rollouts and testing.

By using edge functions to rewrite responses, you can gain more flexibility and control over the content delivered to the user without needing to modify the backend directly.

Modifying the body is a unique feature in Fastly compute

Parsing and modifying the body of a backend response before sending it to the client is a unique feature of Compute. This isn’t possible with VCL, making it a major advantage of Compute over VCL.

Inject a script tag

In my example I would like to inject a script tag right after the closing <head> tag.

To do so, I write my logic in the route I targeted :

router.get("/", async (req, res) => {


});

Now I have to query my backend :

router.get("/", async (req, res) => {

    // Forward the request to a backend.
    let beResp = await fetch(
        "https://fast.antoinebrossault.com", {
            backend: "fast_antoinebrossault_com",
            cacheOverride: new CacheOverride("override", {
                ttl: 50
            })
        }
    );


});

Then read the backend response and if our backend send back a text reponse we replace the closing head tag with our script.
Then we create a new reponse and we send back the result

if( beResp.headers.get("Content-Type").startsWith('text/') ){

    let body = await beResp.text();

    let newBody = body.replace("</head>","<script>console.log('Hello 👋');</script></head>");

    beResp = new Response(newBody, beResp);

}

res.send(beResp);

The final code looks like this :

router.get("/", async (req, res) => {
    // Forward the request to a backend.
    let beResp = await fetch(
        "https://fast.antoinebrossault.com", {
            backend: "fast_antoinebrossault_com",
            cacheOverride: new CacheOverride("override", {
                ttl: 50
            })
        }
    );

    if( beResp.headers.get("Content-Type").startsWith('text/') ){

        let body = await beResp.text();

        let newBody = body.replace("</head>","<script>console.log('Hello 👋');</script></head>");

        beResp = new Response(newBody, beResp);

    }

    res.send(beResp);

});

Troubleshooting : If you get this error "malformed UTF-8 character sequence at offset 1"

✅ Be sure your request to your backend looks like this :

 await fetch(req.url,{
            backend: "fast_antoinebrossault_com",
            cacheOverride: new CacheOverride("override", {
                ttl: 50
            })
        }
    );

Or like so

 await fetch(
        "https://fast.antoinebrossault.com", {
            backend: "fast_antoinebrossault_com",
            cacheOverride: new CacheOverride("override", {
                ttl: 50
            })
        }
    );

❌ As not specifying the full URL for the backend and passing the req object doesn’t work

const beResp = await fetch(req, {
        backend: "fast_antoinebrossault_com",
        cacheOverride: new CacheOverride("override", {
            ttl: 50
        })
    });

Improve performance by using streams

Instead of loading the full response in memory before performing the replacement Andrew Bets (Developer Advocate @ Fastly ) suggest to use a function he wrote.

First add the function to your code :

const streamReplace = (inputStream, targetStr, replacementStr) => {
    let buffer = ""
    const decoder = new TextDecoder()
    const encoder = new TextEncoder()
    const inputReader = inputStream.getReader()
    const outputStream = new ReadableStream({
      start() {
        buffer = ""
      },
      pull(controller) {
        return inputReader.read().then(({ value: chunk, done: readerDone }) => {
          buffer += decoder.decode(chunk)

          if (buffer.length > targetStr.length) {
            buffer = buffer.replaceAll(targetStr, replacementStr)
            controller.enqueue(encoder.encode(buffer.slice(0, buffer.length - targetStr.length)))
            buffer = buffer.slice(0 - targetStr.length)
          }

          // Flush the queue, and close the stream if we're done
          if (readerDone) {
            controller.enqueue(encoder.encode(buffer))
            controller.close()
          } else {
            controller.enqueue(encoder.encode(""))
          }
        })
      },
    })
    return outputStream
  }

Then use it like so :

router.get("/", async (req, res) => {
    // Forward the request to a backend.
    let beResp = await fetch(
        "https://fast.antoinebrossault.com", {
            backend: "fast_antoinebrossault_com",
            cacheOverride: new CacheOverride("override", {
                ttl: 50
            })
        }
    );


    // rewrite the response 
    if( beResp.headers.get("Content-Type").startsWith('text/') ){

        const newRespStream = streamReplace( beResp.body ,"</head>","<script>console.log('Hello 👋');</script></head>")

        beResp = new Response(newRespStream, beResp);

    }

    res.send(beResp);

});

Use Config stores / a key value pair storage

Fastly Config stores act like mini-databases at the edge of their network. They store frequently used configuration settings (like feature flags or A/B testing values) that your edge applications can access quickly and easily. This allows you to manage configurations centrally and deliver them to your services with minimal delay.

Create a config store

Here I created a config store named Features

Here you can add your key value data.

I createad a key html_404 with as a value a bit of HTML representig a 404 page. My goal is to use this HTML code asa synthetic response.

Use the Config Store

It’s important to kepp in mind the local code we run with npm run dev doesn’t have access to our backend in the same way our deployed code does. That’s why for the backend end we have a local one specified

Local

[local_server]
  [local_server.backends]
    [local_server.backends.fast_antoinebrossault_com]
      override_host = "fast.antoinebrossault.com"
      url = "https://fast.antoinebrossault.com"

[setup]
  [setup.backends]
    [setup.backends.fast_antoinebrossault_com]
      address = "fast.antoinebrossault.com"
      port = 443

It’s the same idea for config store, here’s the same data used in my config store but stored locally in my fastly.toml file :

  [local_server.config_stores.Features]
      format = "inline-toml"
    [local_server.config_stores.Features.contents]
      "html_404" = '<!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><title>404 Not Found</title><link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"></head><body class="bg-gray-100"><div class="flex items-center justify-center h-screen"><div class="text-center"><h1 class="text-4xl font-bold text-gray-800">404</h1><p class="text-lg text-gray-600">Page Not Found</p></div></div></body></html>'

Query the config store from our code :

First thing first import the module for Config Store :

import { ConfigStore } from 'fastly:config-store';

Then Initialize the config store and get the value :

  const config = new ConfigStore('Features');
  const value = config.get('html_404')

I created a /404route to demonstrate this :

router.get('/404', async (req,res) => {

  const config = new ConfigStore('Features');

  res.withStatus(404).html(config.get('html_404'));

});

It worth noting I send a 404 status code with https://expressly.edgecompute.app/docs/handling-data/response#reswithstatus And instead of setting the headers for HTML myself I used .html()

You can now deploy it to see the result live.

Use KV store / a key-value store

Fastly KV store is a service that allows you to store data persistently at the edge of Fastly’s network, making it geographically closer to users and resulting in faster data access. This key-value store offers both speed and durability, enabling quick reads and writes while ensuring data survives system restarts. By storing data at the edge, KV store reduces the load on your origin server, improving performance and scalability of your applications. This makes it a valuable tool for caching frequently accessed data, storing configuration settings, or even implementing authorization logic at the edge.

Local setup

As I said before for config store : the local code we run with npm run dev doesn’t have access to our KV Store, so we have to represent our KV store in our fastly.toml file.

[local_server] 
  [local_server.kv_stores]
      [[local_server.kv_stores.main]]
        data = "✅"
        key = "mood"

  [[local_server.kv_stores.main]]
      key = "readme"
      path = "./README.md"

For the second key, the readme one, the goal is tu return the content of the README.md file stored at the root of our project.

Live setup

We can create a KV-store from the UI, API, CLI… in the following lines I decided to go with the CLI.

Create the KV Store :

Here I’m creating a store called main :

fastly kv-store create --name=main

This should reuturn the ID of the nes KV-store

SUCCESS: Created KV Store 'main' (tff04ymfrdkgvvqi00yq0g)

Insert data in the KV Store :

Here I’m adding a key value pair : mood: ✅ to my KV Store

fastly kv-store-entry create --store-id=tff04ymfrdkgvvqi00yq0g --key="mood" --value="✅"

Here I’m adding the content of a document (readme) in the KV Store

fastly kv-store-entry create --store-id=tff04ymfrdkgvvqi00yq0g --key="readme" --value="$(cat ./readme.md)"

Link the KV Store to a compute service :

I didn’t find how it’s possible to link the newly created KV-store with a compute service with the CLI. But in the UI it’s very straitforward.

Interact wih KV store with the API :

If you want to insert an item using the API, there’s something important to keep in mind: the data you send should be encoded in base64. See the following example for clarification:

Buffer.from(`Hello World !`).toString('base64')

Here is a full example and API call:

   const response = await axios.put(
            `https://api.fastly.com/resources/stores/kv/${storeId}/batch`,
            {
                    "key": city,
                    "value": Buffer.from(`Hello World !`).toString('base64')
            }
            , {
                headers: {
                    'Content-Type': 'application/x-ndjson',
                    'Accept': 'application/json',
                    'Fastly-Key': process.env.FASTLY_KEY
                },
            }
        );

Use the KV Store in the code

Import KVStore :

import { KVStore } from "fastly:kv-store";

router.get('/kvstore', async (req, res) => {

   const files = new KVStore('main');

   const entry = await files.get('mood');

   const entryText = await entry.text();

       res.html(`
        <!DOCTYPE html>
        <html class="no-js">

            <head>
                <meta charset="utf-8">
                <meta http-equiv="X-UA-Compatible" content="IE=edge">
                <title></title>
                <meta name="description" content="">
                <meta name="viewport" content="width=device-width, initial-scale=1">
                <!-- <link href=" " rel='"stylesheet" type='"text/css"> -->
                <style>
                    * {
                        font-family: sans-serif
                    }

                    main {
                        max-width: 38rem;
                        padding: 2rem;
                        margin: auto;
                    }

                    h1{
                        text-align: center;
                    }

                    pre{
                        text-align:center;
                        font-size: 3rem;
                    }
                </style>
            </head>

            <body>
                <main>

                    <h1> KVstore </h1>

                    <pre>${entryText}</pre>

                </main>
            </body>

        </html>
    `);


});

This should display the value of our mood key we previously set :

An other example :

router.get('/kvstore/readme', async(req,res) => {

    const store = new KVStore('main');
    const entry = await store.get('readme');

    if (!entry) {
        res.withStatus(404).json( { status: 404 } );
    }

    res.send(entry.body);

});

This returns the content of our readme file :

Add geolocation / personalize the content based on location

Identifying user location through IP can enhance website experience by offering localized content, relevant services, and targeted ads. It enables personalized recommendations based on regional preferences, language, and cultural nuances, fostering user engagement and satisfaction. Additionally, it aids in fraud prevention and compliance with geo-specific regulations.

Import our getGeolocationForIpAddress function from the package :

import { getGeolocationForIpAddress } from "fastly:geolocation"

Add a new route :

router.get("/geo", async (req, res) => {
  if(!req.ip) res.withStatus(403).json({error: "No ip set"});
  res.json(getGeolocationForIpAddress(req.ip));
});

Here’s the available data returned by getGeolocationForIpAddress()

{
"as_name": "netskope inc",
"as_number": 55256,
"area_code": 0,
"city": "paris",
"conn_speed": "broadband",
"conn_type": "wired",
"continent": "EU",
"country_code": "FR",
"country_code3": "FRA",
"country_name": "france",
"gmt_offset": 200,
"latitude": 48.86,
"longitude": 2.34,
"metro_code": 250075,
"postal_code": "75001",
"proxy_description": "cloud-security",
"proxy_type": "hosting",
"region": "IDF",
"utc_offset": 200
}

Detect the device / personalize the content based device type

Dynamic serving or content negotiation is crucial for delivering a tailored user experience across various devices and platforms. By detecting characteristics like user agent or device type, websites can serve different content optimized for each, enhancing usability and performance. For instance, a website might offer a mobile-friendly layout for smartphones, a desktop version for computers, and even an alternative format for accessibility purposes

Import the device lib

import {Device} from "fastly:device"

Use the Device class :

router.get("/device", async (req, res) => {

  const ua = req.headers.get('user-agent')

  const machine = Device.lookup(ua)

  res.json({
    ua,
    machine
  });

});

The route will send back those data when connecting with a Google Pixel 7 user-agent :

{
"ua": "Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36",
"machine": {
"name": "Google Pixel 7",
"brand": "Google",
"model": "Pixel 7",
"hardwareType": null,
"isDesktop": null,
"isGameConsole": null,
"isMediaPlayer": null,
"isMobile": null,
"isSmartTV": null,
"isTablet": null,
"isTouchscreen": null
}

Read environement variables / set by Fastly

You can access those env variables within your compute code :

FASTLY_CACHE_GENERATION
FASTLY_CUSTOMER_ID
FASTLY_HOSTNAME
FASTLY_POP
FASTLY_REGION
FASTLY_SERVICE_ID
FASTLY_SERVICE_VERSION
FASTLY_TRACE_ID

To so so import :

import { env } from "fastly:env";

Then use is like so :

router.get("/env", async (req, res) => {

    const hostname = env("FASTLY_HOSTNAME");
    const serviceId = env('FASTLY_SERVICE_ID');
    const cacheGeneration = env('FASTLY_CACHE_GENERATION');
    const pop = env('FASTLY_POP');
    const region = env('FASTLY_REGION');
    const serviceVersion = env('FASTLY_SERVICE_VERSION');

    res.json({
        hostname,
        serviceId,
        cacheGeneration,
        region,
        pop,
        serviceVersion
    });

});

This code returns :

{
"hostname": "cache-par-lfpg1960091",
"serviceId": "abqGMGiPUkqTNvD7cgRiz4",
"cacheGeneration": "0",
"region": "EU-West",
"pop": "PAR",
"serviceVersion": "37"
}

Edge Rate limiter / set rate limit at the edge

A rate limiter is like a gatekeeper that controls how often people or programs can access your app or website. It helps keep out bad behavior like too many requests from hackers or automated tools trying to grab all your data or slow down your site. By limiting requests, you protect your server, ensure everyone gets a fair chance to use your service, and keep your app or website running smoothly without overloads.

Import the device lib :

import { RateCounter, PenaltyBox, EdgeRateLimiter } from 'fastly:edge-rate-limiter';

Then use is like so :

router.get('/rateLimit', async (req, res) => {

    const rc = new RateCounter('rc')
    const pb = new PenaltyBox('pb')
    const limiter = new EdgeRateLimiter(rc, pb);

    if (!req.ip) res.withStatus(403).json({ error: "No ip set" });

    const shouldBeBlocked = limiter.checkRate(
        req.ip, // The client to rate limit.
        1, // The number of requests this execution counts as.
        10, // The time window to count requests within.
        100, // The maximum average number of requests per second calculated over the rate window.
        1 // The duration in minutes to block the client if the rate limit is exceeded.
    );

    if(shouldBeBlocked) res.withStatus(403).json({ shouldBeBlocked, error: "Number of requests exceeded" });

    res.json({
        shouldBeBlocked
    })

});

See it in action

To test the rate limiter I used Siege to make hundreds of HTTP calls with the following command :

siege "https://likely-destined-satyr.edgecompute.app/rateLimit"

As you can see in the video underneath we get some 200 and then quicly we get only 403 :

How to use a Rate Limit to protect your origin ?

The idea of rate limiting is to protect your origin; a high number of requests against a specific resource could cause service disruption. Not necessarily by taking down a server but most often by degrading performance. We all know how important web performance is in the goal of delivering a high-end experience to all our users.

Here’s a piece of code that rate limits the requester when too many requests are performed in a short amount of time.

There’s something important to understand with rate limiting: the primary goal is to protect your production traffic. In other words, when you get a cache MISS. So when you test an edge rate limit rule, be sure not to hit the cache.

The following code has the sole objective of proxying the origin and adding a rate limit rule.

import { Router } from "@fastly/expressly";
import { RateCounter, PenaltyBox, EdgeRateLimiter } from 'fastly:edge-rate-limiter';
import { CacheOverride } from "fastly:cache-override";

const router = new Router();


router.all("(.*)", async (req, res) => {

  const rc = new RateCounter('rc')
  const pb = new PenaltyBox('pb')
  const limiter = new EdgeRateLimiter(rc, pb);

  if (!req.ip) res.withStatus(403).json({ error: "No ip set" });

  const shouldBeBlocked = limiter.checkRate(
      req.ip, // The client to rate limit.
      3, // The number of requests this execution counts as.
      10, // The time window to count requests within.
      12, // The maximum average number of requests per second calculated over the rate window.
      10 // The duration in minutes to block the client if the rate limit is exceeded.
  );

  if(shouldBeBlocked) res.withStatus(403).json({ shouldBeBlocked, error: "Number of requests exceeded" });

  res.headers.set('reqip', req.ip);

  res.send(await fetch(req, {
    backend: "fast_antoinebrossault_com",
    cacheOverride: new CacheOverride("pass", {})

  }));

});


router.listen();

If you need a tool to test an edge rate limit rule, I previously used Siege in this article, but I discovered another tool that is performant and easy to use. It’s called autocannon. Here’s a command you can use to run a lot of requests and analyze the results of those requests:

npx autocannon  --renderStatusCodes  --duration 30  "https://faaaaast-erl.global.ssl.fastly.net/_nuxt/1dfb24c.js"

How to use modules ?

Import your own code

Because it’s always a good idea to split your code into different files for better readability, I’m going to show you how you can do it with Fastly Compute and JavaScript.

You have to use the ESM syntax like this:

In the « /src/« ` folder

export const hello = () => {
    console.log('👋 Hello, world!');
};

import { hello } from './myModule.js';

hello();

How to use third parties libraries ?

As of today (16-03-2025), only a selection of libraries are available with Fastly Compute in JavaScript. The reason is that the SDK differs from what you have in Node.js. So, if you try to install a package, there’s a high chance that some methods will be undefined and the code will not work.

Here are the available libraries.

exif-js (2.3.0)
@borderless/base64 (1.0.1)
jose (5.6.3)
mustache (4.2.0)
minimatch (10.0.1)
crypto (1.0.1)
openapi-backend (5.10.6)
seedrandom (3.0.5)
js-cookie (3.0.5)
jsonwebtoken (9.0.2)
cookie (0.6.0)
@fastly/expressly (2.3.0)
ipaddr.js (2.2.0)
utf8 (3.0.0)
crypto-js (4.2.0)
nunjucks (3.2.4)
base-64 (1.0.0)
hono (4.4.12)
@upstash/redis (1.32.0)
@tusbar/cache-control (1.0.2)
uuid (10.0.0)
intl (1.2.5)
flight-path (1.0.13)
kewarr (1.2.1)
qrcode-svg (1.1.0)
qrcode-generator (1.4.4)
consistent-hash (1.2.2)
@fastly/esi (0.1.4)
@fastly/js-compute (3.27.2)
unix-checksum (4.4.0)
date-fns (3.6.0)

You can refer to the official documentation for more information here.

Conclusion

With Fastly Compute, JavaScript, and Expressly, we can handle our requests and rules at the edge. I find this method efficient and sleek compared to using VCL. As a JavaScript developer, it feels natural and easy to use. If you’re familiar with PWAs and Service Workers, you’ll notice that Fastly’s approach to handling requests is similar to the Service Worker API they aim to implement.

The idea behind autocomplete

The main goal of this application was to help HubSpot users easily create companies in their CRM with auto-completion features. Currently, when you create a company in HubSpot or any other object like a contact, there’s no auto-completion available. For instance, you can’t use the Google Maps API to auto-complete an address. Moreover, there’s no direct connection to the official company registry of your country to retrieve the correct tax number, etc. So, my idea was straightforward: create a website, app, or page with a form that incorporates auto-completion.

I approached this in a more sophisticated manner. To link the form to the HubSpot portal, you typically need to send the data through an API call. However, to make this app accessible to the general public, including those who may not be comfortable with coding or manipulating API keys, integrating with the HubSpot OAuth flow was the solution

Demo

Diagram

The stack and strategy

The most challenging aspect of this project was creating the public app with the OAuth flow. Managing the various tokens and refreshing them required a bit of effort.

The oAuth flow

To create a public app in HubSpot using OAuth, first register your app in the HubSpot Developer Portal. Then, implement the OAuth 2.0 authorization code grant flow. This involves redirecting users to the HubSpot authorization URL, where they’ll authenticate and authorize your app’s access to their HubSpot data. Upon authorization, HubSpot will redirect users back to your app with an authorization code that you’ll exchange for an access token, enabling your app to make authenticated requests to the HubSpot API on behalf of the user.

Firebase

I started by using the demo OAuth flow in Node.js provided by HubSpot as a foundation. Then, I re-implemented the functions in my own coding style, making the code clearer and cleaner.

For storing the tokens, I chose to use Firebase Firestore, which is a NoSQL database provided by Google.

To handle account creation and sign-in with Google, I chose to rely on Firebase’s functions to save time.

Frontend

For the front end, instead of hosting it on Firebase, I decided to host it on my own server.

The front-end is a Single Page Application (SPA) made with Nuxt. I used Vuex to manage my states, and for styling, I relied on Tailwind CSS.

Backends

I created a Node application using Express to manage the OAuth flow and my backend API. The REST API I built is consumed by a Nuxt.js frontend.

For the data, I created a service with an API which consumes another API with the tax information data.

Docker and deployment

Each backend is containerized with Docker. I encountered an interesting challenge related to the OAuth flow. Since all the OAuth logic is handled by my main backend, but it’s completely detached from my front end, the callback URL required didn’t match.

To address this, I utilized a reverse proxy with Apache. All requests coming from https://autocomplete.antoinebrossault.com/app are forwarded to the main backend, while requests for https://autocomplete.antoinebrossault.com/ are handled outside of the container by the Apache Server installed on the VPS.

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerAdmin webmaster@autocomplete.antoinebrossault.com
    ServerName autocomplete.antoinebrossault.com
    DocumentRoot "/home/where/apps/autocomplete.antoinebrossault.com"
    ErrorLog ${APACHE_LOG_DIR}/.log
    CustomLog ${APACHE_LOG_DIR}/.log combined

    <Directory /home/where/apps/autocomplete.antoinebrossault.com >
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>


     ProxyPass /app http://localhost:3012
     ProxyPassReverse /app http://localhost:3012


SSLCertificateFile /etc/letsencrypt/live/autocomplete.antoinebrossault.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/autocomplete.antoinebrossault.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Create a Dockerfile

Create a Dockerfile in your project :

FROM arm64v8/node:14-alpine


WORKDIR /app
COPY . .

# If you have native dependencies, you'll need extra tools
# RUN apk add --no-cache make gcc g++ python

RUN apk update && apk upgrade

RUN apk add curl

RUN npm install

EXPOSE 8181

CMD [ "node", "/app/server.js" ]

This Dockerfile is used to create a Docker image for running a Node.js application on ARM64 architecture using Alpine Linux as the base image.

FROM arm64v8/node:14-alpine This line specifies the base image for the Dockerfile. It pulls the image tagged 14-alpine from the Docker Hub repository arm64v8/node, which is a Node.js image specifically built for ARM64 architecture

This Dockerfile is used to create a Docker image for running a Node.js application on ARM64 architecture using Alpine Linux as the base image. Let’s break down each instruction:

WORKDIR /app: This sets the working directory inside the container to /app. Any subsequent commands will be executed relative to this directory.

COPY . .: This copies all the files from the host machine’s current directory (where the Dockerfile resides) to the /app directory inside the container.

RUN apk update && apk upgrade: This updates the package index and upgrades installed packages in the Alpine Linux system.

RUN apk add curl: This installs the curl command-line tool on the Alpine Linux system.

RUN npm install: This installs Node.js dependencies specified in the package.json file in the /app directory. It assumes there’s a package.json file in the root directory of the copied files.

EXPOSE 8181: This exposes port 8181 on the container. It doesn’t actually publish the port to the host machine, but it serves as documentation for users of the image to know which port to map when running a container based on this image.

CMD [ "node", "/app/server.js" ]: This specifies the command to run when a container is started from this image. It runs the Node.js application, assuming that the entry point of the application is located at /app/server.js inside the container.

Build the image

docker build -t antoine/shortgame-api . --force-rm;

• docker build: This is the Docker command used to build Docker images.

• -t antoine/shortgame-api: This part of the command tags the image with a specific name and optionally a tag. In this case, the image is tagged as antoine/shortgame-api. Tags are useful for identifying and managing different versions of images. The -t option is short for --tag.

• .: This specifies the build context. The . indicates that the build context is the current directory where the Dockerfile is located. This means that Docker will look for a Dockerfile in the current directory and use it to build the image.

• --force-rm: This is an option that tells Docker to remove intermediate containers after a successful build. Intermediate containers are created during the build process and are removed by default after the build is completed. Using --force-rm ensures that even if the build fails, any intermediate containers are removed. This can help save disk space.

• ;: This is a command separator used to run multiple commands in a single line in a Unix-like shell. In this case, it allows you to run multiple Docker commands sequentially in a single line.

In summary, the docker build command builds a Docker image based on the Dockerfile in the current directory, tags the image with antoine/shortgame-api, and removes intermediate containers after a successful build.

Run the image

• docker run: This is the command that tells Docker to run a container based on a Docker image.

• -d: This option tells Docker to run the container in detached mode. It means the container runs in the background, and you can still use your terminal after starting it.

• -p 3081:8181: This option is for port mapping. It tells Docker to map port 3081 on your local machine to port 8181 inside the container. So, any traffic coming to your machine on port 3081 will be forwarded to port 8181 inside the container.

• -it: This option combines two flags: -i and -t. The -i flag allows you to interact with the container’s stdin (standard input), and the -t flag allocates a pseudo-TTY, which basically means you get a terminal-like interface to interact with the container.

• --name shortgame-api: This option assigns a name to the container. In this case, the name is set to shortgame-api.

• antoine/shortgame-api: This is the name of the Docker image that the container is based on. It’s like the blueprint for the container.

Use a Makefile to save time

A Makefile is a simple text file used to define a set of tasks or instructions for building, compiling, and managing software projects. It’s commonly used in software development to automate repetitive tasks such as compiling code, running tests, and deploying applications

Here’s mine for managing my simple Docker steps :

build:
    docker build -t antoine/shortgame-api . --force-rm;

run:
    docker run -d -p 3081:8181 -it --name shortgame-api antoine/shortgame-api

Serve the App through HTTPS

To do so you need to have apache2 setup and certbot

Install Apache2

apt-get install apache2

Install the proxy mods

These Apache2 mods enable the server to act as a proxy, facilitating routing and forwarding of HTTP requests to other servers.

a2enmod proxy proxy_http

Restart Apache2

systemctl restart apache2

Install Certbot

Certbot is a free, open-source tool designed to automate the process of obtaining and renewing SSL/TLS certificates, which are essential for securing websites and enabling HTTPS encryption. It simplifies the management of SSL certificates by handling the configuration.

Install cert-bot to get let’s encrypt certificates, follow the instructions

Install snap

sudo apt update
sudo apt install snapd

Install certbot :

snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot
sudo snap install --classic certbot

Use this Bash script to automate the Apache config and certification

#!/bin/bash

# Check if the script is run as root
if [ "$EUID" -ne 0 ]; then
  echo "Please run this script as root."
  exit 1
fi

# Prompt for the domain name and directory
read -p "Enter the domain name for your website (e.g., example.com): " domain
read -p "Enter the path to the application directory (e.g., /var/www/nodeapp): " app_dir
read -p "Enter the application port (e.g., 3000): " app_port

# Create the Node.js application directory
mkdir -p $app_dir
chown -R www-data:www-data $app_dir
chmod -R 755 $app_dir

# Create a new Apache configuration file
config_file="/etc/apache2/sites-available/$domain.conf"
touch $config_file

# Define the Apache VirtualHost configuration with reverse proxy
cat > $config_file <<EOL
<VirtualHost *:80>
    ServerAdmin webmaster@$domain
    ServerName $domain

    # ProxyPass for Node.js application
    ProxyPass / http://127.0.0.1:$app_port/
    ProxyPassReverse / http://127.0.0.1:$app_port/

    DocumentRoot $app_dir
    ErrorLog \${APACHE_LOG_DIR}/$domain_error.log
    CustomLog \${APACHE_LOG_DIR}/$domain_access.log combined

    <Directory $app_dir>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>
EOL

# Enable the site configuration
a2ensite $domain

# Reload Apache to apply the changes
systemctl reload apache2

# Obtain SSL certificate and update Apache configuration with Certbot
certbot --apache --agree-tos --redirect --non-interactive --domain $domain

# Provide instructions to the user
echo "Node.js application configuration for $domain has been created and enabled."
echo "You can now start your application and configure it to listen on port $app_port."
echo "Don't forget to update your DNS records to point to this server."

Docker-compose is a powerful tool for managing multi-container Docker applications. In this article, we’ll explore how to set up a MySQL database using Docker-compose and tackle common issues that may arise during setup. We’ll cover troubleshooting steps for errors such as « not allowed to connect to this MySQL server » and dealing with authentication plugin loading problems. By following these steps, you can ensure smooth deployment and management of your MySQL containers.

Docker-compose

version: '3.8'

services:
  mysqldb:
    image: mysql:latest
    container_name: mysql-db
    restart: unless-stopped
    env_file: ./.env
    command: mysqld --default-authentication-plugin=mysql_native_password
    environment:
      - MYSQL_ROOT_PASSWORD=$MYSQLDB_ROOT_PASSWORD
      - MYSQL_DATABASE=$MYSQLDB_DATABASE
    ports:
      - $MYSQLDB_LOCAL_PORT:$MYSQLDB_DOCKER_PORT
    volumes:
      - /home/antoine/databases/mysql/data:/var/lib/mysql

Set .env

.env is at the root of my project

MYSQLDB_USER=root
MYSQLDB_ROOT_PASSWORD=qqddqsqddqqdqdsqds
MYSQLDB_LOCAL_PORT=3306
MYSQLDB_DOCKER_PORT=3306

Troubleshooting

not allowed to connect to this MySQL server

Host '172.18.0.1' is not allowed to connect to this MySQL server 

If you get this error when trying to connect to your fresh MySQL container

Enter on the container :

docker exec -it mysql-db bash

Login to mysql

mysql -u root -p

Query the myql.user database

SELECT host, user FROM mysql.user;

You will get this

+------------+------------------+
| host       | user             |
+------------+------------------+
| %          | root             |
| 127.0.0.1  | root             |
| ::1        | root             |
| localhost  | mysql.sys        |
| localhost  | root             |
| localhost  | sonar            |
+------------+------------------+

Create a new user

CREATE USER 'antoine'@'%' IDENTIFIED WITH mysql_native_password BY '<thePassword>';
grant all on *.* to 'antoine'@'%';

Alternatively :

ALTER USER 'antoine'@'%' IDENTIFIED WITH mysql_native_password BY '<thePassword>';

mysql docker uthentication plugin ‘caching_sha2_password’ cannot be loaded:

If you get an erorr which says :

mysql docker uthentication plugin 'caching_sha2_password' cannot be loaded: dlopen(/usr/local/lib/plugin/caching_sha2_password.s

Add this command to the docker-compose

    command: mysqld --default-authentication-plugin=mysql_native_password

Issue with Sequel Pro

If you encounter issues when trying to connect to your database with Sequel Pro, it’s likely a bug from sequel pro with the latest MySQL version.

Instead uses Sequel Ace

What’s docker-compose ?

Docker Compose is a tool that simplifies the management of multi-container Docker applications. It allows users to define and run multiple Docker containers as a single application, making it easier to orchestrate and manage complex deployments. With Docker Compose, users can describe their application’s services, networks, and volumes in a single file, making it straightforward to replicate environments across different systems.

Find the right binary

Identify your distribution and architecture :

uname -s

Returns :

Linux

Identify architecture :

uname -m

Returns :

aarch64

Now we know our systems is Linux aarch64

Go on the docker-compose release page on github :

https://github.com/docker/compose/releases

Identify the right release in my case :

docker-compose-linux-aarch64

Copy the download link :

https://github.com/docker/compose/releases/download/v2.26.1/docker-compose-linux-aarch64

Download it and place it to the right place :

curl -L "https://github.com/docker/compose/releases/download/v2.26.1/docker-compose-linux-aarch64" -o  /usr/local/bin/docker-compose

Make it executable :

chmod +x /usr/local/bin/docker-compose

Test the installation :

docker-compose -v 

Default DNS Management Simplified

When you buy a domain from places like Google Domains, handling your DNS records is done right there. It’s easy for many folks, but it has its limits. For example, there’s no fancy tool like a REST API for managing your domains. This means you can’t automate tasks such as creating subdomains using scripts or smoothly deploying projects from your command line.

Imagine a scenario where you want a new subdomain every time you start a new project. Sadly, with the basic DNS management provided by default, that’s not possible.

Custom DNS Management: A Better Option

The world of DNS management isn’t just limited to what domain registrars offer. There are third-party services like Cloudflare DNS, Amazon Route 53, or DNSimple. These services bring in a lot more flexibility and tools for managing your DNS records.

One great thing about these custom DNS services is that they come with REST APIs. This means developers can talk to their DNS settings using code, opening up a whole new world of automation. Imagine making a simple request to create a new subdomain or updating your DNS records automatically when your infrastructure changes. That’s the power custom DNS management offers.

Moving to Custom DNS Servers

If you want to switch to a custom DNS service, you’ll need to redirect your domain’s DNS queries from your registrar’s default servers to the servers of your chosen DNS provider.

In my situation, I decided to have Hetzner handle my DNS, but my domain is currently looked after by Google.

To make this change, I had to update the name servers, following the instructions provided in the screenshots.

Hetzner new server name :

Google domain registrar :

How to create a record with the Hetzner API

Here’s a small script to get the zones and to create a new record.

The api key is store in a .env. file at the root of my project.

You need to install dotenv and axios

npm install dotenv 
npm install axios 
touch .env

The .env file looks like this :

hetznerDNSToken = sbfwolkdnazhdhl9833

The script :

const dotenv = require('dotenv');
dotenv.config();

const axios = require('axios');



const axiosConfig = {
    headers: {
        'Auth-API-Token': process.env.hetznerDNSToken
      }
}

/**
 * Retrieves all DNS zones from Hetzner DNS API.
 * @async
 * @returns {Promise<Array>} An array containing information about all DNS zones.
 * @throws {Error} If an error occurs during the fetch process or if no zones are found.
 */
const getZones = async () => {

    const response = await axios.get('https://dns.hetzner.com/api/v1/zones', axiosConfig);

    if(!response.data) throw new Error('Error fetching zones:', response.data);

    const {zones} = response.data;

    if(!zones) throw new Error('No zones found');

    return zones;
};

/**
 * Retrieves all records associated with a specific DNS zone from Hetzner DNS API.
 * @async
 * @param {string} zoneId - The ID of the DNS zone to retrieve records from.
 * @returns {Promise<Object>} An object containing information about DNS records.
 * @throws {Error} If an error occurs during the fetch process.
 */
const getRecords = async (zoneId) => {

    const response = await axios.get(`https://dns.hetzner.com/api/v1/records?zone_id=${zoneId}`, axiosConfig);

    if(!response.data) throw new Error('Error fetching records:', response.data);

    return response.data;
};

/**
 * Creates a DNS record within a specific DNS zone using the provided data.
 * @async
 * @param {object} requestData - An object containing data required to create a DNS record.
 * @param {string} requestData.value - The value of the DNS record.
 * @param {number} requestData.ttl - The Time-To-Live (TTL) of the DNS record.
 * @param {string} requestData.type - The type of the DNS record (e.g., 'A', 'CNAME', 'MX').
 * @param {string} requestData.name - The name of the DNS record.
 * @param {string} requestData.zone_id - The ID of the DNS zone where the record will be created.
 * @returns {Promise<Object>} An object containing information about the created DNS record.
 * @throws {Error} If any required property is missing or if an error occurs during the creation process.
 */
const createRecord = async (requestData) => {

    const requiredProperties = ['value', 'ttl', 'type', 'name', 'zone_id'];

    for (const prop of requiredProperties) {
        if (!(prop in requestData)) {
          throw new Error(`Missing required property: ${prop}`);
        }
      }

    const response = await axios.post('https://dns.hetzner.com/api/v1/records', requestData, axiosConfig);

    return response.data;
};




  (async () => {


    const zones = await getZones();

    console.log(zones);


    const records = await getRecords(zones[0].id);

    console.log(records);


    const response = await createRecord({
        zone_id: zones[0].id,
        ttl: 86400,
        name: 'testa',
        type: 'A',
        value: '51.68.227.24'
    });

    console.log(response);

  })().catch(console.log);

Test your custom DNS server with dig

dig @1.0.0.1 lancieux.antoinebrossault.com. AAAA

This command is like asking a specific phone book (the DNS server at 1.0.0.1) for the phone number (IPv6 address) associated with a particular name (lancieux.antoinebrossault.com).

The part AAAA specifies that we’re specifically looking for an IPv6 address.

So, the command is basically saying: « Hey, DNS server at 1.0.0.1, tell me the IPv6 address for the name lancieux.antoinebrossault.com. »

Result

; <<>> DiG 9.10.6 <<>> @1.0.0.1 lancieux.antoinebrossault.com. AAAA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54149
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;lancieux.antoinebrossault.com. IN  AAAA

;; AUTHORITY SECTION:
antoinebrossault.com.   3600    IN  SOA hydrogen.ns.hetzner.com. dns.hetzner.com. 2024033004 86400 10800 3600000 3600

;; Query time: 113 msec
;; SERVER: 1.0.0.1#53(1.0.0.1)
;; WHEN: Sat Mar 30 22:28:50 CET 2024
;; MSG SIZE  rcvd: 118

Under the Authority section we can see our new servers

;; AUTHORITY SECTION:
antoinebrossault.com.   3600    IN  SOA hydrogen.ns.hetzner.com. dns.hetzner.com. 2024033004 86400 10800 3600000 3600

Create the docker container

docker run --name varnish -p 3456:80 varnish

This command runs a Docker container named « varnish » using the « varnish » image. It maps port 3456 on the host to port 80 in the container, allowing access to the containerized varnish service.

Start the container

docker start varnish

Find your local ip address

ifconfig | grep 192

Returns :

inet 192.168.1.105 netmask 0xffffff00 broadcast 192.168.1.255
inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255

Here is the ip I will use as hostname : 192.168.1.105

Create a default VCL

On my local machine I created : default.vcl

vcl 4.0;

backend default {
  .host = "192.168.64.1";
  .port = "3455";
}

The host and the port are the one from my application behind Varnish

Copy the VCL in the container

docker cp default.vcl varnish:/etc/varnish

Restart the container

docker restart varnish

Start to play with the VCL

vcl 4.0;

backend default {
  .host = "192.168.64.1";
  .port = "3455";
}




sub vcl_recv {

    if (req.url ~ "keep-fresh") {
        return (pass);
    }

}



sub vcl_backend_response {

    # Set a default
    set beresp.ttl = 30s;

    if(bereq.url ~ "data"){
        set beresp.ttl = 8s;
    }

}
sub vcl_deliver {

    set resp.http.Antoine = "Hello";
}

Copy the custom VCL and restart in one command

docker cp default.vcl varnish:/etc/varnish && docker restart varnish

Do not cache a specific url :

sub vcl_recv {

    if (req.url ~ "keep-fresh") {
        return (pass);
    }

}

When I call http://localhost:3456/keep-fresh it’s never cached :

abrossault@macbook ~ % curlHeaders http://localhost:3456/keep-fresh
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 35
ETag: W/"23-gZVVGSUOoG7U6R/CPEAL+l/wRng"
Date: Tue, 26 Mar 2024 13:16:17 GMT
X-Varnish: 492
Age: 0
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Antoine: Hello
Connection: keep-alive

Only cache an url for 8sec

sub vcl_backend_response {

    # Set a default
    set beresp.ttl = 30s;

    if(bereq.url ~ "data"){
        set beresp.ttl = 8s;
    }

}

When I call http://localhost:3456/data it will remain in cache for only 8sec

abrossault@macbook ~ % curlHeaders http://localhost:3456/data
HTTP/1.1 200 OK
X-Powered-By: Express
ETag: W/"5fc7-3rbdk4/NvVpLo6VEDXT1gH26XH8"
Date: Tue, 26 Mar 2024 13:18:44 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 24519
X-Varnish: 98308 32836
Age: 8
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Accept-Ranges: bytes
Antoine: Hello
Connection: keep-alive

Here the Age is 8 Age: 8

Then when I call this URL again the Age header is reset :

abrossault@macbook ~ % curlHeaders http://localhost:3456/data
HTTP/1.1 200 OK
X-Powered-By: Express
ETag: W/"5fc7-3rbdk4/NvVpLo6VEDXT1gH26XH8"
Date: Tue, 26 Mar 2024 13:18:53 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 24519
X-Varnish: 505 98309
Age: 2
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Accept-Ranges: bytes
Antoine: Hello
Connection: keep-alive

Cache static assets for 7 days

All the assets finish with (css|js|png|jpg|jpeg|gif|ico|woff|woff2|svg|otf|ttf|eot) will be cached for 7 days, and we put a cache-control header

sub vcl_backend_response {

    # Set a default
    set beresp.ttl = 30s;

    if (bereq.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|woff|woff2|svg|otf|ttf|eot)$") {
        set beresp.ttl = 7d; // Cache static assets for 7 days
        set beresp.http.Cache-Control = "public, max-age=604800"; // Set cache-control header
    } 

}

As you can see after few minutes my Age header went up by more than the default ttl

abrossault@macbook ~ % curlHeaders http://localhost:3456/js/app.js
HTTP/1.1 200 OK
X-Powered-By: Express
Last-Modified: Tue, 26 Mar 2024 13:39:12 GMT
ETag: W/"15-18e7afc80c4"
Content-Type: application/javascript; charset=UTF-8
Content-Length: 21
Date: Tue, 26 Mar 2024 13:55:58 GMT
Cache-Control: public, max-age=604800
X-Varnish: 77 32771
Age: 669
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Accept-Ranges: bytes
Antoine: Hello
Connection: keep-alive

Redirect a URL

sub vcl_recv {

    if (req.url == "/old-url") {
        return (synth(301, "/new-url")); // Redirect to new URL
    }

}

And

sub vcl_synth {
    if (resp.status == 301) {
        set resp.http.location = resp.reason;
        set resp.reason = "Moved";
        return (deliver);
    }
}

Result :

abrossault@macbook ~ % curlHeaders http://localhost:3456/old-url
HTTP/1.1 301 Moved
Date: Tue, 26 Mar 2024 14:42:28 GMT
Server: Varnish
X-Varnish: 32773
location: /new-url
Content-Length: 0
Connection: keep-alive

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 24
ETag: W/"18-/cE9SPz52ciEALTCHs8hXDUSICk"
Date: Tue, 26 Mar 2024 14:42:28 GMT
X-Varnish: 32774
Age: 0
Via: 1.1 5f6b48482a6f (Varnish/7.5)
Accept-Ranges: bytes
Antoine: Hello
Connection: keep-alive

Purge

If I want to purge a specific URL :

First set an acl with the authorized Ips

acl purge {
    "localhost";
    "192.168.1.105";
    "192.168.65.1";
}


sub vcl_recv {

    # Purge
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method Not Allowed for : "+client.ip));
        }
        return (purge);
    }

}

Then use this curl command to purge

curl  -X PURGE http://192.168.64.1:3456/data -i

Which will return :

HTTP/1.1 200 Purged
Date: Tue, 26 Mar 2024 16:03:53 GMT
Server: Varnish
X-Varnish: 32773
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 240
Connection: keep-alive

<!DOCTYPE html>
<html>
  <head>
    <title>200 Purged</title>
  </head>
  <body>
    <h1>Error 200 Purged</h1>
    <p>Purged</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 32773</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

Dynamic backend

If you want to select a backend based on an header, here’s one solution.

Here I have my two apps running on different ports : 192.168.64.1:3454 and 192.168.64.1:3455

First I define my backends :

backend backend_one {
    .host = "192.168.64.1";
    .port = "3454";
}

backend backend_two {
    .host = "192.168.64.1";
    .port = "3455";
}



backend default {
  .host = "192.168.64.1";
  .port = "3454";
}

Then I select the backend I want based on a header in vcl_recv, with varnish you can set the backend by using req.backend_hint as described in the documentation. Note that if you use Fastly flavored VCL you have to use set req.backend = <backendName> as described in the doc

sub vcl_recv {

    if (req.http.X-Custom-Backend) {
        if (req.http.X-Custom-Backend == "one") {
            set req.backend_hint = backend_one;
        } else if (req.http.X-Custom-Backend == "two") {
            set req.backend_hint = backend_two;
        } else {
            # If the header is set but unrecognized, use the default
            set req.backend_hint = default;
        }
    } else {
        # If no header is present, use the default
        set req.backend_hint = default;
    }
}

To test it :

curl -H "X-Custom-Backend: two" http://192.168.64.1:3456/

or

curl -H "X-Custom-Backend: one" http://192.168.64.1:3456/

VCL_hash

From this diagram we can see vcl_hash runs after receive and there is no way to avoid it.
Indeed, just after hash the hash table look up takes place.

As a reminder each node has a hash table. This hash table tracks what’s in cache on that specific node.

Each request that comes through generates its own hash key.

Varnish tries to match that generated hash key with a cache response on the table.

Steps :

1) Request A arrives
2) Hash is generated from URL for request A
3) On the node we try to match this hash with an existing object mapped with this hash.
4) If this hash exists in the table, then it’s a HIT
5) If this hash does not exist in the table, then it’s a MISS

How requests come in VCL_hash ?

Everything comes from receive, no matter what requests pass through hash. Either VCL_rcv asks for a lookup in the hash table or a pass if we want the request to go to the origin.

How requests exit in VCL_hash ?

The transition call out is just return hash.

The return lookup, can go to either hit or miss.

The VCL_hash subroutine in details

sub vcl_hash {
#--FASTLY HASH BEGIN
  #if unspecified fall back to normal
  {
    set req.hash += req.url;
    set req.hash += req.http.host;
    set req.hash += req.vcl.generation;
    return (hash);
  }
#--FASTLY HASH END
}

The vcl_hash routine is relatively short, and it demonstrates how the hash key is generated.

We have already discussed how it’s generated in the purge article, especially the last part regarding the generation number. This number is used to purge as we bump the generation number by one to make it unreachable…

// www.example.com/filepath1/?abc=123
www. example.com === req.host
"/filepath1/?abc=123" === req.URL

Customize the hash key

It could be useful to customize the hash key, especially if the content varies, but the URL doesn’t.

An example, if the content varies based on the language.

  {
    set req.hash += req.url;
    set req.hash += req.http.host;
    set req.hash += req.vcl.generation;
    set req.hash += req.vcl.Accept-Language;
    return (hash);
  }

⚠️ Warning : Adding an extra parameter to the hash will act as a purge-all at first, as all the requested data after this update will get a new hash.

⚠️ Warning : The accept-language header is a relatively safe use case since it doesn’t have much variability in it. If, on the other hand, you use a header like user agent, which lacks standardization and contains a wide range of inputs, your cache hit ratio will suffer significantly due to the large variance within that header.

👌 Good idea : A good idea would be to use the vary header before trying to tweak the hash key. The vary header will keep up to 200 variations or versions of the response object under one hash key.

VCL_hash recap :

• This is where the hash key is generated.
• Changing the Hash Key acts as a Purge All.
• See if Vary Header is better solution before changing the hash key.
• Changing the hash inputs after vcl_hash can cause issues with shielding

VCL_hit

VCL_hit runs if we find a match in the hashtable for the hashkey, in other words, it means we found the object in the cache.

How requests come in VCL_hit ?

The only way requests go to VCL_hit is from VCL_hash, as I said earlier if we end up in vcl_hit there’s a match.

How requests exit from VCL_hit ?

To exit VCL_hit you can still pass, restart or even error. And the default exit call is return deliver.

The VCL_hit subroutine in details

sub vcl_hit {
#--FASTLY HIT BEGIN
# we cannot reach obj.ttl and obj.grace in deliver, save them when we can in vcl_hit
  set req.http.Fastly-Tmp-Obj-TTL = obj.ttl;
  set req.http.Fastly-Tmp-Obj-Grace = obj.grace;
  {
    set req.http.Fastly-Cachetype = "HIT";
  }
#--FASTLY HIT END
  if (!obj.cacheable) {
    return(pass);
  }
  return(deliver);
}

Code explanation :

# we cannot reach obj.ttl and obj.grace in deliver, save them when we can in vcl_hit
set req.http.Fastly-Tmp-Obj-TTL = obj.ttl;
set req.http.Fastly-Tmp-Obj-Grace = obj.grace;

As the comment says, obj.ttl and obj.grace are not available in delivery, so we have to copy those values in temporary variables. To transfer them from one routine to another, we can use headers.

The fetch process happens on the fetch server using the subroutines hit, pass, miss, and fetch and certain variables, like the object variable, that will not be passed back to deliver.

At the opposite certain variables, like the request headers, will get passed from hit or fetch back to deliver.

  {
    set req.http.Fastly-Cachetype = "HIT";
  }

Here, we simply set an internal header to mark the request as a HIT.

  if (!obj.cacheable) {
    return(pass);
  }

Here if the object in cache is not cachable then we call a return(pass)

The use case here could be that at the end you run a logic that decides to pass, even if we are already in vlc_hit. Then we could set obj.cacheable = false.

  return(deliver);

Then of course we finish up with return deliver, that’s our default call, to exit VCL hit.

Takeaways

In vcl_hit it’s always possible to request a fresh version from the origin by your calling return pass. The cached object still remains in cache.

VCL_miss

VCL_miss like VCL_hit and pass and fetch are on the fetch server.

How requests come in VCL_miss ?

VCL_miss is triggered when :

• We don’t have an entry at all for that hash key
• We have a match for the hash key, but the object has expired ( Its TTL has run out )

NB : It’s always possible to call a PASS in MISSif you change your min at last minute.

How requests exit from VCL_miss ?

Our default exit call is a return(fetch), We are going to go and fetch a response from the backend.

We can call return(pass) to go to the pass sub-routine instead.

We can call return(deliver_stale) when the TTL have expiered but we set rules to allow the delivery of a stale object with deliver stale option or stale while revalidate or stale if error

Last transition call is error, which takes us to VCL error

The VCL_miss subroutine in details

sub vcl_miss {
#--FASTLY MISS BEGIN
# this is not a hit after all, clean up these set in vcl_hit
  unset req.http.Fastly-Tmp-Obj-TTL;
  unset req.http.Fastly-Tmp-Obj-Grace;
  {
    if (req.http.Fastly-Check-SHA1) {
       error 550 "Doesnt exist";
    }
#--FASTLY BEREQ BEGIN
    {
      {
        if (req.http.Fastly-FF) {
          set bereq.http.Fastly-Client = "1";
        }
      }
      {
        # do not send this to the backend
        unset bereq.http.Fastly-Original-Cookie;
        unset bereq.http.Fastly-Original-URL;
        unset bereq.http.Fastly-Vary-String;
        unset bereq.http.X-Varnish-Client;
      }
      if (req.http.Fastly-Temp-XFF) {
         if (req.http.Fastly-Temp-XFF == "") {
           unset bereq.http.X-Forwarded-For;
         } else {
           set bereq.http.X-Forwarded-For = req.http.Fastly-Temp-XFF;
         }
         # unset bereq.http.Fastly-Temp-XFF;
      }
    }
#--FASTLY BEREQ END
 #;
    set req.http.Fastly-Cachetype = "MISS";
  }
#--FASTLY MISS END
  return(fetch);
}

Code explanation :

  unset req.http.Fastly-Tmp-Obj-TTL;
  unset req.http.Fastly-Tmp-Obj-Grace;

If our request went to hit but restarted and ended up in MISS we clean those headers HIT set.

   if (req.http.Fastly-Check-SHA1) {
       error 550 "Doesnt exist";
    }

A condition for Fastly check SHa1, giving an error 550 code. This is for internal testing only for Fastly.

    {
      {
        if (req.http.Fastly-FF) {
          set bereq.http.Fastly-Client = "1";
        }
      }

Checking to see if the Fastly forwarded for header is present. If it is, then we set a back-end request header for the Fastly client. This allows customers and developers to use a header to check if this request is coming from Fastly.

# do not send this to the backend
unset bereq.http.Fastly-Original-Cookie;
unset bereq.http.Fastly-Original-URL;
unset bereq.http.Fastly-Vary-String;
unset bereq.http.X-Varnish-Client;

Those headers are specific to Fastly, and the idea here is to remove them before he reaches the backend.

if (req.http.Fastly-Temp-XFF) {
         if (req.http.Fastly-Temp-XFF == "") {
           unset bereq.http.X-Forwarded-For;
         } else {
           set bereq.http.X-Forwarded-For = req.http.Fastly-Temp-XFF;
         }
         # unset bereq.http.Fastly-Temp-XFF;
      }

Modifications to the Fastly temp X forwarded for header

#--FASTLY BEREQ END
 #;
    set req.http.Fastly-Cachetype = "MISS";
  }
#--FASTLY MISS END
  return(fetch);
}

We are setting again the Fastly cache type as miss for internal use.
Then we have the transition call to default transition call of return fetch that will actually trigger the backend request

Takeways

Last Stop before the request goes to the backend

• Last minute changes to request.
• Last chance to PASS on the request.
• This is where WAF lives, before the request to the backend ( where we plug it )

VCL_pass

Pass is the sub routine we use for the requests we know we don’t want to cache the response.

How requests come in VCL_pass ?

You can get to VCL_pass through, receive, ie, calling return pass there, something we already discussed in this article

You can also endup in PASS through miss or hit as discussed in the above sections.

Hit for pass :

The hit for pass behavior, we execute a return pass in VCL_fetch and the hash table for that node gets an entry for that hash key that says, “Pass for the next default of 120 seconds.” As explained here

How requests exit from VCL_pass ?

Our exit call, our exit transition is return pass.ou can also error out of pass as well.

VCL_pass in detail

THe code is verry similar to VCL_miss ooking at them in a diff checker with VCL miss on the left and VCL pass on the right, we can see that there’s just a couple of things that have been removed for pass.

Unsetting of those temporary TTLs and temporary grace request headers, they’re not being unset in pass. Those may be passed to the origin. Also, we don’t have the Fastly check SHA1 in place. That’s also been removed. The other thing to point out that’s missing is looking at the end of VCL pass, there’s no default return pass present. Even though it was listed on our diagram there, it’s not actually present and it doesn’t need to be. For the case of VCL pass, the default transition call is there in the background. Since there’s literally only one way to transition out of this subroutine except from error, really the only call is going to be return pass

Takeways for VCL_pass

Last Stop before the request goes to the backend

• Last minute changes to request.
• Best Practice: Any change in MISS, should be made in PASS
and vise versa.
• This is where also WAF lives, before the request to the backend.

Takeways Hash, Hit, Miss and Pass Subroutines

• Vcl_hash is for generating the Hash Key. Proceed with caution.
• Changing hash key inputs after vc|_hash combined w/ shielding affects purge.
• You can still PASS from vcl_hit & vcl_miss.
• Last min changes should happen in both vcl_miss & vcl_pass

Varnish state machine flow

When a request comes into Fastly, it’s assigned out from a load balancer to one of the nodes on the POP, and that will act as the delivery node.

Then there is a domain lookup where the request host header, the value there, say example.com is looked up on a giant table on the node to see which service configuration has that matching domain. The attached service configuration is then run and this is what we’re looking at here is all the subroutines that make up that service configuration.

We assume that there’s caching that’s attempting to go on with this request. So starting on our left, when a request comes in, receive is the first subroutine to run. We can follow that dash line as it goes from receive to hash.

After hash is the hash table lookup on the node to see if the item is already cached.

And then it’ll split either between hit or miss

This becomes a decision point:

• If we have the item already, it’s a hit and we run the hit subroutine.
  Following through from hit, we can follow that same dotted line. We have the item in cache. It’s handed over to the deliver subroutine, and deliver runs. And then it’s handed off to the end user as a response. Going back to our hash table after the hash subroutine, if we don’t have the item already on the table, we don’t have a match. It’s not in cache, it’s a miss.

• If do not have the item already, it’s a miss. Then the miss subroutine will run. The second action or process, and this is the fetch from backend. So if we don’t have the item, we of course need to request it from the backend. This is where we fetch the object. After this process, we then run the fetch subroutine.

After we run fetch we have an other process and this is where we actually cache or don’t cache the response object. Either way, whatever comes out of fetch, whether we cache it or not is handed off to then deliver, the deliver subroutine runs, and then we send off the actual response to the end user.

VCL_recv

This is our first subroutine, it’s the first point of contact and it’s used for :

• normalizing clients’ input
• normalize headers
• normalize the request URL
• Strip some query parameters
• Switching between the backends (allocating a specific backend to a request)
• Apply authentication ( check JWT, API Keys, geolocation, Ips, Paywallls…)
• Triggering synthetic responses for known paths
• PASS on request that should not be cahed

VCL_recv is the best place to make those changes to standardize client’s inputs It’s very similar to a an Node.js express Middleware.

How request come and exit from VCL_recv subroutine ?

How they enter in VCL_recv ?

• inbound request : The first way requests enter in VCL_recv
• restart : If restart is called, you leave whatever subroutine you’re in and it returns you to the beginning of the VCL flow

How they exit in VCL_recv ?

• return lookup : The default way to exit, Return lookup is referencing a lookup call on the hash table. You’re looking to see if something is in cache
• return pass : If you don’t want to cache
• error : Calling error takes you to the error subroutine
• restart : If restart is called, you leave whatever subroutine you’re in and it returns you to the beginning of the VCL flow

Where requests go after VCL_recv ?

After receive all requests go to hash, but after they go to hash, they are all going to end up at different places.

Default path : Hash then cache table lookup then eihter a cache HIT or cache MISS

Calling Pass in receive : Hash which generates hash key goes to that hash table, but then goes to pass.

Error : Hash ad then straight to error ( as we don’t want to cache errors )

The VCL_recv subroutine in details

sub vcl_recv {

    #--FASTLY RECV BEGIN

    if (req.restarts == 0) {
        if (!req.http.X-Timer) {
        set req.http.X-Timer = "S" time.start.sec "." time.start.usec_frac;
        }
        set req.http.X-Timer = req.http.X-Timer ",VS0";
    }

    declare local var.fastly_req_do_shield BOOL;
    set var.fastly_req_do_shield = (req.restarts == 0);
    # default conditions
    # end default conditions


    #--FASTLY RECV END

    if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") {
      return(pass);
    }
    return(lookup);
}

Let’s analyze this code :

    if (req.restarts == 0) {
        if (!req.http.X-Timer) {
        set req.http.X-Timer = "S" time.start.sec "." time.start.usec_frac;
        }
        set req.http.X-Timer = req.http.X-Timer ",VS0";
    }

Here is the inclusion of a timer header req.http.X-Timer

This is used for our timing metrics. We add it at the very beginning once we get it in receive, and it gets tied up later on in deliver at the end. This is only initialized, if the number of restarts is zero.

declare local var.fastly_req_do_shield BOOL;
set var.fastly_req_do_shield = (req.restarts == 0);

Then we have some shield variables that are being declared and set.

  if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") {
      return(pass);
   }

If the request method is not a head, and the request method is not a get, and the request is not a Fastly purge call, then it’ll return pass. So if it’s not any of these three things, we are not going to cache it, we call the pass behavior.

return(lookup);

Then at the end of the receive subroutine, we have our default transition call, which is returned lookup.

VCL_fetch

How requests enter in VCL_fetch ?

We go to fetch after a PASS or a MISS, if it’s a PASS then we should get the data from our backend. Same thing for MISS, as we don’t have it in the cache we have to grab the data from our backend and then put it in the cache.

VCL_fetch it’s used for :

• Set a default TTL
• Set specific TTLs for content based on things like file path, content type or other relevant variables.
• Modify or remove response headers set from the origin or adding entirely new ones.
• configure rules for serving stale content, using beresp.stale-while-revalidate and or beresp.stale-if-error

When we get that backend response before fetch runs, Varnish looks for the presence of any cache control headers that Fastly considers valid. If we find them and we find values that we can use, we will use them to set the TTL for this particular response object.

How requests exit in VCL_fetch ?

To exit VCL fetch, the default transition is return deliver.

• return deliver
• return deliver stale : Deliver stale is related to those serving stale content options
• return pass : Will not cache, and it will still head off to deliver.
• restart : goes back to receive
• error : which will bypass receive and go straight to error.

By the way we are expecting to cache the responses that we get from the backend, and this would be the default transition to do so.

The VCL_fetch routine in details

sub vcl_fetch {

  declare local var.fastly_disable_restart_on_error BOOL;
#--FASTLY FETCH BEGIN

    # record which cache ran vcl_fetch for this object and when
  set beresp.http.Fastly-Debug-Path = "(F " server.identity " " now.sec ") " if(beresp.http.Fastly-Debug-Path, beresp.http.Fastly-Debug-Path, "");

    # generic mechanism to vary on something
  if (req.http.Fastly-Vary-String) {
    if (beresp.http.Vary) {
      set beresp.http.Vary = "Fastly-Vary-String, "  beresp.http.Vary;
    } else {
      set beresp.http.Vary = "Fastly-Vary-String, ";
    }
  }

#--FASTLY FETCH END

  if (!var.fastly_disable_restart_on_error) {
    if ((beresp.status == 500 || beresp.status == 503) && req.restarts < 1 && (req.request == "GET" || req.request == "HEAD")) {
      restart;
    }
  }

  if(req.restarts > 0 ) {
    set beresp.http.Fastly-Restarts = req.restarts;
  }

  if (beresp.http.Set-Cookie) {
    set req.http.Fastly-Cachetype = "SETCOOKIE";
    return (pass);
  }

  if (beresp.http.Cache-Control ~ "private") {
    set req.http.Fastly-Cachetype = "PRIVATE";
    return (pass);
  }

  if (beresp.status == 500 || beresp.status == 503) {
    set req.http.Fastly-Cachetype = "ERROR";
    set beresp.ttl = 1s;
    set beresp.grace = 5s;
    return (deliver);
  }

  if (beresp.http.Expires || beresp.http.Surrogate-Control ~ "max-age" || beresp.http.Cache-Control ~"(?:s-maxage|max-age)") {
    # keep the ttl here
  } else {
        # apply the default ttl
    set beresp.ttl = 3600s;
  }

  return(deliver);
}

Let’s explore the basic code in VCL_fetch

set beresp.http.Fastly-Debug-Path = "(F " server.identity " " now.sec ") " if(beresp.http.Fastly-Debug-Path, beresp.http.Fastly-Debug-Path, "");

Fastly debug headers can collect and expose that information when you send those headers as a request header. It’ll add in some additional response debugging headers.

To get this header in your response call one of you url with this header : 'Fastly-Debug': '1'

if (req.http.Fastly-Vary-String) {
    if (beresp.http.Vary) {
      set beresp.http.Vary = "Fastly-Vary-String, "  beresp.http.Vary;
    } else {
      set beresp.http.Vary = "Fastly-Vary-String, ";
    }
  }

A generic mechanism to check if your backend uses the Vary header which is used when the reponse vary based on something ( User-agent for example )

 if (!var.fastly_disable_restart_on_error) {
    if ((beresp.status == 500 || beresp.status == 503) && req.restarts < 1 && (req.request == "GET" || req.request == "HEAD")) {
      restart;
    }
  }

If you get a 500 or a 503 and the number of restarts is less than one and the request type is a get or a head, then it will cause a restart to happen.
This mechanism will be triggered only once (req.restarts < 1)

  if(req.restarts > 0 ) {
    set beresp.http.Fastly-Restarts = req.restarts;
  }

If the number of restarts is greater than zero. So we’ve already restarted. Then we’re going to set a backend response header for Fastly restarts equal to the number of restarts

  if (beresp.http.Set-Cookie) {
    set req.http.Fastly-Cachetype = "SETCOOKIE";
    return (pass);
  }

If there is a set cookie header set by the backend. Then this response is unique to the user and we don’t cache it.

 if (beresp.http.Cache-Control ~ "private") {
    set req.http.Fastly-Cachetype = "PRIVATE";
    return (pass);
  }

If control header has a value of private then we pass.

  if (beresp.status == 500 || beresp.status == 503) {
    set req.http.Fastly-Cachetype = "ERROR";
    set beresp.ttl = 1s;
    set beresp.grace = 5s;
    return (deliver);
  }

If the response from the backend is an error a 500 or a 503 then we’re cashing them for just a second set beresp.ttl = 1s;

This one second gap where we temporarily serve this from cache can help offload the strain to that origin and help it to recover and that’s why we just do a return deliver because we are caching that response, but only for one second.

  if (beresp.http.Expires || beresp.http.Surrogate-Control ~ "max-age" || beresp.http.Cache-Control ~"(?:s-maxage|max-age)") {
    # keep the ttl here
  } else {
        # apply the default ttl
    set beresp.ttl = 3600s;
  }

So if there’s an expires header, if there’s a surrogate control value with max age or a cache control value with either surrogate max age or max age, that implies that we would be able to pull a TTL out of those values and we would keep the TTL

return(deliver);

Then if none of these other conditions trigger a return pass or return deliver, we always have our default return deliver without any condition at the end of the subroutine.

VCL_deliver

Deliver is the last set routine before the log subroutine runs. Deliver runs before the first byte of the response is sent to the user. It’s also the last chance to modify your response, headers…

Deliver runs on every single response no matter if the response comes from the cache etc…

It’s an ideal place for adding debugging information or user specific session data that cannot be shared with other users ( adding debug headers…)

How requests enter in VCL_deliver ?

• hit fetch error
• miss fetch *error**
• fetch : return(pass) :from the fetch subroutine you can also call return pass to not cache and pass off the response to VCL deliver.

How requests exit in VCL_deliver ?

• return deliver : the transition to exit VCL deliver is return deliver. That is the default, it is referencing itself.
• restart : restart go back to the beginning at receive.

Let’s explore the VCL for deliver :

sub vcl_deliver {


#--FASTLY DELIVER BEGIN
# record the journey of the object, expose it only if req.http.Fastly-Debug.
  if (req.http.Fastly-Debug || req.http.Fastly-FF) {
    set resp.http.Fastly-Debug-Path = "(D " server.identity " " now.sec ") "
       if(resp.http.Fastly-Debug-Path, resp.http.Fastly-Debug-Path, "");
    set resp.http.Fastly-Debug-TTL = if(obj.hits > 0, "(H ", "(M ")
       server.identity
       if(req.http.Fastly-Tmp-Obj-TTL && req.http.Fastly-Tmp-Obj-Grace, " " req.http.Fastly-Tmp-Obj-TTL " " req.http.Fastly-Tmp-Obj-Grace " ", " - - ")
       if(resp.http.Age, resp.http.Age, "-")
       ") "
       if(resp.http.Fastly-Debug-TTL, resp.http.Fastly-Debug-TTL, "");
    set resp.http.Fastly-Debug-Digest = digest.hash_sha256(req.digest);
  } else {
    unset resp.http.Fastly-Debug-Path;
    unset resp.http.Fastly-Debug-TTL;
    unset resp.http.Fastly-Debug-Digest;
  }


  # add or append X-Served-By/X-Cache(-Hits)
  {
    if(!resp.http.X-Served-By) {
      set resp.http.X-Served-By  = server.identity;
    } else {
      set resp.http.X-Served-By = resp.http.X-Served-By ", " server.identity;
    }
    set resp.http.X-Cache = if(resp.http.X-Cache, resp.http.X-Cache ", ","") if(fastly_info.state ~ "HIT(?:-|\z)", "HIT", "MISS");
    if(!resp.http.X-Cache-Hits) {
      set resp.http.X-Cache-Hits = obj.hits;
    } else {
      set resp.http.X-Cache-Hits = resp.http.X-Cache-Hits ", " obj.hits;
    }
  }


  if (req.http.X-Timer) {
    set resp.http.X-Timer = req.http.X-Timer ",VE" time.elapsed.msec;
  }


  # VARY FIXUP
  {
    # remove before sending to client
    set resp.http.Vary = regsub(resp.http.Vary, "Fastly-Vary-String, ", "");
    if (resp.http.Vary ~ "^\s*$") {
      unset resp.http.Vary;
    }
  }
  unset resp.http.X-Varnish;

  # Pop the surrogate headers into the request object so we can reference them later
  set req.http.Surrogate-Key = resp.http.Surrogate-Key;
  set req.http.Surrogate-Control = resp.http.Surrogate-Control;


  # If we are not forwarding or debugging unset the surrogate headers so they are not present in the response
  if (!req.http.Fastly-FF && !req.http.Fastly-Debug) {
    unset resp.http.Surrogate-Key;
    unset resp.http.Surrogate-Control;
  }


  if(resp.status == 550) {
    return(deliver);
  }

  #default response conditions
#--FASTLY DELIVER END
  return(deliver);
}

Let’s explore the deliver sub routine in details

  if (req.http.Fastly-Debug || req.http.Fastly-FF) {
    set resp.http.Fastly-Debug-Path = "(D " server.identity " " now.sec ") "
       if(resp.http.Fastly-Debug-Path, resp.http.Fastly-Debug-Path, "");
    set resp.http.Fastly-Debug-TTL = if(obj.hits > 0, "(H ", "(M ")
       server.identity
       if(req.http.Fastly-Tmp-Obj-TTL && req.http.Fastly-Tmp-Obj-Grace, " " req.http.Fastly-Tmp-Obj-TTL " " req.http.Fastly-Tmp-Obj-Grace " ", " - - ")
       if(resp.http.Age, resp.http.Age, "-")
       ") "
       if(resp.http.Fastly-Debug-TTL, resp.http.Fastly-Debug-TTL, "");
    set resp.http.Fastly-Debug-Digest = digest.hash_sha256(req.digest);
  } else {
    unset resp.http.Fastly-Debug-Path;
    unset resp.http.Fastly-Debug-TTL;
    unset resp.http.Fastly-Debug-Digest;
  }

Here we set 3 Fastly debug headers :

resp.http.Fastly-Debug-Path : show which nodes the request used ( the delivery and fetch node)

resp.http.Fastly-Debug-TTL : Includes the TTL information

resp.Fastly-Debug-FF : If shielding is on

  # add or append X-Served-By/X-Cache(-Hits)
  {
    if(!resp.http.X-Served-By) {
      set resp.http.X-Served-By  = server.identity;
    } else {
      set resp.http.X-Served-By = resp.http.X-Served-By ", " server.identity;
    }
    set resp.http.X-Cache = if(resp.http.X-Cache, resp.http.X-Cache ", ","") if(fastly_info.state ~ "HIT(?:-|\z)", "HIT", "MISS");
    if(!resp.http.X-Cache-Hits) {
      set resp.http.X-Cache-Hits = obj.hits;
    } else {
      set resp.http.X-Cache-Hits = resp.http.X-Cache-Hits ", " obj.hits;
    }
  }

« `X-Served-By« «  just tells you which node the request passed through. The X cache is going to tell you whether or not it was a hit or a miss. When we looked in cache, did we already have it? Was it hit? If not, was it a miss?

NB : For the X cache response header are hit or miss even when a PASS behavior is called it gets reported out as a miss here on this X cache header

Keep in mind : Miss might be hiding a pass !

http.X-Cache-Hits: The delivery node will say how many times it’s had a hit for this particular object coming out of cache. Keep in mind the delivery nodes gets load balanced across the POP so the X cache hits cna be different.

  if (req.http.X-Timer) {
    set resp.http.X-Timer = req.http.X-Timer ",VE" time.elapsed.msec;
  }

Pulls in the initial X timer value (from VCL_receive) and adds in the time elapsed in milliseconds

  # VARY FIXUP
  {
    # remove before sending to client
    set resp.http.Vary = regsub(resp.http.Vary, "Fastly-Vary-String, ", "");
    if (resp.http.Vary ~ "^\s*$") {
      unset resp.http.Vary;
    }
  }

if (resp.http.Vary ~ "^\s*$") : if the vary header contains nothing

unset resp.http.X-Varnish;

This is an internal diagnostic header, so it’s just being unset

  # Pop the surrogate headers into the request object so we can reference them later
  set req.http.Surrogate-Key = resp.http.Surrogate-Key;
  set req.http.Surrogate-Control = resp.http.Surrogate-Control;


  # If we are not forwarding or debugging unset the surrogate headers so they are not present in the response
  if (!req.http.Fastly-FF && !req.http.Fastly-Debug) {
    unset resp.http.Surrogate-Key;
    unset resp.http.Surrogate-Control;
  }

Here we put Surrogate-Key and Surrogate-Control from the response ( resp) to the request.

So it can be used by an other POP if this one is not the last one ( if there’s a shield after that one ).

What the doc says about http.Faslty-FF : this header as a simple mechanism to determine whether the current data center is acting as an edge or a shield

VCL deliver conclusion :

To sum it up, VCL deliver completes entirely before sending the response to the user. It occurs for each response separately. This is where you can include debugging details and session data specific to the user that shouldn’t be shared with others. It’s also your final opportunity to restart. The key transition calls here are return, deliver, and restart.

Takeaways

• Receive:
  • Normalize input.
  • Select a backend.
  • Block at the edge.
  • Redirect at the edge.
  • Choose not to cache.
• Fetch:
  • Set custom TTL.
  • Modify headers from the backend.
  • Configure rules for serving stale.
  • Set default TTL.
• Deliver:
  • Runs on every response.
  • Adds debugging information.
  • Handles user-specific session data.
  • Last opportunity to restart.

0 personne / cet article