Nomad
Integrate service mesh and API gateway
Consul service mesh is a feature that provides secure service-to-service communication with Transport Layer Security (TLS). In a service mesh, sidecar proxies run alongside each service instance, allowing the service to send requests to localhost
instead of a defined upstream service address. All traffic sent to each application service travels through an outbound and inbound sidecar proxy. As a result, each service has encrypted communication and Consul enforces application routing rules with service intentions.
The next step of this monolith migration integrates Consul service mesh to allow secure TLS connections between each service. It also sets up an API gateway for external public access.
In this tutorial, you deploy a version of HashiCups that uses Consul service mesh while running on private client nodes. Then you deploy an API gateway that allows external public access the services running on the private client node.
Infrastructure overview
At the beginning of the tutorial you have a Nomad and Consul cluster with three server nodes, three private client nodes, and one publicly accessible client node. Each node runs a Consul agent and a Nomad agent.
This infrastructure matches the end state of the previous tutorial.
Prerequisites
This tutorial uses the infrastructure set up in a previous tutorial in this collection, Set up the cluster. Complete that tutorial to set up the infrastructure if you have not done so.
Review configuration files
In your terminal, navigate to the directory that contains the code from the learn-consul-nomad-vm
code repository.
Navigate to the jobs
directory.
$ cd shared/jobs
Additional files for configuring the API gateway and service intentions are located in the shared/jobs
directory. These files include 04.api-gateway.config.sh
, 04.api-gateway.nomad.hcl
, and 04.intentions.consul.sh
.
Review scripts for API gateway configuration
The Consul API gateway allows external traffic to the internal nginx
service.
The repository provides a script, 04.api-gateway.config.sh
, that automates the initial configuration required for Nomad and Consul.
The set up script starts by cleaning up previous Consul configurations, binding rules, auth methods, and the ingress
namespace in Nomad.
/shared/jobs/04.api-gateway.config.sh
echo -e "${_COL}Clean previous configurations.${_NC}"
# Remove route for NGINX
consul config delete -kind http-route -name hashicups-http-route
# Remove Inline certificate
consul config delete -kind inline-certificate -name api-gw-certicate
# Remove API Gateway Listener
consul config delete -kind api-gateway -name api-gateway
# Remove all existing binding rules
# WARNING: if you have existing binding rules you want to maintain, modify this behavior
for i in `consul acl binding-rule list -format json | jq -r .[].ID`; do
consul acl binding-rule delete -id=$i
done
# Delete Nomad namespace
nomad namespace delete ingress
# Delete Consul auth-method
consul acl auth-method delete -name nomad-workloads
if [ "$1 " == "-clean " ]; then
echo -e "${_ERR}Only cleaning selected...Exiting.${_NC}"
exit 0
fi
# ...
Then the script configures the Consul and Nomad ACL integration, which includes setting up the auth method, creating a binding rule, and creating the ingress
namespace in Nomad.
/shared/jobs/04.api-gateway.config.sh
# ...
tee ${_JWT_FILE} > /dev/null << EOF
{
# ...
}
EOF
# This auth method creates an endpoint for generating Consul ACL tokens
# from Nomad workload identities.
consul acl auth-method create \
-name 'nomad-workloads' \
-type 'jwt' \
-description 'JWT auth-method for Nomad services and workloads' \
-config "@${_JWT_FILE}"
# ...
# The 'ingress' namespace will be used to deploy the API Gateway and to identify
# Nomad Jobs that require a token to be generated automatically.
nomad namespace apply \
-description "namespace for Consul API Gateways" \
ingress
# The binding-rule identifies all Nomad Jobs in the 'ingress' namespaces and
# uses the 'builtin/api-gateway' policy to generate a Consul ACL token for them.
# ...
tee ${_BR_FILE} > /dev/null << EOF
{
# ...
}
EOF
curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--connect-to ${CONSUL_TLS_SERVER_NAME}:8443:${_consul_addr} \
--cacert ${CONSUL_CACERT} \
--data @${_BR_FILE} \
--request PUT \
https://${CONSUL_TLS_SERVER_NAME}:8443/v1/acl/binding-rule | jq
Finally, the script generates TLS certificates, creates a listener and route for the API gateway, and then writes them to Consul configurations.
/shared/jobs/04.api-gateway.config.sh
# ...
# Generate TLS certificates
tee ${_ssl_conf_FILE} > /dev/null << EOF
# ...
EOF
openssl genrsa -out ${_ssl_key_file} 4096 2>/dev/null
openssl req -new \
-key ${_ssl_key_file} \
-out ${_ssl_csr_file} \
-config ${_ssl_conf_FILE} 2>/dev/null
openssl x509 -req -days 3650 \
-in ${_ssl_csr_file} \
-signkey ${_ssl_key_file} \
-out ${_ssl_crt_file} 2>/dev/null
export API_GW_KEY=`cat ${_ssl_key_file}`
export API_GW_CERT=`cat ${_ssl_crt_file}`
# Create 'api-gateway-certificate' inline-certificate"
tee ${_GW_certificate_FILE} > /dev/null << EOF
# ...
EOF
consul config write ${_GW_certificate_FILE}
# Create 'api-gateway' HTTP listener on port 8443
tee ${_GW_config_FILE} > /dev/null << EOF
# ...
EOF
consul config write ${_GW_config_FILE}
# Create 'hashicups-http-route' HTTP route "/" > "nginx"
tee ${_GW_route_FILE} > /dev/null << EOF
# ...
EOF
consul config write ${_GW_route_FILE}
Review the API gateway jobspec
The API gateway job is constrained to a node with the ingress
meta role that also runs in the ingress
namespace. It has four tasks: two are for job set up and clean up, one registers the gateway with Consul, and one starts the gateway. The set up, clean up, and service registration tasks each define a lifecycle
block that enforces the order in which they run.
/shared/jobs/04.api-gateway.nomad.hcl
# ...
job "api-gateway" {
namespace = var.namespace
constraint {
attribute = "${meta.nodeRole}"
operator = "="
value = "ingress"
}
group "api-gateway" {
# ...
task "setup" {
driver = "docker"
config {
image = var.consul_image # image containing Consul
command = "/bin/sh"
args = [
"-c",
"consul connect envoy -gateway api -register -deregister-after-critical 10s -service ${NOMAD_JOB_NAME} -admin-bind 0.0.0.0:19000 -ignore-envoy-compatibility -bootstrap > ${NOMAD_ALLOC_DIR}/envoy_bootstrap.json"
]
}
lifecycle {
hook = "prestart"
sidecar = false
}
# ...
}
task "api-gw" {
# ...
config {
image = var.envoy_image # image containing Envoy
args = [
"--config-path",
"${NOMAD_ALLOC_DIR}/envoy_bootstrap.json",
"--log-level",
"${meta.connect.log_level}",
"--concurrency",
"${meta.connect.proxy_concurrency}",
"--disable-hot-restart"
]
}
}
task "service_change" {
# ...
lifecycle {
hook = "poststart"
}
# ...
config {
command = "consul"
args = ["services", "register", "${NOMAD_TASK_DIR}/svc-api-gateway.hcl"]
}
template {
data = <<EOF
service {
id = "api-gateway"
name = "api-gateway"
kind = "api-gateway"
port = 8443
address = ""
meta = {
public_address = "https://{{ env "attr.unique.platform.aws.public-ipv4" }}:8443"
}
}
EOF
destination = "${NOMAD_TASK_DIR}/svc-api-gateway.hcl"
}
}
task "cleanup" {
# ...
lifecycle {
hook = "poststop"
}
# ...
config {
command = "consul"
args = ["services", "deregister", "-id=api-gateway"]
}
}
}
Review the Consul intentions configuration
In a secure configuration, Consul uses intentions to specify allowed communications across services.
The repository provides a script, 04.intentions.consul.sh
, that automates the creation of intentions.
The Consul intentions script cleans up any previous configurations and then creates intentions for the HashiCups services. Intentions are how Consul enforces routes of communication between services.
/shared/jobs/04.intentions.consul.sh
## ...
##-------------------------------------------------------------------------------
## Clean previous configurations
##-------------------------------------------------------------------------------
echo -e "${_COL}Clean previous configurations.${_NC}"
consul config delete -kind service-intentions -name database
consul config delete -kind service-intentions -name product-api
consul config delete -kind service-intentions -name payments-api
consul config delete -kind service-intentions -name public-api
consul config delete -kind service-intentions -name frontend
consul config delete -kind service-intentions -name nginx
## ...
### ----------------------------------------------------------------------------
### Configure Consul Intentions
### ----------------------------------------------------------------------------
## References:
## - https://developer.hashicorp.com/consul/docs/connect/config-entries/service-intentions
echo -e "${_COL}Create Consul intentions for Hashicups and API Gateway${_NC}"
tee ${_int_DB_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "database"
Sources = [
{
Name = "product-api"
Action = "allow"
}
]
EOF
tee ${_int_PROD_API_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "product-api"
Sources = [
{
Name = "public-api"
Action = "allow"
}
]
EOF
tee ${_int_PAY_API_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "payments-api"
Sources = [
{
Name = "public-api"
Action = "allow"
}
]
EOF
tee ${_int_PUB_API_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "public-api"
Sources = [
{
Name = "nginx"
Action = "allow"
}
]
EOF
tee ${_int_FE_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "frontend"
Sources = [
{
Name = "nginx"
Action = "allow"
}
]
EOF
tee ${_int_NGINX_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "nginx"
Sources = [
{
Name = "api-gateway"
Action = "allow"
}
]
EOF
consul config write ${_int_DB_FILE}
consul config write ${_int_PROD_API_FILE}
consul config write ${_int_PAY_API_FILE}
consul config write ${_int_PUB_API_FILE}
consul config write ${_int_FE_FILE}
consul config write ${_int_NGINX_FILE}
Review the HashiCups jobspec
Open the 04.hashicups.nomad.hcl
jobspec file and review the contents.
This version uses the same constraint
as previous versions to run all of the HashiCups services on private client nodes. It integrates Consul service mesh with upstream configurations and configures services to use mTLS through the Envoy sidecar proxy. These configuration attributes are located in the connect
block.
/shared/jobs/04.hashicups.nomad.hcl
job "hashicups" {
# ...
group "db" {
# ...
service {
# ...
connect {
sidecar_service {}
}
# ...
}
# ...
}
# ...
group "public-api" {
# ...
service {
# ...
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "product-api"
local_bind_port = 9090
}
upstreams {
destination_name = "payments-api"
local_bind_port = 8080
}
}
}
}
# ...
}
# ...
}
# ...
}
Deploy Consul API gateway
Set up and deploy the Consul API gateway before you submit the updated jobspec to Nomad. Set up and deploy the Consul API gateway before you submit the updated jobspec to Nomad.
Run the API setup script and jobspec
Set up the API gateway configurations in Consul.
$ ./04.api-gateway.config.sh
Configure environment.
Clean previous configurations.
Config entry deleted: http-route/hashicups-http-route
Config entry deleted: inline-certificate/api-gw-certicate
Config entry deleted: api-gateway/api-gateway
Error deleting namespace: Unexpected response code: 500 (rpc error: namespace not found)
Error deleting auth method "nomad-workloads": Unexpected response code: 404 (Cannot find auth method to delete)
## ...
Create Nomad namespace 'ingress'
Successfully applied namespace "ingress"!
Create Consul binding-rule 'Nomad API gateway'
## ...
Generate TLS certificate for 'hashicups.hashicorp.com'
Create 'api-gateway-certificate' inline-certificate
Config entry written: inline-certificate/api-gw-certificate
Create 'api-gateway' HTTP listener on port 8443
Config entry written: api-gateway/api-gateway
Create 'hashicups-http-route' HTTP route '/' > 'nginx'
Config entry written: http-route/hashicups-http-route
Submit the API gateway job to Nomad.
$ nomad job run 04.api-gateway.nomad.hcl
==> 2024-11-13T15:21:40+01:00: Monitoring evaluation "f25cae62"
2024-11-13T15:21:40+01:00: Evaluation triggered by job "api-gateway"
2024-11-13T15:21:40+01:00: Allocation "c1f36918" created: node "30b5f033", group "api-gateway"
2024-11-13T15:21:42+01:00: Evaluation within deployment: "796699a7"
2024-11-13T15:21:42+01:00: Evaluation status changed: "pending" -> "complete"
==> 2024-11-13T15:21:42+01:00: Evaluation "f25cae62" finished with status "complete"
==> 2024-11-13T15:21:42+01:00: Monitoring deployment "796699a7"
✓ Deployment "796699a7" successful
2024-11-13T15:22:03+01:00
ID = 796699a7
Job ID = api-gateway
Job Version = 0
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
api-gateway 1 1 1 0 2024-11-13T14:32:00Z
Run the Consul intentions setup script
Set up the service intentions in Consul.
$ ./04.intentions.consul.sh
Configure environment.
Clean previous configurations.
Config entry deleted: service-intentions/database
Config entry deleted: service-intentions/product-api
Config entry deleted: service-intentions/payments-api
Config entry deleted: service-intentions/public-api
Config entry deleted: service-intentions/frontend
Config entry deleted: service-intentions/nginx
Create Consul intentions for Hashicups and API Gateway
Config entry written: service-intentions/database
Config entry written: service-intentions/product-api
Config entry written: service-intentions/payments-api
Config entry written: service-intentions/public-api
Config entry written: service-intentions/frontend
Config entry written: service-intentions/nginx
Verify API gateway deployment
Verify that the API gateway deployed successfully.
Use the nomad job
command to retrieve information about the hashicups
job.
$ nomad job allocs --namespace=ingress api-gateway
ID Node ID Task Group Version Desired Status Created Modified
c1f36918 30b5f033 api-gateway 0 run running 58m7s ago 57m46s ago
Use the consul catalog
command to verify that the services are correctly registered inside Consul's catalog.
$ consul catalog services
api-gateway
consul
nomad
nomad-client
Deploy HashiCups with service mesh
After you deploy the Consul API gateway, deploy the new version of the HashiCups job.
Run the HashiCups jobspec
Submit the HashiCups job to Nomad.
$ nomad job run 04.hashicups.nomad.hcl
==> 2024-11-13T16:35:14+01:00: Monitoring evaluation "746b95fe"
2024-11-13T16:35:14+01:00: Evaluation triggered by job "hashicups"
2024-11-13T16:35:14+01:00: Allocation "04fffc73" created: node "b50f2376", group "product-api"
2024-11-13T16:35:14+01:00: Allocation "23d2a286" created: node "b50f2376", group "public-api"
2024-11-13T16:35:14+01:00: Allocation "444528dc" created: node "b50f2376", group "db"
2024-11-13T16:35:14+01:00: Allocation "5af579d5" created: node "7fb20437", group "payments"
2024-11-13T16:35:14+01:00: Allocation "8d4c039a" created: node "7fb20437", group "nginx"
2024-11-13T16:35:14+01:00: Allocation "ffddd50e" created: node "b50f2376", group "frontend"
2024-11-13T16:35:15+01:00: Evaluation within deployment: "5178e7b3"
2024-11-13T16:35:15+01:00: Evaluation status changed: "pending" -> "complete"
==> 2024-11-13T16:35:15+01:00: Evaluation "746b95fe" finished with status "complete"
==> 2024-11-13T16:35:15+01:00: Monitoring deployment "5178e7b3"
✓ Deployment "5178e7b3" successful
2024-11-13T16:36:03+01:00
ID = 5178e7b3
Job ID = hashicups
Job Version = 0
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
db 1 1 1 0 2024-11-13T15:45:49Z
frontend 1 1 1 0 2024-11-13T15:45:54Z
nginx 1 1 1 0 2024-11-13T15:45:34Z
payments 1 1 1 0 2024-11-13T15:45:53Z
product-api 1 1 1 0 2024-11-13T15:45:53Z
public-api 1 1 1 0 2024-11-13T15:46:01Z
Verify the HashiCups deployment
Use the nomad job
command to retrieve information about the hashicups
job.
$ nomad job allocs hashicups
ID Node ID Task Group Version Desired Status Created Modified
04fffc73 b50f2376 product-api 0 run running 5m25s ago 4m45s ago
23d2a286 b50f2376 public-api 0 run running 5m25s ago 4m37s ago
444528dc b50f2376 db 0 run running 5m25s ago 4m49s ago
5af579d5 7fb20437 payments 0 run running 5m25s ago 4m44s ago
8d4c039a 7fb20437 nginx 0 run running 5m25s ago 5m3s ago
ffddd50e b50f2376 frontend 0 run running 5m25s ago 4m44s ago
Use the consul catalog
command to verify that the services are correctly registered inside Consul's catalog.
$ consul catalog services
api-gateway
consul
database
database-sidecar-proxy
frontend
frontend-sidecar-proxy
nginx
nginx-sidecar-proxy
nomad
nomad-client
payments-api
payments-api-sidecar-proxy
product-api
product-api-sidecar-proxy
public-api
public-api-sidecar-proxy
To view the Hashicups application, navigate to the public IP address of the API gateway.
In the following command, note the --namespace
flag. The API gateway runs in a namespace that is separate from the application. You will need to trust the certificate in your browser.
$ nomad node status -verbose \
$(nomad job allocs --namespace=ingress api-gateway | grep -i running | awk '{print $2}') | \
grep -i public-ipv4 | awk -F "=" '{print $2}' | xargs | \
awk '{print "https://"$1":8443"}'
Output from the above command.
https://3.15.17.40:8443
Visit the address in your web browser to view the application.
Before you proceed, interact with the HashiCups application and then observe the changes in the Nomad and Consul UIs.
Cleanup
Stop the HashiCups deployment when you are ready to move on.
$ nomad job stop -purge hashicups
==> 2024-11-04T17:26:49-05:00: Monitoring evaluation "8ff7d2ad"
2024-11-04T17:26:49-05:00: Evaluation triggered by job "hashicups"
2024-11-04T17:26:49-05:00: Evaluation status changed: "pending" -> "complete"
==> 2024-11-04T17:26:49-05:00: Evaluation "8ff7d2ad" finished with status "complete"
Warning
Do not stop the API gateway job. You will use it in the next tutorial.
Next steps
In this tutorial, you deployed the Consul API gateway to secure access to your application. Then you deployed a version of HashiCups that uses Consul service mesh to fully secure services running on private client nodes.
For more information about to Consul API gateway integration with Nomad, refer to Consul ACL with Nomad Workload Identities and Deploy a Consul API Gateway on Nomad.
In the next tutorial, you will deploy the Nomad Autoscaler and use it to scale up the frontend
service of HashiCups.