Cluster Docker Services¶

Short how-to for putting your app on the Tower Services cluster. Anyone who wants to share a service can follow this — the live Drone / CEM / DIY LULC apps are examples, not the whole cluster. Start with Tower Services if you do not know the setup or why Airflow is here. Tick boxes on the Cluster Service Checklist — this page does not repeat that list.

In one pass: env in .env (never in the image) → three host mounts → deps-only image on GHCR or Docker Hub → Airflow if AIRFLOW_API_BASE is set → return a STAC Item → front page + one demo video → Google Sign-In.

1. Config from env¶

Hostnames, passwords, Airflow URLs, and paths come from the environment. Ship .env.example with a comment on every variable. Do not commit a real .env or put secrets in the Dockerfile.

Empty AIRFLOW_API_BASE turns Airflow off (job runs in this container). Do not add COMPUTE_MODE or AIRFLOW_BASE_API_URL.

Credential files (GEE JSON) need both an env var and a path that exists in the container (usually because code/ is mounted). Add a one-command auth check in the repo.

2. Mount code, models, and data¶

The image is deps-only (OS/Python packages + start command). The host keeps three folders:

Rebuild the image only when requirements change. A laptop can mount . at /app; cluster deploy uses the three mounts.

3. README and VERSION¶

The repo README must be enough to install: purpose, Docker, clone layout, .env, docker pull with a pinned tag, run command, health check, upgrade, proxy notes.

Keep a root VERSION (or changelog). Tag images with the same number (:1.2.0), not only :latest.

5. Push a deps-only image¶

Push to GHCR or Docker Hub. The Dockerfile copies requirements only, not application source:

FROM python:3.11-slim
WORKDIR /app
COPY deploy/requirements-docker.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt
EXPOSE 8000
CMD ["uvicorn", "backend:app", "--app-dir", "src", "--host", "0.0.0.0", "--port", "8000"]

docker build -t <registry>/<name>:1.0.0 .
docker push <registry>/<name>:1.0.0

Also: /health, LOG_LEVEL (debug | info | error), --restart unless-stopped, listen on 0.0.0.0, document CPU/RAM/GPU. Avoid host ports that clash with GeoServer (8080) and the backend (8000).

7. Describe the job in STACD YAML¶

Do not hand-write Airflow Python. Upload three YAMLs in Airflow: STACD → Initialize Workflow. Full YAML guide: STACD §11.

url: / the worker must reach this app from Airflow (CORESTACK_API_BASE — not localhost unless they share a network). Pin the Docker image tag. Docker-mode tasks print a STAC Item between ===RESULT_JSON_START=== / ===RESULT_JSON_END===. Record the new dag_id as AIRFLOW_DAG_ID. Update later with STACD plugin pages, not by editing generated .py.

8. Start the job from your app¶

Why the cluster prefers “set”: Tower Services — Why we use Airflow.

# start
curl -u "$AIRFLOW_USERNAME:$AIRFLOW_PASSWORD" -H "Content-Type: application/json" \
  -X POST "$AIRFLOW_API_BASE/dags/$AIRFLOW_DAG_ID/dagRuns" \
  -d '{"conf": {"param1": "value1"}}'
# poll until success or failed
curl -u "$AIRFLOW_USERNAME:$AIRFLOW_PASSWORD" \
  "$AIRFLOW_API_BASE/dags/$AIRFLOW_DAG_ID/dagRuns/$DAG_RUN_ID"

Env to document: AIRFLOW_API_BASE, AIRFLOW_DAG_ID, AIRFLOW_USERNAME / AIRFLOW_PASSWORD (or AIRFLOW_TOKEN), CORESTACK_API_BASE.

UI → your /api/dag/run and /api/dag/status. Never call Airflow from JavaScript (no CORS; credentials stay off the page). Enable Airflow Basic Auth if REST returns 401. Unpause the DAG before the first trigger.

Details: STACD §9.

9. Return a STAC Item¶

The deliverable is a STAC 1.x Feature, not a bare file path or GeoServer layer name. Shape and field names: STAC Specs and stac.core-stack.org.

Must have: type (Feature), stac_version, id, geometry, bbox, collection, properties (title, description, datetime, start_datetime, end_datetime, keywords), links (root, collection, parent), assets.data with a fetchable href. Vectors also need the table extension and table:columns for every attribute. Add style and thumbnail assets when you can.

API-mode services may wrap the Item as { "status", "asset_id", "stac_items": [ … ] } if STACD expects that envelope.

11. Front page and demo video¶

The service URL (for example /drone) opens a landing page like Drone: what the project is, then login, then the working UI. On that same page: one short video that is both a project intro and a create → run → FileBrowser tutorial. The video is reviewed and approved. Same acceptance as checklist §11.

12. Google Sign-In setup¶

Tower Services apps use Google Sign-In so people open your page, click Sign in with Google, and only then can start a job or change data. This is the same rule as checklist §4.

You do not need a paid Google Cloud account. A normal Gmail (or Google Workspace) account is enough. You do not need billing for Sign-In alone — do not start a free trial unless you also need other Cloud products.

This section is for a student, researcher, or individual putting a new service on the cluster. Work through it once on your laptop, then add the cluster URL before you ask for a deploy.

What you are building¶

Google Earth Engine service-account JSON is a different credential. Do not mix it with Sign-In. GEE keys stay under mounts / env as in §1.

sequenceDiagram
  participant You as Person in the browser
  participant UI as Your app UI
  participant Google as Google Sign-In
  participant API as Your app backend
  You->>UI: open landing page, click Sign in with Google
  UI->>Google: open Google account picker
  Google-->>UI: ID token
  UI->>API: call a protected API with the token
  API->>Google: verify the token
  Google-->>API: email and subject are valid
  API-->>UI: allow the request (start job, save data, …)

Words used below¶

Step 1 — Create a Google Cloud project¶

1. Open Google Cloud Console and sign in with the Google account that will own the app credentials (your personal Gmail is fine for a student project).
2. Open the project picker (top bar) → New project.
3. Name it after your service, for example tower-drone-trees or diy-lulc-signin. Create it and select it.

You do not need to link a billing account for Sign-In.

Step 2 — Configure the OAuth consent screen¶

1. Go to APIs & Services → OAuth consent screen (or Google Auth Platform if the console has moved the menu).
2. Choose External (unless you only serve people inside a Google Workspace you administer).
3. Fill the required fields:

1. Scopes: for Tower Services login you usually only need basic profile / email (the defaults for Sign in with Google). Do not request Drive, Gmail, or Cloud scopes unless the app truly needs them.
2. Test users (while the app is in Testing): add every Google account that must sign in during development — yourself, your advisor, teammates. Until you publish the app, only listed test users can complete Sign-In.
3. Save. You can leave the app in Testing for local work and early demos. Publishing for production is a later Google review step; ask a cluster maintainer when your service is accepted.

Step 3 — Create an OAuth Web client¶

1. Go to APIs & Services → Credentials → Create credentials → OAuth client ID.
2. Application type: Web application.
3. Name: for example tower-service-web.
4. Under Authorized JavaScript origins, add every origin where the UI will run. No path, no trailing slash quirks — match what the browser shows in the address bar.

1. Under Authorized redirect URIs, add a URI only if your code uses a redirect callback (server receives ?code=). Example shapes:

If you use the Sign in with Google button with a JavaScript callback (no full-page redirect), you still need the JavaScript origins; redirect URIs may be optional. Follow whatever your chosen library documents.

1. Click Create. Copy:

2. Client ID → GOOGLE_CLIENT_ID (or VITE_GOOGLE_CLIENT_ID / REACT_APP_GOOGLE_CLIENT_ID if the frontend build inlines it)

3. Client secret → GOOGLE_CLIENT_SECRET in server env only

Store both in a password manager or a local .env that is gitignored. Put placeholders in .env.example only.

Step 4 — Put credentials in env (never in source)¶

In .env.example:

# Google Sign-In (OAuth Web client). Never commit real values.
GOOGLE_CLIENT_ID=your-client-id.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=your-client-secret
# Frontend frameworks that bake env at build time often need a VITE_ / REACT_APP_ copy of the client ID only:
# VITE_GOOGLE_CLIENT_ID=your-client-id.apps.googleusercontent.com

Rules:

• Client ID may appear in the browser. Client secret must not.
• Do not hardcode either string in src/, Dockerfiles, or GitHub Actions logs.
• On the cluster, the maintainer injects real values into the host env / Compose file. Your image stays deps-only (§5).

Step 5 — Add Sign-In to the frontend¶

On the landing page (after the project story, before the working tool — see §11):

1. Load Google’s script on pages that show the button:

<script src="https://accounts.google.com/gsi/client" async defer></script>

1. Render the official Sign in with Google button with your GOOGLE_CLIENT_ID.
2. When Google succeeds, your callback receives a credential (ID token). Send that token to your backend on every protected call, for example:

Authorization: Bearer <id-token>

or a cookie your backend sets after it has verified the token once. Prefer verifying on the server; do not trust a bare email string the browser sends.

Official overview: Sign in with Google for Web.

Step 6 — Validate on the backend¶

Your backend must:

1. Read the token from the header or session.
2. Verify it with Google (signature, audience = your Client ID, expiry). Use a maintained library for your language (for example Google’s token-verify helpers, or google-auth in Python).
3. Reject the request with 401 if verification fails or the token is missing.
4. Only then allow: start compute, call Airflow, write under data/, delete outputs, change settings.

Public GETs that only show the marketing landing page can stay open. POST that starts a job must not.

Quick mental test: open the browser private window, skip Sign-In, call your “run” API with curl and no token → must fail.

Step 7 — Protect the right routes¶

Airflow credentials stay on the server (§8). The browser never gets AIRFLOW_PASSWORD. Sign-In proves who may ask your backend to trigger Airflow.

Step 8 — Local laptop vs cluster URLs¶

Do this in order:

1. Get Sign-In working on http://localhost:<port> with that origin (and redirect, if any) listed in the Google client.
2. When the cluster path is known (for example /act4dws5/your-app), edit the same OAuth client and add the production JavaScript origin and redirect URI. Do not create a second client unless you intentionally separate “dev” and “prod” projects.
3. Wait a few minutes after saving Google Console changes if the browser still shows redirect_uri_mismatch or origin errors.
4. Document both URLs in the service README so the next student on the repo knows what to add in Google Cloud.

Same-origin proxy tip: the browser origin is the public host (for example https://www.cse.iitd.ernet.in), not the Docker service name. Put that public origin in Google’s form.

Step 9 — Check that it works¶

If something fails¶

Done when¶

• Landing page shows Sign in with Google.
• Protected APIs reject unauthenticated calls.
• Secrets live only in env; .env.example documents the names.
• Local origins work; cluster origins are listed before go-live.

Then tick checklist §4.

Copy-this: Custom LULC¶

Working pattern: salil-123/Project and the STACD LULC report.

On a laptop you can mount . at /app. On the cluster use the three mounts in §2. Empty AIRFLOW_API_BASE turns the DAG path off. To add the same glue to another HTTP app: those six files + env, then curl /api/health → work endpoint → Airflow trigger.

IIT Delhi proxy¶

If docker pull times out on campus, set the Docker daemon proxy. Do not put registry-1.docker.io or auth.docker.io in NO_PROXY. Same snippet: Tower Services.

Inside containers, set HTTP_PROXY / HTTPS_PROXY and list every Docker hostname you call in both NO_PROXY and no_proxy. Otherwise Squid returns 503 / ERR_DNS_FAIL for names like corestac-stacd-airflow. Full table, Compose template, Nginx path stripping, deploy and update steps: Deploy, update, and troubleshoot.