Run CoRE Stack Backend with Docker¶

CoRE Stack Docker installation has two ways. Both use the same Compose stack from installation/DOCKER.md. Airflow is installed beside that stack, only on the first way.

The switch is LAYER_GENERATION_SYNC_MODE in nrm_app/.env (False in the template). One request can override it with layer_generation_mode (sync or async). Aliases: layerGenerationMode, layer_mode, mode. SYNC_LAYER is a separate older flag.

First install: Install CoRE Stack. GEE and GCS background: Google Earth Engine, Google Cloud Storage.

Installation in two ways¶

1. CoRE Stack with Airflow integration¶

Do these in order:

1. Read what Airflow is on this path.
2. Install Airflow and STACD.
3. Install CoRE Stack with the steps in installation/DOCKER.md, and set LAYER_GENERATION_SYNC_MODE=True.
4. Point the DAGs at the API, upload them, trigger a run, and use re-exec when a task fails.

What Airflow is¶

Apache Airflow is a workflow scheduler. A workflow is a DAG (directed acyclic graph): tasks are nodes, and edges say which task must finish before the next one starts. The scheduler starts a task only after its upstream tasks succeed. The web UI shows that graph, the logs, and a way to run the workflow again with new parameters.

CoRE Stack uses Airflow when layer generation should be that graph, instead of one API call handed to Celery:

• Two root datasets run first: admin boundary and microwatersheds.
• Each algorithm task POSTs a Django computing API and waits for a finished body. That requires sync mode (LAYER_GENERATION_SYNC_MODE=True, or "layer_generation_mode": "sync" on the request). The body includes asset_id and stac_items.
• The matching *_Asset task registers that STAC payload.
• STACD stores which algorithms succeeded or failed for a state, district, and block, so a later run can retry only the failures.

STACD (SpatioTemporal Asset Catalog for Dataflows) is the YAML layer on Airflow. You upload three files — DAG, algorithm repo, and dataset repo. STACD loads them into its database, generates the Python DAG, and places it in Airflow’s dags/ folder. You then trigger and watch that DAG in the Airflow UI. Framework reference: STACD Framework (dev).

Airflow is a separate install next to Compose. GeoServer already listens on 8080, so the Airflow webserver uses 8081.

Install Airflow and STACD¶

Authoritative install: STACD Framework README (dev). You need Python 3.10+, pip, and Git.

mkdir airflow_stacd && cd airflow_stacd
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip

export AIRFLOW_HOME=$(pwd)/airflow
mkdir -p "$AIRFLOW_HOME"

AIRFLOW_VERSION=2.10.4
PYTHON_VERSION=$(python -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')")
pip install "apache-airflow==${AIRFLOW_VERSION}" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

git clone -b dev https://github.com/SaharshLaud/STACD_framework.git
cp -r STACD_framework/stacd "$AIRFLOW_HOME/"
cp -r STACD_framework/plugins "$AIRFLOW_HOME/"

export PYTHONPATH=$AIRFLOW_HOME:$AIRFLOW_HOME/stacd:$PYTHONPATH
airflow db init
airflow users create \
  --username admin \
  --firstname admin \
  --lastname admin \
  --role Admin \
  --email admin@example.com

airflow users create asks for a password.

Start the scheduler and the webserver in two terminals. In each one, cd to airflow_stacd, activate venv, and export AIRFLOW_HOME and PYTHONPATH as above.

Terminal 1:

airflow scheduler

Terminal 2 — port 8081, because GeoServer uses 8080:

airflow webserver --port 8081

Open http://localhost:8081 and log in. airflow standalone is fine only after you move GeoServer (GEOSERVER_PORT) or Airflow off 8080.

$AIRFLOW_HOME then contains stacd/ (database, DAG generator, YAML configs) and plugins/ (the STACD pages in the Airflow menu). Airflow creates dags/ when the first workflow is uploaded.

Leave this running and continue with the CoRE Stack install below. The auth token and the DAG upload happen after the API is up.

Install CoRE Stack (with Airflow)¶

Follow Docker install steps — the same sequence as installation/DOCKER.md.

In nrm_app/.env, before the first docker compose up, also set:

LAYER_GENERATION_SYNC_MODE=True

Apply a later change with:

docker compose --env-file nrm_app/.env up -d --force-recreate backend

url: values in the algorithm YAML must be reachable from the Airflow worker. The bundled files use http://core-stack:8000/... (the Compose service name). When Airflow runs on the host, change those URLs to http://127.0.0.1:8000/... (or the LAN IP) before you upload.

A single request can force sync even when the env default is False:

{
  "state": "Rajasthan",
  "district": "Bhilwara",
  "block": "Mandalgarh",
  "gee_account_id": 1,
  "layer_generation_mode": "sync"
}

Point Airflow at the API¶

After step 6 of the Docker install returns a token, store it as an Airflow Variable so DAG tasks can call the APIs. In the Airflow UI use Admin → Variables, or:

airflow variables set CORESTACK_AUTH_TOKEN '<access token>'

You can also log in yourself:

curl -s -X POST http://127.0.0.1:8000/api/v1/auth/login/ \
  -H "Content-Type: application/json" \
  -d '{"username":"<admin_user>","password":"<password>"}'

Copy access into CORESTACK_AUTH_TOKEN. Tokens expire (typically 90 days). Reset the variable when calls start returning 401.

Upload the DAG¶

The backend ships the STACD trio for both layer DAGs:

Each folder has three files. Upload them in the Airflow UI:

1. Top menu → STACD → Initialize Workflow
2. DAG YAML → dag.yaml
3. Algorithm Repo YAML → algo_repo.yaml
4. Dataset Repo YAML → dataset_repo.yaml
5. Click Initialize Workflow

Success looks like: Database initialized | DAG script generated | DAG CoreStack_DAG_LocalCompute_StaticLayers deployed to Airflow.

STACD writes the YAMLs under $AIRFLOW_HOME/stacd/yaml_configs/, builds the Python DAG, and drops it in $AIRFLOW_HOME/dags/. Wait up to 30 seconds, then refresh DAGs. Toggle the DAG ON.

Repeat for Dynamic Layers if you need yearly / inter-dependent layers (start_year, end_year, gee_account_id).

Leave the generated .py as STACD wrote it. To change the graph later, use STACD Register Algorithm, Register Dataset, or Update DAG.

Trigger the DAG¶

1. DAGs → CoreStack_DAG_LocalCompute_StaticLayers
2. Click ▶ Trigger DAG
3. Set conf / params, then Trigger

REST uses the same conf. Enable Basic Auth in airflow.cfg ([api] auth_backends = airflow.api.auth.backend.basic_auth) and restart the webserver:

curl -X POST "http://127.0.0.1:8081/api/v1/dags/CoreStack_DAG_LocalCompute_StaticLayers/dagRuns" \
  -H "Content-Type: application/json" \
  -u "admin:admin" \
  -d '{
    "conf": {
      "execution_type": "fullexec",
      "state": "jharkhand",
      "district": "dumka",
      "block": "masalia"
    }
  }'

Watch Grid or Graph. Each algorithm task POSTs a sync layer API; the matching *_Asset task registers STAC from that response.

Full DAG (Airflow Graph)¶

Open the DAG → Graph. That is the full static-layers workflow: one branch, two root datasets, every algorithm, then every output asset.

determine_execution_path is the only branch. It has a direct edge to every algorithm and to the two root datasets so resume_exec / update_* can start in the middle of the graph. fullexec still runs the roots first (Admin_Boundary_Asset, MWS_Boundaries), then the algorithms that consume them, then each *_Asset.

flowchart TB
  B[determine_execution_path]
  B --> Admin[Admin_Boundary_Asset]
  B --> MWS[MWS_Boundaries]
  B --> Livestock
  B --> Antyodaya
  B --> Fac[Facilities_Proximity]
  B --> Aquifer_Vector
  B --> Drainage_Density
  B --> River
  B --> Canal
  B --> DEM[Digital_Elevation_Model]
  B --> Drainage_Lines
  B --> Rest[Restoration_Opportunity]
  B --> SOGE_Vector
  B --> LCW[LCW_Conflict]
  B --> Agro[Agro_Ecological]
  B --> Factory_CSR
  B --> Green_Credit
  B --> Mining
  B --> Nat[Natural_Depression]
  B --> Dist[Dist_to_Drainage]
  B --> Catch[Catchment_Area]
  B --> Slope[Slope_Percentage]
  B --> Conn[MWS_Connectivity]
  B --> Cent[MWS_Centroid]
  B --> Soil[Soil_Type]
  Admin --> Livestock
  Admin --> Antyodaya
  Admin --> Fac
  MWS --> Aquifer_Vector
  MWS --> Drainage_Density
  MWS --> River
  MWS --> Canal
  MWS --> DEM
  MWS --> Drainage_Lines
  MWS --> Rest
  MWS --> SOGE_Vector
  MWS --> LCW
  MWS --> Agro
  MWS --> Factory_CSR
  MWS --> Green_Credit
  MWS --> Mining
  MWS --> Nat
  MWS --> Dist
  MWS --> Catch
  MWS --> Slope
  MWS --> Conn
  MWS --> Cent
  MWS --> Soil
  Livestock --> Livestock_Asset
  Antyodaya --> Antyodaya_Asset
  Fac --> Facilities_Proximity_Asset
  Aquifer_Vector --> Aquifer_Vector_Asset
  Drainage_Density --> Drainage_Density_Asset
  River --> River_Asset
  Canal --> Canal_Asset
  DEM --> Digital_Elevation_Model_Asset
  Drainage_Lines --> Drainage_Lines_Asset
  Rest --> Restoration_Opportunity_Asset
  SOGE_Vector --> SOGE_Vector_Asset
  LCW --> LCW_Conflict_Asset
  Agro --> Agro_Ecological_Asset
  Factory_CSR --> Factory_CSR_Asset
  Green_Credit --> Green_Credit_Asset
  Mining --> Mining_Asset
  Nat --> Natural_Depression_Asset
  Dist --> Dist_to_Drainage_Asset
  Catch --> Catchment_Area_Asset
  Slope --> Slope_Percentage_Asset
  Conn --> MWS_Connectivity_Asset
  Cent --> MWS_Centroid_Asset
  Soil --> Soil_Type_Asset

Re-exec (resume_exec)¶

Re-exec on these DAGs is the STACD primitive execution_type=resume_exec. Airflow’s Clear-and-rerun of the whole Graph is a different action.

On failure, log_algo_failure writes the failed algorithm, region, and years into the STACD database. The next trigger with the same state / district / block (and years, if used) plus resume_exec queries that table and runs the failed algorithm nodes. Successful tasks stay skipped.

Trigger a re-exec from the UI (▶ Trigger DAG) or REST:

curl -X POST "http://127.0.0.1:8081/api/v1/dags/CoreStack_DAG_LocalCompute_StaticLayers/dagRuns" \
  -H "Content-Type: application/json" \
  -u "admin:admin" \
  -d '{
    "conf": {
      "execution_type": "resume_exec",
      "state": "jharkhand",
      "district": "dumka",
      "block": "masalia"
    }
  }'

If resume_exec finds no failed tasks for those params, determine_execution_path raises ValueError and the run fails immediately — use fullexec for a full run. The same happens for update_dag when every algorithm already has a successful execution.

Airflow Clear on selected failed boxes re-runs those task instances in the same dag run. resume_exec starts a new run that STACD can record, so lineage and get_failed_executions stay consistent.

2. CoRE Stack without Airflow¶

Skip Airflow and STACD. Leave the template default:

LAYER_GENERATION_SYNC_MODE=False

Follow Docker install steps. Celery workers run layer jobs in the background. Call a computing API with "layer_generation_mode": "async", or omit the field. The response is initiated. Follow the work here:

docker compose --env-file nrm_app/.env logs -f backend celery-nrm celery-layer-bulk

Example:

{
  "state": "Rajasthan",
  "district": "Bhilwara",
  "block": "Mandalgarh",
  "gee_account_id": 1,
  "layer_generation_mode": "async"
}

The same stack can take the Airflow path later: install Airflow and STACD, set LAYER_GENERATION_SYNC_MODE=True, and recreate backend. You do not reinstall Compose.

Docker install steps¶

These are the steps in installation/DOCKER.md. Run every Compose command from the repository root with --env-file nrm_app/.env. Compose does not auto-load that file.

When it is done you have:

Postgres is on 127.0.0.1:5432 (DB_USER / DB_PASSWORD / DB_NAME from nrm_app/.env).

docker compose --env-file nrm_app/.env down keeps volumes. Adding -v deletes the database, GeoServer catalog, and Redis. Files under CORESTACK_HOST_DATA_DIR stay.

1. Before you start¶

For the GPU jobs you also need an NVIDIA driver and the NVIDIA Container Toolkit. This must print your GPU:

docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi

On a network that only reaches the internet through a proxy (for example a campus network), read Behind a proxy first.

If this machine already has CoRE Stack containers or images from an earlier installation, remove them first.

2. Get the code¶

git clone https://github.com/core-stack-org/core-stack-backend.git
cd core-stack-backend

3. Create the settings file¶

cp installation/docker/env.template nrm_app/.env
chmod 600 nrm_app/.env

nrm_app/.env holds every setting and password. Docker Compose and Django both read it.

4. Edit nrm_app/.env¶

Set the admin account you will log in with:

DJANGO_SUPERUSER_USERNAME=admin
DJANGO_SUPERUSER_EMAIL=you@example.com
DJANGO_SUPERUSER_PASSWORD='choose-a-password'

Way 1 (Airflow): set LAYER_GENERATION_SYNC_MODE=True. Way 2: leave it False.

On a machine with an NVIDIA GPU, uncomment this line (see GPU and long jobs):

COMPOSE_PROFILES=heavy

On a server, set an absolute data root before the first start, for example CORESTACK_HOST_DATA_DIR=/srv/core-stack-data. Replace DB_PASSWORD and GEOSERVER_PASSWORD before any shared host. Leave everything else as it is for a first local run.

5. Optional: download the admin boundaries yourself¶

The first start downloads the admin-boundary archive (about 600 MB) from Google Drive. To use a browser download instead, which is often faster, download it from here and save it as data/dataset.7z in the repository.

6. Build and start¶

docker compose --env-file nrm_app/.env up -d --build

The first run takes 5 to 60 minutes, depending on your connection. The command waits while the database is set up and the data is downloaded. If it is interrupted, run the same command again.

One-shot jobs run before Gunicorn:

1. app-init completes nrm_app/.env and normalizes database, GeoServer, Celery, and runtime values.
2. database-init creates installation-local migrations, applies them with --fake-initial, collects static files, and loads seed data once.
3. geoserver-init reconciles workspaces and bundled styles.
4. data-download downloads only requested or missing source layers.
5. gee-config discovers optional mounted GEE JSON.
6. tehsil-watershed-setup downloads active tehsil watershed layers from the GeoServer mws WFS workspace.
7. Gunicorn and the queue-specific Celery workers start.

7. Check that it works¶

docker compose --env-file nrm_app/.env ps -a

• app-init, database-init, geoserver-init, data-download, gee-config, and tehsil-watershed-setup show Exited (0).
• backend, postgres, redis, and geoserver show Up (healthy).
• The celery-* workers show Up. celery-heavy is there only with COMPOSE_PROFILES=heavy.

If a job shows a non-zero exit code, read its log: docker compose --env-file nrm_app/.env logs <service>.

Log in to the API. This reads the username and password from nrm_app/.env and keeps the token in $TOKEN for Test the APIs:

export no_proxy=localhost,127.0.0.1 NO_PROXY=localhost,127.0.0.1
TOKEN=$(set -a; . nrm_app/.env; set +a; python3 -c '
import json, os, urllib.request
req = urllib.request.Request("http://localhost:8000/api/v1/auth/login/",
    data=json.dumps({"username": os.environ["DJANGO_SUPERUSER_USERNAME"],
                     "password": os.environ["DJANGO_SUPERUSER_PASSWORD"]}).encode(),
    headers={"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req))["access"])')
echo "${TOKEN:0:20}"

It prints the start of a token (eyJhbGci...). You can also log in to Django admin at http://localhost:8000/admin/.

On way 1, continue with Point Airflow at the API.

What to set up next¶

The stack now runs. Set up only what you need:

Other settings are listed in Settings.

Data for local compute¶

Local computation ("compute": "local" in a request) reads its inputs from data/base_layers/. All of it is in this Google Drive folder, in the same layout as data/base_layers/. Download what you need and place it at the same path.

Files can be added while the stack runs; no restart is needed.

Runoff, ET, and pan-India annual hydrology can also be generated with the APIs in Test the APIs. That takes many hours. Tehsil hydrology needs the pan-India annual layer for every year it covers; the API tells you which years are missing.

With S3 credentials for the CoRE Stack datasets bucket, terrain, LULC, aquifer, and microwatersheds can be downloaded automatically: set S3_ACCESS_KEY, S3_SECRET_KEY, S3_REGION, S3_BUCKET, and SKIP_BASE_LAYER_DOWNLOAD=0, then run:

docker compose --env-file nrm_app/.env run --rm data-download

Large-download switches (set in nrm_app/.env, or on one invocation). 1 means skip:

SKIP_ADMIN_BOUNDARY_DOWNLOAD=1 \
SKIP_TEHSIL_WATERSHEDS=1 \
docker compose --env-file nrm_app/.env up -d --build

Tehsil watersheds land at <CORESTACK_HOST_DATA_DIR>/data/base_layers/tehsil_watersheds/<state>/<district>/<tehsil>.gpkg. Retry a failed fetch with docker compose --env-file nrm_app/.env run --rm tehsil-watershed-setup.

Google Earth Engine¶

The stack starts without Earth Engine. Local compute does not need it. Earth Engine jobs need a Google Cloud service account with Earth Engine access and its JSON key.

1. Open http://localhost:8000/admin/gee_computing/geeaccount/add/ and log in.
2. Fill in a name, the client_email from the JSON as the service account email, and upload the JSON as the credentials file. Save.
3. Open the account again, set Helper account to the same account, and save.
4. The account id is the number in the page address (.../geeaccount/1/change/). Put it in nrm_app/.env:

GEE_DEFAULT_ACCOUNT_ID=1
GEE_HELPER_ACCOUNT_ID=1

1. Apply it:

docker compose --env-file nrm_app/.env up -d --force-recreate

The key is stored encrypted in the database and the uploaded file is deleted, so keep your own copy. The encryption key is FERNET_KEY in nrm_app/.env. If it changes, upload the JSON again.

Pass that numeric id as gee_account_id on computing requests. Background: Google Earth Engine.

Optional Google Cloud Storage¶

GEE-backed raster publication also needs a bucket. Create it in us-central1, grant the same service account roles/storage.objectViewer, roles/storage.legacyBucketReader, and roles/storage.objectAdmin, then set GCS_BUCKET_NAME (and GEE_STORAGE_PROJECT if the pipeline reads it) in nrm_app/.env and recreate the stack as above. Details: Google Cloud Storage.

Use SKIP_GEE_CONFIG=1 to disable the gee-config init job.

NASA Earthdata¶

The ET download (/api/v1/et_download/) fetches FLDAS data from NASA GES DISC.

1. Create an account at urs.earthdata.nasa.gov.
2. In your profile, under Applications → Authorized Apps, approve NASA GESDISC DATA ARCHIVE.
3. Put the login in nrm_app/.env. Keep the single quotes if the password contains $:

USERNAME_GESDISC=your-username
PASSWORD_GESDISC='your-password'

1. Apply it:

docker compose --env-file nrm_app/.env up -d --force-recreate

A wrong password or an unapproved application makes the task fail with an HTML page from GES DISC in the celery-heavy log.

GPU and long jobs¶

Four endpoints start jobs that run for hours:

They run on the celery-heavy worker, one at a time, so they never block the other layers. COMPOSE_PROFILES=heavy in nrm_app/.env creates that worker and gives it the GPU. Without it, these four endpoints answer 503.

To run the three CPU jobs on a machine without a GPU, set both:

COMPOSE_PROFILES=heavy
GPU_AVAILABLE=False

After changing the profile:

docker compose --env-file nrm_app/.env up -d --remove-orphans

Check that the worker sees the GPU:

docker compose --env-file nrm_app/.env exec celery-heavy nvidia-smi

Behind a proxy¶

Image pulls are done by the Docker daemon, which needs its own proxy setting: see Docker daemon proxy. docker info | grep -i proxy shows the current one.

Builds and containers use the proxy from your shell. If http_proxy and https_proxy are exported, nothing else is needed. Otherwise uncomment and set these in nrm_app/.env:

HTTP_PROXY=http://proxy.example.org:3128
HTTPS_PROXY=http://proxy.example.org:3128

Service names (postgres, redis, geoserver, backend) stay on NO_PROXY. For your own curl calls to the stack, keep local addresses off the proxy:

export no_proxy=localhost,127.0.0.1 NO_PROXY=localhost,127.0.0.1

Settings¶

All in nrm_app/.env. After a change, run docker compose --env-file nrm_app/.env up -d --force-recreate.

Test the APIs¶

Log in first (check that it works). Base URL: http://127.0.0.1:8000. Computing APIs use the JWT in $TOKEN.

On way 2 each request answers at once and queues a task. Follow it in the worker log, for example docker compose --env-file nrm_app/.env logs -f celery-nrm. On way 1 the same call waits until the layer is finished.

api() { curl -s -X POST "http://localhost:8000/api/v1/$1/" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d "$2"; echo; }

Tehsil hydrology needs start_year 2017. Pan-India requests take one year per call: end_year is start_year + 1.

Admin-boundary generation (used by the Airflow static-layers DAG) is POST /api/v1/generate_block_layer/:

curl -s http://127.0.0.1:8000/api/v1/generate_block_layer/ \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "state": "Rajasthan",
    "district": "Bhilwara",
    "block": "Mandalgarh",
    "gee_account_id": 1,
    "layer_generation_mode": "async"
  }'

More routes: Computing API Endpoints. Auth errors: API Errors.

Postman¶

Everyday use¶

docker compose --env-file nrm_app/.env ps
docker compose --env-file nrm_app/.env logs -f backend
docker compose --env-file nrm_app/.env stop
docker compose --env-file nrm_app/.env up -d

After git pull:

docker compose --env-file nrm_app/.env up -d --build --force-recreate

Commands that write files, such as manage.py commands, should run as your user so the files stay yours:

docker compose --env-file nrm_app/.env exec --user "$(id -u):$(id -g)" backend python manage.py <command>

Celery Beat is opt-in:

docker compose --env-file nrm_app/.env --profile periodic up -d celery-beat

docker compose --env-file nrm_app/.env down -v deletes the database, GeoServer, and Redis data. data/ on the host is kept.

Back up the database:

docker compose --env-file nrm_app/.env --profile maintenance run --rm database-backup

The dump is written to backups/postgres/. GeoServer catalog (stop GeoServer first):

docker compose --env-file nrm_app/.env stop geoserver
docker compose --env-file nrm_app/.env --profile maintenance run --rm geoserver-backup
docker compose --env-file nrm_app/.env start geoserver

Remove an earlier installation¶

This deletes the containers, the database, GeoServer and Redis data, the images, and the build cache. data/ on the host is kept.

docker compose -p core-stack down --volumes --remove-orphans
docker rmi corestack-backend-local:latest postgres:16-bookworm redis:7-alpine kartoza/geoserver:2.24.4 alpine:3.20
docker builder prune -af

No such image for one of the images is fine. Check that nothing is left. Each of the first three commands prints only its header, and the last prints nothing:

docker ps -a --filter name=core-stack
docker volume ls --filter name=core-stack
docker network ls --filter name=corestack
docker images | grep -E 'corestack|postgres|redis|geoserver|alpine'

To also delete the downloaded and computed data, remove data/ in the repository, or under CORESTACK_HOST_DATA_DIR if you set it:

rm -rf data

Use sudo if an older installation left files owned by root. The next start downloads the admin boundaries again. See step 5 to use your own copy.

Running on a server¶

• Change DB_PASSWORD, GEOSERVER_PASSWORD, and the admin password before the first start.
• Set DEBUG=False, and ALLOWED_HOSTS to the server's host name.
• Keep ports bound to 127.0.0.1; put an HTTPS reverse proxy in front.
• Set CORESTACK_HOST_DATA_DIR to a path on a disk with room for the data, for example /srv/core-stack-data.
• For way 1, set LAYER_GENERATION_SYNC_MODE=True and run Airflow on 8081. Leave False for way 2.
• Back up PostgreSQL, the GeoServer catalog, CORESTACK_HOST_DATA_DIR/data, and GEE secrets.
• Pin the Git commit and image tag. Review Beat schedules before enabling the periodic profile.

Troubleshooting¶

To stop a long job that is running on celery-heavy:

docker compose --env-file nrm_app/.env kill celery-heavy
docker compose --env-file nrm_app/.env exec celery-nrm celery -A nrm_app purge -Q heavy -f
docker compose --env-file nrm_app/.env exec redis redis-cli del unacked unacked_index
docker compose --env-file nrm_app/.env up -d celery-heavy

Without the purge and redis-cli steps the job starts again when the worker restarts. The redis-cli step also drops tasks started but not finished on other workers.

More setup fixes: Setup Troubleshooting.