# GCP production deploy — runbook

> Hosted copy for `docs.ikon168.com`. Canonical path in the project: `docs/operations/GCP-DEPLOY-RUNBOOK.md`.

End-to-end steps for **HTTPS on `DOMAIN`**, **Caddy + Let’s Encrypt**, and the **production compose** stack on the **`yaybpo-cs-agent`** VM.

---

## Target environment (reference)

| Item | Value |
|------|--------|
| **GCP project** | `optimistic-yew-491906-t6` |
| **VM name** | `yaybpo-cs-agent` |
| **Zone** | `asia-southeast1-b` |
| **External IPv4** (current) | `34.143.185.136` |
| **SSH user** (OS login) | Your GCP OS Login / Linux username for this VM |
| **App directory on VM** | `~/cs-agent` |
| **Production hostname** (this deployment) | `ikon168.com` (apex A record → VM) |

Use **`gcloud config set project optimistic-yew-491906-t6`** if your default project differs.

---

## Prerequisites

- `gcloud` installed; **`gcloud auth login`** completed for an account with access to the project.
- **`config/.env`** on the VM with API keys, Telegram, LiveChat (never commit this file).
- **DNS:** Apex **`ikon168.com`** should resolve to the VM. Subdomains **`agent`**, **`docs`**, and **`portal`** should use A records to the same IP before enabling the full Caddy vhost set. `portal.ikon168.com` serves the official IKON168 Customer Portal with the embedded LiveChat widget.

---

## Step 1 — Open firewall (TCP 80 and 443)

Let’s Encrypt needs **80** (HTTP redirect / optional HTTP-01) and **443** (HTTPS and **TLS-ALPN-01**, which Caddy uses by default).

From your machine (project root):

```bash
./scripts/gcp_firewall_production.sh optimistic-yew-491906-t6
```

**Rule name:** `allow-yaybpo-cs-agent-http-https`  
**Allows:** `tcp:80`, `tcp:443` from `0.0.0.0/0` on network `default`.

Verify in the [VPC firewall rules](https://console.cloud.google.com/networking/firewalls/list?project=optimistic-yew-491906-t6) console or:

```bash
gcloud compute firewall-rules describe allow-yaybpo-cs-agent-http-https \
  --project=optimistic-yew-491906-t6
```

---

## Step 2 — Sync code to the VM (if the server lacks the current project files)

The production stack needs at least:

- `deploy/docker-compose.production.yml`
- `infra/caddy/Caddyfile` (mounted into Caddy)
- `client_docs/` if serving `docs.ikon168.com`
- `customer_portal/` if serving `portal.ikon168.com`
- `scripts/compose_production.sh`, `scripts/vm_production_deploy.sh`

If the VM does not have these (older tarball-only trees), copy from your laptop:

```bash
PROJECT_ROOT="/path/to/YAYBPO-CS-Agent"
ZONE=asia-southeast1-b
PROJ=optimistic-yew-491906-t6
# Replace YOUR_SSH_USER with your GCP VM Linux username (OS Login).
VM=YOUR_SSH_USER@yaybpo-cs-agent

gcloud compute scp "$PROJECT_ROOT/deploy/docker-compose.production.yml" "$VM:~/cs-agent/" --zone="$ZONE" --project="$PROJ"
gcloud compute scp --recurse "$PROJECT_ROOT/infra" "$VM:~/cs-agent/" --zone="$ZONE" --project="$PROJ"
gcloud compute scp --recurse "$PROJECT_ROOT/client_docs" "$VM:~/cs-agent/" --zone="$ZONE" --project="$PROJ"
gcloud compute scp --recurse "$PROJECT_ROOT/customer_portal" "$VM:~/cs-agent/" --zone="$ZONE" --project="$PROJ"
gcloud compute scp \
  "$PROJECT_ROOT/scripts/compose_production.sh" \
  "$PROJECT_ROOT/scripts/vm_production_deploy.sh" \
  "$PROJECT_ROOT/scripts/verify_production.sh" \
  "$PROJECT_ROOT/scripts/gcp_firewall_production.sh" \
  "$VM:~/cs-agent/scripts/" --zone="$ZONE" --project="$PROJ"
chmod +x ~/cs-agent/scripts/*.sh   # on the VM
```

---

## Step 3 — SSH to the VM

```bash
gcloud compute ssh yaybpo-cs-agent --zone=asia-southeast1-b --project=optimistic-yew-491906-t6
```

---

## Step 4 — Deploy on the VM

```bash
cd ~/cs-agent
chmod +x scripts/*.sh
./scripts/vm_production_deploy.sh
```

The script will:

- Append **`DOMAIN=ikon168.com`** if `DOMAIN` is missing (override: `DOMAIN=agent.ikon168.com ./scripts/vm_production_deploy.sh`).
- Generate **`DASHBOARD_PASSWORD`** once if missing (hex string) — **copy the printed value** to your secure store; it is also in **`~/cs-agent/config/.env`** on the server.
- Stop the dev stack (`deploy/docker-compose.yml` on **8080**).
- Run **`compose_production.sh`** → `docker compose -f deploy/docker-compose.production.yml up -d --build`.

### Docker socket permission

If the login user cannot access `/var/run/docker.sock`, the scripts fall back to **`sudo docker compose`**. To use Docker without `sudo` after the next login:

```bash
sudo usermod -aG docker "$USER"
# then log out and back in, or: newgrp docker
```

---

## Step 5 — Verify

From any machine with the project source tree:

```bash
./scripts/verify_production.sh https://ikon168.com
```

Or:

```bash
curl -fsS https://ikon168.com/health
# expect: {"status":"ok"} and HTTP 200
```

**TLS:** Caddy obtains a Let’s Encrypt certificate for **`DOMAIN`**. The first HTTPS requests may fail for a few seconds while issuance completes; retry.

**Dashboard:** `https://ikon168.com/` uses HTTP Basic Auth when `DASHBOARD_PASSWORD` is set (default username `admin` unless `DASHBOARD_USERNAME` is set). **`/health`** remains unauthenticated for probes.

---

## Step 6 — Post-deploy verification

Run **`./scripts/verify_production.sh`** and **`/health`** as above. For broader manual QA, use checklists in the **source repository** at **`docs/operations/`** (not mirrored on the public docs site).

---

## Rollback to HTTP :8080

On the VM:

```bash
cd ~/cs-agent
sudo docker compose --env-file config/.env -f deploy/docker-compose.production.yml down
sudo docker compose -f deploy/docker-compose.yml up -d
```

---

## Troubleshooting

| Symptom | What to check |
|--------|----------------|
| Port 80/443 timeout from the internet | Firewall rule exists; VM has a public IP; no upstream ISP block |
| TLS handshake error / “internal error” right after deploy | Wait for Let’s Encrypt; check `sudo docker logs cs-agent-caddy-1` |
| Certificate never issues | **DNS** must point to this VM; **443** must reach Caddy; check Caddy logs for ACME errors |
| `permission denied` on Docker socket | Use `sudo` or add user to `docker` group (see above) |
| Wrong vhost | `DOMAIN` in `config/.env` must match the name clients use in the browser |

---

## Execution record — 2026-04-16 (UTC)

The following was performed against **`optimistic-yew-491906-t6`** with an authenticated `gcloud` CLI.

1. **Firewall:** Created **`allow-yaybpo-cs-agent-http-https`** (`tcp:80`, `tcp:443`, ingress `0.0.0.0/0` on `default`).
2. **VM:** **`yaybpo-cs-agent`** in **`asia-southeast1-b`**, external IP **`34.143.185.136`**, **`RUNNING`**.
3. **DNS:** **`ikon168.com`** A record resolved to **`34.143.185.136`** (matches VM).
4. **Files:** **`deploy/docker-compose.production.yml`**, **`infra/`**, and production **`scripts/`** were copied to **`~/cs-agent`** (server tree predated those paths). For **parity with the project source tree** (e.g. dashboard Basic Auth in **`src/dashboard/api.py`**), sync **`src/`** as well: `gcloud compute scp --recurse ./src YOUR_SSH_USER@yaybpo-cs-agent:~/cs-agent/ --zone=asia-southeast1-b --project=optimistic-yew-491906-t6` then rebuild the agent image (`YOUR_SSH_USER` = VM Linux username).
5. **`config/.env`:** **`DOMAIN=ikon168.com`** appended; **`DASHBOARD_PASSWORD`** generated and appended on first run — **store securely**; do not commit. Retrieve on VM with:  
   `grep '^DASHBOARD_PASSWORD=' ~/cs-agent/config/.env`
6. **Docker:** First `vm_production_deploy` failed without `sudo`; production stack was brought up with **`sudo docker compose ...`**. The SSH user was added to the **`docker`** group for future sessions.
7. **Stack:** **`cs-agent-agent-1`** + **`cs-agent-caddy-1`** started; Caddy used **TLS-ALPN-01** with Let’s Encrypt; certificate **CN=ikon168.com** issued successfully.
8. **Verification:** **`./scripts/verify_production.sh https://ikon168.com`** → **OK**; **`curl https://ikon168.com/health`** → **200** and **`{"status":"ok"}`**.
9. **QA:** Manual checklists in **`docs/operations/`** in the repo for the team to run against the live URL when needed.

---

## Related

- [PRODUCTION-READINESS.md](PRODUCTION-READINESS.md)
- Source repository — **`docs/operations/TEST-CASES-CHECKLIST.md`**, **`docs/operations/TESTING_GUIDE.md`**