Scheduled Python on Kubernetes — Without the Kubernetes

We wrote a companion post about running Python on a schedule without servers — why Lambda, Airflow, and GitHub Actions all add friction when all you want is "run this script at 6am." That post is about the scheduling problem. This one is about what happens after you click "Run."

Specifically: where does your code actually execute? How much memory does it get? What happens when it needs 8 GB of RAM to process a large dataset? How do logs get from a container in the cloud to your browser in real time? What happens when a run fails at 3am?

These are infrastructure questions. And if you're considering any platform that runs your code for you — Fastero included — you should know how it works under the hood.

The execution engine

When you trigger a Python run in Fastero — manually, on a cron schedule via the trigger builder, or as a step in a workflow — the platform packages your code into a bundle, uploads it to S3, and dispatches it to one of two execution backends: Kubernetes (batch Jobs) or ECS Fargate (one-shot tasks). The choice depends on the deployment configuration, but from your perspective the behavior is identical. You write Python. It runs.

The Kubernetes path creates a batch/v1 Job with a container spec, resource requests and limits, and a restartPolicy: Never. Your code bundle is downloaded into the pod at /workspace, dependencies are installed, and the entrypoint script runs. The ECS path does the same thing via RunTask on Fargate — same bundle, same environment injection, same log streaming — just on a different orchestrator.

Both paths produce the same observable behavior: your script runs in an isolated container with the dependencies you specified, the environment variables you configured, and the compute resources you selected.

Resource tiers — pick the right size

Not every script needs the same hardware. A 20-line script that hits an API and writes results to Postgres does not need the same resources as one that loads a million-row DataFrame and runs feature engineering.

Fastero gives you four compute tiers:

Each tier has guaranteed minimums (not just limits) — the Small tier guarantees 0.5 vCPU and 1 GB, Medium guarantees 1 vCPU and 2 GB, and so on. This means your script won't get starved by noisy neighbors. The guarantee is a Kubernetes resource request; the limit is the ceiling.

Change the tier per-job. Run your quick API poll on Small, your nightly data warehouse rebuild on X-Large. You're not locked into a single instance size for everything.

Warm pools — skip the cold start

Container cold starts are the silent tax of serverless execution. Pulling a Python image, installing dependencies, initializing the runtime — that can take 30-60 seconds before your script's first line executes.

Fastero maintains a warm execution pool: a long-running ECS service with the runtime already booted and dependencies cached. When you trigger a run, the platform sends a request to a warm runner agent over HTTP, which downloads your code bundle, executes it, and streams results back over WebSocket. No image pull, no pip install, no cold start.

The system-default warm pool stays running continuously. Custom runtime profiles (say, one with specific ML libraries pre-installed) spin up on demand and auto-stop after 10 minutes of inactivity. If the warm path fails for any reason, the system falls back automatically to a cold Fargate task — you get a log line about it, but the run still completes.

This is the kind of infrastructure optimization that's invisible when it works, and impossible to build yourself without managing ECS services, health checks, and fallback logic.

Live log streaming

Every print() in your script shows up in your browser as it happens. Not after the run finishes. Not in a CloudWatch console you have to navigate to. In the same panel where you clicked "Run."

The pipeline works like this: your script's stdout and stderr are captured unbuffered (PYTHONUNBUFFERED=1), relayed through Socket.IO events (pythonRun:log), and rendered in the run detail view. Warm-path runs stream via WebSocket from the runner agent. Cold-path runs tail CloudWatch logs with a polling loop.

Status transitions — queued, starting, running, succeeded, failed — are broadcast as separate pythonRun:update events, so the UI reflects what's happening without you refreshing.

This sounds like a small thing until you've spent 20 minutes in the CloudWatch console searching for the right log group, then the right log stream, then scrolling through container lifecycle noise to find your actual output. Fastero puts the logs where you're already looking.

Connected to everything

Here's where running Python on Fastero differs from running it on Lambda or a random EC2 instance: your script has access to every data source you've already connected.

Fastero's secret manager injects database credentials, API keys, and OAuth tokens as environment variables at runtime. Your script reads os.environ['DATABASE_URL'] the same way it would locally, but the credential is managed centrally, encrypted at rest, and rotated in one place. Every connector you've set up — Postgres, BigQuery, Snowflake, Stripe, or any of the 100+ others — is available to your Python code without hardcoding anything.

This matters most for scripts that bridge systems. Pull from the Stripe API, join with your Postgres billing table, write the reconciled output to Snowflake. That's three connection strings you don't manage in your repo.

A real example

Here's a script that runs nightly on Fastero. It pulls order data from a REST API, transforms it with pandas, loads the results into DuckDB, and triggers a dashboard refresh:

import os
import requests
import pandas as pd
import duckdb
 
# Credentials injected by Fastero at runtime
api_key = os.environ['ORDERS_API_KEY']
db_path = os.environ.get('DUCKDB_PATH', '/workspace/orders.duckdb')
 
# Pull yesterday's orders
resp = requests.get(
    'https://api.example.com/v1/orders',
    headers={'Authorization': f'Bearer {api_key}'},
    params={'since': pd.Timestamp.now().floor('D') - pd.Timedelta(days=1)}
)
orders = pd.DataFrame(resp.json()['data'])
print(f"Pulled {len(orders)} orders")
 
# Transform: calculate net revenue, flag refunds
orders['net_revenue'] = orders['amount'] - orders['refund_amount'].fillna(0)
orders['is_refunded'] = orders['refund_amount'].fillna(0) > 0
orders['order_date'] = pd.to_datetime(orders['created_at']).dt.date
 
# Load into DuckDB — upsert by order_id
con = duckdb.connect(db_path)
con.execute("""
    CREATE TABLE IF NOT EXISTS daily_orders (
        order_id VARCHAR PRIMARY KEY,
        order_date DATE,
        customer_id VARCHAR,
        net_revenue DECIMAL(10,2),
        is_refunded BOOLEAN
    )
""")
con.execute("DELETE FROM daily_orders WHERE order_date = ?",
            [orders['order_date'].iloc[0]])
con.execute("INSERT INTO daily_orders SELECT * FROM orders")
print(f"Loaded {len(orders)} rows into daily_orders")

Set this to run daily at 6am UTC via a cron trigger. Pick the Small tier — it's pulling a few thousand rows, not a million. The output files persist to project storage automatically, so downstream dashboards can query the DuckDB file directly. Chain it into a workflow to trigger a dashboard refresh or a Slack notification when the load completes.

The script is 30 lines. No Dockerfile. No task definition. No Kubernetes manifest. No IAM role. The infrastructure is there — real K8s Jobs, real ECS tasks, real resource isolation — but it's not your problem.

Runtime profiles

The default runtime is Python 3.11 on a slim base image. If your script needs specific system packages, a different Python version, or pre-installed ML libraries, you create a runtime profile. Fastero builds a custom container image (via Kaniko, if you're curious about the build system) with your dependencies baked in. Subsequent runs use the cached image — no pip install on every execution.

This is the escape hatch for scripts that need heavy native dependencies: numpy with specific BLAS bindings, PyTorch, GDAL, or anything that takes five minutes to pip install. Build once, run many times.

When to use this vs. other Fastero features

Fastero has Streamlit hosting for interactive apps, a notebook runner for Jupyter execution, and SQL-based dashboards for queries. The Python execution engine is specifically for headless scripts — batch jobs, ETL, data pulls, scheduled transforms, anything that runs and produces output without a UI.

If your script generates a web app, deploy it as a Streamlit app. If it's an exploratory analysis, run it as a notebook. If it's a recurring data job that should run unattended and alert you when something breaks, this is what it's for.

The infrastructure you didn't build

The whole point is that the infrastructure exists, it's real, and you don't maintain it. Kubernetes resource scheduling, ECS warm pools, S3 code bundling, Socket.IO log streaming, automatic fallback from warm to cold execution, idempotency locks to prevent duplicate runs, watchdog timers that fail stuck jobs instead of letting them hang silently.

This is the kind of plumbing that a platform team spends months building. It's not interesting work. It's not differentiated work. It's the work that exists between "I have a Python script" and "it runs reliably every day in the cloud." That gap is what Fastero closes.

Try Fastero free — upload your Python script, pick a compute tier, set a schedule, and let real infrastructure handle the rest. No credit card required.