From de158fa19bc4bdef43818564c9b97908181e2290 Mon Sep 17 00:00:00 2001 From: Branden Chen Date: Sun, 25 Oct 2026 13:15:50 -0400 Subject: [PATCH 1/2] docs: add verified steps to run training backend locally without AWS Document a verified local run of the Django + Celery + Redis training backend without AWS credentials, plus the two fixes needed on a fresh checkout: - Pin pytorch=2.5.1 / torchvision=0.20.1 in environment.yml (unpinned deps resolve to a build incompatible with Python 3.9: import torch -> ImportError 'TypeIs'). - Fix training Dockerfiles so README.md is available before poetry install. Tested: /api/test -> 200, redis ping -> PONG, pytest 17 passed / 1 failed (CALIFORNIA_HOUSING failure is an environment permission gap, not a pipeline bug). --- README.md | 36 ++++++++++++++++++++++++ training/Dockerfile | 4 +-- training/environment.yml | 4 +-- training/training/core/celery/Dockerfile | 7 +++-- 4 files changed, 44 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 552f7989..6093443b 100644 --- a/README.md +++ b/README.md @@ -149,6 +149,42 @@ poetry run python manage.py runserver 0.0.0.0:8000 That last path will not start Celery/Redis for you. +#### Verified: run the training backend locally without AWS (2026-10-25) + +The Django + Celery + Redis stack **boots without AWS credentials**. AWS is only +reached at two runtime edges: Firebase auth via Secrets Manager (`authenticator.py` +→ `cli.py init_firebase()`) and saving results to S3 (`worker.py`). Booting and the +unauthenticated surface need neither. + +```sh +mkdir -p ~/.aws # lets the $HOME/.aws bind mount succeed even with no AWS +cd training +docker compose up --build +``` + +Verify it is up: + +```sh +curl http://localhost:8000/api/test # -> {"result": "200 Backend surface test successful"} +docker compose exec redis redis-cli ping # -> PONG +docker compose exec web mamba run -n dlp poetry run pytest tests +``` + +Expected: `/api/test` returns 200, Redis replies `PONG`, and pytest reports +`17 passed, 1 failed`. The single failure (`test_columns_endpoint[CALIFORNIA_HOUSING]`) +is an environment gap, not a pipeline bug: `/home/app` is root-owned while the +container runs as user `app`, so `fetch_california_housing` cannot write its download +cache. Run `pytest tests -k "not CALIFORNIA_HOUSING"` for an all-green run. + +Two fixes on a fresh checkout were needed to reach this state (both in this branch): +the unpinned `pytorch`/`torchvision` in `environment.yml` resolve to a build that is +incompatible with Python 3.9 (`import torch` → `ImportError: cannot import name +'TypeIs'`), so they are pinned to `pytorch=2.5.1` / `torchvision=0.20.1`; and the +training `Dockerfile`s needed `README.md` available before `poetry install`. + +A full end-to-end training job (`POST /api/tabular`) still requires AWS for Firebase +auth and the S3 result save, so it cannot complete on a no-AWS setup. + ### 3. Flask API (`backend/`) ```sh diff --git a/training/Dockerfile b/training/Dockerfile index f1b3c0ef..2126dbda 100644 --- a/training/Dockerfile +++ b/training/Dockerfile @@ -5,7 +5,7 @@ WORKDIR /usr/src/build # install dependencies RUN mamba create --name dlp -y -COPY environment.yml pyproject.toml poetry.lock ./ +COPY environment.yml pyproject.toml poetry.lock README.md ./ RUN mamba run --live-stream -n dlp mamba env update --file environment.yml --prune # create directory for the app user @@ -25,7 +25,7 @@ RUN mkdir $APP_HOME FROM base AS development ENV ENVIRONMENT development -RUN mamba run --live-stream -n dlp poetry install +RUN mamba run --live-stream -n dlp poetry install --no-root # we previously mounted training/ to APP_HOME diff --git a/training/environment.yml b/training/environment.yml index e566214a..8b40727f 100644 --- a/training/environment.yml +++ b/training/environment.yml @@ -6,5 +6,5 @@ channels: dependencies: - python=3.9 - poetry - - pytorch - - torchvision \ No newline at end of file + - pytorch=2.5.1 + - torchvision=0.20.1 \ No newline at end of file diff --git a/training/training/core/celery/Dockerfile b/training/training/core/celery/Dockerfile index ece91a2b..ec2ad1dc 100644 --- a/training/training/core/celery/Dockerfile +++ b/training/training/core/celery/Dockerfile @@ -5,7 +5,9 @@ WORKDIR /usr/src/build # install dependencies RUN mamba create --name dlp -y -COPY environment.yml pyproject.toml poetry.lock ./ +COPY environment.yml pyproject.toml poetry.lock README.md ./ +# This is to just see if the readme file exists + RUN mamba run --live-stream -n dlp mamba env update --file environment.yml --prune # create directory for the app user @@ -20,12 +22,11 @@ ARG APP_HOME=/home/app/training RUN mkdir "$APP_HOME" - ######################## development target ######################## FROM base AS development ENV ENVIRONMENT development -RUN mamba run --live-stream -n dlp poetry install +RUN mamba run --live-stream -n dlp poetry install --no-root # we previously mounted training/ to APP_HOME From db7996eacd7f045fdbefc718c3c060cb3c135b26 Mon Sep 17 00:00:00 2001 From: Comment-XD <90992234+Comment-XD@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:45:18 +0000 Subject: [PATCH 2/2] :art: Auto-generated directory tree for repository in Architecture.md --- .github/Architecture.md | 253 ++++++++++++++++++++-------------------- 1 file changed, 127 insertions(+), 126 deletions(-) diff --git a/.github/Architecture.md b/.github/Architecture.md index dc8ac020..e403482d 100644 --- a/.github/Architecture.md +++ b/.github/Architecture.md @@ -5,61 +5,61 @@ ``` 📦 training | |- 📂 training: -| | |- 📂 middleware: -| | | |- 📜 __init__.py -| | | |- 📜 health_check_middleware.py | | |- 📂 routes: -| | | |- 📂 datasets: -| | | | |- 📂 default: -| | | | | |- 📜 schemas.py -| | | | | |- 📜 __init__.py -| | | | | |- 📜 columns.py +| | | |- 📂 tabular: +| | | | |- 📜 schemas.py +| | | | |- 📜 tabular.py | | | | |- 📜 __init__.py | | | |- 📂 image: | | | | |- 📜 schemas.py -| | | | |- 📜 __init__.py | | | | |- 📜 image.py -| | | |- 📂 audio: -| | | | |- 📜 schemas.py | | | | |- 📜 __init__.py +| | | |- 📂 audio: | | | | |- 📜 audio.py -| | | |- 📂 tabular: | | | | |- 📜 schemas.py | | | | |- 📜 __init__.py -| | | | |- 📜 tabular.py +| | | |- 📂 datasets: +| | | | |- 📂 default: +| | | | | |- 📜 columns.py +| | | | | |- 📜 schemas.py +| | | | | |- 📜 __init__.py +| | | | |- 📜 __init__.py | | | |- 📜 schemas.py | | | |- 📜 __init__.py +| | |- 📂 middleware: +| | | |- 📜 __init__.py +| | | |- 📜 health_check_middleware.py | | |- 📂 core: | | | |- 📂 celery: -| | | | |- 📜 Dockerfile -| | | | |- 📜 __init__.py | | | | |- 📜 optimizer.py : what optimizer to use (ie: SGD or Adam for now) | | | | |- 📜 dataset.py : read in the dataset through URL or file upload -| | | | |- 📜 train_types.py | | | | |- 📜 trainer.py | | | | |- 📜 dl_model.py : torch model based on user specifications from drag and drop -| | | | |- 📜 criterion.py | | | | |- 📜 worker.py -| | | |- 📜 __init__.py +| | | | |- 📜 criterion.py +| | | | |- 📜 Dockerfile +| | | | |- 📜 __init__.py +| | | | |- 📜 train_types.py | | | |- 📜 authenticator.py -| | |- 📜 __init__.py +| | | |- 📜 __init__.py +| | |- 📜 celery_app.py | | |- 📜 wsgi.py -| | |- 📜 celeryconfig.py +| | |- 📜 constants.py : list of helpful constants +| | |- 📜 urls.py +| | |- 📜 __init__.py | | |- 📜 settings.py | | |- 📜 asgi.py -| | |- 📜 urls.py -| | |- 📜 celery_app.py -| | |- 📜 constants.py : list of helpful constants -| |- 📜 pytest.ini +| | |- 📜 celeryconfig.py | |- 📜 cli.py -| |- 📜 Dockerfile -| |- 📜 README.md | |- 📜 docker-compose.yml -| |- 📜 docker-compose.prod.yml | |- 📜 pyproject.toml -| |- 📜 environment.yml +| |- 📜 pytest.ini +| |- 📜 docker-compose.prod.yml | |- 📜 poetry.lock +| |- 📜 README.md +| |- 📜 Dockerfile | |- 📜 manage.py +| |- 📜 environment.yml ``` ## Frontend Architecture @@ -68,179 +68,180 @@ 📦 frontend | |- 📂 public: | | |- 📂 images: -| | | |- 📂 learn_mod_images: -| | | | |- 📜 neuralnet.png -| | | | |- 📜 ReLUactivation.png -| | | | |- 📜 binarystepactivation.png -| | | | |- 📜 neuron.png -| | | | |- 📜 sigmoidactivation.png -| | | | |- 📜 robotImage.jpg -| | | | |- 📜 LeakyReLUactivation.png -| | | | |- 📜 neuronWithEquation.png -| | | | |- 📜 lossExampleTable.png -| | | | |- 📜 lossExampleEquation.png -| | | | |- 📜 sigmoidfunction.png -| | | | |- 📜 tanhactivation.png -| | | | |- 📜 lossExample.png +| | | |- 📂 wiki_images: +| | | | |- 📜 batchnorm_diagram.png +| | | | |- 📜 sigmoid_equation.png +| | | | |- 📜 maxpool2d.gif +| | | | |- 📜 tanh_plot.png +| | | | |- 📜 avgpool_maxpool.gif +| | | | |- 📜 softmax_equation.png : PNG file of Softmax equation +| | | | |- 📜 tanh_equation.png +| | | | |- 📜 dropout_diagram.png +| | | | |- 📜 conv2d.gif +| | | | |- 📜 conv2d2.gif | | | |- 📂 logos: | | | | |- 📂 dlp_branding: | | | | | |- 📜 dlp-logo.svg : DLP Logo, duplicate of files in public, but essential as the frontend can't read public | | | | | |- 📜 dlp-logo.png : DLP Logo, duplicate of files in public, but essential as the frontend can't read public -| | | | |- 📜 dsgt-logo-light.png -| | | | |- 📜 pytorch-logo.png +| | | | |- 📜 flask-logo.png +| | | | |- 📜 aws-logo.png +| | | | |- 📜 react-logo.png +| | | | |- 📜 pandas-logo.png | | | | |- 📜 python-logo.png +| | | | |- 📜 dsgt-logo-light.png | | | | |- 📜 google.png -| | | | |- 📜 aws-logo.png | | | | |- 📜 github.png -| | | | |- 📜 flask-logo.png -| | | | |- 📜 dsgt-logo-white-back.png +| | | | |- 📜 pytorch-logo.png | | | | |- 📜 dsgt-logo-dark.png -| | | | |- 📜 react-logo.png -| | | | |- 📜 pandas-logo.png -| | | |- 📂 wiki_images: -| | | | |- 📜 sigmoid_equation.png -| | | | |- 📜 conv2d.gif -| | | | |- 📜 softmax_equation.png : PNG file of Softmax equation -| | | | |- 📜 batchnorm_diagram.png -| | | | |- 📜 dropout_diagram.png -| | | | |- 📜 maxpool2d.gif -| | | | |- 📜 conv2d2.gif -| | | | |- 📜 tanh_equation.png -| | | | |- 📜 avgpool_maxpool.gif -| | | | |- 📜 tanh_plot.png +| | | | |- 📜 dsgt-logo-white-back.png +| | | |- 📂 learn_mod_images: +| | | | |- 📜 ReLUactivation.png +| | | | |- 📜 sigmoidfunction.png +| | | | |- 📜 tanhactivation.png +| | | | |- 📜 lossExample.png +| | | | |- 📜 neuralnet.png +| | | | |- 📜 sigmoidactivation.png +| | | | |- 📜 lossExampleTable.png +| | | | |- 📜 lossExampleEquation.png +| | | | |- 📜 LeakyReLUactivation.png +| | | | |- 📜 binarystepactivation.png +| | | | |- 📜 neuronWithEquation.png +| | | | |- 📜 robotImage.jpg +| | | | |- 📜 neuron.png | | | |- 📜 demo_video.gif : GIF tutorial of a simple classification training session | | |- 📜 manifest.json : Default React file for choosing icon based on -| | |- 📜 robots.txt -| | |- 📜 index.html : Base HTML file that will be initially rendered | | |- 📜 dlp-logo.ico : DLP Logo -| |- 📂 layer_docs: -| | |- 📜 Softmax.md : Doc for Softmax layer -| | |- 📜 softmax_equation.png : PNG file of Softmax equation -| | |- 📜 Linear.md : Doc for Linear layer -| | |- 📜 ReLU.md : Doc for ReLU later +| | |- 📜 index.html : Base HTML file that will be initially rendered +| | |- 📜 robots.txt | |- 📂 src: -| | |- 📂 pages: -| | | |- 📂 train: -| | | | |- 📜 [train_space_id].tsx -| | | | |- 📜 index.tsx -| | | | |- 📜 metrics_to_charts.tsx -| | | |- 📜 settings.tsx -| | | |- 📜 dashboard.tsx -| | | |- 📜 login.tsx -| | | |- 📜 learn.tsx -| | | |- 📜 _app.tsx -| | | |- 📜 _document.tsx -| | | |- 📜 LearnContent.tsx -| | | |- 📜 feedback.tsx -| | | |- 📜 forgot.tsx -| | | |- 📜 wiki.tsx -| | | |- 📜 about.tsx +| | |- 📂 backend_outputs: +| | | |- 📜 model.pkl +| | | |- 📜 model.pt : Last model.pt output +| | | |- 📜 my_deep_learning_model.onnx : Last ONNX file output | | |- 📂 features: | | | |- 📂 Train: +| | | | |- 📂 redux: +| | | | | |- 📜 trainspaceApi.ts +| | | | | |- 📜 trainspaceSlice.ts | | | | |- 📂 components: | | | | | |- 📜 DatasetStepLayout.tsx -| | | | | |- 📜 CreateTrainspace.tsx | | | | | |- 📜 TrainspaceLayout.tsx +| | | | | |- 📜 CreateTrainspace.tsx | | | | |- 📂 features: | | | | | |- 📂 Image: +| | | | | | |- 📂 redux: +| | | | | | | |- 📜 imageActions.ts +| | | | | | | |- 📜 imageApi.ts | | | | | | |- 📂 components: -| | | | | | | |- 📜 ImageFlow.tsx -| | | | | | | |- 📜 ImageParametersStep.tsx | | | | | | | |- 📜 ImageDatasetStep.tsx +| | | | | | | |- 📜 ImageParametersStep.tsx | | | | | | | |- 📜 ImageReviewStep.tsx | | | | | | | |- 📜 ImageTrainspace.tsx +| | | | | | | |- 📜 ImageFlow.tsx | | | | | | |- 📂 constants: | | | | | | | |- 📜 imageConstants.ts | | | | | | |- 📂 types: | | | | | | | |- 📜 imageTypes.ts -| | | | | | |- 📂 redux: -| | | | | | | |- 📜 imageApi.ts -| | | | | | | |- 📜 imageActions.ts | | | | | | |- 📜 index.ts | | | | | |- 📂 Tabular: +| | | | | | |- 📂 redux: +| | | | | | | |- 📜 tabularActions.ts +| | | | | | | |- 📜 tabularApi.ts | | | | | | |- 📂 components: -| | | | | | | |- 📜 TabularParametersStep.tsx | | | | | | | |- 📜 TabularFlow.tsx +| | | | | | | |- 📜 TabularDatasetStep.tsx | | | | | | | |- 📜 TabularTrainspace.tsx | | | | | | | |- 📜 TabularReviewStep.tsx -| | | | | | | |- 📜 TabularDatasetStep.tsx +| | | | | | | |- 📜 TabularParametersStep.tsx | | | | | | |- 📂 constants: | | | | | | | |- 📜 tabularConstants.ts | | | | | | |- 📂 types: | | | | | | | |- 📜 tabularTypes.ts -| | | | | | |- 📂 redux: -| | | | | | | |- 📜 tabularActions.ts -| | | | | | | |- 📜 tabularApi.ts | | | | | | |- 📜 index.ts | | | | |- 📂 constants: | | | | | |- 📜 trainConstants.ts | | | | |- 📂 types: | | | | | |- 📜 trainTypes.ts -| | | | |- 📂 redux: -| | | | | |- 📜 trainspaceApi.ts -| | | | | |- 📜 trainspaceSlice.ts | | | |- 📂 LearnMod: +| | | | |- 📜 ModulesSideBar.tsx +| | | | |- 📜 MCQuestion.tsx | | | | |- 📜 FRQuestion.tsx -| | | | |- 📜 Exercise.tsx | | | | |- 📜 ClassCard.tsx | | | | |- 📜 ImageComponent.tsx -| | | | |- 📜 ModulesSideBar.tsx -| | | | |- 📜 MCQuestion.tsx +| | | | |- 📜 Exercise.tsx | | | | |- 📜 LearningModulesContent.tsx | | | |- 📂 Feedback: | | | | |- 📂 redux: | | | | | |- 📜 feedbackApi.ts -| | | |- 📂 OpenAi: -| | | | |- 📜 openAiUtils.ts | | | |- 📂 Dashboard: +| | | | |- 📂 redux: +| | | | | |- 📜 dashboardApi.ts | | | | |- 📂 components: -| | | | | |- 📜 TrainDataGrid.tsx | | | | | |- 📜 TrainBarChart.tsx | | | | | |- 📜 TrainDoughnutChart.tsx -| | | | |- 📂 redux: -| | | | | |- 📜 dashboardApi.ts +| | | | | |- 📜 TrainDataGrid.tsx +| | | |- 📂 OpenAi: +| | | | |- 📜 openAiUtils.ts +| | |- 📂 pages: +| | | |- 📂 train: +| | | | |- 📜 [train_space_id].tsx +| | | | |- 📜 metrics_to_charts.tsx +| | | | |- 📜 index.tsx +| | | |- 📜 about.tsx +| | | |- 📜 settings.tsx +| | | |- 📜 LearnContent.tsx +| | | |- 📜 _document.tsx +| | | |- 📜 feedback.tsx +| | | |- 📜 forgot.tsx +| | | |- 📜 dashboard.tsx +| | | |- 📜 learn.tsx +| | | |- 📜 login.tsx +| | | |- 📜 wiki.tsx +| | | |- 📜 _app.tsx | | |- 📂 common: +| | | |- 📂 redux: +| | | | |- 📜 train.ts +| | | | |- 📜 hooks.ts +| | | | |- 📜 userLogin.ts +| | | | |- 📜 backendApi.ts +| | | | |- 📜 store.ts | | | |- 📂 components: -| | | | |- 📜 EmailInput.tsx -| | | | |- 📜 ClientOnlyPortal.tsx +| | | | |- 📜 HtmlTooltip.tsx +| | | | |- 📜 Spacer.tsx | | | | |- 📜 Footer.tsx | | | | |- 📜 TitleText.tsx +| | | | |- 📜 ClientOnlyPortal.tsx | | | | |- 📜 NavBarMain.tsx -| | | | |- 📜 Spacer.tsx +| | | | |- 📜 EmailInput.tsx | | | | |- 📜 DlpTooltip.tsx -| | | | |- 📜 HtmlTooltip.tsx | | | |- 📂 utils: | | | | |- 📜 dateFormat.ts -| | | | |- 📜 firebase.ts | | | | |- 📜 dndHelpers.ts +| | | | |- 📜 firebase.ts | | | |- 📂 styles: | | | | |- 📜 globals.css | | | | |- 📜 Home.module.css -| | | |- 📂 redux: -| | | | |- 📜 train.ts -| | | | |- 📜 backendApi.ts -| | | | |- 📜 userLogin.ts -| | | | |- 📜 hooks.ts -| | | | |- 📜 store.ts | | |- 📂 __tests__: | | | |- 📂 common: | | | | |- 📂 components: | | | | | |- 📜 TitleText.test.tsx -| | |- 📂 backend_outputs: -| | | |- 📜 my_deep_learning_model.onnx : Last ONNX file output -| | | |- 📜 model.pt : Last model.pt output -| | | |- 📜 model.pkl +| | |- 📜 constants.ts | | |- 📜 next-env.d.ts -| | |- 📜 iris.csv : Sample CSV data | | |- 📜 GlobalStyle.ts -| | |- 📜 constants.ts -| |- 📜 tsconfig.json +| | |- 📜 iris.csv : Sample CSV data +| |- 📂 layer_docs: +| | |- 📜 softmax_equation.png : PNG file of Softmax equation +| | |- 📜 ReLU.md : Doc for ReLU later +| | |- 📜 Softmax.md : Doc for Softmax layer +| | |- 📜 Linear.md : Doc for Linear layer +| |- 📜 next-env.d.ts +| |- 📜 .eslintrc.json | |- 📜 pnpm-lock.yaml -| |- 📜 package.json | |- 📜 next.config.js +| |- 📜 tsconfig.json | |- 📜 jest.config.ts -| |- 📜 next-env.d.ts +| |- 📜 .env.example +| |- 📜 package.json | |- 📜 .eslintignore -| |- 📜 .eslintrc.json ```