LNA Book Proc API

This project is a FastAPI service that lets you:

Authenticate users with JWT
Check service health
Upload and process EPUB files into:
an HTML “viewer” page (viewer.html) with a navigation sidebar
a RAG-ready folder containing extracted text and images
Convert extracted text chunks (.txt) into audio (.mp3) with Kokoro and concatenate them

The code is intentionally split into routers (HTTP/API layer) and utilities (file processing / templating / audio generation).

1. High-level architecture

Entry point (FastAPI)

main.py creates the FastAPI app, configures CORS, and mounts routers under their prefixes:

app.routers.auth → /auth
app.routers.health → /health
app.routers.epub → /epub
app.routers.audio → /audio

See:

```7:37:/home/paul/Apps/LNA/AI/lna-doc-proc-api/main.py

Include routers

app.include_router(auth.router) app.include_router(health.router) app.include_router(epub.router) app.include_router(audio.router) ... if name == "main": import uvicorn uvicorn.run(app, host="0.0.0.0", port=9000)

### CORS

`main.py` configures permissive CORS:

- `allow_origins=["*"]`
- `allow_credentials=True`
- `allow_methods=["*"]`, `allow_headers=["*"]`

For production, restrict `allow_origins` to your real frontend URLs.

### Authentication / middleware

All “processing” endpoints depend on `get_current_active_user` (JWT bearer token).

JWT logic is implemented in `app/auth.py` using a small in-memory “fake” user database (useful for development).

```9:92:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/auth.py
SECRET_KEY = "your-secret-key-here-change-in-production"
...
fake_users_db = {
    "testuser": {
        "username": "testuser",
        ...
        "hashed_password": bcrypt.hashpw("testpassword".encode('utf-8'), bcrypt.gensalt()),
        "disabled": False,
    }
}
...
async def get_current_active_user(current_user = Depends(get_current_user)):
    """Get current active user"""
    if current_user.get("disabled"):
        raise HTTPException(status_code=400, detail="Inactive user")
    return current_user

EPUB processing pipeline

The EPUB endpoint (/epub/proc) calls process_epub(...) in app/utils/utils.py.

In practice this pipeline is:

Save upload to a temp directory
Unzip EPUB
Locate the OPS/OEBPS/EPUB folder
Parse content.opf / contents.opf to extract:
the book title + navigation items
structured RAG text chunks (.txt) and a copy of images
Copy certain extracted folders to make assets available to the HTML viewer
Generate/overwrite viewer.html by injecting nav + audio filename into html/viewer_template1.html

See the endpoint and the call into the processing function:

```14:111:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/routers/epub.py @router.post("/proc", response_model=EpubProcResponse) async def epub_proc( file: UploadFile = File(...), book_name: str = Form(...), book_id: str = Form(...), current_user = Depends(get_current_active_user) ): ... # Save uploaded file to temp directory temp_dir = tempfile.mkdtemp() ... buffer.write(content) ... # Process the EPUB file result_path, rag_dir_text, rag_dir_images, error_message, success = process_epub(temp_file_path, book_id) ... if success: folder_name = os.path.dirname(result_path) data = { "book_name": book_name, "book_id": book_id, "result_path": result_path, "folder_name": folder_name, "rag_dir_text": rag_dir_text, "rag_dir_images": rag_dir_images }

The core EPUB processing logic is in `process_epub`:

```101:207:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/utils/utils.py
def process_epub(epub_file_path, book_id) -> Tuple[str, str, str,str, bool]:
    ...
    root_dir = os.getenv('ROOT_DIR')
    rag_dir = os.getenv('RAG_DIR')
    template_path = os.getenv('TEMPLATE_PATH')
    ...
    temp_folder = os.path.join(root_dir, book_id)
    ...
    # Unzip the epub
    files = unzip_epub(epub_file_path, temp_folder)
    ...
    # Choose ops_dir (OPS / OEBPS / EPUB)
    if len(matches) > 0:
        ops_dir = os.path.join(temp_folder, 'OPS')
    ...
    # Parse OPF
    if os.path.exists(os.path.join(ops_dir, 'content.opf')):
        with open(os.path.join(ops_dir, 'content.opf'), 'r', encoding='utf-8') as f:
            opf_content = f.read()
            title, nav_items, rag_dir_text, rag_dir_images = parse_opf_xml(opf_content, ops_dir, rag_dir)
    ...
    # Inject nav into viewer template
    viewer_html_path = os.path.join(ops_dir, 'viewer.html')
    success = add_nav_section(template_path, viewer_html_path, title, nav_items, f"{book_id}.mp3", book_id)
    ...
    return viewer_html_path, rag_dir_text, rag_dir_images, error_message, True

Audio pipeline

The audio endpoint (/audio/convert) converts .txt files from a directory into audio segments and concatenates them into a single {book_id}.mp3.

```8:84:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/routers/audio.py router = APIRouter( prefix="/audio", tags=["audio"], responses={404: {"description": "Not found"}}, )

@router.post("/convert", response_model=EpubProcResponse) async def convert_audio( ops_dir: str = Form(...), destination_dir: str = Form(...), book_id: str = Form(...), current_user = Depends(get_current_active_user) ): ... txt_files = [f for f in os.listdir(source_dir) if f.endswith('.txt')] txt_files.sort(key=lambda x: int(os.path.splitext(os.path.basename(x))[0]) if os.path.splitext(os.path.basename(x))[0].isdigit() else float('inf')) ... book_audio_files, page = generate_audio(text, output_dir=destination_dir, page_number=page_number) ... allGood = combine_mp3_files(destination_dir, destination_dir + f'/{name}.mp3') ... return EpubProcResponse(success=True, message=f"Successfully converted {len(txt_files)} text files to audio")

Audio segment generation and concatenation are implemented here:

```14:34:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/utils/audio_utils.py
def generate_audio(..., output_dir='output', page_number=0):
    pipeline = KPipeline(lang_code) 
    generator = pipeline(text, voice=voice, speed=speed, split_pattern=split_pattern)
    ...
    sf.write(f'{output_dir}/{page_number}.mp3', audio, 24000)
    fileList.append(f'{output_dir}/{page_number}.mp3')
    return fileList, page

```38:106:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/utils/audio_utils.py def combine_mp3_files(input_dir: str, output_file: str, ...): mp3_files = glob.glob(os.path.join(input_dir, file_pattern)) ... mp3_files.sort(key=lambda x: int(os.path.splitext(os.path.basename(x))[0]) if os.path.splitext(os.path.basename(x))[0].isdigit() else float('inf')) ... final_audio = np.concatenate(combined_audio) sf.write(output_file, final_audio, sample_rate) return True

---

## 2. Start the service (port, run commands)

### Default port

- When running directly via `python main.py`, the server uses port **9000**:

```34:37:/home/paul/Apps/LNA/AI/lna-doc-proc-api/main.py
if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=9000)

Development scripts

start_server.sh starts the service on port 9001:

```1:3:/home/paul/Apps/LNA/AI/lna-doc-proc-api/start_server.sh source venv/bin/activate uvicorn main:app --host 0.0.0.0 --reload --port 9001

### Run steps

1. Create and activate a virtual environment

   ```bash
   python -m venv venv
   source venv/bin/activate
   ```

2. Install dependencies

   ```bash
   pip install -r requirements.txt
   ```

3. Provide environment variables (see next section)

4. Start the server (pick one):

   ```bash
   # Option A (default port 9000)
   uvicorn main:app --host 0.0.0.0 --port 9000 --reload
   ```

   ```bash
   # Option B (script port 9001)
   ./start_server.sh
   ```

### API docs

FastAPI exposes:

- Swagger UI: `http://localhost:<port>/docs`
- ReDoc: `http://localhost:<port>/redoc`

---

## 3. Configuration (.env)

The service expects a root `.env` at the project root.

Required variables used by EPUB processing and templating:

- `ROOT_DIR`: where EPUB extraction temp folders are created (`ROOT_DIR/<book_id>/...`)
- `RAG_DIR`: where extracted RAG text/images are stored
- `TEMPLATE_PATH`: path to the HTML template used to generate `viewer.html`
- `API_URL`: injected into the viewer template
- `BOOK_SOURCE_DIR`: used by `BookProcessor.py` (optional background indexing)

The project also uses Supabase + Ollama for indexing/embeddings (used by `app/utils/BookProcessor.py` and `app/utils/BookProcessor.py` helpers).

You can also find a second `.env` under `app/utils/.env` (useful for running indexing scripts locally). Keep in mind that modules call `load_dotenv()` at import time, so the working directory matters.

---

## 4. Authentication (JWT)

### Endpoints

- `POST /auth/login`: returns a JWT access token
- `GET /auth/me`: returns current user info
- Protected endpoints require `Authorization: Bearer <token>`

Login endpoint is implemented as:

```13:32:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/routers/auth.py
@router.post("/login", response_model=Token)
async def login_for_access_token(form_data: OAuth2PasswordRequestForm = Depends()):
    user = authenticate_user(form_data.username, form_data.password)
    ...
    access_token = create_access_token(
        data={"sub": user["username"]}, expires_delta=access_token_expires
    )
    return {"access_token": access_token, "token_type": "bearer"}

Default development user

The in-memory user database includes:

Username: testuser
Password: testpassword

5. HTTP API summary (what to call)

All non-auth endpoints require a valid JWT bearer token.

Root

GET / → { "message": "LNA BOOK Proc API is running" }

Health

GET /health

EPUB processing

POST /epub/proc
Content type: multipart/form-data
Form fields:
- file: EPUB file
- book_name: string
- book_id: string

Validations:

File extension must end with .epub
File size must be <= 500MB
Rejects files with suspiciously small payload (< 100 bytes)

See:

```14:63:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/routers/epub.py if not file.filename.lower().endswith('.epub'): return EpubProcResponse( success=False, message=f"Invalid file type. Only EPUB files are allowed. {file.filename}" ) ... if file.size and file.size > MAX_FILE_SIZE: return EpubProcResponse( success=False, message=f"File too large. Maximum file size is {MAX_FILE_SIZE // (10241024)}MB. Your file is {file.size // (10241024)}MB." ) ... if len(content) < 100: return EpubProcResponse( success=False, message=f"File appears to be incomplete. Received {len(content)} bytes, which is too small for a valid EPUB file. Please check your upload." )

Response fields on success include:

- `result_path`: generated `viewer.html` path
- `folder_name`: directory that contains `viewer.html`
- `rag_dir_text`: directory containing extracted numbered `.txt` chunks
- `rag_dir_images`: directory containing images used by the viewer

### Audio conversion

- `POST /audio/convert`
  - Content type: `multipart/form-data`
  - Form fields:
    - `ops_dir`: directory containing numbered `.txt` files
    - `destination_dir`: directory to write `{book_id}.mp3` into (usually alongside `viewer.html`)
    - `book_id`: controls output name `{book_id}.mp3`

---

## 6. EPUB → Viewer + RAG: end-to-end flow

Here is the intended flow based on the endpoint responses:

```text
Client
  |
  | 1) POST /epub/proc (file, book_name, book_id) [JWT]
  v
API
  |
  | 2) unzip EPUB into ROOT_DIR/<book_id>/
  | 3) parse OPF (content.opf/contents.opf)
  | 4) write RAG text/images into RAG_DIR/<identifier>/{text,images}/
  | 5) generate ROOT_DIR/<book_id>/<OPS-or-OEBPS>/viewer.html
  |
  v
Client receives:
  - viewer.html path (result_path)
  - ops folder for assets (folder_name)
  - rag text folder (rag_dir_text)
  - rag images folder (rag_dir_images)
  |
  | (optional) 6) POST /audio/convert using rag_dir_text + folder_name [JWT]
  v
Audio output:
  - {folder_name}/{book_id}.mp3

Key implementation details to know when updating code:

OPS folder detection: process_epub tries OPS, then OEBPS, then EPUB.
OPF parsing: parse_opf_xml(...) scans the OPF manifest/spine and:
creates RAG_DIR/<identifier>/text/
creates RAG_DIR/<identifier>/images/
writes numbered text chunks into .../text/<file_num>.txt
Viewer templating: add_nav_section(...) modifies the HTML template:
injects title + nav_items
sets API URL and AI source book id
attempts to set the audio file name in the viewer

The OPF parsing behavior (directory creation + text writing) lives here:

```72:188:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/utils/epub_utils.py def parse_opf_xml(html_data: str, ops_dir: str, rag_dir: str): ... identifier = soup.find('identifier') ... if len(spine_data) > 0: # create the folder for the RAG if not os.path.exists(os.path.join(rag_dir, identifier )): os.makedirs(os.path.join(rag_dir, identifier )) else: shutil.rmtree(os.path.join(rag_dir, identifier )) os.makedirs(os.path.join(rag_dir, identifier ))

    os.makedirs(os.path.join(rag_dir, identifier, 'text')) ...
    os.makedirs(os.path.join(rag_dir, identifier, 'images')) ...
    rag_dir_text = os.path.join(rag_dir, identifier, 'text')
    rag_dir_images = os.path.join(rag_dir, identifier, 'images')

Text chunk writing:211:254:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/utils/epub_utils.py if item['media-type'] == 'application/xhtml+xml': ... title, heading, data = get_text_from_xhtml(os.path.join(ops_dir,item['href'] )) if data is not None: try: file_name = f"{rag_dir}/{identifier}/text/{file_num}.txt" if "cover" in f"{item['id']}": file_name = f"{rag_dir}/{identifier}/text/0.txt" with open(file_name, 'w', encoding='utf-8') as f: f.write(data) file_num += 1 ... nav_items.append({ 'id': item['id'], 'index': index, 'title': title, 'subtitle': heading, 'url': item['href'] }) ```

Viewer injection is done in html_util.py:

```25:107:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/utils/html_util.py def add_nav_section(template_path: str, html_path: str, section_title: str, nav_items: List[Dict[str, str]], audio_file_path: str, book_id: str) -> bool: ... api_url = soup.find('label', {'id':'API_URL'}) api_url.attrs['value'] = os.getenv('API_URL') ai_source = soup.find('label', ) ai_source.attrs['value'] = book_id ... aside = soup.find('aside', **{'class':'sidebar'}) ... # Add the new nav_section to the sidebar aside.append(new_nav_section) ... with open(html_path, 'w', encoding='utf-8') as file: file.write(str(soup.prettify())) return True

---

## 7. Audio conversion: expected inputs

The audio endpoint sorts `.txt` files by their filename stem (numeric stems come first).

Implementation details:

```48:83:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/routers/audio.py
txt_files = [f for f in os.listdir(source_dir) if f.endswith('.txt')]
txt_files.sort(key=lambda x: int(os.path.splitext(os.path.basename(x))[0]) if os.path.splitext(os.path.basename(x))[0].isdigit() else float('inf'))
...
with open(os.path.join(source_dir, file), 'r') as f:
    text = f.read()
book_audio_files, page = generate_audio(text, output_dir=destination_dir, page_number=page_number)
...
allGood = combine_mp3_files(destination_dir, destination_dir + f'/{name}.mp3')

Practical usage pattern:

Call POST /epub/proc
Use the response values:
rag_dir_text → pass as ops_dir
folder_name → pass as destination_dir
book_id → pass as book_id

This causes the service to write:

{folder_name}/{book_id}.mp3

so the viewer can reference the correct audio file.

8. Optional: background indexing into Supabase (BookProcessor)

app/utils/BookProcessor.py is a standalone script that:

reads directories from BOOK_SOURCE_DIR
uses Ollama to:
generate titles/summaries (get_title_and_summary)
generate embeddings (get_embedding)
inserts processed chunks into Supabase table book_vector_pages

Key configuration:

```25:43:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/utils/BookProcessor.py load_dotenv() llm_host = os.getenv('OLLAMA_HOST') ollama_client = AsyncClient(host=llm_host) ... def get_supabase_client() -> Client: supabase_url = os.getenv("SUPABASE_URL") supabase_key = os.getenv("SUPABASE_KEY") ... return create_client(supabase_url, supabase_key)

Embeddings + title/summary helpers:

```220:255:/home/paul/Apps/LNA/AI/lna-doc-proc-api/app/utils/BookProcessor.py
async def get_title_and_summary(chunk: str, url: str) -> Dict[str, str]:
    ...
async def get_embedding(text: str) -> List[float]:
    response = await ollama_client.embeddings(
        model="nomic-embed-text:latest",
        prompt=text
    )
    return response['embedding']

If you update RAG/chunking behavior, start here first.

9. Notes when updating code

Keep endpoint contracts stable:
/epub/proc request fields and response field names are relied upon by downstream consumers (audio conversion).
/audio/convert expects numbered .txt files for ordering.
Treat .env values as secrets:
Supabase keys and service keys must not be committed.
Be mindful of “working directory” effects:
several modules call load_dotenv() at import time.