Skip to content

Documentation Cleanup and Standardization Plan

Document Version: 1.0 Created: 2025-11-01 Status: Draft Ticket: PIQUE-618

1. Executive Summary & Feature Overview

1.1 Feature Description

  • Feature Name: Documentation Cleanup and Standardization
  • Feature Type: Refactor/Improvement
  • Priority Level: Medium

1.2 Problem Statement

  • Current State: The /docs directory contains 27 markdown files with inconsistent naming conventions, mixed organizational patterns, and unclear categorization. Files use a combination of naming styles including:
  • ALL_CAPS with underscores (e.g., DATABASE_SCHEMA.md)
  • kebab-case (e.g., playwright-e2e-testing-plan.md)
  • Mixed case with prefixes (e.g., PIQUE-506-feature-flag-integration-plan.md)

  • Date suffixes (e.g., PRODUCER_PORTAL_BUG_AUDIT_2025-10-13.md)

  • Pain Points:

  • Difficult to locate specific documentation
  • No clear distinction between active vs archived documentation
  • Inconsistent organization across subdirectories
  • Some documentation may be outdated relative to current codebase

  • Redundant or overlapping content (e.g., multiple performance optimization docs)

  • User Impact: Developers, new team members, and stakeholders struggle to find relevant documentation quickly, leading to wasted time and potential misunderstandings about system architecture and processes.

  • Business Value: Standardized, well-organized documentation improves developer productivity, reduces onboarding time, and ensures accurate understanding of system architecture and processes.

1.3 Expected Outcomes

  • Success Metrics:
  • 100% of documentation files follow consistent naming conventions
  • Clear directory structure with logical categorization
  • All documentation validated against current codebase

  • Documentation navigation time reduced by 50%

  • User Experience Goals:

  • Developers can quickly locate relevant documentation
  • Clear separation between active and archived documentation

  • Consistent formatting and structure across all docs

  • Technical Goals:

  • Implement standardized naming conventions
  • Create clear directory structure
  • Update outdated documentation
  • Remove or archive obsolete documentation

2. Stakeholder Analysis & Requirements

2.1 Affected Users & Systems

  • Primary Users: Development team members (API, Frontend, Producer Portal, Check-in teams)
  • Secondary Users: New hires, project managers, stakeholders
  • System Components: All documentation in /docs directory
  • Integration Points: None (documentation only)

2.2 Functional Requirements

Must-Have Features: - Consistent naming convention across all documentation files - Logical directory structure organized by topic/system - Updated content that reflects current codebase state - Clear README.md in /docs root to guide navigation - Distinction between plans, guides, architecture docs, and testing docs

Should-Have Features: - Archive directory for historical/completed plans - Template files for different documentation types - Cross-referencing between related documents - Table of contents in longer documents

Could-Have Features: - Documentation linter/validator - Automated checks for broken links - Documentation search functionality

Won't-Have Features: - Complete rewrite of all documentation content - Migration to different documentation platform (staying with markdown) - Auto-generated API documentation (separate effort)

2.3 Non-Functional Requirements

  • Performance: N/A (static documentation)
  • Security: No sensitive information in documentation files
  • Accessibility: Markdown format ensures accessibility
  • Browser/Platform Support: Standard markdown rendering
  • Reliability: Documentation should be version controlled and backed up via Git

3. Current State Analysis

3.1 Codebase Research Methodology

Primary Tools: - Bash commands for directory exploration and file listing - Read tool for examining document content and structure - Grep for pattern matching across documentation files - Manual review of document headers and organization

Secondary Tools: - Git history analysis (not performed but recommended for understanding doc evolution) - Codebase comparison to validate documentation accuracy

3.2 Existing Documentation Structure

Current Directory Organization: 

docs/
├── api/                    # API-specific documentation (7 files)
├── plans/                  # Feature planning documents (11 files)
├── portal/                 # Producer portal documentation (3 files)
└── [root level]           # Mixed general documentation (6 files)

Current Naming Patterns: - ALL_CAPS_UNDERSCORE: DATABASE_SCHEMA.md, LOAD_TESTING_PLAN.md, CLOUDFLARE_MULTI_ENV_DEPLOYMENT.md - kebab-case: playwright-e2e-testing-plan.md, knowledge-base-planning.md, producer-feature-flags-plan.md - Mixed with prefixes: PIQUE-506-feature-flag-integration-plan.md, BUG-004-SQL-INJECTION-FIX-PLAN.md - Date suffixes: PRODUCER_PORTAL_BUG_AUDIT_2025-10-13.md

3.3 Documentation Inventory

Root Level Documentation (6 files)

  1. CLOUDFLARE_MULTI_ENV_DEPLOYMENT.md - Deployment guide
  2. DATABASE_SCHEMA.md - Architecture documentation
  3. E2E_TESTING_PLAN.md - Testing plan
  4. LOAD_TESTING_PLAN.md - Testing plan
  5. playwright-e2e-testing-plan.md - Testing plan (duplicate topic)
  6. QUEUE_BASED_CHECKOUT_PROPOSAL.md - Architecture proposal

API Directory (7 files)

  1. CELERY_BEAT_TESTING.md - Testing guide
  2. CHECKOUT_TESTING_PLAN.md - Testing plan
  3. password_reset_api.md - API documentation
  4. PRODUCTION_CONNECTION_MONITORING.md - Operations guide
  5. QUICK_TEST.md - Testing guide
  6. TEST_DATABASE_CONNECTION_FIX.md - Fix documentation
  7. TEST_FIXES_SUMMARY.md - Summary document

Plans Directory (11 files)

  1. BUG-004-SQL-INJECTION-FIX-PLAN.md - Bug fix plan
  2. CDN_EDGE_CACHING_IMPLEMENTATION.md - Implementation plan
  3. knowledge-base-planning.md - Feature plan
  4. MEMORY_CODE_OPTIMIZATIONS.md - Optimization plan
  5. MEMORY_OPTIMIZATION_PLAN.md - Optimization plan (similar to #4)
  6. MEMORY_REDUCTION_STRATEGIES.md - Optimization strategies (similar to #4, #5)
  7. NEXTJS_CACHE_MANAGEMENT_PLAN.md - Implementation plan
  8. PERFORMANCE_OPTIMIZATION_PLAN.md - Optimization plan
  9. PERFORMANCE_OPTIMIZATION.md - Optimization doc (similar to #8)
  10. PIQUE-506-feature-flag-integration-plan.md - Feature plan
  11. producer-feature-flags-plan.md - Feature plan (related to #10)

Portal Directory (3 files)

  1. AUTH_CONFIG.md - Configuration guide
  2. KNOWLEDGE_BASE_FRONTEND_GUIDE.md - Implementation guide
  3. PRODUCER_PORTAL_BUG_AUDIT_2025-10-13.md - Audit report

3.4 Identified Issues

Naming Inconsistencies: - Multiple naming conventions in use (ALL_CAPS, kebab-case, mixed) - Some files have ticket prefixes (PIQUE-506, BUG-004), others don't - Date suffixes only on some files - No consistent pattern for differentiating document types

Organizational Issues: - Duplicate/overlapping content (multiple performance optimization docs, multiple memory optimization docs, two E2E testing plans) - Testing documentation split between root and /api subdirectory - No clear distinction between active plans and completed work - Architecture documents mixed with testing plans at root level

Content Issues (requires deeper review): - Potential outdated information (needs validation against codebase) - Some documents may be obsolete (completed plans, fixed bugs) - Unclear which documents are authoritative when duplicates exist

4. Research & Best Practices

4.1 Industry Standards Research

Documentation Organization Best Practices: - Organized by type: Separate architecture, guides, references, tutorials, and planning docs - Consistent naming: Use lowercase with hyphens (kebab-case) for cross-platform compatibility - Semantic naming: File names should clearly indicate content without needing to open the file - Archival strategy: Move completed/historical docs to archive rather than deletion - README-driven: Each directory should have a README explaining its contents

Recommended Documentation Types: 1. Architecture: System design, database schemas, technical overviews 2. Guides: How-to documents, configuration guides, operational procedures 3. API Reference: Endpoint documentation, request/response formats 4. Planning: Feature plans, proposals, design documents 5. Testing: Test plans, testing guides, QA documentation 6. Operations: Deployment, monitoring, maintenance guides 7. Archive: Completed plans, historical documents, obsolete information

4.2 Framework/Library Standards

Markdown Best Practices: - Use ATX-style headers (# syntax) - Include table of contents for documents over 500 lines - Use relative links for cross-referencing - Include metadata at document top (Created, Updated, Status, Authors) - Use consistent formatting for code blocks, tables, and lists

Monorepo Documentation Patterns: - Keep app-specific docs in app directories (apps/*/docs/) - Keep shared/system docs in root /docs - Maintain a central index/README for navigation - Version documentation alongside code when possible

5. Solution Design

5.1 Proposed Documentation Structure

docs/
├── README.md                           # Main documentation index and navigation guide
├── architecture/                       # System architecture and design
│   ├── database-schema.md
│   ├── checkout-queue-architecture.md
│   └── multi-environment-deployment.md
├── guides/                            # How-to and configuration guides
│   ├── api/
│   │   ├── password-reset.md
│   │   └── celery-beat-setup.md
│   ├── frontend/
│   │   ├── knowledge-base-implementation.md
│   │   └── cache-management.md
│   └── producer-portal/
│       └── authentication-configuration.md
├── operations/                        # Deployment, monitoring, maintenance
│   ├── cloudflare-deployment.md
│   ├── production-monitoring.md
│   └── database-connection-management.md
├── testing/                           # Test plans and testing guides
│   ├── e2e-testing-strategy.md
│   ├── load-testing-plan.md
│   ├── api-testing-guide.md
│   └── checkout-testing-plan.md
├── planning/                          # Active feature plans and proposals
│   ├── feature-flags-integration.md
│   ├── knowledge-base-feature.md
│   └── performance-optimization.md
├── archive/                           # Completed plans and historical docs
│   ├── 2025-10/
│   │   ├── bug-004-sql-injection-fix.md
│   │   ├── memory-optimization-completed.md
│   │   └── producer-portal-audit-2025-10-13.md
│   └── README.md                      # Archive organization explanation
└── templates/                         # Documentation templates
    ├── feature-plan-template.md
    ├── architecture-doc-template.md
    └── testing-plan-template.md

5.2 Naming Convention Standards

Standard Format: 

{topic}-{subtopic}-{type}.md

Rules: 1. Use kebab-case (lowercase with hyphens) for all file names 2. Be descriptive but concise (aim for 2-5 words) 3. No dates in file names (use git history and document metadata instead) 4. No ticket numbers in file names (reference tickets in document metadata) 5. Avoid redundant suffixes like "PLAN", "DOC", "GUIDE" when directory already indicates type

Examples: - ✅ architecture/database-schema.md - ✅ guides/api/password-reset.md - ✅ testing/load-testing.md - ✅ planning/feature-flags-integration.md - ❌ DATABASE_SCHEMA.md - ❌ PIQUE-506-feature-flag-integration-plan.md - ❌ PRODUCER_PORTAL_BUG_AUDIT_2025-10-13.md

5.3 Document Metadata Standards

Required Metadata Header: 

# Document Title

**Type:** [Architecture | Guide | Testing | Planning | Operations]
**Status:** [Draft | Active | Completed | Archived | Deprecated]
**Created:** YYYY-MM-DD
**Last Updated:** YYYY-MM-DD
**Related Tickets:** PIQUE-XXX, BUG-YYY (if applicable)
**Authors:** [Names or GitHub handles]

---

5.4 Archive Strategy

Criteria for Archival: 1. Completed feature plans (implementation finished) 2. Fixed bug documentation (after fix is deployed) 3. Historical audit reports 4. Superseded documentation (when new version created)

Archive Organization: - Organize by year/month: archive/YYYY-MM/ - Preserve original file names but apply naming convention - Include README in archive explaining organization - Maintain references from active docs to archived versions when relevant

6. Implementation Plan

6.1 Development Phases

Phase 1: Structure and Prepare (2-3 hours) - [ ] Create new directory structure (architecture/, guides/, operations/, testing/, planning/, archive/, templates/) - [ ] Create README.md in /docs root with navigation guide - [ ] Create README.md in /archive explaining archival strategy - [ ] Create documentation templates in /templates - [ ] Deliverable: New directory structure with README files and templates

Phase 2: Categorize and Move Files (3-4 hours) - [ ] Review each document to determine appropriate category - [ ] Determine which documents should be archived vs remain active - [ ] Rename files according to new naming convention - [ ] Move files to appropriate directories - [ ] Update internal cross-references between documents - [ ] Deliverable: All documentation files organized in new structure with consistent naming

Phase 3: Content Review and Updates (4-6 hours) - [ ] Add metadata headers to all documents - [ ] Review and consolidate duplicate/overlapping documents - [ ] Validate technical accuracy against current codebase - [ ] Update outdated information - [ ] Add cross-references between related documents - [ ] Deliverable: Validated, deduplicated, and updated documentation

Phase 4: Quality Assurance (1-2 hours) - [ ] Verify all internal links work correctly - [ ] Ensure consistent formatting across all documents - [ ] Spell check and grammar review - [ ] Remove .DS_Store and other non-documentation files - [ ] Get team review and feedback - [ ] Deliverable: High-quality, validated documentation set

6.2 Detailed File Migration Plan

Current File New Location Action Notes
DATABASE_SCHEMA.md architecture/database-schema.md Move + Rename Core architecture doc
QUEUE_BASED_CHECKOUT_PROPOSAL.md architecture/checkout-queue.md Move + Rename Architecture proposal
CLOUDFLARE_MULTI_ENV_DEPLOYMENT.md operations/cloudflare-deployment.md Move + Rename Operations guide
E2E_TESTING_PLAN.md testing/e2e-testing-strategy.md Move + Rename Active testing plan
playwright-e2e-testing-plan.md Archive or Merge Review Potential duplicate of above
LOAD_TESTING_PLAN.md testing/load-testing.md Move + Rename Active testing plan
api/password_reset_api.md guides/api/password-reset.md Move + Rename API guide
api/CELERY_BEAT_TESTING.md guides/api/celery-beat-setup.md Move + Rename Setup guide
api/PRODUCTION_CONNECTION_MONITORING.md operations/database-monitoring.md Move + Rename Operations guide
api/CHECKOUT_TESTING_PLAN.md testing/checkout-testing.md Move + Rename Testing plan
api/QUICK_TEST.md guides/api/quick-testing.md Move + Rename Testing guide
api/TEST_DATABASE_CONNECTION_FIX.md archive/2025-10/database-connection-fix.md Archive Historical fix
api/TEST_FIXES_SUMMARY.md archive/2025-10/test-fixes-summary.md Archive Historical summary
portal/AUTH_CONFIG.md guides/producer-portal/authentication.md Move + Rename Config guide
portal/KNOWLEDGE_BASE_FRONTEND_GUIDE.md guides/producer-portal/knowledge-base.md Move + Rename Implementation guide
portal/PRODUCER_PORTAL_BUG_AUDIT_2025-10-13.md archive/2025-10/producer-portal-audit.md Archive Historical audit
plans/knowledge-base-planning.md planning/knowledge-base-feature.md Move + Rename Active plan or archive if completed
plans/PIQUE-506-feature-flag-integration-plan.md Review vs producer-feature-flags-plan.md Merge or Archive Potential duplicate
plans/producer-feature-flags-plan.md planning/feature-flags.md Move + Rename Active plan or archive if completed
plans/PERFORMANCE_OPTIMIZATION_PLAN.md Review vs PERFORMANCE_OPTIMIZATION.md Merge Consolidate
plans/PERFORMANCE_OPTIMIZATION.md planning/performance-optimization.md Merge + Move Consolidate with above
plans/MEMORY_OPTIMIZATION_PLAN.md Review with other memory docs Merge Consolidate memory docs
plans/MEMORY_CODE_OPTIMIZATIONS.md Review with other memory docs Merge Consolidate memory docs
plans/MEMORY_REDUCTION_STRATEGIES.md archive/2025-10/memory-optimization.md Merge + Archive Consolidate all memory docs
plans/NEXTJS_CACHE_MANAGEMENT_PLAN.md guides/frontend/cache-management.md Move + Rename Implementation guide
plans/CDN_EDGE_CACHING_IMPLEMENTATION.md guides/frontend/cdn-caching.md Move + Rename Implementation guide
plans/BUG-004-SQL-INJECTION-FIX-PLAN.md archive/2025-10/sql-injection-fix.md Archive Completed bug fix

6.3 Consolidation Strategy

Documents to Merge:

  1. E2E Testing Documents:
  2. Merge: E2E_TESTING_PLAN.md + playwright-e2e-testing-plan.md
  3. Result: testing/e2e-testing-strategy.md

  4. Keep most recent and comprehensive content

  5. Performance Optimization:

  6. Merge: plans/PERFORMANCE_OPTIMIZATION_PLAN.md + plans/PERFORMANCE_OPTIMIZATION.md
  7. Result: planning/performance-optimization.md or archive/2025-10/performance-optimization.md (if completed)

  8. Consolidate into single comprehensive document

  9. Memory Optimization:

  10. Merge: plans/MEMORY_OPTIMIZATION_PLAN.md + plans/MEMORY_CODE_OPTIMIZATIONS.md + plans/MEMORY_REDUCTION_STRATEGIES.md
  11. Result: archive/2025-10/memory-optimization.md (likely completed work)

  12. Create single consolidated document covering all aspects

  13. Feature Flags:

  14. Review: plans/PIQUE-506-feature-flag-integration-plan.md + plans/producer-feature-flags-plan.md
  15. Result: Determine if these are separate features or same feature
  16. If same: merge into planning/feature-flags.md or archive if completed
  17. If different: keep separate with clear naming

7. Testing Strategy

7.1 Documentation Validation

Manual Review Checklist: - [ ] All files follow naming convention - [ ] All files in appropriate directories - [ ] All metadata headers present and complete - [ ] Cross-references updated and working - [ ] No broken internal links - [ ] Consistent formatting (headers, lists, code blocks) - [ ] Technical accuracy validated against codebase - [ ] No duplicate content

Automated Checks (Nice to Have): - Markdown linting (e.g., markdownlint) - Link checking (e.g., markdown-link-check) - Spell checking

7.2 Team Review Process

  1. Self Review: Implementer reviews all changes
  2. Peer Review: At least one team member reviews structure and content
  3. SME Review: Subject matter experts review technical accuracy of their domain docs
  4. Final Approval: Team lead approves final structure

7.3 Acceptance Criteria

Definition of Done: - [ ] All 27 existing documentation files processed (moved, renamed, or archived) - [ ] New directory structure created with README files - [ ] All files follow kebab-case naming convention - [ ] All files include metadata headers - [ ] Duplicate documents consolidated - [ ] Internal links and cross-references updated - [ ] Non-documentation files (.DS_Store) removed - [ ] Documentation index (main README.md) created - [ ] Team review completed and feedback addressed - [ ] No broken links or references

8. Risk Assessment & Mitigation

8.1 Technical Risks

Risk Probability Impact Mitigation Strategy
Breaking existing documentation links from external sources Low Medium Create redirect/reference doc at old locations pointing to new locations
Accidentally deleting important information during consolidation Medium High Use git for all changes; review diffs carefully; have team review before merging
Misunderstanding document purpose leading to wrong categorization Medium Low Review with document authors when possible; can be corrected later
Merge conflicts if others are editing docs simultaneously Medium Low Communicate cleanup work to team; work in dedicated branch

8.2 Resource Risks

  • Schedule Risks: Work may take longer if many documents need content updates
  • Mitigation: Phase 3 (content updates) is optional for initial cleanup; can be iterative
  • Skill Gaps: Need understanding of all system components to validate technical accuracy
  • Mitigation: Involve subject matter experts for technical validation; focus on structure first
  • External Dependencies: None

8.3 Rollback Strategy

  • Git-based rollback: All work done in git branch; can revert if needed
  • Incremental approach: Work directory by directory; can pause/revert at any point
  • Branch deployment: Keep old structure in main branch until new structure validated

9. Deployment & Operations

9.1 Deployment Plan

  • Environment Progression: Work in feature branch → Team review → Merge to main
  • Configuration Updates: None required
  • Monitoring Setup: None required (documentation only)

Steps: 1. Create feature branch: pique-618-documentation-cleanup 2. Implement changes in phases 3. Self-review all changes 4. Create PR with before/after directory structure 5. Team review 6. Merge to main

9.2 Communication Plan

Before Starting: - [ ] Notify team of documentation cleanup effort - [ ] Ask team to avoid documentation changes during cleanup period - [ ] Request input on document categorization questions

During Work: - [ ] Post updates on progress - [ ] Ask clarifying questions about specific documents

After Completion: - [ ] Announce new documentation structure - [ ] Provide navigation guide (main README) - [ ] Gather feedback for future improvements

10. Success Measurement

10.1 Success Metrics

Immediate Metrics: - ✅ 100% of files follow naming convention - ✅ All files categorized into appropriate directories - ✅ All duplicate content consolidated - ✅ All files have metadata headers - ✅ Zero broken internal links

Long-term Metrics: - Developer feedback on documentation discoverability (survey) - Time to find documentation (before/after comparison) - Reduction in documentation-related questions - Documentation update frequency (easier to maintain = more updates)

10.2 Review Schedule

  • Initial Review: Immediately after Phase 4 completion
  • Team Feedback: 1 week after merge to main
  • Retrospective: 1 month after merge to assess effectiveness

11. Appendices

Appendix A: Current Documentation Summary

Total Files: 27 markdown files + 1 .DS_Store file Current Categories: - API-related: 7 files - Planning: 11 files - Portal-specific: 3 files - Root level: 6 files

Naming Patterns: - ALL_CAPS_UNDERSCORE: 15 files - kebab-case: 4 files - Mixed: 8 files

Appendix B: Proposed Directory Structure

docs/
├── README.md
├── architecture/       (2-3 files)
├── guides/            (10-12 files)
│   ├── api/
│   ├── frontend/
│   └── producer-portal/
├── operations/        (3-4 files)
├── testing/          (4-5 files)
├── planning/         (3-5 files, active only)
├── archive/          (8-10 files)
│   └── 2025-10/
└── templates/        (3 template files)

Appendix C: Documentation Templates

Templates will be created for: 1. Feature Planning Document - Based on existing plan structure 2. Architecture Document - For system design documentation 3. Testing Plan - For test strategy and planning documents

Appendix D: Team Communication

Announcement Template: 

📚 Documentation Cleanup Initiative - PIQUE-618

We're reorganizing the /docs directory to improve discoverability and consistency.

What's changing:
- New directory structure organized by document type
- Consistent kebab-case naming convention
- Consolidated duplicate documents
- Archive for completed plans

Timeline: [dates]

Please avoid making documentation changes during this period. Questions or concerns? Reply to this thread.

Quick Checklist for Implementation

Research Phase

  • Analyzed existing documentation structure
  • Identified naming patterns and inconsistencies
  • Catalogued all documentation files
  • Identified duplicate and overlapping content
  • Researched documentation best practices

Planning Phase

  • Defined new directory structure
  • Established naming conventions
  • Created migration plan for each file
  • Identified consolidation opportunities
  • Defined acceptance criteria

Implementation Phase (To Do)

  • Create new directory structure
  • Create README and template files
  • Migrate and rename files
  • Consolidate duplicate content
  • Add metadata headers
  • Update cross-references
  • Validate technical accuracy
  • Team review

Quality Assurance (To Do)

  • Verify all files migrated
  • Check all links working
  • Validate naming consistency
  • Remove non-doc files
  • Final team review and approval

Document Information: - Created: 2025-11-01 - Last Updated: 2025-11-01 - Version: 1.0 - Author(s): Development Team - Status: Draft - Awaiting Review