Video Processing Flow Documentation

This document describes the step-by-step flow of the video processing system, from the upload of a file to its final processing and storage. The flow involves several key components including AWS services like Lambda, DynamoDB, MediaConvert, and S3.

Process Diagram

For better readability, the process has been split into two diagrams.

1. Upload and Initialization Flow

2. Processing and Finalization Flow

This process illustrates the complete media processing flow from initial upload to final storage, divided into two main phases for clarity:

1. Upload and Initialization:

   • Server requests a signed URL from S3
   • File is uploaded to S3
   • Lambda Upload function is triggered
   • Process is initiated in DynamoDB
   • MediaConvert processing is triggered

2. Processing and Finalization:

   • MediaConvert processes the file
   • Progress updates are tracked
   • Processed media are stored
   • Process completion is recorded
   • Final media information is stored in database

1. Request Signed URL for Upload

• Server -> S3: The process starts with the server requesting a signed URL from Amazon S3 to allow users to upload video files securely to the uploads/* path in the S3 bucket.
• This signed URL is used to ensure that only authorized requests can upload content to the storage location.

2. File Upload to S3

• S3 -> Lambda (Upload Trigger): Once the file is uploaded to the designated S3 location (uploads/*), the S3 bucket triggers a Lambda function (referred to as LambdaUpload), indicating the successful file upload.

3. Add File to Media Process Table (DynamoDB)

• LambdaUpload -> DynamoDB (Media Process Table): The LambdaUpload function adds an entry for the uploaded file into the DynamoDB table (named Media Process Table).
• The entry includes the file metadata and sets the status to PENDING, indicating that the file is ready for processing.

4. Trigger MediaConvert Processing

• LambdaUpload -> MediaConvert: The next step is for LambdaUpload to trigger AWS MediaConvert to start processing the uploaded video file.
• MediaConvert handles video transcoding, format conversion, based on the provided settings.

5. Progress Updates from MediaConvert

• MediaConvert -> Lambda (Progress Update): As MediaConvert processes the video file, it sends progress updates back to the LambdaProgress function. These updates typically include information about the current stage of the conversion (e.g., transcoding, encoding, etc.).

6. Update Media Process Table (DynamoDB)

• LambdaProgress -> DynamoDB (Media Process Table): The LambdaProgress function takes the progress updates and updates the Media Process Table in DynamoDB.
• The status of the media file is updated to reflect the current stage of processing (e.g., In Progress, Transcoding, etc.).

7. Store Converted Media in S3

• MediaConvert -> S3: After the video file is processed and converted by MediaConvert, the converted media are stored back in S3 under the appropriate locations.
• These files are now ready to be accessed or further processed if needed.

8. Processing Completion Notification

• MediaConvert -> Lambda (Processing Complete): Once MediaConvert finishes processing the file, it triggers the LambdaFinish function, signaling the completion of the media processing.

9. Add Media Info to Media Data Table (DynamoDB)

• LambdaFinish -> DynamoDB (Media Data Table): The LambdaFinish function adds important media information to another DynamoDB table (referred to as Media Data Table). This table includes the media file's unique ID, the S3 key to access the file.
• This step concludes the media processing workflow.