Auto-cleanup courses for ended semesters #455

[MithishR]

Description

This PR implements an automated course cleanup service to manage server space usage and removing course documents from ended semesters. The cronjob runs monthly (1st of the month for warnings and 15th for archival), and it sends warning emails to professors 2 weeks before archival. The archival is an automated process which cleans up associated data and archives the courses. For the sake of robustness this service will only process semesters ending after January 1, 2023.

Closes #455

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• This requires a run of yarn install
• This change requires an addition/change to the production .env variables. These changes are below:
• This change requires developers to add new .env variables. The file and variables needed are below:
• This change requires a database query to update old data on production. This query is below:

How Has This Been Tested?

Testing the warning process by running the cron job every minute, and the archival process every two minutes. Verified the sent emails in the email log, and created test courses across various test semesters to see if the archival actually happened.

Checklist:

• I have performed a code review of my own code (under the "Files Changed" tab on github) to ensure nothing is committed that shouldn't be (e.g. leftover console.logs, leftover unused logic, or anything else that was accidentally committed)
• I have commented my code where needed
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have checked that new and existing tests pass locally with my changes
• Any work that this PR is dependent on has been merged into the main branch
• Any UI changes have been checked to work on desktop, tablet, and mobile

[AdamFipke]

Gave it an initial pass, see comments

[AdamFipke]

idk if this is all you have to do to cleanup a connection. @bhunt02 you will want to review this step

[AdamFipke]

erm, this only deletes the full documents (used for citations). You should also delete the corresponding document (and its chunks) in the chatbot DB, basically copying the logic for deleteDocument() endpoint inside chatbot.controller.ts.

 @Delete('document/:courseId/:docId')
 @UseGuards(CourseRolesGuard)
 @Roles(Role.PROFESSOR, Role.TA)
 async deleteDocument(
   @Param('courseId', ParseIntPipe) courseId: number,
   @Param('docId') docId: string,
   @User({ chat_token: true }) user: UserModel,
 ) {
   handleChatbotTokenCheck(user);
   const chatbotDeleteResponse = await this.chatbotApiService.deleteDocument(
     docId,
     courseId,
     user.chat_token.token,
   );
   // if that succeeded (an error would have been thrown if it didn't), then delete the document from database
   await ChatbotDocPdfModel.delete({
     docIdChatbotDB: docId,
   });
   return chatbotDeleteResponse;
 }

I'm thinking you can just loop over all ChatbotDocPdfModel entities for the course and use that docIdChatbotDB to delete the corresponding documents using chatbotApiService.deleteDocument, and then if that succeeds, delete the ChatbotDocPdfModel

[AdamFipke]

Actually, you also need to figure out how to delete any LMS chunks too. Which means we might need a new endpoint in the chatbot repo for deleting all chunks and documents (including LMS ones)? Unless I'm missing something which is very likely.

@bhunt02 easiest way to delete all LMS chunks for a course?

[AdamFipke]

Actually wait, deleting all chunks is also a bad idea since some of the chunks are ones you would really want to keep (like custom made ones, or ones made from question, or ones that have been edited (i forget if we have an attribute for that but i don't think so)).

So maybe just deleting all chunks from uploaded documents and LMS chunk is best

[AdamFipke]

Okay I just realised that you haven't actually requested my review just yet (idk why i thought that) and it wasn't until course-cleanup.service.ts that I realised, sorry in advance, but I did manage to find some important things (and some less important) that might help you.

[AdamFipke]

This is partially just a note for me (and @bhunt02 ) since I think our backend on production uses a different timezone so this might be an issue idk.

Maybe a fix for this is to grab the timezone from the course (course.timezone)